> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Topics

> Create, manage, and monitor Kafka topics and messages

Kafka Topics are the fundamental unit of data organization in Kafka. Messages are published to topics and consumed from topics. Each topic is divided into partitions for parallel processing and replicated for fault tolerance.

## Key Concepts

<CardGroup cols={2}>
  <Card title="Partitions" icon="database">
    Topics are split into partitions for parallel processing. Each partition is an ordered, immutable sequence of messages.
  </Card>

  <Card title="Replication" icon="copy">
    Partitions are replicated across brokers for fault tolerance. The replication factor determines how many copies exist.
  </Card>

  <Card title="Retention" icon="clock">
    Messages are retained for a configurable time period or until a size limit is reached, then deleted or compacted.
  </Card>

  <Card title="Consumer Groups" icon="users">
    Consumer groups track which messages have been read. Each partition is consumed by one consumer in a group.
  </Card>
</CardGroup>

## Required Permissions

| Action                     | Permission                                 |
| -------------------------- | ------------------------------------------ |
| View topics                | `iam:project:infrastructure:kafka:read`    |
| Create topics              | `iam:project:infrastructure:kafka:write`   |
| Update topic configuration | `iam:project:infrastructure:kafka:write`   |
| Delete topics              | `iam:project:infrastructure:kafka:delete`  |
| Clear messages             | `iam:project:infrastructure:kafka:delete`  |
| Produce messages           | `iam:project:infrastructure:kafka:produce` |

## How to Create a Topic

<Steps>
  <Step title="Select Kafka Cluster">
    Choose the target Kafka cluster from the dropdown.
  </Step>

  <Step title="Click Create Topic">
    Click the **Create Topic** button in the header.
  </Step>

  <Step title="Enter Basic Settings">
    Configure the topic fundamentals:

    * **Topic Name** (required): Unique identifier
    * **Partitions**: Number of partitions (1-1000)
    * **Replication Factor**: Number of replicas (1-10)
  </Step>

  <Step title="Configure Advanced Settings">
    Optionally set retention and cleanup policies:

    * **Retention (ms)**: How long to keep messages (-1 for unlimited)
    * **Cleanup Policy**: delete, compact, or both
    * **Min In-Sync Replicas**: Minimum replicas for acks=all
  </Step>

  <Step title="Create">
    Click **Create Topic** to create the topic.
  </Step>
</Steps>

### Topic Name Rules

Topic names must follow these rules:

| Rule           | Description                                                    |
| -------------- | -------------------------------------------------------------- |
| **Length**     | 1-255 characters                                               |
| **Characters** | Letters, numbers, dots (`.`), underscores (`_`), hyphens (`-`) |
| **Reserved**   | Names starting with `_` are reserved for internal Kafka topics |

<Warning>
  Topic names cannot be changed after creation. Choose names carefully using a consistent naming convention.
</Warning>

### Partition Guidelines

| Scenario             | Recommended Partitions |
| -------------------- | ---------------------- |
| Low throughput       | 1-3                    |
| Medium throughput    | 6-12                   |
| High throughput      | 12-50                  |
| Very high throughput | 50+                    |

<Info>
  Partition count can only be **increased**, never decreased. Start with fewer partitions and scale up as needed.
</Info>

## How to View Topic Details

<Steps>
  <Step title="Find the Topic">
    Locate the topic in the list using search or scrolling.
  </Step>

  <Step title="Click to Open">
    Click on the topic row to open the detail page.
  </Step>

  <Step title="Explore Tabs">
    The detail page has four tabs:

    * **Overview**: Partition details, replicas, ISR status
    * **Messages**: Browse and search messages
    * **Consumers**: Consumer groups consuming this topic
    * **Settings**: Topic configuration
  </Step>
</Steps>

## How to Browse Messages

<Steps>
  <Step title="Open Topic Detail">
    Navigate to the topic detail page.
  </Step>

  <Step title="Go to Messages Tab">
    Select the **Messages** tab.
  </Step>

  <Step title="Configure Filters">
    Set filters to find specific messages:

    * **Partition**: Filter by specific partition
    * **Offset**: Start from a specific offset
    * **Key Contains**: Search by message key
    * **Value Contains**: Search by message value
  </Step>

  <Step title="Browse Results">
    Messages are displayed with partition, offset, timestamp, key, value, and headers.
  </Step>
</Steps>

## How to Produce a Message

<Steps>
  <Step title="Open Topic Detail">
    Navigate to the topic detail page.
  </Step>

  <Step title="Go to Messages Tab">
    Select the **Messages** tab.
  </Step>

  <Step title="Click Produce Message">
    Click the **Produce Message** button.
  </Step>

  <Step title="Enter Message Details">
    Configure the message:

    * **Key** (optional): Message key for partitioning
    * **Value** (required): Message content
    * **Partition** (optional): Target partition (-1 for auto)
    * **Headers** (optional): Key-value metadata
  </Step>

  <Step title="Send">
    Click **Produce** to send the message.
  </Step>
</Steps>

## How to Update Topic Configuration

<Steps>
  <Step title="Open Topic Detail">
    Navigate to the topic detail page.
  </Step>

  <Step title="Go to Settings Tab">
    Select the **Settings** tab.
  </Step>

  <Step title="Modify Configuration">
    Update the configuration values you want to change.
  </Step>

  <Step title="Save Changes">
    Click **Save** to apply the new configuration.
  </Step>
</Steps>

### Common Configuration Options

| Configuration         | Description                        | Default            |
| --------------------- | ---------------------------------- | ------------------ |
| `retention.ms`        | How long to retain messages        | 604800000 (7 days) |
| `cleanup.policy`      | delete, compact, or delete,compact | delete             |
| `min.insync.replicas` | Min replicas for acks=all          | 1                  |
| `segment.bytes`       | Log segment size                   | 1073741824 (1 GB)  |
| `max.message.bytes`   | Max message size                   | 1048588 (\~1 MB)   |

## How to Increase Partitions

<Steps>
  <Step title="Open Topic Detail">
    Navigate to the topic detail page.
  </Step>

  <Step title="Go to Settings Tab">
    Select the **Settings** tab.
  </Step>

  <Step title="Enter New Partition Count">
    Set the new partition count. Must be greater than current count.
  </Step>

  <Step title="Apply">
    Click **Increase Partitions** to apply.
  </Step>
</Steps>

<Warning>
  Increasing partitions affects message ordering. Messages with the same key may be distributed across different partitions after the change.
</Warning>

## How to Clear Messages

Clearing messages deletes all data from a topic while keeping the topic itself.

<Steps>
  <Step title="Open Topic Detail">
    Navigate to the topic detail page.
  </Step>

  <Step title="Click Clear Messages">
    Click the **Clear Messages** button.
  </Step>

  <Step title="Confirm">
    Confirm the action. This cannot be undone.
  </Step>
</Steps>

## How to Delete a Topic

<Steps>
  <Step title="Find the Topic">
    Locate the topic in the list or open the detail page.
  </Step>

  <Step title="Click Delete">
    Click the **Delete Topic** button.
  </Step>

  <Step title="Confirm">
    Confirm the deletion. This permanently removes the topic and all messages.
  </Step>
</Steps>

<Warning>
  Deleting a topic is irreversible. All messages and consumer group offsets for this topic are permanently lost.
</Warning>

## How to Use Favorites

Mark frequently used topics as favorites for quick access.

<Steps>
  <Step title="Find the Topic">
    Locate the topic in the list.
  </Step>

  <Step title="Click the Star">
    Click the star icon next to the topic name to add/remove from favorites.
  </Step>

  <Step title="Filter by Favorites">
    Click the **Favorites** button in the toolbar to show only favorited topics.
  </Step>
</Steps>

## Understanding Partition Details

Each partition displays:

| Field            | Description                                        |
| ---------------- | -------------------------------------------------- |
| **Leader**       | Broker ID handling reads/writes for this partition |
| **Replicas**     | List of broker IDs holding copies                  |
| **ISR**          | In-Sync Replicas - replicas caught up with leader  |
| **Start Offset** | Lowest available offset (oldest message)           |
| **End Offset**   | Highest offset (next message position)             |

### Under-Replicated Partitions (URP)

A partition is under-replicated when ISR count \< replica count. This indicates:

* A broker is down or unreachable
* A replica is falling behind
* Network issues between brokers

<Warning>
  Under-replicated partitions reduce fault tolerance. Investigate and resolve URP issues promptly.
</Warning>

## Cleanup Policies

| Policy             | Behavior                                    |
| ------------------ | ------------------------------------------- |
| **delete**         | Delete messages older than retention period |
| **compact**        | Keep only the latest value for each key     |
| **delete,compact** | Compact first, then delete old segments     |

### When to Use Compact

Use compaction for:

* State stores (latest user profile, settings)
* Changelog topics
* Deduplication scenarios

<Info>
  Compacted topics keep messages indefinitely until a newer message with the same key arrives. Set `delete.retention.ms` to control how long tombstones are retained.
</Info>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Topic creation failed" icon="circle-question">
    * Topic name may already exist
    * Invalid characters in topic name
    * Replication factor exceeds available brokers
    * Cluster may be in maintenance mode
  </Accordion>

  <Accordion title="Messages not appearing" icon="circle-question">
    * Messages may be in a different partition
    * Check if you're looking at the correct offset range
    * Consumer may have already committed past these offsets
    * Topic may have low retention and messages expired
  </Accordion>

  <Accordion title="Cannot produce messages" icon="circle-question">
    * You need produce permission
    * Topic may be read-only (internal topics)
    * Message size may exceed `max.message.bytes`
    * Cluster may be unavailable
  </Accordion>

  <Accordion title="Cannot delete topic" icon="circle-question">
    * You need delete permission
    * Topic may be in use by active consumers
    * Cluster's `delete.topic.enable` may be false
  </Accordion>

  <Accordion title="Partition count cannot be decreased" icon="circle-question">
    This is a Kafka limitation. Partition count can only increase, never decrease. To reduce partitions, create a new topic with fewer partitions and migrate data.
  </Accordion>

  <Accordion title="High consumer lag" icon="circle-question">
    * Consumer may be processing slowly
    * Consumer group may have fewer members than partitions
    * Consider increasing partition count and consumer instances
    * Check for processing errors in consumer application
  </Accordion>
</AccordionGroup>

## FAQ

<AccordionGroup>
  <Accordion title="What's the difference between partitions and replicas?">
    **Partitions** enable parallel processing - each partition can be consumed by a different consumer. **Replicas** provide fault tolerance - they're copies of partitions on different brokers. A topic with 6 partitions and replication factor 3 has 18 total partition replicas distributed across brokers.
  </Accordion>

  <Accordion title="How do I choose the right number of partitions?">
    Consider: (1) Expected throughput - more partitions enable higher parallelism, (2) Number of consumers - each partition can only be consumed by one consumer in a group, (3) Ordering requirements - messages are only ordered within a partition. Start small and increase as needed.
  </Accordion>

  <Accordion title="What happens when a broker fails?">
    If a broker fails, leadership for its partitions moves to another broker with a replica. If the failed broker had the only replica (replication factor 1), those partitions become unavailable. Use replication factor >= 3 for production.
  </Accordion>

  <Accordion title="How does message ordering work?">
    Messages are ordered within a partition only. Messages with the same key go to the same partition (by default), ensuring order for that key. Messages with different keys may be processed out of order across partitions.
  </Accordion>

  <Accordion title="What is min.insync.replicas?">
    This setting defines the minimum number of replicas that must acknowledge a write before it's considered successful (when producer uses `acks=all`). Set to `replication_factor - 1` for best availability while maintaining durability.
  </Accordion>

  <Accordion title="Can I change a topic's replication factor?">
    Not directly through this UI. Changing replication factor requires Kafka's reassignment tool. You would need to create a reassignment plan and execute it using Kafka CLI tools.
  </Accordion>

  <Accordion title="What are internal topics?">
    Topics starting with `_` or `__` are internal Kafka topics (like `__consumer_offsets`, `__transaction_state`). They're managed by Kafka and shouldn't be modified directly. By default, they're hidden from the topic list.
  </Accordion>

  <Accordion title="How long are messages retained?">
    Controlled by `retention.ms` (time-based) and `retention.bytes` (size-based). Messages are deleted when either limit is exceeded. Default is 7 days. Set to `-1` for unlimited time-based retention.
  </Accordion>
</AccordionGroup>

## Best Practices

### Naming Conventions

Use consistent naming patterns:

```
<domain>.<entity>.<action>
<team>.<service>.<event-type>

Examples:
orders.payments.completed
users.profiles.updated
inventory.stock.low
```

### Partition Strategy

* **Key-based**: Messages with same key go to same partition (default)
* **Round-robin**: No key means random partition assignment
* **Custom**: Use custom partitioner for special requirements

### Production Settings

For production topics:

```
partitions: >= 6 (for parallelism)
replication.factor: 3 (for fault tolerance)
min.insync.replicas: 2 (for durability)
retention.ms: based on use case
```

### Monitoring

Monitor these metrics:

* Under-replicated partitions (should be 0)
* Consumer lag (should be stable or decreasing)
* Message rate (for capacity planning)
* Partition distribution (should be balanced)