Skip to main content
Kafka Topics are the fundamental unit of data organization in Kafka. Messages are published to topics and consumed from topics. Each topic is divided into partitions for parallel processing and replicated for fault tolerance.

​
Key Concepts

Partitions

Topics are split into partitions for parallel processing. Each partition is an ordered, immutable sequence of messages.

Replication

Partitions are replicated across brokers for fault tolerance. The replication factor determines how many copies exist.

Retention

Messages are retained for a configurable time period or until a size limit is reached, then deleted or compacted.

Consumer Groups

Consumer groups track which messages have been read. Each partition is consumed by one consumer in a group.

​
Required Permissions

ActionPermission
View topicsiam:project:infrastructure:kafka:read
Create topicsiam:project:infrastructure:kafka:write
Update topic configurationiam:project:infrastructure:kafka:write
Delete topicsiam:project:infrastructure:kafka:delete
Clear messagesiam:project:infrastructure:kafka:delete
Produce messagesiam:project:infrastructure:kafka:produce

​
How to Create a Topic

1

Select Kafka Cluster

Choose the target Kafka cluster from the dropdown.
2

Click Create Topic

Click the Create Topic button in the header.
3

Enter Basic Settings

Configure the topic fundamentals:
  • Topic Name (required): Unique identifier
  • Partitions: Number of partitions (1-1000)
  • Replication Factor: Number of replicas (1-10)
4

Configure Advanced Settings

Optionally set retention and cleanup policies:
  • Retention (ms): How long to keep messages (-1 for unlimited)
  • Cleanup Policy: delete, compact, or both
  • Min In-Sync Replicas: Minimum replicas for acks=all
5

Create

Click Create Topic to create the topic.

​
Topic Name Rules

Topic names must follow these rules:
RuleDescription
Length1-255 characters
CharactersLetters, numbers, dots (.), underscores (_), hyphens (-)
ReservedNames starting with _ are reserved for internal Kafka topics
Topic names cannot be changed after creation. Choose names carefully using a consistent naming convention.

​
Partition Guidelines

ScenarioRecommended Partitions
Low throughput1-3
Medium throughput6-12
High throughput12-50
Very high throughput50+
Partition count can only be increased, never decreased. Start with fewer partitions and scale up as needed.

​
How to View Topic Details

1

Find the Topic

Locate the topic in the list using search or scrolling.
2

Click to Open

Click on the topic row to open the detail page.
3

Explore Tabs

The detail page has four tabs:
  • Overview: Partition details, replicas, ISR status
  • Messages: Browse and search messages
  • Consumers: Consumer groups consuming this topic
  • Settings: Topic configuration

​
How to Browse Messages

1

Open Topic Detail

Navigate to the topic detail page.
2

Go to Messages Tab

Select the Messages tab.
3

Configure Filters

Set filters to find specific messages:
  • Partition: Filter by specific partition
  • Offset: Start from a specific offset
  • Key Contains: Search by message key
  • Value Contains: Search by message value
4

Browse Results

Messages are displayed with partition, offset, timestamp, key, value, and headers.

​
How to Produce a Message

1

Open Topic Detail

Navigate to the topic detail page.
2

Go to Messages Tab

Select the Messages tab.
3

Click Produce Message

Click the Produce Message button.
4

Enter Message Details

Configure the message:
  • Key (optional): Message key for partitioning
  • Value (required): Message content
  • Partition (optional): Target partition (-1 for auto)
  • Headers (optional): Key-value metadata
5

Send

Click Produce to send the message.

​
How to Update Topic Configuration

1

Open Topic Detail

Navigate to the topic detail page.
2

Go to Settings Tab

Select the Settings tab.
3

Modify Configuration

Update the configuration values you want to change.
4

Save Changes

Click Save to apply the new configuration.

​
Common Configuration Options

ConfigurationDescriptionDefault
retention.msHow long to retain messages604800000 (7 days)
cleanup.policydelete, compact, or delete,compactdelete
min.insync.replicasMin replicas for acks=all1
segment.bytesLog segment size1073741824 (1 GB)
max.message.bytesMax message size1048588 (~1 MB)

​
How to Increase Partitions

1

Open Topic Detail

Navigate to the topic detail page.
2

Go to Settings Tab

Select the Settings tab.
3

Enter New Partition Count

Set the new partition count. Must be greater than current count.
4

Apply

Click Increase Partitions to apply.
Increasing partitions affects message ordering. Messages with the same key may be distributed across different partitions after the change.

​
How to Clear Messages

Clearing messages deletes all data from a topic while keeping the topic itself.
1

Open Topic Detail

Navigate to the topic detail page.
2

Click Clear Messages

Click the Clear Messages button.
3

Confirm

Confirm the action. This cannot be undone.

​
How to Delete a Topic

1

Find the Topic

Locate the topic in the list or open the detail page.
2

Click Delete

Click the Delete Topic button.
3

Confirm

Confirm the deletion. This permanently removes the topic and all messages.
Deleting a topic is irreversible. All messages and consumer group offsets for this topic are permanently lost.

​
How to Use Favorites

Mark frequently used topics as favorites for quick access.
1

Find the Topic

Locate the topic in the list.
2

Click the Star

Click the star icon next to the topic name to add/remove from favorites.
3

Filter by Favorites

Click the Favorites button in the toolbar to show only favorited topics.

​
Understanding Partition Details

Each partition displays:
FieldDescription
LeaderBroker ID handling reads/writes for this partition
ReplicasList of broker IDs holding copies
ISRIn-Sync Replicas - replicas caught up with leader
Start OffsetLowest available offset (oldest message)
End OffsetHighest offset (next message position)

​
Under-Replicated Partitions (URP)

A partition is under-replicated when ISR count < replica count. This indicates:
  • A broker is down or unreachable
  • A replica is falling behind
  • Network issues between brokers
Under-replicated partitions reduce fault tolerance. Investigate and resolve URP issues promptly.

​
Cleanup Policies

PolicyBehavior
deleteDelete messages older than retention period
compactKeep only the latest value for each key
delete,compactCompact first, then delete old segments

​
When to Use Compact

Use compaction for:
  • State stores (latest user profile, settings)
  • Changelog topics
  • Deduplication scenarios
Compacted topics keep messages indefinitely until a newer message with the same key arrives. Set delete.retention.ms to control how long tombstones are retained.

​
Troubleshooting

  • Topic name may already exist
  • Invalid characters in topic name
  • Replication factor exceeds available brokers
  • Cluster may be in maintenance mode
  • Messages may be in a different partition
  • Check if you’re looking at the correct offset range
  • Consumer may have already committed past these offsets
  • Topic may have low retention and messages expired
  • You need produce permission
  • Topic may be read-only (internal topics)
  • Message size may exceed max.message.bytes
  • Cluster may be unavailable
  • You need delete permission
  • Topic may be in use by active consumers
  • Cluster’s delete.topic.enable may be false
This is a Kafka limitation. Partition count can only increase, never decrease. To reduce partitions, create a new topic with fewer partitions and migrate data.
  • Consumer may be processing slowly
  • Consumer group may have fewer members than partitions
  • Consider increasing partition count and consumer instances
  • Check for processing errors in consumer application

​
FAQ

Partitions enable parallel processing - each partition can be consumed by a different consumer. Replicas provide fault tolerance - they’re copies of partitions on different brokers. A topic with 6 partitions and replication factor 3 has 18 total partition replicas distributed across brokers.
Consider: (1) Expected throughput - more partitions enable higher parallelism, (2) Number of consumers - each partition can only be consumed by one consumer in a group, (3) Ordering requirements - messages are only ordered within a partition. Start small and increase as needed.
If a broker fails, leadership for its partitions moves to another broker with a replica. If the failed broker had the only replica (replication factor 1), those partitions become unavailable. Use replication factor >= 3 for production.
Messages are ordered within a partition only. Messages with the same key go to the same partition (by default), ensuring order for that key. Messages with different keys may be processed out of order across partitions.
This setting defines the minimum number of replicas that must acknowledge a write before it’s considered successful (when producer uses acks=all). Set to replication_factor - 1 for best availability while maintaining durability.
Not directly through this UI. Changing replication factor requires Kafka’s reassignment tool. You would need to create a reassignment plan and execute it using Kafka CLI tools.
Topics starting with _ or __ are internal Kafka topics (like __consumer_offsets, __transaction_state). They’re managed by Kafka and shouldn’t be modified directly. By default, they’re hidden from the topic list.
Controlled by retention.ms (time-based) and retention.bytes (size-based). Messages are deleted when either limit is exceeded. Default is 7 days. Set to -1 for unlimited time-based retention.

​
Best Practices

​
Naming Conventions

Use consistent naming patterns: 
<domain>.<entity>.<action>
<team>.<service>.<event-type>

Examples:
orders.payments.completed
users.profiles.updated
inventory.stock.low

​
Partition Strategy

  • Key-based: Messages with same key go to same partition (default)
  • Round-robin: No key means random partition assignment
  • Custom: Use custom partitioner for special requirements

​
Production Settings

For production topics: 
partitions: >= 6 (for parallelism)
replication.factor: 3 (for fault tolerance)
min.insync.replicas: 2 (for durability)
retention.ms: based on use case

​
Monitoring

Monitor these metrics:
  • Under-replicated partitions (should be 0)
  • Consumer lag (should be stable or decreasing)
  • Message rate (for capacity planning)
  • Partition distribution (should be balanced)