Skip to main content
Consumer Groups coordinate multiple consumers reading from the same topics. Kafka tracks which messages each group has consumed, enabling parallel processing while ensuring each message is processed once per group.

​
Key Concepts

Consumer Group

A logical grouping of consumers that work together to consume messages from topics. Each partition is assigned to one consumer in the group.

Offset

A position marker indicating which messages have been consumed. Kafka stores offsets per partition per consumer group.

Lag

The difference between the latest message in a partition and the last consumed message. High lag indicates consumers are falling behind.

Coordinator

A broker responsible for managing the consumer group’s membership and partition assignments.

​
Required Permissions

ActionPermission
View consumer groupsiam:project:infrastructure:kafka:read
Reset offsetsiam:project:infrastructure:kafka:write
Delete consumer groupsiam:project:infrastructure:kafka:delete

​
Consumer Group States

StateDescription
StableGroup is active with assigned partitions. Members are consuming normally.
EmptyGroup exists but has no active members. Offsets are still retained.
PreparingRebalanceGroup is preparing to reassign partitions due to member changes.
CompletingRebalanceGroup is finalizing partition assignments after rebalance.
DeadGroup has been deleted or expired due to inactivity.
Groups in Stable state cannot be deleted or have their offsets reset. Stop all consumers first.

​
How to View Consumer Groups

1

Select Kafka Cluster

Choose the target Kafka cluster from the dropdown.
2

Browse Groups

The list shows all consumer groups with their state, member count, and lag.
3

Filter by State

Use the state filter to show only groups in specific states (Stable, Empty, Rebalancing, Dead).
4

Open Details

Click on a group to view detailed information including members, topics, and offsets.

​
How to View Consumer Group Details

The detail page provides three tabs:

​
Overview Tab

Shows subscribed topics and general group information:
  • Group ID
  • Current state
  • Member count
  • Coordinator broker

​
Members Tab

Lists all active consumers in the group:
  • Client ID: Application identifier
  • Client Host: IP address or hostname
  • Member ID: Unique member identifier assigned by Kafka
  • Assigned Partitions: Topic:partition pairs assigned to this member

​
Offsets Tab

Shows consumption progress for each partition:
  • Current Offset: Last committed offset
  • Log End Offset: Latest message position in the partition
  • Lag: Messages waiting to be consumed

​
Understanding Lag

Lag indicates how far behind a consumer group is from the latest messages.
Lag LevelMeaningAction
0-1,000HealthyConsumers are keeping up
1,000-10,000WarningMonitor closely, may need attention
> 10,000CriticalConsumers falling behind significantly
Consistently high lag can cause data processing delays and may lead to data loss if retention expires before messages are consumed.

​
Common Causes of High Lag

  • Consumer processing is too slow
  • Not enough consumers for the number of partitions
  • Consumer application errors causing reprocessing
  • Network issues between consumers and brokers
  • Unbalanced partition distribution

​
How to Reset Consumer Offsets

Resetting offsets allows you to reprocess messages or skip ahead.
1

Stop All Consumers

Ensure no consumers are actively running. The group must not be in Stable state.
2

Open Consumer Group Details

Navigate to the consumer group detail page.
3

Choose Reset Method

Select how to reset offsets:
  • Earliest: Reprocess all available messages from the beginning
  • Latest: Skip to the end, only consume new messages
  • Timestamp: Reset to a specific point in time
  • Offset: Reset to a specific offset number
4

Apply Reset

Click Reset Offsets to apply the changes.
5

Restart Consumers

Start your consumer applications. They will begin consuming from the new offset positions.

​
Reset Types

TypeUse Case
EarliestReprocess all data (disaster recovery, data replay)
LatestSkip backlog, only process new messages
TimestampReplay from a specific point in time
OffsetPrecise control for specific partition offsets
Resetting offsets can cause message reprocessing or data loss. Ensure your application handles duplicate messages appropriately (idempotent processing).

​
How to Delete a Consumer Group

1

Stop All Consumers

Ensure no consumers are actively running. The group must be Empty or Dead.
2

Click Delete

Click the Delete Group button on the detail page.
3

Confirm

Confirm the deletion. This removes all offset data for the group.
Deleting a consumer group removes all committed offsets. If consumers restart with the same group ID, they will begin from the configured auto.offset.reset position (usually earliest or latest).

​
Troubleshooting

  • A consumer may be taking too long to process messages
  • Check for consumer application errors or crashes
  • Increase session.timeout.ms and heartbeat.interval.ms
  • Reduce max.poll.records if processing is slow
  • Check network connectivity between consumers and brokers
  • Add more consumer instances (up to partition count)
  • Optimize message processing in your application
  • Increase partition count to enable more parallelism
  • Check if consumers are committing offsets correctly
  • Verify no consumer application errors
  • Consumer group must be Empty or Dead, not Stable
  • Stop all consumer applications first
  • Wait for heartbeat timeout (default 10 seconds)
  • Verify no lingering consumer connections
  • Consumer group must not be in Stable state
  • Stop all consumers before resetting
  • You need write permission
  • Verify the topic and partitions exist
  • Consumer group may not have committed offsets yet
  • Consumer may be using manual offset management
  • Topic may have been recreated
  • Check if enable.auto.commit is configured correctly
  • Consumers may be using dynamic group membership
  • Short session.timeout.ms causes frequent disconnects
  • Application may be crashing and restarting
  • Check for resource constraints (memory, CPU) on consumer hosts
  • Default partition assignment strategies may not balance evenly
  • Consider using StickyAssignor or CooperativeStickyAssignor
  • Ensure consumer count doesn’t exceed partition count
  • Check if some consumers have different subscription patterns

​
FAQ

Maximum useful consumers equals the number of partitions across all subscribed topics. Extra consumers remain idle. For high availability, use N-1 consumers where N is partition count, so one consumer can handle failover.
After session.timeout.ms (default 10 seconds), the coordinator marks the consumer as dead and triggers a rebalance. Partitions are reassigned to remaining consumers. Some messages may be reprocessed depending on when offsets were last committed.
Auto-commit is simpler but may lose messages if the consumer crashes after processing but before commit. Manual commit gives precise control but requires careful implementation. Use manual commit for exactly-once semantics.
A rebalance redistributes partitions among consumers. It happens when: (1) Consumer joins or leaves, (2) Consumer fails heartbeat, (3) Topic partition count changes, (4) Subscription changes. During rebalance, consumption pauses briefly.
Controlled by broker config offsets.retention.minutes (default 7 days). If a consumer group is inactive for longer than this period, all offsets are deleted. Consumers will restart from auto.offset.reset position.
Yes. Each consumer group maintains its own offsets independently. This is useful for different applications processing the same data stream, or for fan-out patterns where multiple systems need the same messages.
Lag typically refers to the offset difference (messages behind). Consumer lag may also refer to time-based lag (how old the unprocessed messages are). Both indicate processing delays but from different perspectives.
Implement idempotent processing: (1) Use unique message IDs to detect duplicates, (2) Use database transactions with deduplication, (3) Store processed message IDs in a cache or database, (4) Design operations to be safe when repeated.

​
Best Practices

​
Consumer Configuration

# Recommended settings for production
session.timeout.ms=30000
heartbeat.interval.ms=10000
max.poll.records=500
max.poll.interval.ms=300000
enable.auto.commit=false
auto.offset.reset=earliest

​
Monitoring Strategy

Monitor these metrics continuously:
  • Consumer lag per partition (should be stable or decreasing)
  • Rebalance frequency (frequent rebalances indicate issues)
  • Commit rate (should match processing rate)
  • Consumer group state (should be Stable during normal operation)

​
High Availability

  • Run at least 2 consumers per group for failover
  • Don’t exceed partition count with consumers
  • Use rack-aware consumer assignment in multi-datacenter setups
  • Configure appropriate timeouts for your network environment

​
Offset Management

  • Commit offsets after successful processing, not before
  • Use synchronous commits for critical data
  • Consider storing offsets externally for exactly-once semantics
  • Implement dead letter queues for failed messages