> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Consumer Groups

> Monitor consumer group health, lag, and manage offsets

Consumer Groups coordinate multiple consumers reading from the same topics. Kafka tracks which messages each group has consumed, enabling parallel processing while ensuring each message is processed once per group.

## Key Concepts

<CardGroup cols={2}>
  <Card title="Consumer Group" icon="users">
    A logical grouping of consumers that work together to consume messages from topics. Each partition is assigned to one consumer in the group.
  </Card>

  <Card title="Offset" icon="bookmark">
    A position marker indicating which messages have been consumed. Kafka stores offsets per partition per consumer group.
  </Card>

  <Card title="Lag" icon="gauge">
    The difference between the latest message in a partition and the last consumed message. High lag indicates consumers are falling behind.
  </Card>

  <Card title="Coordinator" icon="server">
    A broker responsible for managing the consumer group's membership and partition assignments.
  </Card>
</CardGroup>

## Required Permissions

| Action                 | Permission                                |
| ---------------------- | ----------------------------------------- |
| View consumer groups   | `iam:project:infrastructure:kafka:read`   |
| Reset offsets          | `iam:project:infrastructure:kafka:write`  |
| Delete consumer groups | `iam:project:infrastructure:kafka:delete` |

## Consumer Group States

| State                   | Description                                                               |
| ----------------------- | ------------------------------------------------------------------------- |
| **Stable**              | Group is active with assigned partitions. Members are consuming normally. |
| **Empty**               | Group exists but has no active members. Offsets are still retained.       |
| **PreparingRebalance**  | Group is preparing to reassign partitions due to member changes.          |
| **CompletingRebalance** | Group is finalizing partition assignments after rebalance.                |
| **Dead**                | Group has been deleted or expired due to inactivity.                      |

<Info>
  Groups in **Stable** state cannot be deleted or have their offsets reset. Stop all consumers first.
</Info>

## How to View Consumer Groups

<Steps>
  <Step title="Select Kafka Cluster">
    Choose the target Kafka cluster from the dropdown.
  </Step>

  <Step title="Browse Groups">
    The list shows all consumer groups with their state, member count, and lag.
  </Step>

  <Step title="Filter by State">
    Use the state filter to show only groups in specific states (Stable, Empty, Rebalancing, Dead).
  </Step>

  <Step title="Open Details">
    Click on a group to view detailed information including members, topics, and offsets.
  </Step>
</Steps>

## How to View Consumer Group Details

The detail page provides three tabs:

### Overview Tab

Shows subscribed topics and general group information:

* Group ID
* Current state
* Member count
* Coordinator broker

### Members Tab

Lists all active consumers in the group:

* **Client ID**: Application identifier
* **Client Host**: IP address or hostname
* **Member ID**: Unique member identifier assigned by Kafka
* **Assigned Partitions**: Topic:partition pairs assigned to this member

### Offsets Tab

Shows consumption progress for each partition:

* **Current Offset**: Last committed offset
* **Log End Offset**: Latest message position in the partition
* **Lag**: Messages waiting to be consumed

## Understanding Lag

Lag indicates how far behind a consumer group is from the latest messages.

| Lag Level        | Meaning  | Action                                 |
| ---------------- | -------- | -------------------------------------- |
| **0-1,000**      | Healthy  | Consumers are keeping up               |
| **1,000-10,000** | Warning  | Monitor closely, may need attention    |
| **> 10,000**     | Critical | Consumers falling behind significantly |

<Warning>
  Consistently high lag can cause data processing delays and may lead to data loss if retention expires before messages are consumed.
</Warning>

### Common Causes of High Lag

* Consumer processing is too slow
* Not enough consumers for the number of partitions
* Consumer application errors causing reprocessing
* Network issues between consumers and brokers
* Unbalanced partition distribution

## How to Reset Consumer Offsets

Resetting offsets allows you to reprocess messages or skip ahead.

<Steps>
  <Step title="Stop All Consumers">
    Ensure no consumers are actively running. The group must not be in **Stable** state.
  </Step>

  <Step title="Open Consumer Group Details">
    Navigate to the consumer group detail page.
  </Step>

  <Step title="Choose Reset Method">
    Select how to reset offsets:

    * **Earliest**: Reprocess all available messages from the beginning
    * **Latest**: Skip to the end, only consume new messages
    * **Timestamp**: Reset to a specific point in time
    * **Offset**: Reset to a specific offset number
  </Step>

  <Step title="Apply Reset">
    Click **Reset Offsets** to apply the changes.
  </Step>

  <Step title="Restart Consumers">
    Start your consumer applications. They will begin consuming from the new offset positions.
  </Step>
</Steps>

### Reset Types

| Type          | Use Case                                            |
| ------------- | --------------------------------------------------- |
| **Earliest**  | Reprocess all data (disaster recovery, data replay) |
| **Latest**    | Skip backlog, only process new messages             |
| **Timestamp** | Replay from a specific point in time                |
| **Offset**    | Precise control for specific partition offsets      |

<Warning>
  Resetting offsets can cause message reprocessing or data loss. Ensure your application handles duplicate messages appropriately (idempotent processing).
</Warning>

## How to Delete a Consumer Group

<Steps>
  <Step title="Stop All Consumers">
    Ensure no consumers are actively running. The group must be **Empty** or **Dead**.
  </Step>

  <Step title="Click Delete">
    Click the **Delete Group** button on the detail page.
  </Step>

  <Step title="Confirm">
    Confirm the deletion. This removes all offset data for the group.
  </Step>
</Steps>

<Warning>
  Deleting a consumer group removes all committed offsets. If consumers restart with the same group ID, they will begin from the configured `auto.offset.reset` position (usually earliest or latest).
</Warning>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Consumer group stuck in rebalancing" icon="circle-question">
    * A consumer may be taking too long to process messages
    * Check for consumer application errors or crashes
    * Increase `session.timeout.ms` and `heartbeat.interval.ms`
    * Reduce `max.poll.records` if processing is slow
    * Check network connectivity between consumers and brokers
  </Accordion>

  <Accordion title="Consumer lag keeps increasing" icon="circle-question">
    * Add more consumer instances (up to partition count)
    * Optimize message processing in your application
    * Increase partition count to enable more parallelism
    * Check if consumers are committing offsets correctly
    * Verify no consumer application errors
  </Accordion>

  <Accordion title="Cannot delete consumer group" icon="circle-question">
    * Consumer group must be Empty or Dead, not Stable
    * Stop all consumer applications first
    * Wait for heartbeat timeout (default 10 seconds)
    * Verify no lingering consumer connections
  </Accordion>

  <Accordion title="Cannot reset offsets" icon="circle-question">
    * Consumer group must not be in Stable state
    * Stop all consumers before resetting
    * You need write permission
    * Verify the topic and partitions exist
  </Accordion>

  <Accordion title="Offsets show as 0 or negative" icon="circle-question">
    * Consumer group may not have committed offsets yet
    * Consumer may be using manual offset management
    * Topic may have been recreated
    * Check if `enable.auto.commit` is configured correctly
  </Accordion>

  <Accordion title="Consumer group appears and disappears" icon="circle-question">
    * Consumers may be using dynamic group membership
    * Short `session.timeout.ms` causes frequent disconnects
    * Application may be crashing and restarting
    * Check for resource constraints (memory, CPU) on consumer hosts
  </Accordion>

  <Accordion title="Partitions not evenly distributed" icon="circle-question">
    * Default partition assignment strategies may not balance evenly
    * Consider using `StickyAssignor` or `CooperativeStickyAssignor`
    * Ensure consumer count doesn't exceed partition count
    * Check if some consumers have different subscription patterns
  </Accordion>
</AccordionGroup>

## FAQ

<AccordionGroup>
  <Accordion title="How many consumers should I have?">
    Maximum useful consumers equals the number of partitions across all subscribed topics. Extra consumers remain idle. For high availability, use N-1 consumers where N is partition count, so one consumer can handle failover.
  </Accordion>

  <Accordion title="What happens when a consumer crashes?">
    After `session.timeout.ms` (default 10 seconds), the coordinator marks the consumer as dead and triggers a rebalance. Partitions are reassigned to remaining consumers. Some messages may be reprocessed depending on when offsets were last committed.
  </Accordion>

  <Accordion title="Should I use auto-commit or manual commit?">
    **Auto-commit** is simpler but may lose messages if the consumer crashes after processing but before commit. **Manual commit** gives precise control but requires careful implementation. Use manual commit for exactly-once semantics.
  </Accordion>

  <Accordion title="What is a rebalance and why does it happen?">
    A rebalance redistributes partitions among consumers. It happens when: (1) Consumer joins or leaves, (2) Consumer fails heartbeat, (3) Topic partition count changes, (4) Subscription changes. During rebalance, consumption pauses briefly.
  </Accordion>

  <Accordion title="How long are offsets retained?">
    Controlled by broker config `offsets.retention.minutes` (default 7 days). If a consumer group is inactive for longer than this period, all offsets are deleted. Consumers will restart from `auto.offset.reset` position.
  </Accordion>

  <Accordion title="Can I have multiple consumer groups reading the same topic?">
    Yes. Each consumer group maintains its own offsets independently. This is useful for different applications processing the same data stream, or for fan-out patterns where multiple systems need the same messages.
  </Accordion>

  <Accordion title="What's the difference between lag and consumer lag?">
    **Lag** typically refers to the offset difference (messages behind). **Consumer lag** may also refer to time-based lag (how old the unprocessed messages are). Both indicate processing delays but from different perspectives.
  </Accordion>

  <Accordion title="How do I handle message reprocessing after offset reset?">
    Implement idempotent processing: (1) Use unique message IDs to detect duplicates, (2) Use database transactions with deduplication, (3) Store processed message IDs in a cache or database, (4) Design operations to be safe when repeated.
  </Accordion>
</AccordionGroup>

## Best Practices

### Consumer Configuration

```properties theme={null}
# Recommended settings for production
session.timeout.ms=30000
heartbeat.interval.ms=10000
max.poll.records=500
max.poll.interval.ms=300000
enable.auto.commit=false
auto.offset.reset=earliest
```

### Monitoring Strategy

Monitor these metrics continuously:

* **Consumer lag** per partition (should be stable or decreasing)
* **Rebalance frequency** (frequent rebalances indicate issues)
* **Commit rate** (should match processing rate)
* **Consumer group state** (should be Stable during normal operation)

### High Availability

* Run at least 2 consumers per group for failover
* Don't exceed partition count with consumers
* Use rack-aware consumer assignment in multi-datacenter setups
* Configure appropriate timeouts for your network environment

### Offset Management

* Commit offsets after successful processing, not before
* Use synchronous commits for critical data
* Consider storing offsets externally for exactly-once semantics
* Implement dead letter queues for failed messages