Key Concepts
Broker
A Kafka server node that stores topic partitions and handles produce/consume requests from clients.
Controller
One broker in the cluster is elected as controller. It manages partition leader elections and replica assignments.
Rack Awareness
Brokers can be assigned to racks for fault tolerance. Kafka distributes replicas across racks when possible.
Endpoints
Each broker exposes one or more listener endpoints with different security protocols (PLAINTEXT, SSL, SASL_SSL).
Required Permissions
| Action | Permission |
|---|---|
| View brokers | iam:project:infrastructure:kafka:read |
| Update broker configuration | iam:project:infrastructure:kafka:admin |
How to View Brokers
View Broker List
The table shows all brokers in the cluster with their host, port, and controller status.
How to View Broker Configuration
Configuration Sources
Broker configurations come from different sources with varying precedence:| Source | Description |
|---|---|
| Dynamic | Changed at runtime via Kafka Admin API |
| Dynamic Default | Dynamic config applied to all brokers |
| Static | Set in server.properties file |
| Default | Kafka’s built-in default value |
Dynamic configurations take precedence over static configurations. Changes made through the UI create dynamic configurations.
Understanding Configuration Metadata
Each configuration entry includes metadata:- Read-Only: Cannot be modified (Kafka internal settings)
- Sensitive: Value is hidden for security (passwords, keys)
- Source: Where the current value comes from
Common Broker Configurations
Performance Tuning
| Configuration | Purpose |
|---|---|
num.io.threads | Number of threads for network I/O |
num.network.threads | Number of threads for request processing |
socket.receive.buffer.bytes | Socket receive buffer size |
socket.send.buffer.bytes | Socket send buffer size |
Log Management
| Configuration | Purpose |
|---|---|
log.retention.hours | How long to retain messages |
log.segment.bytes | Maximum size of a log segment |
log.cleanup.policy | How to handle old segments (delete/compact) |
Replication
| Configuration | Purpose |
|---|---|
default.replication.factor | Default replicas for new topics |
min.insync.replicas | Minimum ISR for acks=all writes |
Broker Detail Information
The detail page shows:Connection Details
- Host: Broker hostname or IP address
- Port: Listener port number
- Rack: Physical rack assignment (if configured)
- Role: Controller or Follower status
Partition Statistics
- Leader Partitions: Number of partitions where this broker is leader
- Follower Partitions: Number of partitions where this broker is a replica
Endpoints
List of listener endpoints showing security protocol for each (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL).Troubleshooting
Broker not appearing in list
Broker not appearing in list
- Verify the broker is running and connected to the cluster
- Check network connectivity between the API server and brokers
- Ensure the bootstrap servers configuration includes reachable addresses
- The broker may be in a failed state; check Kafka logs
Cannot view broker configurations
Cannot view broker configurations
- You need read permission for Kafka
- The broker may be unreachable
- Check if the Kafka cluster is healthy
Configuration change failed
Configuration change failed
- You need admin permission to modify broker configurations
- Some configurations are read-only and cannot be changed
- The value may be invalid for the configuration type
- Check if the broker is accepting connections
Broker shows as offline
Broker shows as offline
- The broker process may have crashed
- Network partition between brokers
- Check disk space on the broker node
- Review broker logs for errors
High leader partition count on one broker
High leader partition count on one broker
- Run partition reassignment to balance leaders
- Check if other brokers are healthy
- Consider using Kafka’s
kafka-leader-election.shtool
FAQ
Can I add or remove brokers from this UI?
Can I add or remove brokers from this UI?
No. Broker membership is managed by Kafka itself. To add brokers, start new Kafka instances pointing to the same ZooKeeper/KRaft cluster. To remove brokers, gracefully shut them down after reassigning their partitions.
What does the Controller broker do?
What does the Controller broker do?
The Controller is elected among brokers and handles administrative tasks: partition leader elections, tracking cluster membership, and managing topic/partition metadata. If the controller fails, another broker is elected.
Why are some configurations read-only?
Why are some configurations read-only?
Read-only configurations are either Kafka internals or require a broker restart to change. These must be modified in the broker’s server.properties file and require a restart.
What are sensitive configurations?
What are sensitive configurations?
Configurations marked as sensitive contain security-related values like passwords or API keys. Their values are hidden in the UI and API responses for security.
How do dynamic configurations work?
How do dynamic configurations work?
Dynamic configurations are stored in ZooKeeper/KRaft and applied without broker restart. They override static configurations in server.properties. Changes take effect immediately.
What is rack awareness?
What is rack awareness?
Rack awareness lets Kafka spread partition replicas across different failure domains (racks, availability zones). Configure
broker.rack in server.properties and ensure topics have enough replicas to span racks.How do I identify performance issues on a broker?
How do I identify performance issues on a broker?
Check leader partition count (unbalanced distribution increases load), monitor network I/O thread configuration, and review follower partition lag. High partition counts on a single broker may indicate need for rebalancing.
Best Practices
Monitor Broker Balance
Ensure leader partitions are evenly distributed across brokers. An unbalanced cluster puts excessive load on certain brokers.Use Rack Awareness
In multi-datacenter or multi-AZ deployments, configurebroker.rack to ensure replicas are distributed for fault tolerance.
Review Configuration Changes
Before modifying broker configurations:- Understand the impact on cluster performance
- Test changes in a non-production environment
- Monitor broker metrics after changes
- Document configuration changes for audit purposes
Regular Health Checks
Monitor broker health regularly:- Check that all brokers are online
- Verify controller election is stable
- Monitor partition leader distribution
- Watch for under-replicated partitions