Kafka consumer group rebalancing: causes and fixes

Your consumer lag climbs, logs fill with rebalance messages, and messages get processed twice. Constant rebalancing is one of the most common Kafka problems in Java microservices — and usually one of four causes.

What a rebalance is

Consumers in the same consumer group share a topic’s partitions. Whenever group membership changes — a consumer joins, leaves, or is considered dead — the group coordinator redistributes partitions. With the classic “eager” protocol, every consumer gives up all its partitions during the rebalance, so processing pauses for the whole group.

Symptoms

• Consumer lag rising even though consumers are running.
• Log lines such as “consumer poll timeout has expired” or “Attempt to heartbeat failed since group is rebalancing”.
• Duplicate processing: offsets for in-flight work weren’t committed before partitions moved.
• Rebalances that line up with deployments, autoscaling or traffic spikes.

The four common causes

1. Processing is slower than max.poll.interval.ms. If the time between two poll() calls exceeds this limit (five minutes by default), the consumer is removed from the group. A large max.poll.records (500 by default) combined with slow per-record work — a database call, an HTTP request — is the classic trigger.
2. Heartbeats stop arriving within session.timeout.ms. Long garbage-collection pauses, CPU starvation in a throttled container, or network trouble can make a healthy consumer look dead.
3. Restarts and scaling. Every rolling deployment or autoscaling event changes membership — twice per pod, once when it leaves and once when it comes back.
4. Crash loops. A consumer that throws on a “poison” message and restarts triggers a rebalance every time.

Fixes, in order of impact

1. Match the batch size to your processing time. Lower max.poll.records so a batch reliably finishes well inside max.poll.interval.ms, or raise the interval if slow processing is expected. Measure your real per-record time first.
2. Use cooperative rebalancing. The CooperativeStickyAssignor moves only the partitions that need to move, so the rest of the group keeps processing during a rebalance.
3. Use static membership for restarts. Giving each consumer a stable group.instance.id lets it restart within the session timeout without triggering a rebalance at all.
4. Handle poison messages. Catch per-record failures, retry a bounded number of times, then send the record to a dead-letter topic instead of crashing.
5. Make processing idempotent and commit after processing. Rebalances will still happen; idempotent handlers make the occasional duplicate harmless.

Newer Kafka versions also introduce a new consumer group protocol (KIP-848, available from Kafka 4.0) that moves assignment to the broker and makes rebalances incremental by design. It requires both broker and client support — worth evaluating if you control your cluster version.

Spring Kafka configuration

In a Spring Boot service, these settings live in your application configuration:

spring:
  kafka:
    consumer:
      group-id: order-service
      max-poll-records: 100
      properties:
        max.poll.interval.ms: 600000
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
        group.instance.id: ${HOSTNAME}

group.instance.id must be unique per consumer and stable across restarts — a StatefulSet pod name works well; a random pod name from a Deployment does not survive restarts. If your listener container runs several consumers (concurrency > 1), Spring Kafka adds a suffix per consumer so the IDs stay unique.

How to diagnose it

• Describe the group to see members, assignments and lag: kafka-consumer-groups.sh --bootstrap-server … --describe --group order-service
• Watch consumer metrics such as rebalance rate and time since the last rebalance; alert when rebalances spike outside deployments.
• Time your handler. Log processing time per batch; compare it with max.poll.interval.ms.
• Correlate with events — deployments, autoscaling, GC pauses, CPU throttling.