Kafka KIP-848: the new consumer rebalance protocol explained

Kafka 4.0 made the next-generation consumer rebalance protocol — KIP-848 — generally available. It moves partition assignment from the clients to the broker and replaces the group-wide “stop, rejoin, sync” cycle with small, per-member updates. For large or frequently scaling consumer groups, rebalance storms largely disappear. This guide explains how it works, what configuration changes, and how to migrate a running group safely.

What was wrong with classic rebalancing

In the classic protocol, every rebalance is a group-wide barrier. All members send JoinGroup, wait until everyone has joined, one elected leader computes the assignment on the client side, and the result is distributed with SyncGroup. Consequences:

• One slow member stalls everyone — the group waits for the slowest consumer to rejoin.
• Eager assignors stop the world: every member revokes every partition before the new assignment. Cooperative-sticky assignment reduced this, but still needs multiple rounds.
• Deploys and autoscaling trigger cascades: each pod restart in a rolling deploy can cause another full rebalance.
• Assignment logic lives in clients, so every client library must implement it identically.

If you are fighting these symptoms today, our guide to diagnosing consumer rebalancing covers the classic protocol’s tuning knobs.

How KIP-848 works

• The broker owns the assignment. The group coordinator runs a server-side assignor and computes a target assignment whenever membership or subscriptions change.
• One RPC replaces JoinGroup/SyncGroup/Heartbeat. Each consumer sends periodic ConsumerGroupHeartbeat requests; the response tells it which partitions it should own.
• Members reconcile independently. A consumer revokes only partitions that are moving away, and receives a new partition only after its previous owner has released it. Everyone else keeps consuming.
• Epochs replace generations. The group epoch increases when the target assignment changes; each member’s epoch tracks how far it has reconciled.

There is no global barrier anymore: a slow consumer only delays the partitions it is giving up, not the whole group.

Enabling it

Requirements: Kafka 4.0+ brokers (KRaft mode — ZooKeeper is gone in 4.0) and a client that supports the protocol (the Java client 4.0+, or a recent librdkafka-based client). The broker side is on by default in 4.0. The client still defaults to the classic protocol, so you opt in per consumer:

group.id=billing
group.protocol=consumer
# Optional: choose the server-side assignor (uniform or range)
group.remote.assignor=uniform

The subscription API doesn’t change — subscribe(), poll() and ConsumerRebalanceListener work as before. Regex subscriptions can use the new SubscriptionPattern type, which the broker evaluates with RE2J syntax.

Configs that change

Leaving one of the disallowed properties in place makes the consumer fail at startup with a configuration error — the most common first problem teams hit. Per-group timeouts can be set without a broker restart:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type groups --entity-name billing \
  --add-config consumer.session.timeout.ms=30000

Spring Kafka

With Spring Boot, pass the protocol through the consumer properties and remove the settings that are now broker-side:

spring:
  kafka:
    consumer:
      group-id: billing
      properties:
        group.protocol: consumer
        group.remote.assignor: uniform
        # remove these if present — they now fail startup:
        # session.timeout.ms, heartbeat.interval.ms, partition.assignment.strategy

Use a Spring for Apache Kafka version built on the 4.x client. Listener containers, error handlers and ConsumerAwareRebalanceListener callbacks keep working; you’ll simply see them fire for the few partitions that actually move instead of for the whole assignment.

Migrating an existing group

A running classic group can be converted online with a rolling restart: each restarted consumer joins with group.protocol=consumer, and the coordinator converts the group once the first new-protocol member joins, while still serving the remaining classic members. Before starting:

1. Confirm the broker’s group.consumer.migration.policy allows upgrades (and downgrades, if you want a rollback path).
2. Remove the disallowed client properties from the new configuration.
3. Move session and heartbeat timeouts to group or broker configs so behaviour stays the same after the switch.
4. Roll consumers one at a time and watch lag and assignment as you go:

# Group type and protocol for every group
kafka-groups.sh --bootstrap-server localhost:9092 --list

# Members, their assignments and lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group billing --members --verbose

To roll back, restart consumers with group.protocol=classic; the group converts back once the last new-protocol member leaves, if the migration policy permits downgrades.

Gotchas

• Managed Kafka lags upstream. Amazon MSK, Confluent Cloud and others enable KIP-848 on their own schedule — check your provider’s supported versions before planning.
• Custom client-side assignors don’t carry over. If you wrote your own ConsumerPartitionAssignor, you must pick a server-side assignor or stay on the classic protocol.
• Mixed client fleets. Every consumer in a group must eventually support the new protocol; an old client version in one service keeps the group in mixed mode.
• Kafka Streams is separate. Streams applications use their own rebalance protocol work (KIP-1071); group.protocol=consumer applies to plain consumers.
• Long processing still hurts. Exceeding max.poll.interval.ms still removes the member; the new protocol makes the resulting reassignment cheaper, not unnecessary.

Should you switch?

If you run Kafka 4.x with Java clients, yes — especially for groups with many members, frequent deploys or autoscaling consumers on Kubernetes, where classic rebalances caused visible lag spikes. Start with one non-critical group, compare rebalance duration and lag during a rolling deploy, then roll it out group by group. If you’re on Kafka 3.x or a managed service that doesn’t offer it yet, cooperative-sticky assignment plus static membership remains the best classic configuration.