What is partition rebalancing in Kafka?

Partition rebalancing is the process of redistributing topic partition assignments across consumers in a group when the group membership changes (consumer joins, leaves, or crashes). During a rebalance: all consumers in the group stop processing (the "stop the world" problem) while the Group Coordinator broker orchestrates the new assignment. This causes latency spikes and potential duplicate processing (uncommitted offsets). Rebalance triggers: new consumer joins, consumer crashes or exceeds max.poll.interval.ms (took too long to poll), topic partition count changes. Improvements: Cooperative incremental rebalancing (introduced in Kafka 2.4): only revokes partitions that need to move, allowing other partitions to continue processing during the rebalance — dramatically reduces disruption. Enable with partition.assignment.strategy=CooperativeStickyAssignor.