📨 Apache Kafka Intermediate

How does Kafka handle back pressure?

Answer

Kafka's pull-based consumer model is its primary back pressure mechanism — consumers fetch messages at their own pace, so a slow consumer does not overwhelm itself; it simply accumulates lag. This is fundamentally different from push-based systems where a fast producer can overwhelm a slow consumer. Consumer-side controls: max.poll.records: limit messages per poll (default 500). fetch.min.bytes/fetch.max.wait.ms: control polling frequency. max.partition.fetch.bytes: limit data per partition per fetch. Producer-side controls: producers buffer messages and apply linger.ms and batch.size. If the producer's send buffer fills up, send() blocks until space is available (controlled by max.block.ms). For Kafka Streams: buffered.records.per.partition limits in-memory buffering. The key insight: Kafka's durability (disk-backed buffer) means the back pressure mechanism is simply the natural lag between producer and consumer, which is visible and measurable.