How do you tune Kafka for ultra-low latency?
Answer
For ultra-low latency (sub-millisecond to single-millisecond end-to-end): Producer: set linger.ms=0 (send immediately, no batching wait), batch.size=1 (minimize batching), compression.type=none (no compression overhead), acks=1 (leader-only ack). Broker: use fast NVMe SSDs, disable log.flush.interval.messages (rely on OS page cache), tune JVM GC (G1GC with MaxGCPauseMillis=20), use dedicated CPUs to reduce context switching. Consumer: set fetch.min.bytes=1 (return immediately even for one message), fetch.max.wait.ms=0. Network: place producers and consumers in the same data center/AZ as the broker leader. Enable socket.keepalive.enable=true. OS tuning: increase file descriptor limits, set vm.swappiness=1, use a high-performance network (10Gb+). Low-latency configurations sacrifice throughput and durability — apply only where latency is the primary constraint.
Previous
How does Kafka handle back pressure?
Next
What is Kafka's ISR (In-Sync Replicas) management and unclean leader election?
More Apache Kafka Questions
View all →- Advanced What is Kafka's ISR (In-Sync Replicas) management and unclean leader election?
- Advanced What is Kafka's controller and how is leader election handled in KRaft mode?
- Advanced How do you implement a dead letter queue (DLQ) pattern in Kafka?
- Advanced What is Kafka's exactly-once semantics in multi-broker transactions?
- Advanced What strategies exist for handling Kafka consumer failures in production?