How do you implement a dead letter queue (DLQ) pattern in Kafka?
Answer
A dead letter queue (DLQ) in Kafka handles messages that cannot be processed after exhausting retries, preventing them from blocking the consumer or being silently dropped. Implementation: Consumer retry loop: catch processing exceptions, retry N times. After N failures, produce the failed message to a dedicated topic-name.DLQ topic. Include the original message plus error metadata (exception, timestamp, consumer group, original offset) in headers or envelope. Kafka Connect DLQ: built-in DLQ support in connectors — set errors.deadletterqueue.topic.name=topic.DLQ and errors.tolerance=all. Spring Kafka: configure DefaultErrorHandler with a DeadLetterPublishingRecoverer. Monitor the DLQ and alert when messages appear — they indicate a systemic processing failure. Build a DLQ replay tool to reprocess messages after fixing the underlying bug.
Previous
What is Kafka's controller and how is leader election handled in KRaft mode?
Next
What is Kafka's exactly-once semantics in multi-broker transactions?
More Apache Kafka Questions
View all →- Advanced How do you tune Kafka for ultra-low latency?
- Advanced What is Kafka's ISR (In-Sync Replicas) management and unclean leader election?
- Advanced What is Kafka's controller and how is leader election handled in KRaft mode?
- Advanced What is Kafka's exactly-once semantics in multi-broker transactions?
- Advanced What strategies exist for handling Kafka consumer failures in production?