What is the Kafka idempotent producer?
Answer
The idempotent producer, enabled with enable.idempotence=true, prevents duplicate messages when the producer retries after a transient failure. Without it, a retry after an acknowledged message (where the ack was lost in transit) creates a duplicate. The idempotent producer assigns each message a sequence number and a Producer ID (PID). Brokers track the last sequence number from each producer for each partition. If a broker receives a message with a sequence number it has already seen, it discards the duplicate silently. This provides exactly-once semantics for a single producer session on a single topic partition. Enabling idempotence requires acks=all, max.in.flight.requests.per.connection ≤ 5, and retries > 0. It is the foundation for Kafka Transactions.