What is Kafka's exactly-once semantics in multi-broker transactions?
Answer
Kafka transactions for exactly-once span multiple brokers via a Transaction Coordinator — a broker elected to coordinate a specific producer's transactions (determined by hashing the transactional.id to a partition of the __transaction_state topic). Flow: 1. InitProducerId: producer contacts transaction coordinator, gets a PID + epoch. 2. beginTransaction(): client-side only. 3. AddPartitionsToTxn: producer registers participating partitions with the coordinator. 4. Produce: messages are written to partition leaders with the transaction ID. 5. EndTransaction: coordinator writes PREPARE_COMMIT to its log, then sends WriteTxnMarkers to each partition leader to write a COMMIT marker. 6. CompleteTransaction: coordinator writes COMMITTED. Consumers with read_committed buffer messages until they see the commit marker. If the coordinator crashes, the new coordinator can recover in-flight transactions from the transaction log.
Previous
How do you implement a dead letter queue (DLQ) pattern in Kafka?
Next
What strategies exist for handling Kafka consumer failures in production?
More Apache Kafka Questions
View all →- Advanced How do you tune Kafka for ultra-low latency?
- Advanced What is Kafka's ISR (In-Sync Replicas) management and unclean leader election?
- Advanced What is Kafka's controller and how is leader election handled in KRaft mode?
- Advanced How do you implement a dead letter queue (DLQ) pattern in Kafka?
- Advanced What strategies exist for handling Kafka consumer failures in production?