Top 44 Apache Kafka Interview Questions & Answers (2026)

Apache Kafka is an open-source, distributed event streaming platform originally developed at LinkedIn and donated to the Apache Software Foundation. It is designed for high-throughput, fault-tolerant, real-time data streaming and can handle millions of messages per second. Kafka acts as a distributed commit log — messages are written to disk and replicated, enabling durability and replay. It decouples data producers from consumers, enabling asynchronous, scalable architectures. Kafka is used for real-time analytics, event sourcing, log aggregation, stream processing, and as a backbone for microservice communication. It is one of the most widely deployed pieces of infrastructure in modern data-intensive systems.

Kafka consists of several core components. Broker: a Kafka server that stores messages and serves producer/consumer requests. A Kafka cluster is a group of brokers. Topic: a named category/feed to which records are published. Analogous to a table in a database. Partition: a topic is divided into partitions for parallelism and scalability. Producer: a client that publishes (writes) messages to a topic. Consumer: a client that subscribes to topics and reads messages. Consumer Group: a group of consumers that jointly consume a topic, with each partition assigned to exactly one consumer in the group. ZooKeeper (legacy): managed cluster metadata. KRaft (modern): Kafka's built-in metadata consensus, replacing ZooKeeper. Offset: an integer identifying a message's position within a partition.

A Kafka Topic is a named, ordered, and persistent log of messages. It is the central abstraction in Kafka — producers write to topics and consumers read from topics. Think of a topic as a database table or a message queue channel. Key properties: Name: identifies the stream (e.g., user-events, order-created). Partitions: topics are split into partitions for parallel processing. Replication factor: how many copies of each partition exist across brokers. Retention: how long messages are kept (time-based, e.g., 7 days, or size-based). Unlike traditional queues, Kafka topics are durable and replayable — messages are not deleted after consumption; multiple consumers can read the same messages independently. This immutable log is the foundation of Kafka's "event store" capability.

A Partition is the unit of parallelism in Kafka. Each topic is divided into one or more ordered, immutable sequences of messages. Key properties: Ordering: messages within a partition are strictly ordered by offset. Parallelism: different partitions can be read/written by different consumers simultaneously — more partitions = higher throughput. Scalability: partitions are distributed across brokers in the cluster. Assignment: each partition is assigned to one broker as the leader (handles all reads/writes) and replicated to other brokers as followers. Choosing the right number of partitions involves trade-offs: more partitions allow higher throughput but increase end-to-end latency, memory usage, and failover time. You cannot decrease the number of partitions after creation.

A Kafka Producer is a client that publishes (writes) messages to Kafka topics. Key concepts: Key: optional message attribute used to determine partition routing — messages with the same key always go to the same partition, enabling per-key ordering. Value: the message payload (bytes). Partition selection: if a key is provided, a hash of the key selects the partition; if no key, partitions are assigned in round-robin. Batching: producers buffer messages and send in batches for efficiency (batch.size, linger.ms). Acknowledgements: acks=0 (fire-and-forget), acks=1 (leader confirms), acks=all (all in-sync replicas confirm — most durable). Compression: producers can compress messages (gzip, snappy, lz4) to reduce network bandwidth.

A Kafka Consumer reads messages from one or more topic partitions. Key concepts: Offset: the consumer tracks its position in each partition via the offset — the sequence number of the last processed message. Consumers commit offsets to the __consumer_offsets internal topic. Auto-commit: by default, offsets are committed automatically every 5 seconds. Manual commit: commit after processing for exactly-once-like behavior. Pull model: Kafka uses a pull model — consumers request messages from the broker at their own pace, preventing overload. Consumer Group: consumers in a group share the topic partitions. Rebalancing: when a consumer joins or leaves a group, partitions are redistributed. Seek: consumers can reposition their offset to replay or skip messages.

A Consumer Group is a set of Kafka consumers that cooperate to consume a topic. Each partition in a topic is assigned to exactly one consumer in the group at any time — this is the key rule. This enables parallel processing: with N partitions and N consumers in a group, all consumers process simultaneously. If there are more consumers than partitions, some consumers sit idle. If there are fewer consumers than partitions, some consumers handle multiple partitions. Multiple independent consumer groups can read the same topic simultaneously — each group maintains its own offsets and receives all messages independently. This is Kafka's publish-subscribe model: one topic, many independent consumer groups. Consumer groups enable both load balancing (multiple consumers in one group) and broadcasting (multiple groups).

A Kafka Broker is a Kafka server — a node in the Kafka cluster. It stores messages in topic partitions on disk, serves producer writes and consumer reads, and replicates data to other brokers. Each broker in a cluster has a unique broker.id. Key responsibilities: Partition leadership: each partition has one broker as the leader (handles all reads and writes) and others as followers (replicate data). Message storage: messages are stored in segment files on disk, organized by topic and partition. Client coordination: brokers provide metadata (which broker is the leader for each partition) to clients. A typical production cluster has 3–12 brokers. One broker acts as the Controller (elected leader among brokers) responsible for partition leader elections and cluster metadata.

An Offset is a unique, monotonically increasing integer that identifies a message's position within a partition. Offsets start at 0. Key uses: Consumer position tracking: consumers commit their current offset to know where to resume after a restart or rebalance. Message addressing: you can fetch a specific message by partition + offset. Replay: reset a consumer's offset to an earlier position to reprocess historical messages. Consumers can seek to specific offsets: seekToBeginning() for the oldest available message, seekToEnd() for new messages only, or a specific offset for replay. The __consumer_offsets internal topic stores committed offsets durably. Understanding offsets is fundamental to Kafka's delivery semantics — offset management determines whether you get at-least-once, at-most-once, or exactly-once processing.

Kafka replicates each partition across multiple brokers for fault tolerance. The replication factor (typically 3) determines how many copies exist. One broker is the partition leader — all reads and writes go to the leader. Other brokers are followers that replicate the leader's log. In-Sync Replicas (ISR): the set of replicas fully caught up with the leader. If the leader fails, a new leader is elected from the ISR. A message is considered "committed" when all ISR replicas have written it. This ensures no data loss when any ISR member becomes the new leader. Configure min.insync.replicas=2 with acks=all to require at least 2 ISR acknowledgements — providing strong durability guarantees. If an ISR falls behind, it is removed from the ISR set until it catches up.

ZooKeeper was Kafka's metadata management system for many years, responsible for: Broker registration: tracking which brokers are alive. Controller election: selecting which broker acts as the cluster controller. Topic and partition metadata: storing topic configurations and partition leader assignments. Consumer group coordination: (in older Kafka versions) tracking consumer offsets. ZooKeeper was a dependency that added operational complexity and scaling limitations. Starting with Kafka 2.8 (preview) and stable in Kafka 3.3+, KRaft (Kafka Raft) replaces ZooKeeper — Kafka manages its own metadata using a Raft-based consensus protocol internally. KRaft dramatically simplifies deployment (one system to manage), improves startup time, and enables clusters with millions of partitions (ZooKeeper struggled beyond ~200K).

Kafka and RabbitMQ serve different use cases. Retention: Kafka retains messages for a configurable period (days/weeks) regardless of consumption — messages are replayable. RabbitMQ deletes messages after they are consumed. Throughput: Kafka handles millions of messages/second through sequential disk I/O and batching. RabbitMQ is optimized for lower throughput with complex routing. Consumer model: Kafka uses a pull model; RabbitMQ pushes messages to consumers. Ordering: Kafka guarantees ordering per partition. RabbitMQ guarantees ordering per queue with some exceptions. Use cases: Kafka excels for event streaming, log aggregation, and replay. RabbitMQ excels for task queues, complex routing (direct, topic, fanout exchanges), and RPC patterns. Scale: Kafka is designed for large-scale distributed streaming; RabbitMQ is simpler to operate at moderate scale.

A Kafka message (record) consists of several fields: Key (optional, bytes): used for partition routing and log compaction. Messages with the same key always go to the same partition. Value (bytes): the actual message payload. Can be any format — JSON, Avro, Protobuf, plain text. Topic: the topic it belongs to. Partition: which partition it is stored in. Offset: its position in the partition. Timestamp: either the producer-set creation time (CreateTime) or the broker-set ingestion time (LogAppendTime). Headers (optional, key-value pairs): metadata about the message — correlation IDs, content type, source system. Serialization: keys and values must be serialized to bytes — use Avro with Schema Registry for schema evolution or JSON for simplicity.

Log compaction is a Kafka retention policy that retains the latest value for each message key indefinitely, discarding older records with the same key. Enable it with cleanup.policy=compact on a topic. Unlike time-based or size-based retention (which deletes old messages based on age/size), compaction keeps at least the last message for every key — like a key-value store embedded in Kafka. Compaction runs in the background as a periodic cleanup job. Use cases: Database change data capture (CDC) — a compacted topic of userId → userRecord gives the latest state for every user. Application configuration topics. Event sourcing materialized views. A key with a null value is a tombstone — after compaction, both the tombstone and the key are deleted, effectively deleting the entry. Compaction and time-based retention can be combined.

The acks producer configuration controls how many broker acknowledgements the producer requires before considering a write successful. acks=0: fire-and-forget. The producer sends the message without waiting for any acknowledgement. Maximum throughput, zero durability. acks=1 (default): the partition leader acknowledges after writing to its local log. If the leader fails after acknowledgement but before followers replicate, the message is lost. Balanced throughput and durability. acks=all (or -1): the leader waits until all in-sync replicas (ISR) have written the message before acknowledging. Combined with min.insync.replicas=2, this provides the strongest durability — no message loss unless a majority of replicas fail simultaneously. Use acks=all for financial, order, and critical data. Lower acks settings sacrifice durability for throughput.

Kafka topic retention defines how long messages are kept before being deleted. Two retention strategies: Time-based: retention.ms — delete segments older than the configured duration (e.g., 604800000 = 7 days). Default is 7 days. Size-based: retention.bytes — delete the oldest segments when the total partition size exceeds the limit. Both can be combined — whichever limit is hit first triggers deletion. Key nuances: messages are deleted at the segment level, not individually — an entire segment file is deleted once all its messages are past the retention threshold. The active segment (being written to) is never deleted. Log compaction (cleanup.policy=compact) is an alternative — keep the latest value per key instead of time/size-based deletion. For compliance scenarios, use very long retention or S3 tiered storage.

The Confluent Schema Registry (standard in the Kafka ecosystem) is a centralized service that stores and enforces schemas for Kafka messages. It supports Avro, Protobuf, and JSON Schema. How it works: producers register a schema before producing; the Schema Registry assigns a schema ID. The producer embeds the schema ID (4 bytes) in each message. Consumers retrieve the schema from the registry using this ID to deserialize the message. Benefits: Schema evolution: the registry enforces compatibility rules (backward, forward, full) when schemas change — preventing breaking consumers by adding fields without defaults. Documentation: the registry is a catalog of all message formats. Compact messages: schemas are shared via ID rather than inlined per message, reducing message size. Schema Registry is essential in production systems where many teams share Kafka topics.

Kafka Connect is a scalable, reliable framework for streaming data between Kafka and other systems without writing custom code. It provides a plugin-based connector system. Source Connectors: read data from an external system and write it to Kafka topics. Examples: Debezium (CDC from databases), S3 Source, JDBC Source (read from any SQL database). Sink Connectors: read from Kafka topics and write to external systems. Examples: S3 Sink, Elasticsearch Sink, JDBC Sink (write to SQL databases), BigQuery Sink. Workers: Connect runs as distributed workers that execute connectors. Converters: serialize/deserialize data (Avro, JSON, Protobuf). Hundreds of connectors are available from Confluent Hub as open-source or commercial plugins. Connect eliminates most ETL pipeline boilerplate code.

Kafka Streams is a client library for building real-time stream processing applications using Kafka as both input and output. Unlike batch processing (process, then output), Kafka Streams processes data as it arrives. Key features: Topology: a graph of processing nodes (source, processor, sink). High-level DSL: operations like filter(), map(), groupBy(), aggregate(), join(). State stores: local, fault-tolerant RocksDB state for stateful operations (counts, aggregates). Windowing: time-window operations (tumbling, hopping, session). No separate cluster: Kafka Streams is a library that runs inside your application — no external processing cluster (unlike Flink or Spark). Fault tolerance: state is backed up to Kafka changelog topics. Kafka Streams is ideal for Java/Scala applications needing stream processing without operational complexity.

Kafka supports three delivery guarantees. At-most-once: messages may be lost but never duplicated. Use acks=0 or acks=1 without retries. Consumer commits offsets before processing — if processing fails, the message is lost. Acceptable for metrics/logging where loss is tolerable. At-least-once: messages are never lost but may be duplicated. Use acks=all with retries. Consumer commits offsets after processing — if crash before commit, the message is reprocessed. Requires idempotent consumers. Default production pattern. Exactly-once: no loss, no duplication. Use Kafka's idempotent producer (enable.idempotence=true) and transactions for atomic read-process-write. Required for financial transactions. Kafka Streams implements exactly-once semantics with processing.guarantee=exactly_once_v2. Exactly-once has performance overhead but is necessary for correctness in some scenarios.

The idempotent producer, enabled with enable.idempotence=true, prevents duplicate messages when the producer retries after a transient failure. Without it, a retry after an acknowledged message (where the ack was lost in transit) creates a duplicate. The idempotent producer assigns each message a sequence number and a Producer ID (PID). Brokers track the last sequence number from each producer for each partition. If a broker receives a message with a sequence number it has already seen, it discards the duplicate silently. This provides exactly-once semantics for a single producer session on a single topic partition. Enabling idempotence requires acks=all, max.in.flight.requests.per.connection ≤ 5, and retries > 0. It is the foundation for Kafka Transactions.

Kafka Transactions enable atomic writes across multiple partitions and topics — all messages in a transaction are either all committed or all aborted. Use cases: read from one topic, process, write results to another topic atomically (no partial writes visible to consumers). Setup: configure the producer with a unique transactional.id, then: producer.initTransactions(); producer.beginTransaction(); write to multiple topics; producer.commitTransaction(); or producer.abortTransaction(); on error. Consumers must set isolation.level=read_committed to only see committed transactional messages (not aborted or in-progress). Transactions are the building block for exactly-once processing in Kafka Streams (exactly_once_v2 mode). They come with throughput overhead — use only when exactly-once is required.

Consumer lag is the difference between the producer's latest offset (log-end offset) and the consumer group's committed offset for a partition. It indicates how far behind a consumer is from the latest messages. High lag means: slow consumers, a traffic spike, consumer failure, or partition rebalance. Monitor with: kafka-consumer-groups.sh --describe: shows lag per consumer, per partition. Kafka JMX metrics: kafka.consumer:type=consumer-fetch-manager-metrics,records-lag-max. Kafka Lag Exporter: Prometheus exporter for consumer lag metrics. Confluent Control Center or Cruise Control: visual dashboards. Alert on lag when it exceeds a threshold. Root causes: under-provisioned consumers (add more), slow processing (optimize code), large batches causing GC pauses, or Kafka broker issues. Monitor lag per consumer group, not just total.

Partition rebalancing is the process of redistributing topic partition assignments across consumers in a group when the group membership changes (consumer joins, leaves, or crashes). During a rebalance: all consumers in the group stop processing (the "stop the world" problem) while the Group Coordinator broker orchestrates the new assignment. This causes latency spikes and potential duplicate processing (uncommitted offsets). Rebalance triggers: new consumer joins, consumer crashes or exceeds max.poll.interval.ms (took too long to poll), topic partition count changes. Improvements: Cooperative incremental rebalancing (introduced in Kafka 2.4): only revokes partitions that need to move, allowing other partitions to continue processing during the rebalance — dramatically reduces disruption. Enable with partition.assignment.strategy=CooperativeStickyAssignor.

ksqlDB is a streaming SQL database built on Kafka Streams that enables real-time stream processing using SQL syntax. Write SQL to process Kafka streams without writing Java code. Core abstractions: Streams: an unbounded, immutable sequence of events from a Kafka topic. Tables: a mutable, changelog-based view (materialized from a compacted topic). Operations: SELECT (continuous queries), CREATE STREAM AS SELECT (create a derived stream), CREATE TABLE AS SELECT (aggregate into a table), JOIN streams and tables. Example: CREATE STREAM fraud_alerts AS SELECT userId, amount FROM transactions WHERE amount > 10000; — creates a new Kafka topic with transactions over $10K. ksqlDB has a REST API and CLI, enabling non-Java developers to build stream processing pipelines. It runs as a managed service in Confluent Cloud.

MirrorMaker 2 (MM2) is Kafka's built-in cluster replication tool for mirroring topics between Kafka clusters — for geo-replication, disaster recovery, or multi-cloud setups. Built on Kafka Connect, MM2 runs as Connect connectors. Key features: Active-active: both clusters can accept writes and replicate to each other (conflict-free for non-overlapping partitions). Active-passive: one cluster is primary, the other is a standby replica. Offset translation: MM2 maintains a mapping between source and target offsets so consumers can resume correctly after a failover. Configuration sync: replicate topic configurations and consumer group offsets. Renaming: replicated topics are prefixed with the source cluster alias (e.g., us-east.orders) to avoid conflicts. Deploy MM2 on the source or target cluster, or on a dedicated Connect cluster for isolation.

Kafka topic design for high throughput involves several decisions. Partition count: more partitions = higher throughput (more parallel producers/consumers). Rule of thumb: start with max(target throughput / single-partition throughput, desired consumer parallelism). A single partition typically handles 10–100 MB/s. Replication factor: 3 for production (balance durability and performance). Producer batching: increase batch.size (512KB+) and linger.ms (5–20ms) for larger batches, reducing per-message overhead. Compression: use lz4 or zstd for a good speed/compression ratio. Consumer parallelism: match consumer count to partition count. Avoid hotspots: choose partition keys that distribute load evenly — high-cardinality keys like UUIDs are safe; low-cardinality keys (country code, boolean) create hot partitions. Tiered storage: offload old data to S3/GCS to reduce broker disk pressure.

Debezium is an open-source CDC (Change Data Capture) platform that captures row-level changes from databases (MySQL, PostgreSQL, MongoDB, Oracle, SQL Server) and streams them as events to Kafka topics. How it works: Debezium connects to the database's transaction log (binlog for MySQL, WAL for PostgreSQL, oplog for MongoDB) and reads every INSERT, UPDATE, and DELETE event. Each change event is published to a Kafka topic (one topic per table by default) with the full before/after row image, operation type, timestamp, and transaction ID. This enables: Event-driven architecture (react to DB changes without polling), CQRS (build read models from write events), Cache invalidation, Microservice sync. Debezium runs as a Kafka Connect source connector — deploy on a Connect cluster and configure with the database connection details.

Kafka Tiered Storage (introduced in Kafka 3.6 stable) allows Kafka to store older log segments in cheap remote storage (S3, Azure Blob, GCS) while keeping recent data on local broker disk. Brokers retain only recent segments locally for fast consumption; older segments are transparently fetched from remote storage when needed. Benefits: Cost reduction: remote storage costs 1/10th of local SSD. Broker scalability: broker disks only need to hold recent data, enabling smaller/cheaper brokers. Longer retention: retain data for months or years affordably. Independent scaling: add partitions for throughput without expanding storage. Before tiered storage, the only option for long-term retention was large, expensive disks or running a separate S3 sink connector alongside Kafka. Tiered storage makes Kafka a viable long-term event store without operational workarounds.

Kafka's exactly-once semantics (EOS) for stream processing consists of three components working together. 1. Idempotent producer: prevents duplicate messages from producer retries using sequence numbers and Producer IDs per partition. 2. Transactions: atomic multi-partition writes — consumers see either all or none of a transaction's messages (with read_committed isolation). 3. Transactional consumer-processor-producer: in Kafka Streams, the atomic pattern is: consume from input partition + process + produce to output partition + commit input offset — all in one transaction. If any step fails, the entire transaction is aborted; on retry, the input offset is back to its pre-transaction position. Kafka Streams handles this automatically with processing.guarantee=exactly_once_v2. EOS adds ~20–40% latency overhead. Use for financial transactions, billing, and any domain where duplicate processing has monetary or compliance consequences.

Schema evolution — changing the message format of a Kafka topic — without breaking producers or consumers. The Schema Registry enforces compatibility rules: Backward compatible: new schema can read data written by old schema. Safe for consumer upgrades first (add optional fields with defaults). Forward compatible: old schema can read data written by new schema. Safe for producer upgrades first (remove fields, add fields without defaults). Full compatible: both backward and forward. None: no compatibility check. Rules for safe Avro evolution: adding a field with a default value is backward AND forward compatible. Removing a required field breaks both. For Protobuf, adding fields with new field numbers is safe. Best practice: upgrade consumers before producers for backward-compatible changes. Use the FULL_TRANSITIVE compatibility mode for the strictest safety in high-traffic shared topics.

Kafka Cruise Control is an open-source project from LinkedIn for automating Kafka operations at scale. Key features: Partition rebalancing: automatically rebalance partitions across brokers to equalize CPU, network, and disk usage — solves the common problem of hot brokers after adding new nodes. Anomaly detection: detect under-replicated partitions, broker failures, and disk saturation. Self-healing: automatically fix detected issues without operator intervention. Capacity estimation: predict when resources will run out based on current growth. Goal-based optimization: define goals (balanced rack distribution, resource utilization) and Cruise Control proposes partition moves to achieve them. In large Kafka clusters (100+ brokers), manual partition rebalancing is impractical — Cruise Control automates it. It exposes a REST API for triggering and monitoring rebalances.

Kafka Streams is a Java client library embedded in your application. No separate cluster needed. Best for: Java/Kotlin applications, simpler topologies, teams that want minimal operational overhead. Limitations: scales by adding application instances; no cross-application coordination; complex event time handling. Apache Flink is a full distributed stream processing framework with its own cluster (JobManager + TaskManagers). Best for: complex event processing, ML inference pipelines, sub-second latency requirements, non-Kafka sources, large-scale stateful processing. Flink supports multiple sources (Kafka, Kinesis, files, databases) and sinks. It has more powerful windowing, better exactly-once support for complex graphs, and a richer SQL layer. Choose Kafka Streams for standard Kafka-to-Kafka stream processing in microservices. Choose Flink for complex analytics, multiple data sources, or when your team needs the full distributed processing framework capabilities.

For ultra-low latency (sub-millisecond to single-millisecond end-to-end): Producer: set linger.ms=0 (send immediately, no batching wait), batch.size=1 (minimize batching), compression.type=none (no compression overhead), acks=1 (leader-only ack). Broker: use fast NVMe SSDs, disable log.flush.interval.messages (rely on OS page cache), tune JVM GC (G1GC with MaxGCPauseMillis=20), use dedicated CPUs to reduce context switching. Consumer: set fetch.min.bytes=1 (return immediately even for one message), fetch.max.wait.ms=0. Network: place producers and consumers in the same data center/AZ as the broker leader. Enable socket.keepalive.enable=true. OS tuning: increase file descriptor limits, set vm.swappiness=1, use a high-performance network (10Gb+). Low-latency configurations sacrifice throughput and durability — apply only where latency is the primary constraint.

The ISR (In-Sync Replicas) is the dynamic set of replicas that are fully caught up with the partition leader (within replica.lag.time.max.ms, default 30 seconds). Only ISR members are eligible for leader election by default. Unclean leader election (unclean.leader.election.enable): if the leader fails and no ISR member is available, Kafka can elect an out-of-sync replica as the new leader (default: false). This risks data loss — messages acknowledged by the old leader but not replicated to the new leader are permanently lost. For systems where data loss is unacceptable (banking, payments), keep unclean.leader.election.enable=false. For systems where availability is more important than durability (metrics collection), enabling it prevents an outage when all ISR members fail. Monitor UnderReplicatedPartitions JMX metric — it should always be 0 in steady state.

In KRaft mode (Kafka Raft, without ZooKeeper), a subset of Kafka brokers are designated as controllers (typically 3 nodes). These controller nodes form a Raft quorum — they elect a single active controller using the Raft consensus protocol. The active controller maintains the metadata log: a Kafka topic (__cluster_metadata) containing all cluster state (broker registration, topic/partition metadata, leader assignments). When a partition leader fails, the active controller detects it (via heartbeat timeout), selects a new leader from the ISR, writes the change to the metadata log, and pushes the update to all brokers. Brokers maintain a local cache of the metadata log for fast local decisions without querying the controller for every request. KRaft enables Kafka clusters with millions of partitions, sub-second controller failover, and simpler operations compared to ZooKeeper.

A dead letter queue (DLQ) in Kafka handles messages that cannot be processed after exhausting retries, preventing them from blocking the consumer or being silently dropped. Implementation: Consumer retry loop: catch processing exceptions, retry N times. After N failures, produce the failed message to a dedicated topic-name.DLQ topic. Include the original message plus error metadata (exception, timestamp, consumer group, original offset) in headers or envelope. Kafka Connect DLQ: built-in DLQ support in connectors — set errors.deadletterqueue.topic.name=topic.DLQ and errors.tolerance=all. Spring Kafka: configure DefaultErrorHandler with a DeadLetterPublishingRecoverer. Monitor the DLQ and alert when messages appear — they indicate a systemic processing failure. Build a DLQ replay tool to reprocess messages after fixing the underlying bug.

Kafka transactions for exactly-once span multiple brokers via a Transaction Coordinator — a broker elected to coordinate a specific producer's transactions (determined by hashing the transactional.id to a partition of the __transaction_state topic). Flow: 1. InitProducerId: producer contacts transaction coordinator, gets a PID + epoch. 2. beginTransaction(): client-side only. 3. AddPartitionsToTxn: producer registers participating partitions with the coordinator. 4. Produce: messages are written to partition leaders with the transaction ID. 5. EndTransaction: coordinator writes PREPARE_COMMIT to its log, then sends WriteTxnMarkers to each partition leader to write a COMMIT marker. 6. CompleteTransaction: coordinator writes COMMITTED. Consumers with read_committed buffer messages until they see the commit marker. If the coordinator crashes, the new coordinator can recover in-flight transactions from the transaction log.

Production Kafka consumer resilience strategies: Idempotent processing: design processing logic to be safely retried — use idempotency keys to detect and skip duplicates. Retry logic with backoff: catch transient errors (DB timeouts), retry with exponential backoff before DLQ. Circuit breaker: if downstream service is consistently failing, pause consumption temporarily to allow recovery. Poison pill handling: if one specific message always fails (bad format, unexpected data), send to DLQ after N retries without blocking healthy messages. Offset management: commit offsets only after successful processing (enable.auto.commit=false); track per-message status if partial batch processing is needed. Consumer group monitoring: alert on group rebalances, partition assignment timeouts (max.poll.interval.ms exceeded), and lag growth. Graceful shutdown: call consumer.wakeup() on SIGTERM, finish current batch, commit, then close — avoiding unnecessary rebalances and offset gaps.

Kafka's ordering guarantees are nuanced. Within a partition: messages are strictly ordered — offset 0 is before offset 1 — always. Across partitions: no ordering guarantee. The fundamental design decision: messages with the same key always go to the same partition (via key hash), ensuring per-key ordering. Producer complications: with max.in.flight.requests.per.connection > 1 and retries enabled, a failed batch can be overtaken by a later batch, breaking in-partition ordering. Fix: enable idempotent producer (enable.idempotence=true) which preserves ordering with up to 5 in-flight requests. Consumer complications: a consumer processing messages from the same partition in parallel threads can process them out of order. Always process one partition serially per consumer thread or use a per-key processing queue. Exactly-once with ordering: Kafka Streams with exactly_once_v2 processes records in order per partition while maintaining exactly-once guarantees.

Kafka uses a custom binary TCP protocol for all client-broker communication. Key characteristics: Request/Response model: clients send requests; brokers respond with the same correlation_id to match responses to requests. API versioning: each API call has a version number — clients and brokers negotiate the highest mutually supported version. This enables rolling upgrades without breaking compatibility. Common APIs: Produce (write messages), Fetch (read messages), Metadata (get cluster topology), FindCoordinator (find consumer group coordinator), JoinGroup/SyncGroup/Heartbeat (consumer group protocol), ListOffsets, CreateTopics, AlterConfigs. Bootstrap discovery: clients connect to one or more bootstrap.servers, fetch cluster metadata (all broker addresses, topic-partition-leader mapping), then connect directly to the relevant broker leaders. Authentication: supports SASL (PLAIN, SCRAM, GSSAPI/Kerberos, OAUTHBEARER) and TLS for transport encryption.