What is Kafka's controller and how is leader election handled in KRaft mode?

In KRaft mode (Kafka Raft, without ZooKeeper), a subset of Kafka brokers are designated as controllers (typically 3 nodes). These controller nodes form a Raft quorum — they elect a single active controller using the Raft consensus protocol. The active controller maintains the metadata log: a Kafka topic (__cluster_metadata) containing all cluster state (broker registration, topic/partition metadata, leader assignments). When a partition leader fails, the active controller detects it (via heartbeat timeout), selects a new leader from the ISR, writes the change to the metadata log, and pushes the update to all brokers. Brokers maintain a local cache of the metadata log for fast local decisions without querying the controller for every request. KRaft enables Kafka clusters with millions of partitions, sub-second controller failover, and simpler operations compared to ZooKeeper.