What is the Kafka protocol and how do clients communicate with brokers?

Kafka uses a custom binary TCP protocol for all client-broker communication. Key characteristics: Request/Response model: clients send requests; brokers respond with the same correlation_id to match responses to requests. API versioning: each API call has a version number — clients and brokers negotiate the highest mutually supported version. This enables rolling upgrades without breaking compatibility. Common APIs: Produce (write messages), Fetch (read messages), Metadata (get cluster topology), FindCoordinator (find consumer group coordinator), JoinGroup/SyncGroup/Heartbeat (consumer group protocol), ListOffsets, CreateTopics, AlterConfigs. Bootstrap discovery: clients connect to one or more bootstrap.servers, fetch cluster metadata (all broker addresses, topic-partition-leader mapping), then connect directly to the relevant broker leaders. Authentication: supports SASL (PLAIN, SCRAM, GSSAPI/Kerberos, OAUTHBEARER) and TLS for transport encryption.