What is consumer lag in Kafka and how do you monitor it?
Answer
Consumer lag is the difference between the producer's latest offset (log-end offset) and the consumer group's committed offset for a partition. It indicates how far behind a consumer is from the latest messages. High lag means: slow consumers, a traffic spike, consumer failure, or partition rebalance. Monitor with: kafka-consumer-groups.sh --describe: shows lag per consumer, per partition. Kafka JMX metrics: kafka.consumer:type=consumer-fetch-manager-metrics,records-lag-max. Kafka Lag Exporter: Prometheus exporter for consumer lag metrics. Confluent Control Center or Cruise Control: visual dashboards. Alert on lag when it exceeds a threshold. Root causes: under-provisioned consumers (add more), slow processing (optimize code), large batches causing GC pauses, or Kafka broker issues. Monitor lag per consumer group, not just total.