What is the difference between message queues and event streaming platforms?
Answer
Message queues (RabbitMQ, AWS SQS): designed for task distribution — a message is consumed by ONE consumer and then deleted from the queue. Pull or push delivery. Consumers compete for messages (competing consumer pattern). Messages are typically short-lived (consumed and gone). Use for: task queues (send one email per event), request-response patterns, work distribution among workers. Event streaming platforms (Apache Kafka, AWS Kinesis): designed as a distributed, persistent log — events are retained for a configured retention period (days to forever). Multiple independent consumer groups each read ALL events from their own position (offset) in the log — events are not deleted after consumption. Producers write to the end of the log; consumers read from any offset. Use for: event sourcing, audit logs, real-time analytics, multiple systems consuming the same event stream, event replay, stream processing (Kafka Streams, Apache Flink). Key differences: message queue — messages disappear after consumption, consumers are equivalent (one processes each message); event log — events persist, multiple consumers independently process all events. Kafka mechanics: topics split into partitions (parallel processing); each partition is an ordered, immutable sequence of records; consumers track their offset per partition; consumer groups — each partition is consumed by one consumer in the group (scalability); replication for durability. When to use each: need task distribution with one processor per task → message queue; need multiple systems to react to the same events, event replay, audit trail → Kafka.
Previous
What is a time-series database and when should you use one?
Next
How would you design a chat application like WhatsApp?