Basics

• a queue that holds messages to be processed asynchronously, but sequentially
• example uses:
  • sending data from one service to another to be processed
  • distributing tasks across multiple servers
  • processing orders and payments on an online store
  • processing bank transactions
• messages can be held even when the services consuming them are offline or busy
• messages are in a standardized format, so it doesn’t matter what language or platform the connected services run on
• producer/sender/publisher: a service that creates messages and puts them in the queue
• consumer/receiver/subscriber: a service that takes messages from the queue and processes them
• topic: a way to group messages, so that consumers can subscribe to only the subset of messages that are relevant to them
• message broker: the service that manages the message queue
• dead letter queue (DLQ) or dead letter exchange (DLX): a queue which holds messages that have expired or failed to process, letting them be inspected before moving back into a source queue

Publisher-Subscriber (pub-sub) model

• allows communication between multiple publishers and multiple subscribers using topics
• loosely coupled - publishers don't have to know anything about the subscribers, and vice versa

Examples

RabbitMQ

• simple linear queue
• push-based - messages are pushed to consumers (but a limit can be configured)
• "smart broker/dumb consumer" - the broker pushes messages to consumers, at which point they are removed from the queue
• very fast and reliable, can be distributed across clusters of nodes with fault tolerance
• supports standardized protocols like AMQP, MQTT, and STOMP

Kafka

• uses the pub-sub model (topic-based)
• pull-based - consumers pull messages from queues in batches
• "dumb broker/smart consumer" - the broker doesn't track which messages are read, all messages are kept for a certain period of time (so Kafka can act as a log)