Message Queues & Event-Driven Architecture

Synchronous request-response communication works well for simple interactions, but it creates tight coupling and struggles with variable load. Message queues and event-driven architecture decouple producers from consumers, enabling resilient, scalable systems.

---

Why Message Queues?

graph TD
  subgraph Sync["Synchronous (tightly coupled)"]
    SA["Service A"] -->|"HTTP"| SB["Service B"]
    SB -->|"HTTP"| SC["Service C"]
    snote["If B is slow A waits; if C is down B fails. Everything is coupled."]
  end
  subgraph Async["Asynchronous (decoupled via queue)"]
    AA["Service A"] -->|"publish"| Q["Message Queue"]
    Q -->|"consume"| AB["Service B"]
    anote["A does not wait for B. B processes at its own pace. If B is down, messages wait in the queue."]
  end

Benefits

Benefit	Description
Decoupling	Producer and consumer operate independently
Buffering	Queue absorbs traffic spikes
Reliability	Messages persist until processed
Scalability	Add more consumers to increase throughput
Fault tolerance	Consumer failures do not affect the producer

---

Pub/Sub vs Point-to-Point

Point-to-Point (Queue)

One message is consumed by exactly one consumer.

graph LR
  P["Producer"] --> Q["Queue"]
  Q --> A["Consumer A"]
  Q --> B["Consumer B"]
  note["Load balanced - each consumer gets different messages"]

Publish/Subscribe

One message is delivered to all subscribers.

graph LR
  P["Publisher"] --> T["Topic"]
  T --> A["Subscriber A"]
  T --> B["Subscriber B"]
  T --> C["Subscriber C"]
  note["All subscribers get the same message"]

Model	Delivery	Use Case
Point-to-Point	One consumer	Task processing, work queues
Pub/Sub	All subscribers	Event broadcasting, notifications

---

Message Brokers Compared

Apache Kafka

graph TD
  Producer["Producer"] --> Kafka["Kafka Cluster"]
  subgraph Kafka2["Topic: orders"]
    P0["Partition 0: msg1, msg3, msg5"]
    P1["Partition 1: msg2, msg4, msg6"]
    P2["Partition 2: msg7, msg8"]
  end
  Kafka --> Kafka2
  P0 --> C1["Consumer 1 (Part. 0)"]
  P1 --> C2["Consumer 2 (Part. 1)"]
  P2 --> C3["Consumer 3 (Part. 2)"]

Comparison

Feature	Kafka	RabbitMQ	AWS SQS
Model	Log-based pub/sub	Message broker	Managed queue
Throughput	Very high (millions/sec)	High (100K+/sec)	High (managed)
Message retention	Configurable (days/weeks)	Until consumed/TTL	Up to 14 days
Ordering	Per partition	Per queue (FIFO)	FIFO option available
Replay	Yes (consumers re-read)	No (once consumed)	No
Consumer groups	Yes	Yes (exchanges)	No
Operational overhead	High (self-managed)	Medium	None (fully managed)
Best for	Event streaming, logs	Task queues, routing	Simple cloud queues

---

Event Sourcing

Instead of storing the current state, event sourcing stores every state change as an immutable event.