Message Queues & Event-Driven Architecture

Synchronous request-response communication works well for simple interactions, but it creates tight coupling and struggles with variable load. Message queues and event-driven architecture decouple producers from consumers, enabling resilient, scalable systems.

---

Why Message Queues?

Synchronous (tightly coupled):

┌──────────┐    HTTP     ┌──────────┐    HTTP     ┌──────────┐
│ Service A│────────────▶│ Service B│────────────▶│ Service C│
└──────────┘             └──────────┘             └──────────┘
  If B is slow, A waits. If C is down, B fails. Everything is coupled.

Asynchronous (decoupled via queue):

┌──────────┐  publish   ┌───────────┐  consume   ┌──────────┐
│ Service A│───────────▶│  Message  │───────────▶│ Service B│
└──────────┘            │   Queue   │            └──────────┘
                        └───────────┘
  A does not wait for B. B processes at its own pace.
  If B is down, messages wait in the queue.

Benefits

Benefit	Description
Decoupling	Producer and consumer operate independently
Buffering	Queue absorbs traffic spikes
Reliability	Messages persist until processed
Scalability	Add more consumers to increase throughput
Fault tolerance	Consumer failures do not affect the producer

---

Pub/Sub vs Point-to-Point

Point-to-Point (Queue)

One message is consumed by exactly one consumer.

Producer ──▶ [Queue] ──▶ Consumer A
                    ──▶ Consumer B  (load balanced — each gets different messages)

Publish/Subscribe

One message is delivered to all subscribers.

                          ┌──▶ Subscriber A
Publisher ──▶ [Topic] ────┼──▶ Subscriber B
                          └──▶ Subscriber C
                          (all get the same message)

Model	Delivery	Use Case
Point-to-Point	One consumer	Task processing, work queues
Pub/Sub	All subscribers	Event broadcasting, notifications

---

Message Brokers Compared

Apache Kafka

┌──────────┐     ┌─────────────────────────────────────┐
│ Producer │────▶│           Kafka Cluster              │
└──────────┘     │  ┌─────────────────────────────────┐ │
                 │  │ Topic: orders                    │ │
                 │  │  Partition 0: [msg1][msg3][msg5] │ │
                 │  │  Partition 1: [msg2][msg4][msg6] │ │
                 │  │  Partition 2: [msg7][msg8]       │ │
                 │  └─────────────────────────────────┘ │
                 └────────────────┬────────────────────┘
                                  │
                    ┌─────────────┼─────────────┐
                    ▼             ▼             ▼
              ┌──────────┐ ┌──────────┐ ┌──────────┐
              │Consumer 1│ │Consumer 2│ │Consumer 3│
              │(Part. 0) │ │(Part. 1) │ │(Part. 2) │
              └──────────┘ └──────────┘ └──────────┘

Comparison

Feature	Kafka	RabbitMQ	AWS SQS
Model	Log-based pub/sub	Message broker	Managed queue
Throughput	Very high (millions/sec)	High (100K+/sec)	High (managed)
Message retention	Configurable (days/weeks)	Until consumed/TTL	Up to 14 days
Ordering	Per partition	Per queue (FIFO)	FIFO option available
Replay	Yes (consumers re-read)	No (once consumed)	No
Consumer groups	Yes	Yes (exchanges)	No
Operational overhead	High (self-managed)	Medium	None (fully managed)
Best for	Event streaming, logs	Task queues, routing	Simple cloud queues

---

Event Sourcing

Instead of storing the current state, event sourcing stores every state change as an immutable event.

Traditional (State):            Event Sourcing: