Message Ordering & Deduplication

By default, Google Cloud Pub/Sub delivers messages in a best-effort order and guarantees at-least-once delivery, which means messages may arrive out of order and duplicates are possible. For many use cases this is acceptable, but some applications require strict ordering or exactly-once processing. This lesson covers Pub/Sub's ordering and deduplication features and how to use them effectively.

---

Message Ordering

The Problem

Without ordering, Pub/Sub may deliver messages out of the order they were published. This happens because Pub/Sub distributes messages across multiple servers for scalability. Consider a scenario where you publish three status updates for an order:

1. Order created
2. Payment processed
3. Order shipped

Without ordering guarantees, a subscriber might receive "Order shipped" before "Payment processed", leading to incorrect state transitions.

Ordering Keys

Pub/Sub supports ordered delivery using ordering keys. When you publish messages with the same ordering key, Pub/Sub guarantees that the subscriber receives them in the order they were published.

Concept	Description
Ordering key	A string included with each message that defines the ordering group
Scope	Ordering is guaranteed within a single ordering key, not across keys
Per-region	Ordering is guaranteed when publishing to a single region
Performance	Ordered delivery may reduce throughput compared to unordered

Enabling Ordered Delivery

To use message ordering, you must:

1. Enable ordering on the subscription
2. Include an ordering key when publishing

# Create a subscription with ordering enabled
gcloud pubsub subscriptions create ordered-sub \
  --topic=orders-topic \
  --enable-message-ordering

Publishing with Ordering Keys

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-project", "orders-topic")

# All messages with the same ordering key are delivered in order
publisher.publish(topic_path, b"Order created", ordering_key="order-123")
publisher.publish(topic_path, b"Payment processed", ordering_key="order-123")
publisher.publish(topic_path, b"Order shipped", ordering_key="order-123")

# Messages with different ordering keys can be delivered independently
publisher.publish(topic_path, b"Order created", ordering_key="order-456")

How Ordering Works Internally

When ordering is enabled, Pub/Sub routes all messages with the same ordering key to the same subscriber instance. This means:

• Messages within a key are delivered sequentially
• Messages across different keys can be delivered in parallel
• If a message fails (nack), subsequent messages with the same key are paused until the failed message is retried and acknowledged

Ordering Key Design

Choosing the right ordering key is critical for both correctness and performance:

Strategy	Example Key	Trade-off
Per entity	order-123	Good parallelism, ordering per entity
Per user	user-456	All events for a user are ordered
Per partition	region-eu	Coarse grouping, less parallelism
Single key	all	Strict global ordering but NO parallelism

The best practice is to use the most granular ordering key that satisfies your requirements. Using a single ordering key forces all messages through a single subscriber, eliminating parallelism.

---

Message Deduplication

The Problem

Pub/Sub guarantees at-least-once delivery. This means a message may be delivered more than once. Duplicates can occur when: