Messaging Best Practices

Building reliable messaging systems on Google Cloud requires more than choosing the right service. This lesson consolidates the best practices for Pub/Sub, Cloud Tasks, and Cloud Scheduler into a comprehensive guide covering architecture, reliability, security, performance, cost optimisation, and operational excellence.

---

Architecture Best Practices

Choose the Right Service

Requirement	Service
Fan-out events to multiple services	Pub/Sub
Dispatch tasks to a single handler with rate control	Cloud Tasks
Schedule recurring jobs	Cloud Scheduler
High-volume, cost-sensitive streaming	Pub/Sub Lite
Complex stream processing with windowing	Dataflow
Simple event-driven reactions	Cloud Functions + Pub/Sub
Containerised event handlers	Cloud Run + Pub/Sub (push)

Design for Failure

1. Always configure dead-letter topics for production Pub/Sub subscriptions
2. Set appropriate retry policies — distinguish transient from permanent failures
3. Implement circuit breakers in subscribers to prevent cascade failures
4. Use acknowledgement deadline extension for long-running processors
5. Monitor backlog growth and alert before messages expire

Design for Idempotency

Every message consumer should be idempotent — producing the same result whether a message is processed once or multiple times.

Strategy	Description
Natural idempotency	Operations like "set status to X" are inherently idempotent
Deduplication table	Store processed message IDs and skip duplicates
Database upserts	Use INSERT ON CONFLICT to handle duplicate writes
Conditional updates	Only update if the current state matches expected state
Exactly-once delivery	Enable on Pub/Sub pull subscriptions where available

---

Reliability Best Practices

Acknowledgement Management

Practice	Description
Acknowledge after processing	Only ack after successful processing, never before
Extend deadlines	Use modifyAckDeadline for long-running tasks
Set realistic deadlines	Match the ack deadline to your processing time
Nack explicitly	Nack messages you cannot process for immediate redelivery
Handle partial failures	In batch processing, ack successes and nack failures individually

Message Retention

Setting	Recommendation
Topic retention	Enable 7-31 days for replay capability
Subscription retention	Default 7 days is sufficient for most workloads
Dead-letter retention	14-31 days to allow investigation time
Snapshot frequency	Create snapshots before major deployments for rollback

Ordering and Consistency

1. Use ordering keys when message sequence matters
2. Choose granular ordering keys (per entity, not global) for parallelism
3. Handle nack carefully with ordering — a nacked message blocks all subsequent messages in that key
4. Consider eventual consistency — most systems do not need strict ordering

---

Security Best Practices

Authentication and Authorisation

Practice	Description
Use service accounts	Dedicated service accounts for each publisher and subscriber
Least privilege	Grant only the specific roles needed
Avoid user credentials	Use Workload Identity for GKE, service accounts for Compute Engine
OIDC for push	Always configure push authentication with OIDC tokens

IAM Roles

Role	Grants
roles/pubsub.publisher	Publish messages to topics
roles/pubsub.subscriber	Create subscriptions and consume messages
roles/pubsub.viewer	View topics and subscriptions (read-only)
roles/pubsub.editor	Create and manage topics and subscriptions
roles/pubsub.admin	Full access including IAM policy management

Network Security

1. Use VPC Service Controls to restrict Pub/Sub access to authorised networks
2. Enable CMEK (Customer-Managed Encryption Keys) for sensitive data
3. Do not put secrets in message attributes — attributes may be logged
4. Encrypt sensitive payloads at the application level before publishing
5. Use Private Google Access for subscribers in private VPC networks

---

Performance Best Practices

Publishing

Practice	Description
Batch messages	Client libraries batch messages automatically — configure batch size and delay
Use regional endpoints	Publish to the nearest region to reduce latency
Compress large messages	Compress payloads over 1 KB to reduce costs and latency
Set ordering keys wisely	Too few keys limit parallelism; too many eliminate ordering benefits
Handle publish failures	Retry with exponential backoff on transient errors