SNS Fan-Out Pattern with SQS

The SNS fan-out pattern is one of the most important architectural patterns in AWS event-driven systems. By combining SNS topics with SQS queues, you can broadcast a single event to multiple independent consumers, each processing the event at its own pace with its own retry logic and dead-letter queue.

---

What Is Fan-Out?

Fan-out means taking a single input and distributing it to multiple outputs. In the context of messaging:

• A producer publishes one message to an SNS topic
• The SNS topic delivers a copy of that message to multiple SQS queues
• Each SQS queue has its own independent consumer

                              +——> [SQS: email-queue]    ——> Email Service
                              |
Producer ——> [SNS Topic] ——>  +——> [SQS: analytics-queue] ——> Analytics Service
                              |
                              +——> [SQS: inventory-queue] ——> Inventory Service

---

Why SNS + SQS? Why Not SNS Alone?

SNS pushes messages to subscribers immediately. If the subscriber is unavailable or slow, the message may be lost (for some subscriber types) or create backpressure. Adding SQS queues as subscribers provides:

Benefit	Explanation
Durability	Messages are stored in SQS until successfully processed — they will not be lost
Independent processing	Each consumer processes at its own rate, independent of others
Retry and DLQ	Each queue can have its own retry strategy and dead-letter queue
Backpressure handling	If a consumer is slow, messages buffer in its queue without affecting other consumers
Scaling	Each consumer can scale independently based on its own queue depth

---

Setting Up the Fan-Out Pattern

Step 1: Create the SNS Topic

import boto3

sns = boto3.client('sns')

topic = sns.create_topic(Name='order-events')
topic_arn = topic['TopicArn']

Step 2: Create SQS Queues for Each Consumer

sqs = boto3.client('sqs')

email_queue = sqs.create_queue(QueueName='order-email-notifications')
analytics_queue = sqs.create_queue(QueueName='order-analytics')
inventory_queue = sqs.create_queue(QueueName='order-inventory-update')

Step 3: Subscribe Each Queue to the SNS Topic

sns.subscribe(
    TopicArn=topic_arn,
    Protocol='sqs',
    Endpoint='arn:aws:sqs:eu-west-2:123456789012:order-email-notifications'
)

sns.subscribe(
    TopicArn=topic_arn,
    Protocol='sqs',
    Endpoint='arn:aws:sqs:eu-west-2:123456789012:order-analytics'
)

sns.subscribe(
    TopicArn=topic_arn,
    Protocol='sqs',
    Endpoint='arn:aws:sqs:eu-west-2:123456789012:order-inventory-update'
)

Step 4: Grant SNS Permission to Send to SQS

Each SQS queue needs a policy that allows the SNS topic to send messages:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "sns.amazonaws.com"},
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:eu-west-2:123456789012:order-email-notifications",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:eu-west-2:123456789012:order-events"
        }
      }
    }
  ]
}

Step 5: Publish and Fan Out

sns.publish(
    TopicArn=topic_arn,
    Message='{"orderId": "ORD-12345", "customerId": "CUST-789", "total": 149.99}',
    MessageAttributes={
        'orderType': {'DataType': 'String', 'StringValue': 'express'}
    }
)

This single publish delivers the message to all three SQS queues simultaneously.

---

Fan-Out with Message Filtering

Not every consumer needs every message. Use SNS subscription filter policies to route specific messages to specific queues.

Example: Order Processing System

Producer ——> [SNS: order-events]
                 |
                 |——> [SQS: express-orders]   (filter: orderType = "express")
                 |——> [SQS: standard-orders]  (filter: orderType = "standard")
                 |——> [SQS: all-orders]       (no filter — receives everything)

Apply a filter policy when subscribing:

sns.subscribe(
    TopicArn=topic_arn,
    Protocol='sqs',
    Endpoint=express_queue_arn,
    Attributes={
        'FilterPolicy': '{"orderType": ["express"]}'
    }
)

Now the express-orders queue only receives messages where orderType is "express".

---