SQS Message Handling and Visibility Timeout

Reliably processing messages from an SQS queue requires understanding how visibility timeouts work, how to handle failures gracefully, and how to design consumers that are idempotent. In this lesson, we dive deep into the message lifecycle and the patterns that make SQS-based systems production-ready.

---

The Message Lifecycle

Every message in SQS moves through a defined lifecycle:

1. Sent — the producer sends the message to the queue
2. Available — the message is sitting in the queue, waiting to be received
3. In-flight — a consumer has received the message, and it is temporarily invisible to other consumers
4. Deleted — the consumer successfully processed the message and deleted it
5. Re-available — if the consumer fails to delete the message before the visibility timeout expires, it becomes visible again

[Sent] ——> [Available] ——> [In-flight] ——> [Deleted]
                              |
                              v  (visibility timeout expires)
                          [Re-available]

---

Understanding Visibility Timeout

The visibility timeout is the period during which a message, once received by a consumer, is invisible to all other consumers. This prevents multiple consumers from processing the same message simultaneously.

How It Works

1. Consumer A receives Message X from the queue
2. SQS starts the visibility timeout clock (default: 30 seconds)
3. While the clock is running, no other consumer can see Message X
4. Consumer A processes the message and deletes it — success
5. If Consumer A crashes or fails to delete the message before the timeout expires, Message X becomes visible again and can be processed by another consumer

Choosing the Right Visibility Timeout

Scenario	Recommendation
Processing takes < 10 seconds	Set timeout to 30 seconds (default is fine)
Processing takes 1-5 minutes	Set timeout to 6-10 minutes
Processing time varies widely	Use a shorter default and extend dynamically
Processing takes > 12 hours	Consider a different architecture (Step Functions)

Rule of thumb: Set the visibility timeout to at least 6x the average processing time to allow for retries and variability.

Changing Visibility Timeout Dynamically

If a consumer realises it needs more time, it can extend the visibility timeout using the ChangeMessageVisibility API call:

sqs.change_message_visibility(
    QueueUrl=queue_url,
    ReceiptHandle=message['ReceiptHandle'],
    VisibilityTimeout=120  # Extend by 120 seconds
)

This is useful when processing time is unpredictable — the consumer can periodically "heartbeat" to extend the timeout while it works.

---

Idempotent Message Processing

Because SQS Standard queues offer at-least-once delivery, your consumer may receive the same message more than once. Your processing logic must be idempotent — processing the same message twice should produce the same result as processing it once.

Strategies for Idempotency

Strategy	How It Works
Idempotency key	Include a unique ID in the message. Before processing, check if this ID has already been processed (e.g. in a database)
Database constraints	Use unique constraints or conditional writes to prevent duplicate records
Upsert operations	Use INSERT ... ON CONFLICT or equivalent to safely handle duplicates
Deduplication window	FIFO queues provide a 5-minute deduplication window using MessageDeduplicationId

Example: Idempotency Check

def process_order(message):
    order_id = message['orderId']

    # Check if already processed
    existing = db.query("SELECT 1 FROM processed_orders WHERE id = %s", order_id)
    if existing:
        print(f"Order {order_id} already processed — skipping")
        return