Reliability Pillar

The Reliability pillar focuses on ensuring that a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle, detect failures, and automatically recover.

---

Design Principles

The Reliability pillar is guided by five design principles:

1. Automatically Recover from Failure

Monitor key performance indicators (KPIs) of your workload and trigger automated recovery when a threshold is breached. This allows you to detect and fix failures before they affect your users.

2. Test Recovery Procedures

In the cloud, you can test how your workload fails and validate your recovery procedures. Use automation to simulate different failures or recreate scenarios that led to failures before. This exposes failure pathways that you can test and fix before a real failure scenario occurs.

3. Scale Horizontally to Increase Aggregate Workload Availability

Replace one large resource with multiple small resources to reduce the impact of a single point of failure on the overall workload. Distribute requests across multiple, smaller resources to ensure that they don't share a common point of failure.

4. Stop Guessing Capacity

Monitor demand and workload utilisation, and automate the addition or removal of resources to maintain the optimal level to satisfy demand without over- or under-provisioning.

5. Manage Change Through Automation

Changes to your infrastructure should be made using automation. The changes that need to be managed include changes to the automation itself, which can be tracked and reviewed.

---

Best Practice Areas

Foundations

Before you can build reliable systems, you need solid foundations:

• Account-level limits — understand and monitor your AWS service quotas. Request increases before you need them. Use AWS Service Quotas to view and manage limits
• Network topology — plan your VPC design carefully. Use multiple Availability Zones, ensure adequate IP address space, and design for connectivity between environments
• Service limits planning — for critical workloads, document the service limits that could constrain your architecture and set alarms when usage approaches those limits

Workload Architecture

Design your workload to be resilient from the start:

• Service-oriented architecture (SOA) — decompose monolithic applications into smaller, independent services. Each service should have its own data store, scale independently, and be deployable without affecting other services
• Distributed systems — understand the challenges of distributed systems: network latency, partial failures, and eventual consistency. Design your architecture to handle these realities
• Idempotency — design operations so that performing the same action multiple times produces the same result. This is critical for retries and recovery scenarios

# Example: Idempotent DynamoDB write using a conditional expression
import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Orders')

table.put_item(
    Item={
        'OrderId': 'ORD-12345',
        'Status': 'Processing',
        'Amount': 99.99
    },
    ConditionExpression='attribute_not_exists(OrderId)'
)

• Loose coupling — use message queues (SQS), event buses (EventBridge), and load balancers (ALB) to decouple components. When one component fails, the others can continue operating
• Throttling and backoff — implement exponential backoff with jitter for retries. This prevents cascading failures when downstream services are overloaded

Change Management

Manage changes to your workload safely: