Monitoring and DevOps Best Practices

Throughout this course we have explored individual AWS monitoring and DevOps services in depth. In this final lesson we bring everything together into a cohesive set of best practices that will help you build observable, automated, and resilient cloud workloads.

---

1. Start With the Four Golden Signals

When setting up monitoring for any workload, begin with the four golden signals:

Signal	CloudWatch Implementation
Latency	ALB TargetResponseTime, API Gateway IntegrationLatency, custom app metrics
Traffic	ALB RequestCount, API Gateway Count, custom request-per-second metrics
Errors	ALB HTTPCode_Target_5XX_Count, Lambda Errors, custom error count metrics
Saturation	EC2 CPUUtilization, RDS FreeableMemory, SQS ApproximateNumberOfMessagesVisible

If you only have time to set up four alarms, make them these. They will catch the vast majority of production issues.

---

2. Instrument the Three Pillars of Observability

Metrics

• Publish custom application metrics in addition to AWS-provided metrics.
• Use the Embedded Metric Format in Lambda and containers to emit metrics with zero API overhead.
• Choose percentiles (p95, p99) over averages for latency metrics.

Logs

• Emit structured JSON logs with fields like requestId, userId, level, and duration.
• Set retention policies on every log group — never accept the default of indefinite retention.
• Use metric filters to extract key indicators (error count, slow queries) from logs and alarm on them.
• Forward critical logs to a centralised monitoring account using cross-account log subscriptions.

Traces

• Enable X-Ray active tracing on API Gateway, Lambda, and ECS.
• Add the X-Ray SDK to your application code to capture outbound HTTP calls, database queries, and AWS SDK calls.
• Use annotations for fields you want to search on (customerId, orderId, environment).
• Configure sampling rules to balance trace coverage against cost.

---

3. Build Effective Dashboards

A dashboard should answer one question: "Is this workload healthy right now?"

Dashboard Template

Widget	Metric	Purpose
Alarm status	All alarms for this workload	At-a-glance health
Line chart	Request count over time	Traffic trends
Line chart	p99 latency over time	User experience
Number	Error rate (last 5 min)	Current error level
Line chart	CPU / memory utilisation	Saturation
Log table	Recent ERROR-level log events	Quick diagnosis
Text	Runbook links, on-call contacts	Operational context

Dashboard Guidelines

• Create one dashboard per workload, not one mega-dashboard.
• Keep dashboards to 10–15 widgets — more than that and they lose focus.
• Automate dashboard creation using CloudFormation or CDK so dashboards stay in sync with infrastructure changes.
• Review dashboards quarterly and remove widgets that no one looks at.

---

4. Implement a Tiered Alerting Strategy

Not every alarm should wake someone up at 3 a.m.

Tier	Criteria	Channel	Response Time
P1 — Critical	Customer-facing outage, data loss risk	PagerDuty / phone	Immediate
P2 — High	Degraded performance, partial outage	Slack #incidents + email	Within 30 min
P3 — Warning	Early warning, capacity trending	Email / ticket	Next business day
P4 — Info	Informational, non-urgent	Dashboard only	Review weekly

Reducing Alert Fatigue

• Use composite alarms to correlate signals before paging.
• Apply the M of N evaluation model to smooth out transient spikes.
• If an alarm has not fired in 6 months, either lower the threshold or delete it.
• Every alarm must have a documented runbook — if the responder does not know what to do, the alarm is not useful.

---

5. Audit and Secure Everything

CloudTrail

• Enable a multi-region, organisation-wide trail.
• Turn on log file integrity validation.
• Encrypt logs with a customer-managed KMS key.
• Create metric filters for high-risk API calls: ConsoleLogin with root, StopLogging, DeleteTrail, AuthorizeSecurityGroupIngress 0.0.0.0/0.

Least-Privilege IAM

• Scope CodeBuild, CodeDeploy, and CodePipeline roles to only the actions they need.
• Use separate IAM roles for build, deploy, and pipeline orchestration.
• Rotate credentials and audit unused roles regularly.

---

6. Automate the Entire Release Process

Pipeline Structure

A mature CI/CD pipeline on AWS typically follows this pattern: