CloudWatch Logs and Log Insights

Metrics tell you what is happening; logs tell you why. CloudWatch Logs is a fully managed service for ingesting, storing, and querying log data from AWS services and your own applications. Paired with CloudWatch Logs Insights — a purpose-built query language — you can search gigabytes of logs in seconds.

---

Core Concepts

Log Groups

A log group is a container for log streams that share the same retention, monitoring, and access-control settings. You typically create one log group per application or per environment:

• /aws/lambda/order-processor
• /ecs/production/payment-service
• /custom/my-web-app

Log Streams

Within a log group, a log stream represents a single source of log events — for example, one Lambda invocation, one EC2 instance, or one ECS container. Streams are created automatically by most AWS integrations.

Log Events

A log event is a single record containing a timestamp and a raw message. Events are immutable once written.

---

Sending Logs to CloudWatch

AWS Services

Many AWS services send logs to CloudWatch automatically or with minimal configuration:

Service	Log Group Pattern	Configuration
Lambda	/aws/lambda/<function-name>	Automatic — requires IAM permission
ECS / Fargate	User-defined	Use the awslogs log driver
API Gateway	/aws/apigateway/<api-name>	Enable in stage settings
RDS	/aws/rds/instance/<db-id>/<log-type>	Enable in parameter group
VPC Flow Logs	User-defined	Create a flow log with destination CloudWatch

Custom Applications on EC2

Install the CloudWatch Agent on your EC2 instance. The agent reads log files from disk and streams them to CloudWatch Logs. A typical agent configuration specifies:

• The file path to monitor (e.g. /var/log/myapp/app.log)
• The destination log group and stream
• The timestamp format

Structured Logging

Whenever possible, emit logs in structured JSON format rather than unstructured plain text. Structured logs are far easier to query in Logs Insights:

Unstructured:

2024-03-15 10:23:45 ERROR OrderService - Failed to process order 12345: timeout

Structured:

{
  "timestamp": "2024-03-15T10:23:45Z",
  "level": "ERROR",
  "service": "OrderService",
  "message": "Failed to process order",
  "orderId": "12345",
  "reason": "timeout"
}

The structured version lets you query by orderId, filter by level, and aggregate by reason without writing complex regular expressions.

---

Log Retention

By default, CloudWatch Logs retains data indefinitely. This can become expensive. Set a retention policy on every log group:

Retention Period	Typical Use
1 day	Development / debugging
7–30 days	Short-lived environments, staging
60–90 days	Production application logs
1–10 years	Compliance and audit logs

You can also export logs to S3 for long-term, cost-effective archival and query them later with Athena.

---

Metric Filters

A metric filter scans incoming log events for a pattern and increments a CloudWatch metric each time a match is found. This turns log data into metrics that you can alarm on.

Example: Counting Errors

Create a metric filter on your application log group that matches the pattern "level": "ERROR". Each match increments a custom metric ErrorCount in the Custom/MyApp namespace. You can then create an alarm that fires when ErrorCount exceeds 10 in a 5-minute period.

Filter Pattern Syntax

CloudWatch supports three pattern types: