Logging Fundamentals

Logs are timestamped, immutable records of discrete events in your system. They provide the detailed context that metrics cannot — the exact error message, the stack trace, the user ID, the request payload. Effective logging is essential for debugging, auditing, and understanding system behaviour.

---

Structured vs Unstructured Logging

Unstructured Logs

Traditional plain-text logs are difficult to parse and search:

2025-03-15 10:23:45 ERROR PaymentService - Failed to charge card for user 12345: insufficient funds

Structured Logs

Structured logs use a consistent format (typically JSON) with key-value pairs:

{
  "timestamp": "2025-03-15T10:23:45Z",
  "level": "ERROR",
  "service": "payment-service",
  "message": "Failed to charge card",
  "user_id": "12345",
  "error": "insufficient funds",
  "trace_id": "abc123def456",
  "amount": 49.99,
  "currency": "GBP"
}

Always use structured logging. It enables:

• Efficient filtering and searching
• Automated parsing and indexing
• Correlation with metrics and traces (via trace_id)
• Machine-readable analysis

---

Log Levels

Standard log levels provide severity classification:

Level	Usage	Example
TRACE	Very fine-grained debugging	Function entry/exit
DEBUG	Detailed diagnostic information	Variable values, query parameters
INFO	Normal operational events	Request completed, job started
WARN	Potentially harmful situations	Retry attempt, deprecated API usage
ERROR	Error events that allow the application to continue	Failed database query, API timeout
FATAL	Severe errors causing application shutdown	Cannot connect to database on startup

Best Practices for Log Levels

• Use INFO for happy-path operational events
• Use WARN for recoverable issues that need attention
• Use ERROR for failures requiring investigation
• Avoid DEBUG in production unless temporarily enabled for investigation
• Never log sensitive data (passwords, tokens, PII) at any level

---

Log Aggregation Architecture

In a distributed system, logs must be collected from many sources and centralised:

┌───────────┐     ┌────────────┐     ┌────────────┐     ┌──────────┐
│   App 1   │────►│            │     │            │     │          │
├───────────┤     │  Collector │────►│   Index /  │────►│  Query   │
│   App 2   │────►│  (Fluentd, │     │   Store    │     │   UI     │
├───────────┤     │  Fluent    │     │(Elasticsearch│    │(Grafana, │
│   App 3   │────►│  Bit,      │     │  Loki, S3) │     │ Kibana)  │
└───────────┘     │  Logstash) │     └────────────┘     └──────────┘
                  └────────────┘

---

The ELK Stack

The ELK stack (Elasticsearch, Logstash, Kibana) is the traditional log aggregation solution: