Building an Observability Stack

In this final lesson, we bring together everything covered in the course to design and build a complete observability stack. We will walk through architecture decisions, tool selection, implementation, and the organisational practices that make observability effective.

---

Architecture Overview

A production observability stack has four layers:

┌──────────────────────────────────────────────────────────┐
│                    Visualisation Layer                    │
│              Grafana (dashboards, explore, alerts)        │
└──────────────────────┬───────────────────────────────────┘
                       │
┌──────────────────────┼───────────────────────────────────┐
│                  Storage Layer                            │
│   Prometheus/Mimir    │    Loki       │    Tempo          │
│   (metrics)           │    (logs)     │    (traces)       │
└──────────────────────┬───────────────────────────────────┘
                       │
┌──────────────────────┼───────────────────────────────────┐
│               Collection Layer                            │
│  OTel Collector  │  Promtail/Fluent Bit  │  Prometheus    │
└──────────────────────┬───────────────────────────────────┘
                       │
┌──────────────────────┼───────────────────────────────────┐
│             Instrumentation Layer                         │
│   Application code with OTel SDK, client libraries       │
└──────────────────────────────────────────────────────────┘

---

Tool Selection

Open Source Stack (Grafana LGTM)

Layer	Tool	Purpose
Metrics	Prometheus + Mimir	Collection + long-term storage
Logs	Loki + Promtail	Aggregation + collection
Traces	Tempo + OTel Collector	Storage + collection
Visualisation	Grafana	Dashboards, alerting, exploration
Alerting	Alertmanager	Routing and notification

Commercial Alternatives