Prometheus Fundamentals

Prometheus is an open-source monitoring and alerting toolkit originally built at SoundCloud and now a graduated project of the Cloud Native Computing Foundation (CNCF). It has become the de facto standard for metrics collection in cloud-native environments.

---

A Brief History

• 2012 — Prometheus development begins at SoundCloud, inspired by Google's Borgmon
• 2015 — Prometheus is open-sourced
• 2016 — Prometheus joins the CNCF as the second hosted project (after Kubernetes)
• 2018 — Prometheus graduates from the CNCF
• Today — Prometheus is the most widely adopted open-source monitoring system

---

Architecture Overview

Prometheus follows a pull-based architecture:

┌─────────────┐     scrape      ┌──────────────┐
│  Prometheus  │ ◄───────────── │  Application  │
│   Server     │     /metrics   │  (exporter)   │
└──────┬──────┘                 └──────────────┘
       │
       │  store
       ▼
┌─────────────┐     query      ┌──────────────┐
│   TSDB      │ ◄───────────── │   Grafana    │
│ (local disk)│    PromQL      │  Dashboard   │
└─────────────┘                └──────────────┘
       │
       │  evaluate
       ▼
┌─────────────┐     notify     ┌──────────────┐
│   Rules     │ ──────────────►│ Alertmanager │
│  (alerts)   │                └──────────────┘

Key Components

Component	Role
Prometheus Server	Scrapes targets, stores time series, evaluates rules
Exporters	Expose metrics in Prometheus format from third-party systems
Pushgateway	Accepts metrics from short-lived batch jobs
Alertmanager	Handles alert routing, deduplication, and notification
Client libraries	Instrument application code (Go, Java, Python, etc.)

---

Pull vs Push

Prometheus uses a pull (scrape) model:

Aspect	Pull (Prometheus)	Push (StatsD, Datadog Agent)
Discovery	Prometheus discovers targets	Targets must know the collector
Health check	If a scrape fails, the target is down	No built-in health signal
Firewall	Prometheus needs access to targets	Targets need access to collector
Short-lived jobs	Use Pushgateway	Naturally supported

---

Configuration

Prometheus is configured via prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'my-app'
    metrics_path: /metrics
    static_configs:
      - targets: ['app:8080']

Service Discovery

Prometheus supports dynamic target discovery for cloud and container environments:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

Supported discovery mechanisms include: Kubernetes, Consul, EC2, Azure, GCE, DNS, file-based, and many more.

---