Prometheus Fundamentals

Prometheus is an open-source monitoring and alerting toolkit originally built at SoundCloud and now a graduated project of the Cloud Native Computing Foundation (CNCF). It has become the de facto standard for metrics collection in cloud-native environments.

---

A Brief History

• 2012 — Prometheus development begins at SoundCloud, inspired by Google's Borgmon
• 2015 — Prometheus is open-sourced
• 2016 — Prometheus joins the CNCF as the second hosted project (after Kubernetes)
• 2018 — Prometheus graduates from the CNCF
• Today — Prometheus is the most widely adopted open-source monitoring system

---

Architecture Overview

Prometheus follows a pull-based architecture:

graph TD
  Prometheus["Prometheus Server"] -->|"scrape /metrics"| App["Application (exporter)"]
  Prometheus -->|"store"| TSDB["TSDB (local disk)"]
  Grafana["Grafana Dashboard"] -->|"query / PromQL"| TSDB
  TSDB -->|"evaluate"| Rules["Rules (alerts)"]
  Rules -->|"notify"| Alertmanager["Alertmanager"]

Key Components

Component	Role
Prometheus Server	Scrapes targets, stores time series, evaluates rules
Exporters	Expose metrics in Prometheus format from third-party systems
Pushgateway	Accepts metrics from short-lived batch jobs
Alertmanager	Handles alert routing, deduplication, and notification
Client libraries	Instrument application code (Go, Java, Python, etc.)

---

Pull vs Push

Prometheus uses a pull (scrape) model:

Aspect	Pull (Prometheus)	Push (StatsD, Datadog Agent)
Discovery	Prometheus discovers targets	Targets must know the collector
Health check	If a scrape fails, the target is down	No built-in health signal
Firewall	Prometheus needs access to targets	Targets need access to collector
Short-lived jobs	Use Pushgateway	Naturally supported

---

Configuration

Prometheus is configured via prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'my-app'
    metrics_path: /metrics
    static_configs:
      - targets: ['app:8080']

Service Discovery

Prometheus supports dynamic target discovery for cloud and container environments:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

Supported discovery mechanisms include: Kubernetes, Consul, EC2, Azure, GCE, DNS, file-based, and many more.

---

PromQL — Prometheus Query Language

PromQL is a powerful, functional query language for selecting and aggregating time series data.

Instant Vectors

http_requests_total{job="api", status="200"}

Range Vectors

http_requests_total{job="api"}[5m]