Skip to content
Irregularpedia Logo Irregularpedia

Monitoring with Prometheus and Grafana

Monitoring with Prometheus and Grafana

Section titled “Monitoring with Prometheus and Grafana”

Prometheus (metrics + alerting) paired with Grafana (dashboards) is the de-facto open-source monitoring stack. Prometheus scrapes numeric time-series from your services, you query and alert on them, and Grafana visualizes them. This guide gets you from zero to a working, alert-capable stack.

Prometheus architecture

Section titled “Prometheus architecture”

Prometheus is pull-based: it scrapes an HTTP /metrics endpoint on each target on an interval, rather than receiving pushed data. Each sample is a time series identified by a metric name plus key/value labels.

prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["server1:9100", "server2:9100"]

Prometheus stores data locally (its TSDB) with a default retention of ~15 days — fine for operational alerting, not for long-term history (see below).

node_exporter

Section titled “node_exporter”

node_exporter exposes host metrics (CPU, memory, disk, network, filesystem) on port 9100. Run it on every server (systemd or a container), point a scrape job at it, and you immediately have host visibility. Sibling exporters cover other systems — cadvisor for Docker containers, postgres_exporter for PostgreSQL.

PromQL

Section titled “PromQL”

Query with PromQL. The most important rule: use rate() on counters over a range to get per-second rates; never graph a raw counter.

rate(node_network_receive_bytes_total[5m])              # bytes/sec, averaged over 5m
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)  # CPU %
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))         # p95 latency

Use recording rules to precompute expensive queries. Watch the gotchas: rate() vs irate(), counter resets (handled by rate), and histogram_quantile needing a _bucket series.

Grafana dashboards

Section titled “Grafana dashboards”

Add Prometheus as a data source, then build panels or import community dashboards from grafana.com/dashboards (e.g. “Node Exporter Full”). Use template variables (e.g. a $instance dropdown) so one dashboard serves all hosts instead of one-per-server.

Alerting with Alertmanager

Section titled “Alerting with Alertmanager”

Alert rules live in Prometheus; Alertmanager handles routing, grouping, inhibition, silencing, and delivery (email, Slack, PagerDuty).

alert.rules.yml
groups:
  - name: host
    rules:
      - alert: DiskWillFillSoon
        expr: predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0
        for: 1h
        labels: { severity: warning }
        annotations: { summary: "Disk on {{ $labels.instance }} fills within 24h" }

Alert on symptoms users feel (latency, errors, saturation) plus leading indicators (disk filling) — not on every metric, or you’ll train yourself to ignore alerts.

Long-term storage

Section titled “Long-term storage”

Prometheus’s local TSDB isn’t built for months/years of retention or global querying. When you need that, add Thanos or Grafana Mimir (or Cortex) — they provide durable object-storage-backed retention, downsampling, and a global query view across many Prometheus servers. Adopt them only when you actually need long retention or multi-cluster aggregation; they add real operational complexity.

Section titled “Related”