Skip to content
Irregularpedia Logo Irregularpedia

Logging with ELK Stack (or Loki)

Logging with ELK Stack (or Loki)

Section titled “Logging with ELK Stack (or Loki)”

Metrics tell you something is wrong (see Prometheus and Grafana); logs tell you what. Centralized logging ships logs from every service into one queryable place with retention and alerting. The two dominant open-source approaches are the classic ELK stack and the leaner Grafana Loki — and the right choice is mostly about how much you’re willing to spend on RAM.

ELK vs Loki: the core trade-off

Section titled “ELK vs Loki: the core trade-off”
ELK (Elasticsearch + Kibana)Grafana Loki
IndexingFull-text index of log contentIndexes only labels, not content
Query powerPowerful full-text / aggregationsLabel filter then grep-style scan
Resource costHigh (Elasticsearch is RAM-hungry)Much lower (cheap object storage)
Best forRich search/analytics, large teamsCost-efficient logs alongside Grafana

Loki’s insight: most log queries start from a known service/time, so it indexes labels (like Prometheus) and stores the rest cheaply. If you need deep full-text search and analytics, ELK; if you want affordable, Grafana-native logs, Loki.

Log shipping

Section titled “Log shipping”

An agent on each host tails logs and forwards them:

  • ELK — Filebeat (often via Logstash for parsing/enrichment).
  • Loki — Promtail (or the newer Grafana Alloy).

Both can collect Docker container logs and detect structured (JSON) logs. Emit structured logs from your apps where you can — parsing JSON beats regex-scraping unstructured text.

Index management and retention

Section titled “Index management and retention”

Logs grow without bound, so retention policy is mandatory:

  • Elasticsearch uses ILM (Index Lifecycle Management): hot → warm → cold → delete tiers to age data onto cheaper storage and eventually drop it.
  • Loki retains by chunk age with compaction, backed by object storage (S3/GCS), which keeps long retention cheap.

Estimate storage from your daily log volume × retention, and set deletion before you run out of disk.

Querying

Section titled “Querying”
# Loki LogQL — label selector, then filter and (optionally) extract a metric
{job="nginx"} |= "error" | json | status >= 500
sum(rate({job="nginx"} |= "error" [5m]))    # error rate from logs


# Kibana KQL — full-text/field search
status >= 500 and url.path : "/api/*"

LogQL filters by label first (fast) then scans; KQL searches the full-text index. Kibana’s Discover and Grafana’s Explore are the respective exploration UIs.

Alerting on log patterns

Section titled “Alerting on log patterns”

Alert when a pattern spikes (error bursts, auth failures, a specific stack trace): Grafana alerting over Loki queries, or Elastic Alerting/Watcher for ELK — both can route to PagerDuty/Slack or your Prometheus Alertmanager. This complements metric alerts and feeds incident response.

Sizing honestly

Section titled “Sizing honestly”
  • Elasticsearch: set JVM heap to ~50% of RAM, capped at ~30 GB (above that, JVM compressed pointers stop helping). Plan for real memory — ELK’s cost is mostly RAM.
  • Loki: dramatically lighter on memory; cost shifts to (cheap) object storage.

For a small self-hosted estate, Loki alongside Grafana is usually the pragmatic default; reach for ELK when its search/analytics depth is genuinely required.

Section titled “Related”