Skip to content
Irregularpedia Logo Irregularpedia

Disaster Recovery

Disaster Recovery

Section titled “Disaster Recovery”

Backups answer “can I get the data back?” Disaster recovery (DR) answers “can I get the service back, fast enough, when something goes badly wrong?” — hardware failure, a region outage, ransomware, or a fat-fingered DROP TABLE. DR is a plan, exercised regularly; backups are one input to it.

RPO and RTO

Section titled “RPO and RTO”

Two numbers drive every DR decision:

  • RPO — Recovery Point Objective: how much data you can afford to lose, measured in time. Nightly backups → up to 24h RPO. Continuous replication → seconds.
  • RTO — Recovery Time Objective: how long you can afford to be down while restoring. Restoring 2 TB from cold object storage might be a 6-hour RTO; a warm standby might be minutes.

Set these per service before choosing architecture — they tell you how much to spend. A hobby blog tolerates a 24h RPO / 1-day RTO; a payments service does not.

StrategyTypical RPOTypical RTOCost
Nightly off-site backupsUp to 24hHours$
Frequent snapshots + off-siteMinutes–hoursHours$$
Warm standby (active-passive)MinutesMinutes$$$
Active-active replication~Zero~Zero$$$$

Backups vs DR

Section titled “Backups vs DR”

A backup is necessary but not sufficient. DR also needs: documented restore procedures, somewhere to restore to, the credentials/keys to do it, and a tested order of operations (DNS, database, app, cache). Make backups themselves resilient — encrypted and off-site, per Backups with Borg or Backups with Restic — and immutable/append-only where possible so ransomware can’t take them with you.

Failover patterns

Section titled “Failover patterns”
  • Active-passive (warm standby): a second environment is kept updated (data replicated, config in sync) but idle. On failure you promote it and repoint DNS or a floating IP. Lower cost; RTO is “time to promote + DNS TTL.”
  • Active-active: traffic is served from two or more environments simultaneously behind a load balancer / GeoDNS. Near-zero RTO/RPO, but you must handle data consistency across sites. Highest cost and complexity.
  • Pilot light: only the irreplaceable core (e.g. the database) runs in the standby region; the rest is provisioned on demand during recovery. A cost middle ground.

Runbooks

Section titled “Runbooks”

A runbook is a step-by-step recovery procedure that someone tired and stressed — or someone other than you — can follow at 03:00. Each should include:

  • Trigger / scope (what failed, how to confirm it).
  • Pre-checks (is it actually down, or is it the monitoring?).
  • Exact recovery steps in order, with the precise commands.
  • Where the credentials, keys, and backups live (not the secrets themselves).
  • Validation steps and a rollback path.

Store runbooks outside the systems they recover — a recovery doc only reachable on the down server is useless. Version them in git and keep an offline copy.

Tabletop exercises

Section titled “Tabletop exercises”

A DR plan is unproven until rehearsed. A tabletop walks the team through a scenario (“the primary database host is unrecoverable”) and follows the runbook on real-ish infrastructure — ideally an actual restore into a scratch environment. Run them on a cadence (quarterly is common). They surface the gaps that matter: a missing key, an out-of-date runbook, an RTO that’s fantasy. See the restore-drill sections in the backup guides for the data half of this.

Post-mortems

Section titled “Post-mortems”

After any real incident (and after tabletop failures), write a blameless post-mortem: timeline, impact, root cause, what went well, what didn’t, and concrete action items with owners. The goal is fixing systems, not assigning blame. A simple template:

# Incident: <title>            Date: <date>   Severity: <SEV>
## Summary        (2-3 sentences: what happened, impact, duration)
## Timeline       (UTC timestamps: detection -> mitigation -> resolution)
## Root cause     (the actual underlying cause, not just the trigger)
## What went well / What went poorly
## Action items   (owner + due date for each)
Section titled “Related”