The runbook nobody reads.

Every environment I have inherited came with a documentation wiki, and in every one of them the page that mattered was last edited by someone who left two years ago. The pages were not wrong exactly. They were written for a reader who has time. At three in the morning nobody has time.

Documentation optimises for the wrong reader

Most internal docs are written as if the audience is a new hire on their second week, calm, curious, reading top to bottom. The actual audience is a tired engineer who has been paged, has one hand on a terminal, and needs a single answer in under thirty seconds. Those two readers want completely different artifacts, and we keep serving the first one.

If the on-call engineer has to read a paragraph before they can type a command, the document has already failed.

Write the failure, not the system

An architecture overview does not help during an incident. What helps is a list of the ways this thing actually breaks, and for each one, the exact command that confirms it and the exact command that fixes it. Organise by symptom, because symptom is all the reader has.

## Symptom: API returns 502 for all routes

Confirm:
    systemctl status api
    journalctl -u api --since "10 min ago" | tail -50

Most likely cause:
    Worker pool exhausted. Happens under sustained
    > 200 req/s because the pool is fixed at 32.

Fix:
    systemctl restart api        # buys ~20 minutes
    Then scale: edit WORKERS in /etc/api.env, restart.

Escalate if:
    Restart does not clear it within 2 minutes.
    Page the database owner - this pattern also
    appears when Postgres connections are saturated.

Four headings. No prose. A reader can act on this while half asleep, which is the only test that counts.

Verification beats description

The most common defect in a runbook is a step that says "check that the service is healthy." Healthy according to what? Replace every subjective instruction with a command and an expected output. If you cannot write the expected output, you do not actually know what healthy means, and that is worth discovering on a Tuesday afternoon instead of during an outage.

#!/usr/bin/env bash
set -euo pipefail

# Not "check the service is up" - assert it, exit non-zero if not.
code=$(curl -s -o /dev/null -w '%{http_code}' \
        --max-time 5 http://127.0.0.1:8080/healthz)

if [ "$code" != "200" ]; then
  echo "UNHEALTHY: /healthz returned $code (expected 200)" >&2
  exit 1
fi

echo "OK: /healthz 200"

The three tests

Before a runbook counts as finished I put it through three checks, and most drafts fail at least one:

• The stranger test. Someone who did not build the system follows it end to end, alone, without asking questions. If they get stuck, the document is wrong — not the person.
• The one-page test. If the recovery path does not fit on one screen, split it. Nobody scrolls during an incident.
• The expiry test. Every runbook names an owner and a review date. An unowned document decays into a lie with a timestamp.

Automate the parts you wrote down twice

A runbook step that gets executed the same way every single time is not documentation, it is an unimplemented script. Once you have written the same three commands into two different pages, that is the signal to turn it into a command with a name and a rollback path.

# Before: three lines in a wiki that everyone
# copies slightly differently.
#
# After: one verb, one rollback, in version control.

ops restart api --reason "worker pool exhausted"
ops rollback api --to previous

What remains in the document is only the part that requires judgment. That is the part worth writing well, and it is usually a lot shorter than the wiki page it replaces.

The honest version

Documentation is not a deliverable you finish. It is a running cost you either pay in small amounts continuously, or pay all at once during an incident with interest. I have paid it both ways. The second way is significantly more expensive and considerably less fun.

Joseph Ali — Director of IT & Technology. hello@josephali.com