002 — — 5 min
Building for the long run.
Essay
Every system I have watched fail slowly failed the same way. Not a dramatic outage, not a breach — just a gradual accumulation of clever decisions that nobody remaining could explain. The code still ran. The understanding was gone.
Novelty has a maintenance bill
Choosing an unusual tool is not free. The bill arrives later, as a smaller hiring pool, thinner documentation, fewer people who have already hit your bug, and an upgrade path that one maintainer decides on a weekend. None of that shows up in the proof of concept, which is exactly why the proof of concept is a bad place to make the decision.
Pick the technology that will still be answerable on a Tuesday in three years, by someone who has not met you.
Boring is a design constraint, not a mood
Boring does not mean bad, slow, or unambitious. It means the interesting part of your system should be the part genuinely unique to your problem. Everything else — storage, queues, auth, deployment — should be the option a competent stranger would guess. Spend the novelty budget in one place and you get to keep it.
The practical version is a rule I apply on every project: if I cannot articulate why the obvious choice fails for us, we take the obvious choice. That single question kills most bad architecture before it exists.
Write down the reason, not just the result
Configuration records what was decided. It almost never records why. Six months later the why is the only thing anyone needs, because without it nobody dares change anything — and a system nobody dares change is already dying.
Decision: single Postgres instance, no read replicas.
Why: peak load is 40 req/s; replicas add failover
complexity we cannot staff at current headcount.
Rejected: managed multi-AZ cluster - 6x cost for a
durability guarantee we do not yet need.
Revisit: if sustained load passes ~300 req/s, or if
RPO requirement drops below 15 minutes.
The Revisit line is the important one. It converts a permanent judgment into a temporary one with an expiry condition, which is what it always was. Ten minutes to write, years of confident edits bought.
Operability beats elegance
The question that predicts whether a system lasts is not "is this well designed." It is "at three in the morning, with no context, can someone find out what is wrong and fix it." That means real logs, obvious failure modes, a deploy that is one command, and a rollback that is the same command with a different argument.
# Good: one verb, one inverse, both in version control.
deploy web --version 1.24.0
deploy web --rollback
# Bad: a deploy script and a 9-step wiki page
# titled "Emergency Rollback Procedure (draft)".
- Every service states plainly what it is doing and what it just failed to do.
- Restoring from backup has been practised, not merely configured.
- The runbook fits on one page and has been read by someone who did not write it.
Dependencies are relationships, not downloads
Each dependency is a standing commitment to somebody else's release schedule, security posture, and mood. That is often a good trade — I am not writing my own TLS. But it should be a decision with a name attached, not a side effect of a tutorial. Before adding one I ask three questions: who maintains it, what happens the day they stop, and how much of it am I actually using.
The third question kills more dependencies than the first two combined. A great deal of what we pull in is forty lines of behaviour behind two megabytes of surface area.
The compounding part
None of this is impressive on any given day. It is impressive in aggregate, two years later, when the thing is still running, still being changed without fear, and still explainable to a new hire in an afternoon. That is the whole return. It only pays out if you commit before you need it.
Joseph Ali — Director of IT & Technology. hello@josephali.com