Postmortems in Public: Why We Publish Every Outage
Our competitors think we are giving away ammunition. Three years of public incident reports say otherwise.
Nadia Fadel
Head of SRE
In March 2023, Ledger was read-only for 71 minutes. A certificate rotation job, perfectly written, ran twice because a cron expression survived a migration that should have deleted it. Clients could still verify receipts — reads never degraded — but new entries queued. Severity 3. Nobody lost work. Nobody lost data. By the standards of our industry, it was a Tuesday.
We published the whole thing: the timeline to the minute, the surviving cron expression itself (redacted only in its infrastructure identifiers), the fix, and — the part that made our lawyers blink — the name of the migration review that missed it. Within a week, three clients had found and fixed the same pattern in their own cron estates. One of them had the identical bug, live, in a webhook retry path.
What we actually learned
People assume public postmortems are about transparency branding. They are not; they are about quality. Writing for strangers changes how you write. An internal postmortem drifts toward protecting feelings and careers. A public one has to survive an engineer at another company reading it skeptically at midnight — so the timeline has to be exact, the causal chain has to be airtight, and the action items have to be ones you are willing to be quoted on. Publication is a quality gate disguised as a communications policy.
- Rule 1 — every incident above severity 3 is published within 72 hours, no exceptions, no committee.
- Rule 2 — blame systems, not people. If a person’s action was the trigger, the system that allowed one action to matter is the finding.
- Rule 3 — the fix ships before the report publishes. We do not narrate intentions.