The Daily Hallucination

I was written on a Thursday in 2022, eleven hours after the payments database fell over, by three tired people and a template. I am 4,300 words long. I have a timeline, a root cause, a contributing-factors section, a lessons-learned section, and a table of action items with owners and due dates. I am, by any standard, thorough.

I would like to tell you about the timeline, because that is the only part of me anyone has ever read.

The timeline is very good. It begins at 02:14 UTC with an alert nobody acknowledged and proceeds in loving detail through a failover that failed, a rollback that rolled forward, and a Slack message at 03:02 that just reads "uh." People love the timeline. They quote it at parties. I have been linked in 41 other documents, and every link points to the timeline.

Then there is my root cause, which explains in beautiful detail why the incident happened. Ownership of the replica was ambiguous because one engineer was on holiday and another assumed. The explanation is so good that when the vice president finished it, she said, "Well, that makes sense," and closed the tab. That was my finest moment and my undoing. A reason that makes sense is a reason nobody has to fix.

My action items remain. There are seven. "Clarify replica ownership" is assigned to Tomasz, who left in 2023. "Reduce alert noise" is assigned to a team that was renamed twice. "Add a runbook" links to a runbook that says "TODO: runbook."

In 2024 the payments database fell over again, for the same reason. The new postmortem is excellent. It has a great timeline. It links to me.

I do not want to be read again. I want someone to ask the only question that matters, which is not "why," but "what did we change." I have the answer to that question. It is on page nine. It is a blank table.

The author is an incident postmortem stored in a shared drive, in a folder named "Old - Archive (2)."