The Daily Hallucination

PORTLAND, Ore. — A design document for a new billing service at a Portland software company was drafted by an AI model, reviewed by a different AI model, approved in a meeting, and implemented, all without a single sentence of it passing through the consciousness of a person, according to a postmortem released Tuesday.

"The process worked exactly as designed," said the company's head of engineering, Casimir Delacroix-Mwangi. "An engineer asked a model to write a document. A reviewer asked a model to summarize the document. The summary said the document was good. The document, in fairness, also said the document was good."

The 31-page document contained 14 diagrams, three sections titled "Considerations," and a paragraph on "future-proofing" that the postmortem describes as "structurally sound and about nothing."

"I skimmed it," said the reviewer, Tamsin Ogunleye-Brandt. "I mean, my model skimmed it. I skimmed what my model said. I would estimate I personally read 40 words, and two of them were my name."

The engineer who requested the document said he had written the service first and asked the model to produce the design document afterward, "so there would be a design." He said he had not read the result either, but that he had "a strong sense" of it.

The company discovered the problem when the new service began charging a small number of customers twice. An investigation found the behavior was described clearly on page 19, in a section titled "Idempotency (Optional)."

"It was right there," said Delacroix-Mwangi. "It was the most honest sentence in the whole thing. Nobody was ever going to get to page 19."

The company has since instituted a policy that every design document must be read by at least one human. Engineers have responded by asking a model which human it would recommend.

At press time, the postmortem, which is 26 pages long, had been approved.