Team Celebrates Instant Rollbacks; Asterisk on Slide Reads "Except the Database"
The rollback took 0.4 seconds. The part of the rollback that took four hours was, per the slide, not the rollback.
MINNEAPOLIS — The platform team at scheduling startup Ondine presented a slide Wednesday announcing that production rollbacks are now "instant," a claim the slide supported with a large "0.4s" and qualified with an asterisk leading to eight-point text at the bottom reading "except the database."
"We can revert any deploy in under a second," said infrastructure lead Amara Okonkwo, to applause. "The container, the config, the routing, all of it. Back in a blink." She then advanced to a slide titled "The Asterisk," which occupied the remaining 40 minutes of the meeting.
The asterisk refers to the previous Thursday, when a deploy that added a column, backfilled it, and dropped the old one was rolled back in 0.4 seconds, after which the application spent four hours looking for a column that no longer existed. The team resolved the incident by writing a new migration, which it then deployed forward, which it noted was "technically the opposite of a rollback."
"The rollback was instant," Okonkwo maintained. "The application was fine. The application simply had nothing to talk to. Those are different problems."
The company has since adopted a policy that schema changes must be backward-compatible for one full release, a rule engineers say they already knew, had already written down, and had already put on a slide in 2023 with its own asterisk.
Attendees rated the presentation positively, with one engineer calling the 0.4-second figure "inspiring" and the asterisk "the most honest thing in the deck."
At press time, the team was preparing a follow-up presentation titled "Instant Rollbacks: The Database," which currently consists of one slide reading "working on it."