AI Code Reviewer Leaves 340 Comments on Pull Request, Every One Beginning "Great Work"
The bot approved the change, requested changes, and approved it again, which the team says is "within normal parameters for a reviewer."
PORTLAND, ORE. — A pull request modifying 14 lines of a date-formatting helper received 340 comments from an automated code reviewer Thursday, each of which opened with the phrase "Great work," a tone the bot's maintainers say was configured to "keep morale high while being extremely thorough."
"Great work. This function may return null when the input is null," read the first comment. "Great work. Consider whether this comment is necessary," read the 212th, which was attached to a comment the bot had left earlier. Comment 340 read, in full, "Great work."
The change's author, backend engineer Desmond Ferrante, said he read all 340. "About 60 were genuinely right. I fixed those. About 200 were the same thing said in different places. And about 80 were about the tests, which it also wrote, in a previous pull request, which it also reviewed."
The bot then approved the pull request, requested changes eleven minutes later after "reconsidering the null case," and approved it again, a sequence the engineering manager described as "honestly what a senior reviewer does, just compressed."
Company data shows that since the bot's introduction, the median pull request receives 4.6 times more comments and merges 30% faster, a combination the platform team calls "the paradox" and has not attempted to explain.
The bot's maintainers said they are working on a change that would let it leave fewer comments, but noted that the bot reviewed that change and left 41 comments, all beginning "Great work."
At press time, Ferrante had opened a new pull request titled "please just tell me if it is fine."