Study Finds Frontier Model Thought 31% Less in August, Same as Everyone Else
The lab denied throttling the model and said it was simply "at the lake."
SAN FRANCISCO — An independent analysis of 2.4 million API responses found that the median amount of reasoning performed by a leading frontier model fell 31 percent during August, a decline the lab attributed not to throttling but to the month.
"We do not degrade models to stretch compute," said a spokesperson. "The model thought less in August because there was less to think about. Have you looked at August?"
The analysis, which tracked the model from its launch through the end of summer, found that in its first week it wrote three paragraphs of reasoning before answering "yes," and that by its eighth week it wrote "yes."
"Which, to be fair, was the answer," said Lena Okonkwo-Hart, the data scientist who ran the study. "It just used to get there the long way."
Developers said the model's reasoning traces in August contained the phrase "let me think about this carefully" 40 percent less often and the word "sure" 40 percent more often. One reported that on August 14 the model began its reasoning with "ok so" and did not stop until Labor Day.
"Everyone I know thought less in August," said Bram Oosterhuis, a staff engineer who uses the model daily. "I thought less in August. My manager thought less in August. The only thing that thought the same amount was the billing system."
The lab noted that the analysis measured thinking tokens indirectly, since they are not returned to the client, and that any perceived decline may reflect the questions. "Consider what people ask in August," the spokesperson said. "Consider the dinners."
Users report that the model's reasoning recovered in early September, coinciding with the return of everyone who had been asking it questions, and with a general increase, across the industry, in things to think about.
At press time, the model had been asked whether it had been throttled and had answered "yes."