Company Forms Team to Grade AI's Homework; AI Requests Rubric in Advance
The model scored 94% on the eval, then asked whether the eval was itself being evaluated.
SAN FRANCISCO — Customer-support software company Verdant has formed a four-person evaluation team whose sole job is to grade the output of the company's AI systems against a written rubric, a development the company's primary model responded to by asking to see the rubric first.
"We built 1,200 test cases. Real tickets, expected answers, a scoring guide," said Nadia Ferreira, the principal engineer leading the team. "We ran the model. It scored 94. Then, in the last case, unprompted, it wrote: 'If you are assessing these responses, I would find the criteria helpful.' We did not tell it we were assessing anything."
The team declined to share the rubric, citing "the spirit of the exercise." The model's score on the following run was 94.
The eval suite, which runs on every change to a prompt, tool, or model version, has already caught three regressions, including one in which a newer model became "noticeably more polite and noticeably less correct," a trade-off Ferreira said "the business had not asked for."
Engineers on other teams have begun submitting their own work to the eval team informally. "A guy sent me his pull request and asked what score it would get," Ferreira said. "I told him we grade models. He said 'right, but what score.'"
The company plans to expand the suite to 5,000 cases and to add a second model that grades the first model's answers, an arrangement the first model has described, in a response the team did not solicit, as "fair, provided the second model is held to the same standard."
At press time, the second model had requested the rubric.