What are evals, and why is it always the evals’ fault

Black mug with a grey robot at a desk holding a cup next to a burning clipboard in a bin, between the lines THE MODEL IS FINE and THE EVALS ARE WRONG

One sentence gets said in machine learning meetings in a particular tone of voice, usually just after a chart has gone the wrong way: the model is fine, the evals are wrong. Sometimes the speaker is joking. Sometimes they are right. The fun is that nobody in the room can tell which, at first.

What an eval is

An eval, short for evaluation, is a test for an AI model. It is a set of questions or tasks plus a way of scoring the answers. Some are public benchmarks that everyone runs, so models can be compared: maths problems, coding tasks, exam-style questions. Others are private, built by a team to check the specific thing their product needs, such as whether a support bot gives the right refund policy.

Scoring is the tricky part. Sometimes there is one right answer and you check for it. For code, you can run the program against tests. For writing, you need judgement, so either people rate the answers or, increasingly, another AI model does the grading.

How evals go wrong

Quite a few ways, which is why the sentence is never entirely a joke.

The answers leaked. Public benchmarks live on the internet, and models are trained on the internet. If test questions ended up in the training data, a high score may mean the model remembers the answers rather than knows how to work them out. This is called contamination.

The answer key is wrong. Benchmarks are made by people, and some contain mislabelled questions. A model that gives the correct answer gets marked down.

The test is too easy. Once the best models all score close to full marks, the benchmark stops telling them apart.

The grader has opinions. Models used as judges can favour longer answers, or answers written in a style like their own.

The score and the users disagree. A model can improve on every chart while the people using it say it got worse, because the evals measure something slightly different from what people care about.

And the reverse failure: teams who check the same eval too often can end up tuning the model to that test, which is overfitting by another route.

Why it is funny

Because everyone has done the human version. The exam was unfair. The referee was blind. The performance review missed the important work. When the result is bad, questioning the measurement is the oldest move there is.

What makes it particularly good in machine learning is that the move is often legitimate. Evals really are that fragile. So the person saying it may be defending a broken model, or may have found a genuine flaw in the test, and the only way to know is to go and check. Which nobody wants to do at 6 p.m.

On a mug, and a shirt

The Model Is Fine, the Evals Are Wrong is a €16 mug, and the line also comes as a tee and a hoodie. The robot badge version, €26, has a grey robot shrugging with a report card full of red marks pinned to its chest.

For the person who builds these tests, the machine learning gift guide has more. Reward hacking is the related problem where the model games the score, and the rest of the range sits in Agents at work.

Shopping Cart
Scroll to Top