“When a measure becomes a target, it ceases to be a good measure.”Marilyn Strathern · «Improving ratings»: audit in the British University system · 1997 · «Improving ratings»: audit in the British University system, European Review 5(3), 1997 — formularea legii lui Goodhart
An LLM judge is calibrated on humans and kept out of the optimisation loop.
Agent evaluations have three levels: the task (did it finish? in how many attempts?), the trajectory (took the right steps, called no dangerous tools) and cost with latency. Non-determinism calls for repeated runs and distributions, not a single score. An LLM judge scales, but it is biased (it favours long answers, position, its own phrasing): calibrate it on a human-labelled set, use it pairwise, rotate the models. Goodhart: if the agent or your prompt iterations are optimised against the judge, the score rises without the task being done any better; keep a held-out set, refresh it and watch for reward hacking (the agent edits the test instead of the code). The regression suite runs on every prompt or model change — it is the scaffold's unit tests.
Why it matters The model changes every three months; without an evaluation suite, every change is a surprise in production.