Book
«Improving ratings»: audit in the British University system
by Marilyn Strathern · 1997 · 1 reading card
1 card
«Improving ratings»: audit in the British University system · 1997
An LLM judge is calibrated on humans and kept out of the optimisation loop.
Agent evaluations have three levels: the task (did it finish? in how many attempts?), the trajectory (took the right steps, called no dangerous tools) and cost with latency. Non-determinism calls for repeated runs and distributions, not a single score. An LLM judge scales, but it is biased (it favours long answers, position, its own phrasing): calibrate it on a human-labelled set, use it pairwise, rotate the models. Goodhart: if the agent or your prompt iterations are optimised against the judge, the score rises without the task being done any better; keep a held-out set, refresh it and watch for reward hacking (the agent edits the test instead of the code). The regression suite runs on every prompt or model change — it is the scaffold's unit tests.
“When a measure becomes a target, it ceases to be a good measure.”