“Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?”Brian W. Kernighan, P. J. Plauger · The Elements of Programming Style · 1974 · The Elements of Programming Style, ed. a 2-a (1978), cap. 2 «Expression»
Every model and tool call is a span with a parent, a cost and a prompt version.
The observability stack: one trace per run, with a span per model call (model, tokens in and out, latency, cost) and a span per tool call (arguments, result size, error); the OpenTelemetry semantic conventions for GenAI standardise the attribute names; content capture is gated (personal data, lesson 19); replay a single step with an edited prompt; dashboards on cost per task, iterations per stage, stuck runs; sample trajectories for human review weekly; version prompts like code and tag traces with the version. The debugging heuristic: read the observation before the thought — most "reasoning" failures are context failures (lesson 2).
Kernighan and Plauger insisted that good programs are written for the reader, not the machine. The same rule now applies to an agent's traces. A good trace is not just a recording; it is a text you can read and compare. When a run fails, you don't look at it alone, but next to a successful run with the same prompt. The difference between the two spans shows exactly where behavior changed: different context sent, different tool called, different reply. It's like when the fridge stops cooling and you compare this month's bill with last month's to see what got added. The next question is what to do with the difference you found: how to turn it into a test that stays.
Once the difference is found, the step Kernighan and Plauger demanded of every programmer follows: replay the case, don't guess it. For agents, replaying means taking the exact span that failed, with its real context, and running it again with a single change, say an edited prompt. If the result is fixed, you've isolated the cause; if not, the cause lies further down the chain. It's like when a bulb won't light and you swap only the bulb, not the whole house wiring, to know who's at fault. Each successful replay then becomes a regression test, kept next to the trace that produced it. But one fixed run says nothing about the rest of the traffic; the next question is what you choose to look at when you can't read it all.
Why it matters Without a trace, a failed run is an anecdote; with one, it is a regression test.