“Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?”Brian W. Kernighan, P. J. Plauger · The Elements of Programming Style · 1974 · The Elements of Programming Style, ed. a 2-a (1978), cap. 2 «Expression»
Every model and tool call is a span with a parent, a cost and a prompt version.
The observability stack: one trace per run, with a span per model call (model, tokens in and out, latency, cost) and a span per tool call (arguments, result size, error); the OpenTelemetry semantic conventions for GenAI standardise the attribute names; content capture is gated (personal data, lesson 19); replay a single step with an edited prompt; dashboards on cost per task, iterations per stage, stuck runs; sample trajectories for human review weekly; version prompts like code and tag traces with the version. The debugging heuristic: read the observation before the thought — most "reasoning" failures are context failures (lesson 2).
Why it matters Without a trace, a failed run is an anecdote; with one, it is a regression test.