“The very best that intelligence can do is to make the most of the evidence without making more of the evidence than it deserves.”Jack Zlotnick (CIA, Center for the Study of Intelligence) · Bayes' Theorem for Intelligence Analysis · 1972 · Jack Zlotnick, «Bayes' Theorem for Intelligence Analysis», Studies in Intelligence vol. 16, nr. 2 (1972), paragrafele introductive
An item of evidence matters only to the extent that it is more likely under one hypothesis than under the other.
The CIA sponsored in-house research on applying Bayes' theorem to intelligence analysis, and Jack Zlotnick, an analyst who took part, described the method in Studies in Intelligence. It uses the odds form: the revised odds (R) equal the prior odds (P) multiplied by the likelihood ratio (L). Analysts never judged the conclusion directly; they judged only L for each new item — how many times more likely, say, a troop deployment to a border is if war is coming than if it is not. If twice as likely, L = 2 and the odds double. R then becomes the P for the next item. The steps: frame two mutually exclusive hypotheses; fix the starting odds explicitly; for each item of evidence, estimate separately how likely it is under each hypothesis; multiply and move on. It pays off when evidence arrives in a stream and the temptation is to react to the latest item. Zlotnick names a limit: close to the climax, much incoming evidence becomes undiagnostic — equally likely under both hypotheses. A second classic pitfall is treating two reports from the same source as independent; multiplied as if they were, they inflate the conclusion. The typical error remains mistaking evidence that is consistent with a hypothesis for evidence that is diagnostic.
Why it matters Without an explicit updating rule, the latest item of evidence gets the most weight merely because it is the latest, not because it says more.