Go deeper
Intelligence analysis methods — 2026
The methods an analyst uses in 2026, from the ICD 203 standards and probabilistic language to ACH, indicators, red teaming, scenarios, calibrated forecasting and AI-assisted analysis. Every lesson has a quiz.
The reading track
- 01
Analysis is a mental process that runs through simplified models of reality; structure exists to bring the model into the open, where it can be checked.
Psychology of Intelligence Analysis · Richards J. Heuer Jr. · 1999
Heuer starts from Herbert Simon's idea of bounded rationality: the mind cannot take in the world's complexity directly, so it builds a simplified model and behaves rationally only inside it. For an analyst, that model — Heuer also calls it a mind-set — is the lens through which every new report is read. It is unavoidable and useful, since without it no information would make sense. The problem, Heuer says, is that such a model is quick to form but resistant to change, and the analyst does not see it, because he is looking through it. The practical steps follow. Before writing a judgement, note what you already believe and why; list the alternative explanations; ask which evidence would make you change your mind. Structure — a list, a matrix, a written hypothesis — does not make you cleverer, but it moves your thinking onto paper, where others can challenge it. It pays most on high-stakes issues with ambiguous information. The typical trap is believing that experience exempts you: Heuer warns that the experienced specialist may be among the last to see what is really happening when events take an unexpected turn.
Intelligence analysts should be self-conscious about their reasoning process. They should think about how they make judgments and reach conclusions, not just about the judgments and conclusions themselves.Richards J. Heuer Jr., «Psychology of Intelligence Analysis» (CIA, Center for the Study of Intelligence, 1999), capitolul 4 «Strategies for Analytical Judgment», prima frază după rezumatul capitolului
Why it matters An analyst who cannot state his mental model out loud has no way of noticing the moment reality has left it.
- 02
ICD 203 asks five things of every analytic product — objectivity, independence from politics, timeliness, all sources and tradecraft — and independence protects the other four.
Intelligence Community Directive 203: Analytic Standards · Office of the Director of National Intelligence (ODNI) · 2015
Directive 203 of the US Intelligence Community sets five standards for every analytic product: it must be objective, independent of political consideration, timely, based on all available sources, and must implement the nine analytic tradecraft standards. The text published by ODNI is the one signed on 2 January 2015, with a 2022 technical amendment that mainly concerns the role of the analytic ombuds, not the standards themselves. Objectivity means more than lack of bias: analysts must be aware of their own assumptions, use techniques that reveal bias, consider contrary information, and not stay tied to earlier judgements once the facts have changed. In practice it works as a pre-publication checklist: have we weighed alternative perspectives? Will the assessment reach the decision-maker before the decision? Have we used sources we would rather ignore? Have we said how sure we are, and why? The standards fit anywhere an assessment is written for someone who decides — a ministry, a bank or a newsroom. The typical trap is not lying but tailoring: an assessment written to please an audience, an agenda or a policy. That is why the text forbids both verbs — neither distorted by advocacy nor shaped for it.
Analytic assessments must not be distorted by, nor shaped for, advocacy of a particular audience, agenda, or policy viewpoint. Analytic judgments must not be influenced by the force of preference for a particular policy.ODNI, ICD 203 «Analytic Standards» (semnată la 2 ianuarie 2015, cu amendament tehnic din 2022), standardul analitic b «Independent of political consideration»
Why it matters An assessment tailored to its reader no longer says anything about the world, only about what the reader wanted to hear.
- 03
A probability word with no number behind it is read differently by every reader; the remedy is an agreed scale with numerical ranges.
Words of Estimative Probability · Sherman Kent · 1964
In March 1951 National Intelligence Estimate 29-51 called an attack on Yugoslavia that year a serious possibility. Kent, a member of the Board of National Estimates that had approved the text, had odds of about 65 to 35 in favour of an attack in mind; a State Department reader had understood a much lower chance, and Kent's colleagues each held a different figure, from 20 to 80 up to 80 to 20. His answer was a table: a few expressions, each tied to a range — probable around 75 per cent, almost certain around 93 per cent. Today ICD 203 requires a seven-step scale, from almost no chance (1–5 per cent) to almost certain (95–99 per cent). The British PHIA yardstick also has seven steps, from remote chance (up to about 5 per cent) to almost certain (from about 95 per cent), but deliberately leaves gaps between them. In practice: pick the number first, then the word from the scale; do not mix sets of terms; if you cannot defend the word, you do not yet have a judgement. The trap is what Kent called the poets' camp: phrases such as may well or distinctly possible, which suggest odds without committing to them.
It was another jolt to find that each Board member had had somewhat different odds in mind and the low man was thinking of about 20 to 80, the high of 80 to 20.Sherman Kent, «Words of Estimative Probability», Studies in Intelligence, vol. 8, nr. 4 (1964), secțiunea «Early Brush with Ambiguity»
Why it matters If writer and reader put different numbers under the same word, the assessment has said something other than intended, and neither notices.
- 04
Probability says how likely the event is; confidence says how solid the basis for saying so is. They are two axes and are written separately.
Explaining Uncertainty in UK Intelligence Assessment · Professional Head of Intelligence Assessment (PHIA), Guvernul Regatului Unit · 2025
These are two different questions: how likely the event is, and how solid your basis for saying so is. You can judge an attack likely on the strength of a single new source — high probability, low confidence — or unlikely on ten years of consistent data — low probability, high confidence. The British guidance grades analytical confidence (the Analytical Confidence Rating) as one of three levels — high, moderate, low — against three criteria: information base, analytical rigour, and complexity and volatility. Confidence also tells the reader how easily the judgement might change. The practical rule: express probability with a word from the scale, and explain confidence separately, with its reason — which source is missing, what could change the picture. ICD 203 goes down to grammar: a product using a confidence level must not combine it with a degree of likelihood in the same sentence. The reason is a phrase such as we have high confidence that it is unlikely, which the reader hears as a single number. The trap: lowering the probability when what you really lack is evidence. Missing evidence lowers confidence; it does not automatically move the event towards unlikely.
Whereas probability reflects the likelihood that a statement is true, analytical confidence reflects the soundness and stability of the foundations on which the assessment of likelihood has been made.PHIA, «Explaining Uncertainty in UK Intelligence Assessment» (GOV.UK, publicat pe 24 martie 2025), secțiunea «Analytical Confidence Rating (AnCR) Framework»
Why it matters A decision-maker given only the probability cannot tell whether the judgement will hold until tomorrow or flip at the next report.
- 05
Every report gets two independent grades: a letter for the source, based on its track record, and a figure for the information, based on corroboration and logic.
FM 2-22.3 Human Intelligence Collector Operations · Headquarters, Department of the Army (SUA) · 2006
Every report gets two grades, not one. The letter, from A to F, says how reliable the source is, judged on what it has reported before: A reliable, B usually reliable, C fairly reliable, D not usually reliable, E unreliable, and F cannot be judged, for a source with no track record. The figure, from 1 to 6, says how credible the information in this particular report is: 1 confirmed by other independent sources, 2 probably true, 3 possibly true, 4 doubtfully true, 5 improbable, and 6 cannot be judged. The grid is known as the Admiralty code, after its British origin, and the same logic of two separate grades appears in NATO intelligence doctrine. In practice: grade the source on what it has delivered so far, then the information on what it says now — internal logic, corroboration, consistency with everything else. The grades are kept apart because they can diverge: a B source can send a 5 report, and a new F source can bring something already confirmed. The method pays wherever many heterogeneous sources flow in, open sources included. The traps: reading F as liar — the manual says explicitly it only means no track record — and letting the source's reputation contaminate the grade of the information.
An “F” rating does not necessarily mean that the source cannot be trusted, but that there is no reporting history and therefore no basis for making a determination.FM 2-22.3 «Human Intelligence Collector Operations» (6 septembrie 2006), anexa B «Source and Information Reliability Matrix», paragraful B-1
Why it matters A single grade mixes two different questions and hides exactly the dangerous case: the good source that, this time, is wrong.
- 06
The quality of information check periodically tests whether the source base really supports the judgement; the number of sources is no substitute for their quality.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
The quality of information check assesses how complete and sound the source base behind a judgement is. The 2009 primer describes it as a continuous process, not a one-off step: otherwise important judgements stay anchored to weak information, and caveats once attached to them are forgotten. The steps: keep an inventory of sources by type and date, with their strengths and weaknesses; identify the most critical or compelling sources; check that critical reporting has sufficient and strong corroboration; re-examine previously dismissed information in the light of new facts; flag recalled reporting and review any analysis built on it. The result is an honest picture of what we know and what we do not, and it feeds straight into the confidence level. The method pays before any major assessment and whenever an analytic line has run for months. It is also where an adversary's deception strategies can be caught. The main trap is counting: five articles repeating the same source are not five confirmations but one, copied. In the finished product, ICD 203 requires the quality of sources to be described, and source summary statements are strongly encouraged.
Having multiple sources on an issue is not a substitute for having good information that has been thoroughly examined.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (Guvernul SUA, martie 2009), secțiunea «Quality of Information Check», rubrica «When to Use», p. 10
Why it matters A year-old judgement can rest on a source recalled months ago, and nobody notices unless someone periodically looks underneath it.
- 07
Every assessment rests on premises accepted without discussion; the key assumptions check writes them down, challenges them and says in advance what would overturn them.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
A key assumption is any hypothesis the analyst has accepted as true and on which the assessment rests — often without ever having stated it. The primer proposes four steps: write down the current analytic line where everyone can see it; list all the premises, stated or unstated, that must be true for the line to hold; challenge each one — why it must be true and whether it holds under all conditions; keep only the assumptions that really carry the conclusion, and note the conditions under which they would fail. The primer's example is the 2002 sniper case around Washington, where it was assumed the perpetrator was acting alone. At the CIA in the 1990s, Douglas MacEachin promoted a variant called linchpin analysis: assumptions about the key drivers become linchpins, the premises the argument stands on, and must be spelled out. The method pays at the start of a project, when it costs an hour or two, and again before judgements are finalised. The trap: the list turns into an inventory of platitudes, while the dangerous premise — the one nobody sees because everybody shares it — stays unwritten. The question that brings it out: if this proved wrong, would the conclusion change?
The goal is not to undermine or abandon key assumptions; rather, it is to make them explicit and identify what information or developments would demand rethinking them.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (Guvernul SUA, martie 2009), secțiunea «Key Assumptions Check», p. 7
Why it matters Assessments rarely fail because a fact was wrong; they fail because a premise nobody wrote down stopped being true.
- 08
Strategic intelligence failures cannot be removed by reorganisation: evidence is ambiguous, judgement ambivalent, and the great misses often happen at the decision-maker.
Analysis, War, and Decision: Why Intelligence Failures Are Inevitable · Richard K. Betts · 1978
In 1978 Betts reached an uncomfortable conclusion: strategic intelligence failures cannot be prevented by organisational solutions to problems of analysis and communication. Analytic certainty is precluded by ambiguity of evidence, ambivalence of judgement and the atrophy of reforms meant to avert failure. Many sources of error are unresolvable paradoxes and dilemmas, not curable pathologies. And the major misses — attack warning, operational evaluation, intelligence for strategic planning — stem mainly from leaders' psychological attributes rather than from analysts failing to detect the relevant data. The practical consequence is twofold. First, no method in this course guarantees you will not be wrong; methods reduce avoidable errors and make the rest visible. Second, analysis does not end at hand-over: a correct estimate that is heard and dismissed is still a failure. That is why it matters how you write — key judgement first, explicit probability, what would change it — and how you stay in dialogue with the decision-maker. The lesson is an antidote to two illusions: that the next reorganisation will solve the problem, and that the analyst alone is to blame. The symmetric trap is fatalism: inevitable does not mean indifferent.
Since analysis and decision are interactive rather than sequential processes, and authorities often hear but dismiss correct estimates, intelligence failure is inseparable from policy failure.Richard K. Betts, World Politics, vol. 31, nr. 1 (octombrie 1978), pp. 61–89 — rezumatul articolului, în forma publicată de editor (Cambridge University Press)
Why it matters Whoever believes a reorganisation eliminates failure will build yet another structure and be surprised again, by the same dilemmas.
- 09
Analysis of competing hypotheses makes every reasonable explanation compete on the same evidence and keeps the one the evidence refutes least.
Psychology of Intelligence Analysis · Richards J. Heuer Jr. (CIA, Center for the Study of Intelligence) · 1999
The method starts from an awkward observation about how we work: we pick the likely answer by intuition and then look for evidence that supports it. The problem is not that the evidence is false but that most of it fits other explanations just as well. Analysis of competing hypotheses (ACH) has eight steps in Heuer's version, and their core fits into three moves: write down every reasonable hypothesis, ideally with colleagues who think differently; build a matrix with hypotheses across the top and evidence down the side; mark, for each item, which hypotheses it is consistent with and which it is not. The key word is diagnosticity. A fever tells a doctor that the patient is ill, not which illness it is; evidence consistent with every hypothesis does not help you choose and drops out of the calculation. What remains are the few items that discriminate, and those are the ones you re-check first. Then you try to disprove rather than prove, report every hypothesis, and state in advance what would change your mind. The typical trap is confusing an unproven hypothesis with a disproved one. No indication that India will test soon does not mean India will not test — Heuer's example from 1998.
The most probable hypothesis is usually the one with the least evidence against it, not the one with the most evidence for it.Richards J. Heuer Jr., «Psychology of Intelligence Analysis» (CIA Center for the Study of Intelligence, 1999), capitolul 8 «Analysis of Competing Hypotheses», secțiunea «Summary and Conclusion», p. 108
Why it matters A report that counts the evidence for the favoured hypothesis looks solid right up to the moment the same evidence turns out to fit the alternative nobody tested.
- 10
Strategic surprise comes not from missing information but from faint signals buried in louder, more plausible noise.
Pearl Harbor: Warning and Decision · Roberta Wohlstetter · 1962
Roberta Wohlstetter's Bancroft Prize-winning study moved the Pearl Harbor question from who was asleep to why it was not seen. Her answer: the United States failed to anticipate the attack not for want of the relevant materials but because of a plethora of irrelevant ones. The signals were there, but they were few and faint, buried in a larger and louder mass of indications pointing elsewhere. In December 1941 there was ample evidence for all the wrong interpretations, and they looked wrong only after the event. The problem survives in the profession's vocabulary as the signal-to-noise ratio. For an analyst the lesson is practical. More collection does not solve the problem and can make it worse, because noise grows with volume. What helps is knowing in advance which signal you are listening for — explicit hypotheses and indicators defined before the crisis — and judging each clue in its own context rather than with hindsight. The typical trap is the retrospective verdict: it was obvious is almost always the illusion of someone who knows the ending. Her most uncomfortable conclusion: strategic warning cannot be counted on, so defences must be designed to work without it.
Signals that are characterized today as absolutely unequivocal warnings of surprise air attack on Pearl Harbor become, on analysis in the context of December 1941, not merely ambiguous but occasionally inconsistent with such an attack.Roberta Wohlstetter, «Pearl Harbor: Warning and Decision» (Stanford University Press, 1962), pp. 387–388 — pasaj reprodus identic în «Nuclear Heuristics: Selected Writings of Albert and Roberta Wohlstetter» (Strategic Studies Institute, 2009), introducerea editorilor
Why it matters A system that collects everything and does not know what it is looking for reproduces 1941: the signal is in the file but never rises above the noise.
- 11
Warning is not information you either have or lack but a judgement built from an indicator list prepared in advance.
Anticipating Surprise: Analysis for Strategic Warning · Cynthia M. Grabo (Joint Military Intelligence College) · 2002
Grabo wrote the text in the early 1970s as a classified textbook for warning analysts, drawing on some twenty-five years of experience. The US Defense Intelligence Agency published it in three classified volumes between 1972 and 1974, and the Joint Military Intelligence College later published the declassified, condensed version. Her starting distinction is the one non-specialists most often miss: an indicator is a known or theoretical step the adversary should or may take in preparing for hostilities — something anticipated and put on a watch list. An indication is information that the step is actually being taken. The method works in three moves: build the indicator list in advance for the scenario that worries you (a call-up of reservists, a deployment of forces, a military alert), track what gets ticked off, and judge the pattern rather than isolated pieces. The warning judgement concerns the probability of action, not its imminence, and it can come weeks or months ahead. It pays off wherever surprise is very costly and the adversary's preparations leave traces. The trap is treating warning as a commodity you either have or lack. It is a hypothesis, so it has to be issued before certainty arrives.
Warning is an intangible, an abstraction, a theory, a deduction, a perception, a belief. It is the product of reasoning or of logic, a hypothesis whose validity can be neither confirmed nor refuted until it is too late.Cynthia M. Grabo, «Anticipating Surprise: Analysis for Strategic Warning» (Joint Military Intelligence College, 2002), secțiunea «What Is Warning?» — «Warning Is Not a Commodity», p. 4
Why it matters Without an indicator list written before the crisis, every new clue is read however suits the moment, and the pattern only shows afterwards.
- 12
Devil's advocacy tests a single dominant consensus; Team A/Team B sets two equally strong positions against each other.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
Both techniques are contrarian: they do not build an analysis from scratch but attack an existing one. The difference between them is diagnostic, and the primer warns that analysts often confuse them. Devil's advocacy is used when there is ONE dominant consensus: an analyst with doubts, or one designated by a manager, builds the best possible case for an alternative explanation. The steps: summarise the mainline judgement and key assumptions; pick the assumptions most open to challenge; check whether any information is questionable, whether deception is possible, or whether major gaps exist; present the findings and, if needed, draft a contrarian paper clearly labelled as an exercise. Team A/Team B is used when TWO equally strong positions coexist: each side writes out its full argument, with assumptions and data in plain view, and the decision-maker judges which is stronger. It is costly, so it is kept for long-running disputes or far-reaching decisions; analysts are sometimes made to argue the position they do not hold, to expose their own mind-set. The traps: a challenger with no real mandate who merely performs, and an unlabelled contrarian product the reader mistakes for the official view. Reaffirming the original line, with more confidence, is also a valid outcome.
Its primary value is to serve as a check on a dominant mind-set that can develop over time among even the best analysts who have followed an issue and formed strong consensus that there is only one way of looking at their issue.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (US Government, martie 2009), secțiunea «Devil’s Advocacy» — «Value Added», p. 17
Why it matters A consensus years old quietly filters out contrary evidence; without a mandated challenger, nobody checks the assumptions it rests on any more.
- 13
Red teaming is the independent, structured challenge of your own plan in service of a better decision — not a team playing the enemy.
Red Teaming Handbook (3rd Edition) · UK Ministry of Defence — Development, Concepts and Doctrine Centre (DCDC) · 2021
The British handbook separates four notions that are easily blurred. Red teaming is the activity: the independent application of structured, creative and critical thinking techniques so the decision-maker chooses better. A red team is a team formed specifically to subject an organisation's plans, programmes, ideas and assumptions to rigorous analysis and challenge. A red cell is something else: it adopts the viewpoint or even the persona of an adversary or key stakeholder, estimates what they would do, and can role-play them in wargames. NATO calls a closely related approach alternative analysis: the deliberate application of independent, critical thought and alternative perspective. The third edition adds the red team mindset — quick techniques used routinely by anyone who decides, not only by a separate team. How it is applied: the team works for a specific decision-maker, the end user in the definition; it stays independent of the plan's authors; it picks the techniques that fit the question; it hands over findings the decision-maker can actually use. It pays off before large, hard-to-reverse decisions. The traps: a red team is not immune to groupthink — the handbook devotes a section to it — and a team that wins by demolishing the plan, instead of making it more robust, has missed the point.
Red teaming is defined as: the independent application of a range of structured, creative and critical thinking techniques to assist the end user make a better-informed decision or produce a more robust product.UK Ministry of Defence, DCDC, «Red Teaming Handbook», ediția a 3-a (iunie 2021), capitolul 1 «Introduction», paragraful 1.9
Why it matters A plan's authors cannot see its blind spots on their own; an independent team with explicit techniques finds them before the adversary does.
- 14
A premortem asks the team to assume the plan has already failed and explain why, so objections surface while there is still time.
Performing a Project Premortem · Gary Klein (Harvard Business Review) · 2007
The premortem uses a perspective trick that Klein calls prospective hindsight: you imagine the event has already happened. Research from 1989, cited in the article, found that this raises people's ability to correctly identify reasons for future outcomes by 30%. The steps are simple. The team is briefed on the plan. The leader announces that the project has failed spectacularly. For a few minutes, everyone independently writes down every reason they can think of — especially the ones they would ordinarily not mention for fear of being impolitic. Then each person in turn reads one new reason, starting with the project manager, until the lists are exhausted. Finally, the manager reviews them and strengthens the plan. The difference from a conventional risk review is social, not just logical: asking what might go wrong asks someone to oppose a plan that is still alive, whereas asking why it failed only asks them to be clever. The technique also tempers the overconfidence of people too invested in the project. It is worth doing at the start of any important plan. The traps: open discussion before individual writing, where the loudest voices win, and a list of reasons that changes nothing in the plan.
Unlike a typical critiquing session, in which project team members are asked what might go wrong, the premortem operates on the assumption that the “patient” has died, and so asks what did go wrong.Gary Klein, «Performing a Project Premortem», Harvard Business Review, septembrie 2007 (reprint F0709A), partea introductivă a articolului
Why it matters People with reservations stay silent in the planning phase, exactly when it matters most; the premortem turns their objection from an act of opposition into a requested contribution.
- 15
When everyone is sure something will not happen, it is worth writing down how it could happen after all and what you would see first.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
These are two related contrarian techniques for the moment when everyone is sure something will not happen. High-Impact/Low-Probability (HILP) analysis asks: if it does happen, what are the consequences, and by what path would we get there? What If? analysis accepts from the outset that the event has occurred and moves straight to how it came about. The primer recalls events once considered unlikely — the fall of the Shah, the collapse of the Soviet Union, the reunification of Germany — and a State Department assessment of 27 November 1941 that gave odds of five to one against the United States and Japan being at war by 15 December. The steps: state the dominant analytic line and the high-impact outcome clearly; assume it has happened; choose plausible triggers, such as the death of a leader, a natural disaster or an economic shock; think backwards, step by step, about what must have occurred; identify one or more pathways; write observable indicators for each and monitor them periodically. It matters most when a judgement rests on limited information or unproven assumptions. The trap: the exercise does not change the probability; it produces signposts to watch and hedging options. Taken out of context, it turns into alarmism.
By shifting the focus from whether an event could occur to how it may happen, analysts allow themselves to suspend judgment about the likelihood of the event and focus more on what developments—even unlikely ones—might enable such an outcome.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (US Government, martie 2009), secțiunea «What If? Analysis» — «Value Added», p. 24
Why it matters Low probability is exactly why nobody prepares the warning signposts; when the event arrives, there is no list that could have announced it.
- 16
Scenarios do not guess the future: they separate what is already determined from what is truly uncertain and change the decision-maker's mental model.
Scenarios: Uncharted Waters Ahead · Pierre Wack (Harvard Business Review) · 1985
Wack headed the business environment division of the Royal Dutch/Shell Group's planning department, and his article opens with an uncomfortable observation: forecasts are dangerous precisely because they are usually right. They assume tomorrow will look like today and fail exactly when everything changes. The answer is not a better forecast but accepting uncertainty as a structural feature of the environment. The method separates two things: predetermined elements — what has already happened but whose consequences have not yet unfolded, like monsoon rains in the upper Ganges basin that will certainly show downstream — and critical uncertainties. The former are treated as facts; scenarios are built only on the latter. Today's practice has standardised the form. You pick, by consensus, the two most critical and uncertain forces, turn them into axes, and the four quadrants of the 2×2 matrix become four plausible worlds; the 2009 Tradecraft Primer describes exactly this step. For each world you write the indicators that would show it is becoming reality. The trap, in Wack's terms: first-generation scenarios that merely combine obvious uncertainties, such as one oil price or another, do not help decisions. A good scenario changes the decision-maker's mental model; otherwise it is water on a stone.
Our real target was the microcosms of our decision makers: unless we influenced the mental image, the picture of reality held by critical decision makers, our scenarios would be like water on a stone.Pierre Wack, «Scenarios: Uncharted Waters Ahead», Harvard Business Review, septembrie 1985, secțiunea despre «microcosmosul» decidentului
Why it matters Shell's scenarios did not guess the timing of the 1973 oil crisis, but they prepared management for its possibility — something a single forecast could not do.
- 17
A forecast becomes measurable when it is a number on a question with a deadline, and the Brier score shows who was calibrated, not who sounded convincing.
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · Barbara Mellers, Philip E. Tetlock et al. · 2015
Between 2011 and 2015 IARPA, the research agency of the US intelligence community, ran geopolitical forecasting tournaments: hundreds of questions with a verifiable answer and a fixed deadline, on which participants gave probabilities they could revise at any time. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, won, and the book Superforecasting (Tetlock and Gardner, 2015) tells how. The yardstick was the Brier score: the sum of squared differences between the probability given and what happened (1 if the event occurred, 0 if not). Zero is perfect; always saying 50% earns 0.5 on a two-outcome question. The method has four steps. Phrase the question so that a stranger could decide at the deadline whether it happened; give a number, not a maybe; update often, in small steps, as evidence arrives; at resolution, record the score and check calibration — of the claims made at 70%, did roughly 70% come true? It pays off for any judgement that recurs. The trap is choosing vague questions that can never be lost: with no deadline and no criterion there is no score, and so no learning.
This worldview predisposes superforecasters to treat their beliefs more as testable hypotheses and less as sacred possessionsMellers, Tetlock et al., «Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions», Perspectives on Psychological Science (2015), p. 273, secțiunea despre stilurile cognitive ale superprognozatorilor
Why it matters An analyst who keeps no score cannot know whether their ‘likely’ means 60% or 90% — and their reader knows even less.
- 18
Before judging a case by its details, ask how similar cases ended: the base rate is the starting point, and the details only adjust it.
Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking · Daniel Kahneman & Dan Lovallo · 1993
Kahneman and Lovallo start from an anecdote. A team writing a school curriculum was estimating when it would finish. Asked how comparable teams had fared, the group's own expert admitted that about 40% had given up and none had finished in under seven years. Everyone had spontaneously taken the inside view: the plan, the obstacles, the scenario. The outside view ignores the details and asks what happened across a class of cases similar in relevant respects. The authors' point is that, applied with equal skill, the outside view is far more likely to yield a realistic estimate. The steps: choose a reference class of similar cases; find the distribution of outcomes, the base rate; place your case within that distribution, asking why it should do better or worse than average; only then adjust, moderately, for the specifics. It applies to any estimate of duration, cost or chance of success — how long coalition talks last, how often sanctions change a regime's behaviour. The trap is a reference class chosen to deliver the answer you wanted, or the claim that ‘our case is unique’, which the authors describe as the almost moral preference for the inside view.
In contrast, the outside view is essentially statistical and comparative, and involves no attempt to divine future history at any level of detail.Kahneman & Lovallo, «Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking», Management Science 39(1), ianuarie 1993, secțiunea «Inside and Outside Views», p. 24
Why it matters Most bad estimates do not come from missing information about the case but from nobody asking how cases of this kind usually end.
- 19
Bayes does not ask you to guess the truth, only to make one judgement per item of evidence: how many times more likely it is if the hypothesis is true than if it is false.
Bayes' Theorem for Intelligence Analysis · Jack Zlotnick (CIA, Center for the Study of Intelligence) · 1972
The CIA sponsored in-house research on applying Bayes' theorem to intelligence analysis, and Jack Zlotnick, an analyst who took part, described the method in Studies in Intelligence. It uses the odds form: the revised odds (R) equal the prior odds (P) multiplied by the likelihood ratio (L). Analysts never judged the conclusion directly; they judged only L for each new item — how many times more likely, say, a troop deployment to a border is if war is coming than if it is not. If twice as likely, L = 2 and the odds double. R then becomes the P for the next item. The steps: frame two mutually exclusive hypotheses; fix the starting odds explicitly; for each item of evidence, estimate separately how likely it is under each hypothesis; multiply and move on. It pays off when evidence arrives in a stream and the temptation is to react to the latest item. Zlotnick names a limit: close to the climax, much incoming evidence becomes undiagnostic — equally likely under both hypotheses. A second classic pitfall is treating two reports from the same source as independent; multiplied as if they were, they inflate the conclusion. The typical error remains mistaking evidence that is consistent with a hypothesis for evidence that is diagnostic.
The very best that intelligence can do is to make the most of the evidence without making more of the evidence than it deserves.Jack Zlotnick, «Bayes' Theorem for Intelligence Analysis», Studies in Intelligence vol. 16, nr. 2 (1972), paragrafele introductive
Why it matters Without an explicit updating rule, the latest item of evidence gets the most weight merely because it is the latest, not because it says more.
- 20
A close, friendly group is in the greatest danger of no longer thinking critically, and the remedy is a procedure that makes objection a duty rather than a betrayal.
Victims of Groupthink: A Psychological Study of Foreign-Policy Decisions and Fiascoes · Irving L. Janis · 1972
Janis studied foreign-policy fiascoes, starting with the Bay of Pigs, and set them beside better-run decisions such as the Cuban missile crisis and the Marshall Plan. His conclusion, offered in the spirit of Parkinson's laws, is the quote above: the danger grows with cohesion. The symptoms are easy to recognise — an illusion of invulnerability, self-censorship of doubts, an illusion of unanimity, self-appointed mindguards who shield the leader from awkward information. The result is alternatives left unexamined and risks never weighed. Yet Janis does not conclude that cohesion is bad: he notes that groups lacking it can produce even worse fiascoes. The remedies are procedural, not moral. The role of critical evaluator is given to every member, so that objecting is an obligation rather than disloyalty; outside experts are invited to challenge the group's views; after a preliminary decision, a ‘second chance’ meeting lets anyone voice remaining doubts. For an analytic team this means a peer review with an explicit mandate to find weaknesses, and a dissenting note that travels with the product. The typical trap is ritual dissent: a role everyone knows is play-acted, so nobody listens to it any more.
The more amiability and esprit de corps among the members of an in-group of policy-makers, the greater is the danger that independent critical thinking will be replaced by groupthink, which is likely to result in irrational and dehumanizing actions directed at out-groups.Irving L. Janis, «Victims of Groupthink» (Houghton Mifflin, 1972), secțiunea «Are cohesive groups doomed to be victims?», p. 198–199
Why it matters Most analytic teams are small, stable and collegial — exactly the profile Janis describes as most exposed.
- 21
Effective deception does not change your belief, it confirms it; the most dangerous set of clues is the one that fits perfectly with what you already believed.
Deception Maxims: Fact and Folklore · Central Intelligence Agency, Office of Research and Development · 1980
In 1980 a CIA research paper distilled historical deception cases into maxims for analysts. The first, Magruder's Principle, holds that reinforcing an existing belief is easier than building a new one. The classic case is Normandy: Hitler and most of his senior military advisers expected the landing in the Pas de Calais, the Allies knew this from intercepts, and the cover plan fed exactly that belief. Jones' Lemma, named after the British physicist R. V. Jones, adds that deception grows harder as the victim's channels of information multiply — but, within limits, the more channels are controlled, the more believable the story. Counter-deception follows from both. Write down what you already believe and what the adversary would want you to believe; if they coincide, raise your guard. Check how many genuinely independent channels the confirmation comes from, since controlled channels confirm one another. Look for evidence that should exist if the story is true and is missing. In a hypothesis matrix, carry deception as an explicit hypothesis. The opposite trap is paranoia: a remark attributed to Jones, known as Crabtree's bludgeon, warns that for any set of inconsistent observations someone can devise a coherent explanation, however complicated — so a deception story that explains everything proves nothing on its own.
It is generally easier to induce an opponent to maintain a preexisting belief than to present notional evidence to change that belief.CIA, «Deception Maxims: Fact and Folklore» (aprilie 1980, declasificat în 2015), Maxima 1 «Magruder's Principle — the Exploitation of Preconceptions», p. 5
Why it matters An analyst is easiest to deceive exactly where they feel most certain, because there the adversary need not persuade them of anything new.
- 22
An adversary is not analysed in a vacuum: first the environment — political, military, economic, social, information and infrastructure systems — then what the adversary can do within it.
Joint Publication 2-01.3: Joint Intelligence Preparation of the Operational Environment · US Joint Chiefs of Staff · 2014
US joint doctrine calls the process JIPOE, joint intelligence preparation of the operational environment. The 21 May 2014 edition of JP 2-01.3, which superseded the 2009 one, sets out four steps: define the operational environment, describe its impact, evaluate the adversary and other relevant actors, and determine their courses of action. The environment covers the physical domains, the information environment (including cyberspace) and the PMESII systems — political, military, economic, social, information and infrastructure. The civilian variant used in business analysis is STEEP: social, technological, economic, environmental, political. To apply it, list the elements, links and pressure points of each system, then ask how they constrain each actor's options — an energy embargo read as an economic event has political and social effects in another country. The companion technique in the 2009 Tradecraft Primer is outside-in thinking: start from the external forces you do not control and only then narrow to your subject, so that no variable is missed. It pays off at the start of a new analysis or when the subject shifts. The trap is the inventory: a complete PMESII list with no conclusion about what matters. The 2014 edition is the latest public one; the current joint doctrine list sits behind a restricted portal.
The JIPOE process provides a disciplined methodology for applying a holistic view of the OE to the analysis of adversary capabilities and intentions.US Joint Chiefs of Staff, JP 2-01.3 «Joint Intelligence Preparation of the Operational Environment» (21 mai 2014), Executive Summary, «The Operational Environment (OE) — A Holistic View»
Why it matters Most surprises come not from the adversary's capabilities but from systems the analysis never looked at — a power grid, a diaspora, a messaging platform.
- 23
Open source material becomes evidence only when you can show where it came from, that it has not been altered, and what you did with it from the first search onwards.
Berkeley Protocol on Digital Open Source Investigations · OHCHR & Human Rights Center, UC Berkeley School of Law · 2022
The Berkeley Protocol, published by the UN human rights office (OHCHR) together with the Human Rights Center at Berkeley, sets international standards for digital open source investigations. It grew out of a courtroom problem: smartphone photographs and videos posted online, some compromised or misattributed, cannot serve as evidence if they were collected and kept with unsound methods. The Protocol divides an investigation into six phases: online inquiries, preliminary assessment, collection, preservation, verification and investigative analysis. In practice the steps look like this: log every search (term, platform, date); capture the material in full, with its metadata, and record a cryptographic hash at collection; keep the original untouched and work on copies; verify the source (who posted it, when, from where) separately from the content (geolocation, chronology, shadows, weather); only then interpret. It pays off whenever a finding may end up before a court, a regulator or a hostile journalist. The trap is the reverse order: interpreting first and preserving later — by which time the post has been deleted or edited, and you can no longer show what you saw.
For open source information to be admissible as evidence in court, prosecutors and counsel must typically be able to establish its authenticity and chain of custody.«Berkeley Protocol on Digital Open Source Investigations» (ONU, New York și Geneva, 2022), capitolul I «Introduction»
Why it matters An open source finding that nobody can retrace step by step is an opinion, however good the image.
- 24
A wargame does not predict, it discovers: it shows what is plausible and where the plan gives way when someone genuinely opposes it.
Wargaming Handbook · UK Ministry of Defence, Development, Concepts and Doctrine Centre (DCDC) · 2017
The 2017 British handbook defines wargaming as a decision-making technique that offers structured but intellectually liberating, safe-to-fail environments — a place to find out what works and what does not, usually at low cost. At its core are the players, the decisions they take, the narrative they create, their shared experiences and the lessons they carry away. The handbook separates analytical, discovery wargames from training, learning ones; for analysis the first kind matters, from planning games to games that inform real executive decisions. The steps: frame the question (what you want to find out, not what you want to prove); choose the players, including a capable opposing side that is free to win; set the rules and an umpire who adjudicates the effects of moves; run the game, then conduct a rigorous post-game analysis. The handbook is explicit that wargames are not predictive and that a single run risks false lessons, which is why several games are played. It pays off when the adversary reacts to what you do — an interaction a static analysis cannot see. The trap is the game staged to confirm the plan, with an opponent who is not allowed to win.
Wargames can illustrate that something is plausible, but will not be able to definitively predict that it is probable.UK MoD, DCDC, «Wargaming Handbook» (august 2017), capitolul 1, paragraful 1.19
Why it matters A plan never contradicted by a free opponent is an untested plan, however good it looks on paper.
- 25
A product written with the help of a model is still an intelligence product: the same standards and the same question — how do you know — and the answer must lead to real sources, not to the model.
Artificial Intelligence Ethics Framework for the Intelligence Community (v1.0) · Office of the Director of National Intelligence (ODNI) · 2020
In 2020 the US intelligence community published six principles of AI ethics and an implementation framework written as a list of questions. The human-centred principle asks that technological guidance be tempered with human judgement; the framework asks who the accountable human is, what they must know about the model to judge its reliability, and how AI outputs are marked. ICD 505, signed on 17 January 2025, requires data traceability from AI inputs to AI-derived outputs and systems that let analysts meet ICD 203 and ICD 206, the analytic and sourcing standards. The US AI Action Plan of July 2025 also called for an AI assurance standard under ICD 505. Applied in 2026 to a product drafted with a language model, this means: every factual claim leads to a real source that a human has opened and read, not to the model's answer; quotes and figures are checked at the source, because models invent plausible references; probability and confidence remain the analyst's judgement; and whatever the machine produced is marked. The main trap is automation bias: accepting a fluent machine's output without the checks you would demand of a new colleague. The framework says it plainly: machine errors may differ from human errors.
How might you respond to an intelligence consumer asking “How do you know this?” How will you describe the dataset(s) and tools used to make the output?ODNI, «Artificial Intelligence Ethics Framework for the Intelligence Community», v1.0 (iunie 2020), secțiunea «Transparency: Explainability and Interpretability»
Why it matters A model writes fluently and convincingly exactly where it has no sources — and fluency is the signal readers most easily mistake for truth.