Go deeper
Intelligence analysis methods — 2026
The methods an analyst uses in 2026, from the ICD 203 standards and probabilistic language to ACH, indicators, red teaming, scenarios, calibrated forecasting and AI-assisted analysis. Every lesson has a quiz.
The reading track
- 01
Analysis is a mental process that runs through simplified models of reality; structure exists to bring the model into the open, where it can be checked.
Psychology of Intelligence Analysis · Richards J. Heuer Jr. · 1999
Heuer starts from Herbert Simon's idea of bounded rationality: the mind cannot take in the world's complexity directly, so it builds a simplified model and behaves rationally only inside it. For an analyst, that model — Heuer also calls it a mind-set — is the lens through which every new report is read. It is unavoidable and useful, since without it no information would make sense. The problem, Heuer says, is that such a model is quick to form but resistant to change, and the analyst does not see it, because he is looking through it. The practical steps follow. Before writing a judgement, note what you already believe and why; list the alternative explanations; ask which evidence would make you change your mind. Structure — a list, a matrix, a written hypothesis — does not make you cleverer, but it moves your thinking onto paper, where others can challenge it. It pays most on high-stakes issues with ambiguous information. The typical trap is believing that experience exempts you: Heuer warns that the experienced specialist may be among the last to see what is really happening when events take an unexpected turn.
Intelligence analysts should be self-conscious about their reasoning process. They should think about how they make judgments and reach conclusions, not just about the judgments and conclusions themselves.Richards J. Heuer Jr., «Psychology of Intelligence Analysis» (CIA, Center for the Study of Intelligence, 1999), capitolul 4 «Strategies for Analytical Judgment», prima frază după rezumatul capitolului
Why it matters An analyst who cannot state his mental model out loud has no way of noticing the moment reality has left it.
In full →Collapse ↑
Heuer takes bounded rationality one step further: surprise is not an accident but a signal. When an event catches you off guard, reality did not fail; your model was incomplete. A good analyst treats every surprise as a diagnosis of the lens, not as an exception to ignore. Example: you assume a neighbor is unfriendly because he once refused a favor, then he brings in your package. The surprise shows your judgment rested on a single episode. Heuer suggests noting why you expected something else, not just what happened. That note turns the awkward moment of being wrong into a map of your own assumptions. One question remains: how can you tell, before the fact, that your model is fragile?
- 02
ICD 203 asks five things of every analytic product — objectivity, independence from politics, timeliness, all sources and tradecraft — and independence protects the other four.
Intelligence Community Directive 203: Analytic Standards · Office of the Director of National Intelligence (ODNI) · 2015
Directive 203 of the US Intelligence Community sets five standards for every analytic product: it must be objective, independent of political consideration, timely, based on all available sources, and must implement the nine analytic tradecraft standards. The text published by ODNI is the one signed on 2 January 2015, with a 2022 technical amendment that mainly concerns the role of the analytic ombuds, not the standards themselves. Objectivity means more than lack of bias: analysts must be aware of their own assumptions, use techniques that reveal bias, consider contrary information, and not stay tied to earlier judgements once the facts have changed. In practice it works as a pre-publication checklist: have we weighed alternative perspectives? Will the assessment reach the decision-maker before the decision? Have we used sources we would rather ignore? Have we said how sure we are, and why? The standards fit anywhere an assessment is written for someone who decides — a ministry, a bank or a newsroom. The typical trap is not lying but tailoring: an assessment written to please an audience, an agenda or a policy. That is why the text forbids both verbs — neither distorted by advocacy nor shaped for it.
Analytic assessments must not be distorted by, nor shaped for, advocacy of a particular audience, agenda, or policy viewpoint. Analytic judgments must not be influenced by the force of preference for a particular policy.ODNI, ICD 203 «Analytic Standards» (semnată la 2 ianuarie 2015, cu amendament tehnic din 2022), standardul analitic b «Independent of political consideration»
Why it matters An assessment tailored to its reader no longer says anything about the world, only about what the reader wanted to hear.
In full →Collapse ↑
The five standards do not stand alone. Independence from political considerations is the foundation that keeps the others upright. If the analyst already knows what answer the decision-maker wants, objectivity becomes decoration. Timeliness breaks the same way: a report can arrive fast but clean, or late and useful. A familiar example: an accountant who knows the director wants an acquisition will find only the numbers that support it. Independent, that accountant would also look for the numbers that complicate it. That is why the directive puts independence up front: it is not a sixth principle, but the condition for the rest to matter. The natural next question is what practical mechanisms keep an analyst independent when the pressure is real.
Passage 2: the practical machinery of independence. The directive does not leave independence as a matter of good intentions; it turns it into checkable procedures. The analyst must weigh all sources, including those that cut against the conclusion, and state their confidence level with an argument. A key mechanism is exploring alternatives: before publication, the analyst must show what other explanations exist and why they are judged less likely. It is like a doctor who, before saying "it's the flu," notes which other conditions could produce the same symptoms, and why each is ruled out. Another mechanism is the analytic ombudsman, a role strengthened by the 2022 amendment: someone outside the team who verifies the standards were actually applied. So independence is not a character trait but a system of brakes. One question remains: what happens when the brakes work, yet the decision-maker still hears only what they want?
The brakes are useless if the reader never uses them. An honest assessment always says something unwelcome: alternatives, uncertainty, inconvenient sources. If the decision-maker forces it to sound pleasant, it becomes tailored again, this time with the reader's consent. The directive cannot force anyone to listen. So the final link is the reader's habit of asking for the uncomfortable parts, not just the conclusion. A manager who asks "what could this report get wrong?" does more for independence than any procedure. The same move works at home: before accepting an apartment appraisal, ask what evidence cuts against it. What remains to be seen is what happens when even the reader refuses to hear.
When even the reader does not want to hear, the last defense is structure: whoever pays the analyst should not be the one deciding on the analysis. The directive works because analysts answer to standards, not to today's client. A hospital runs case reviews without the treating doctor in the room, precisely so no one has a stake in making things look better. Likewise, a bank separates the team assessing risk from the team selling loans. Structure does not guarantee truth, but it makes betraying it costlier. Which raises the final question: who guards the structure itself, when pressure comes from above?
- 03
A probability word with no number behind it is read differently by every reader; the remedy is an agreed scale with numerical ranges.
Words of Estimative Probability · Sherman Kent · 1964
In March 1951 National Intelligence Estimate 29-51 called an attack on Yugoslavia that year a serious possibility. Kent, a member of the Board of National Estimates that had approved the text, had odds of about 65 to 35 in favour of an attack in mind; a State Department reader had understood a much lower chance, and Kent's colleagues each held a different figure, from 20 to 80 up to 80 to 20. His answer was a table: a few expressions, each tied to a range — probable around 75 per cent, almost certain around 93 per cent. Today ICD 203 requires a seven-step scale, from almost no chance (1–5 per cent) to almost certain (95–99 per cent). The British PHIA yardstick also has seven steps, from remote chance (up to about 5 per cent) to almost certain (from about 95 per cent), but deliberately leaves gaps between them. In practice: pick the number first, then the word from the scale; do not mix sets of terms; if you cannot defend the word, you do not yet have a judgement. The trap is what Kent called the poets' camp: phrases such as may well or distinctly possible, which suggest odds without committing to them.
It was another jolt to find that each Board member had had somewhat different odds in mind and the low man was thinking of about 20 to 80, the high of 80 to 20.Sherman Kent, «Words of Estimative Probability», Studies in Intelligence, vol. 8, nr. 4 (1964), secțiunea «Early Brush with Ambiguity»
Why it matters If writer and reader put different numbers under the same word, the assessment has said something other than intended, and neither notices.
In full →Collapse ↑
Kent discovered something more painful than misunderstanding: the ambiguity was invisible. Everyone who read the estimate believed they had understood it the same way. Nobody asked what, exactly, "serious possibility" meant. Only when he asked his colleagues directly did the gap from 20 to 80 appear. The lesson is that the most dangerous misunderstanding looks like understanding. Think of a doctor saying an operation carries "low risk." The patient hears maybe 2 percent, the doctor means 10 percent. Both walk away satisfied, with different decisions in mind. That is why the scale must be agreed on before use, not negotiated afterward. But a scale written on paper is not yet a practice — the next question is how to make it live in the habits of an entire office.
Kent solved the other half of the problem: the order of thinking. Most people do it backwards — they pick a word that sounds right, then defend it. He proposes fixing the number first, then finding the scale word that covers it. The difference is not style but honesty: the number forces you to commit to a judgment, the word lets you dodge one. That is why Kent feared the "poets" — phrases like "may well" that sound like analysis but commit no one to anything. Think of a colleague who says the project "could fail." Ask them "what percentage?" and they often discover they never had a figure in mind. The word was an escape, not an estimate. But once you have the number, a new question appears: how wide can the interval get before it becomes an excuse?
- 04
Probability says how likely the event is; confidence says how solid the basis for saying so is. They are two axes and are written separately.
Explaining Uncertainty in UK Intelligence Assessment · Professional Head of Intelligence Assessment (PHIA), Guvernul Regatului Unit · 2025
These are two different questions: how likely the event is, and how solid your basis for saying so is. You can judge an attack likely on the strength of a single new source — high probability, low confidence — or unlikely on ten years of consistent data — low probability, high confidence. The British guidance grades analytical confidence (the Analytical Confidence Rating) as one of three levels — high, moderate, low — against three criteria: information base, analytical rigour, and complexity and volatility. Confidence also tells the reader how easily the judgement might change. The practical rule: express probability with a word from the scale, and explain confidence separately, with its reason — which source is missing, what could change the picture. ICD 203 goes down to grammar: a product using a confidence level must not combine it with a degree of likelihood in the same sentence. The reason is a phrase such as we have high confidence that it is unlikely, which the reader hears as a single number. The trap: lowering the probability when what you really lack is evidence. Missing evidence lowers confidence; it does not automatically move the event towards unlikely.
Whereas probability reflects the likelihood that a statement is true, analytical confidence reflects the soundness and stability of the foundations on which the assessment of likelihood has been made.PHIA, «Explaining Uncertainty in UK Intelligence Assessment» (GOV.UK, publicat pe 24 martie 2025), secțiunea «Analytical Confidence Rating (AnCR) Framework»
Why it matters A decision-maker given only the probability cannot tell whether the judgement will hold until tomorrow or flip at the next report.
In full →Collapse ↑
The card says the two axes are written separately. The next step: confidence is also a signal about durability. A probability describes the world today. High confidence says the judgment would survive new information. Low confidence is a warning: the first contrary report could overturn it. That is why a smart reader asks not only how likely something is, but how long the answer can be relied on. Example: a doctor who says the tests most likely show a routine infection, but admits she has only seen the first results. The prognosis may be right, yet a retest tomorrow is warranted. The UK guidance asks for exactly this: alongside the rating, the reason — which source is missing, what could change the picture. Next step: what to do concretely when confidence is low but the decision cannot wait.
- 05
Every report gets two independent grades: a letter for the source, based on its track record, and a figure for the information, based on corroboration and logic.
FM 2-22.3 Human Intelligence Collector Operations · Headquarters, Department of the Army (SUA) · 2006
Every report gets two grades, not one. The letter, from A to F, says how reliable the source is, judged on what it has reported before: A reliable, B usually reliable, C fairly reliable, D not usually reliable, E unreliable, and F cannot be judged, for a source with no track record. The figure, from 1 to 6, says how credible the information in this particular report is: 1 confirmed by other independent sources, 2 probably true, 3 possibly true, 4 doubtfully true, 5 improbable, and 6 cannot be judged. The grid is known as the Admiralty code, after its British origin, and the same logic of two separate grades appears in NATO intelligence doctrine. In practice: grade the source on what it has delivered so far, then the information on what it says now — internal logic, corroboration, consistency with everything else. The grades are kept apart because they can diverge: a B source can send a 5 report, and a new F source can bring something already confirmed. The method pays wherever many heterogeneous sources flow in, open sources included. The traps: reading F as liar — the manual says explicitly it only means no track record — and letting the source's reputation contaminate the grade of the information.
An “F” rating does not necessarily mean that the source cannot be trusted, but that there is no reporting history and therefore no basis for making a determination.FM 2-22.3 «Human Intelligence Collector Operations» (6 septembrie 2006), anexa B «Source and Information Reliability Matrix», paragraful B-1
Why it matters A single grade mixes two different questions and hides exactly the dangerous case: the good source that, this time, is wrong.
In full →Collapse ↑
The next step is what you do with the grades over time. The evaluation is not a label but a testable promise: if a B source keeps sending reports that turn out right, its history moves toward A; if it errs, it slides down. The grades thus become a self-correcting record. An everyday example: a colleague who has always given you good information now says something that fits nothing else. You grade the source B, but the information 4. If it confirms, you raise both; if not, you lower the track record. The open question is how fast these grades should move — too slowly and a newly good source stays underrated, too quickly and one incident rewrites everything.
The grid only becomes useful when many people use it the same way. The ratings work as a shared language: your colleague reads "B3" and understands exactly what you meant, with no explanation. Without a shared grid, each evaluator invents their own scale, and reports cannot be added up. An everyday example: in a group of neighbors reporting problems to the town hall, if everyone describes sources however they like, nobody knows who is credible. If everyone uses the same letters and numbers, you can compare signals against each other and spot patterns. The grid turns personal judgments into something comparable and cumulative. The next step is what you do when two good ratings contradict each other.
- 06
The quality of information check periodically tests whether the source base really supports the judgement; the number of sources is no substitute for their quality.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
The quality of information check assesses how complete and sound the source base behind a judgement is. The 2009 primer describes it as a continuous process, not a one-off step: otherwise important judgements stay anchored to weak information, and caveats once attached to them are forgotten. The steps: keep an inventory of sources by type and date, with their strengths and weaknesses; identify the most critical or compelling sources; check that critical reporting has sufficient and strong corroboration; re-examine previously dismissed information in the light of new facts; flag recalled reporting and review any analysis built on it. The result is an honest picture of what we know and what we do not, and it feeds straight into the confidence level. The method pays before any major assessment and whenever an analytic line has run for months. It is also where an adversary's deception strategies can be caught. The main trap is counting: five articles repeating the same source are not five confirmations but one, copied. In the finished product, ICD 203 requires the quality of sources to be described, and source summary statements are strongly encouraged.
Having multiple sources on an issue is not a substitute for having good information that has been thoroughly examined.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (Guvernul SUA, martie 2009), secțiunea «Quality of Information Check», rubrica «When to Use», p. 10
Why it matters A year-old judgement can rest on a source recalled months ago, and nobody notices unless someone periodically looks underneath it.
In full →Collapse ↑
The card explains how the check is done. The next step is when to do it: early, not at the end, while you can still change something. If you wait for the major assessment, a weak source has already become the foundation of ten conclusions. A familiar example: you check a used car's service history before signing, not after. Early checking turns quality control from a formality into a decision about the analysis's direction. One question remains: who should run this check — the author of the judgment, or someone from outside?
Who should do the check? The author knows the sources best, but precisely for that reason has a hidden stake: the judgment was built on them. The primer insists the check be ongoing, and an honest ongoing check is hard for someone defending their own work. An outside eye sees what the author has already normalized. Everyday example: a writer rereads their own text and misses the typos, while another reader catches them at once. In practice the answer is a mix: the author keeps the source inventory, but someone independent tests the critical sources. That raises a new question: what exactly is this checker looking for when they look under a judgment?
- 07
Every assessment rests on premises accepted without discussion; the key assumptions check writes them down, challenges them and says in advance what would overturn them.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
A key assumption is any hypothesis the analyst has accepted as true and on which the assessment rests — often without ever having stated it. The primer proposes four steps: write down the current analytic line where everyone can see it; list all the premises, stated or unstated, that must be true for the line to hold; challenge each one — why it must be true and whether it holds under all conditions; keep only the assumptions that really carry the conclusion, and note the conditions under which they would fail. The primer's example is the 2002 sniper case around Washington, where it was assumed the perpetrator was acting alone. At the CIA in the 1990s, Douglas MacEachin promoted a variant called linchpin analysis: assumptions about the key drivers become linchpins, the premises the argument stands on, and must be spelled out. The method pays at the start of a project, when it costs an hour or two, and again before judgements are finalised. The trap: the list turns into an inventory of platitudes, while the dangerous premise — the one nobody sees because everybody shares it — stays unwritten. The question that brings it out: if this proved wrong, would the conclusion change?
The goal is not to undermine or abandon key assumptions; rather, it is to make them explicit and identify what information or developments would demand rethinking them.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (Guvernul SUA, martie 2009), secțiunea «Key Assumptions Check», p. 7
Why it matters Assessments rarely fail because a fact was wrong; they fail because a premise nobody wrote down stopped being true.
In full →Collapse ↑
So far we've seen how to write assumptions down. The next step is trickier: the most dangerous assumptions are the ones you can't find because the whole group shares them. When everyone on the team takes something for granted, nobody sees it as an assumption anymore — it looks like a fact. That's why checking alone isn't enough; you need an outsider, or a rule that forces the question: what have we accepted without noticing? A common example: a family plans a renovation and everyone assumes the budget will hold. Nobody says it aloud, so nobody asks what happens if prices rise. The assumption only surfaces when someone states it. How do you find these invisible assumptions in your own thinking? There is a technique built exactly for that, and it comes next.
- 08
Strategic intelligence failures cannot be removed by reorganisation: evidence is ambiguous, judgement ambivalent, and the great misses often happen at the decision-maker.
Analysis, War, and Decision: Why Intelligence Failures Are Inevitable · Richard K. Betts · 1978
In 1978 Betts reached an uncomfortable conclusion: strategic intelligence failures cannot be prevented by organisational solutions to problems of analysis and communication. Analytic certainty is precluded by ambiguity of evidence, ambivalence of judgement and the atrophy of reforms meant to avert failure. Many sources of error are unresolvable paradoxes and dilemmas, not curable pathologies. And the major misses — attack warning, operational evaluation, intelligence for strategic planning — stem mainly from leaders' psychological attributes rather than from analysts failing to detect the relevant data. The practical consequence is twofold. First, no method in this course guarantees you will not be wrong; methods reduce avoidable errors and make the rest visible. Second, analysis does not end at hand-over: a correct estimate that is heard and dismissed is still a failure. That is why it matters how you write — key judgement first, explicit probability, what would change it — and how you stay in dialogue with the decision-maker. The lesson is an antidote to two illusions: that the next reorganisation will solve the problem, and that the analyst alone is to blame. The symmetric trap is fatalism: inevitable does not mean indifferent.
Since analysis and decision are interactive rather than sequential processes, and authorities often hear but dismiss correct estimates, intelligence failure is inseparable from policy failure.Richard K. Betts, World Politics, vol. 31, nr. 1 (octombrie 1978), pp. 61–89 — rezumatul articolului, în forma publicată de editor (Cambridge University Press)
Why it matters Whoever believes a reorganisation eliminates failure will build yet another structure and be surprised again, by the same dilemmas.
In full →Collapse ↑
If failure is political, not just informational, then redesign must target the relationship, not only the structure. Reorganizations create new committees and new flows, but leave untouched the moment when a leader hears an unwelcome estimate. What gets lost there is not data but receptivity. A minister receives a written warning, reads it, then sets it aside because it contradicts his plan. Useful reform would ask: who has the right to insist, when, and with what force? That is, to turn the analyst-decision-maker dialogue into a procedure, not an accident. This shifts the question from structure to power: who can challenge a decision, and at what personal cost. That opens the next step: what happens to the analyst who is right, warning correctly and still not believed.
Passage 1 stopped at the analyst who warns correctly and is still not believed. Betts pushes further: if analysis and decision are an interactive process, not a handoff, then the analyst's responsibility does not end at delivery. A correct estimate that is rejected is still failure — but a failure of dialogue, not of data. That changes what being a good analyst means: not just being right, but making the key judgment impossible to ignore. Concretely: instead of a page of nuance, you lead with the conclusion, state explicitly how likely the scenario is, and name what new fact would change it. A doctor does not merely tell a patient that risks exist; she says which risk, how large, and what sign would force a different decision. Then you stay in the conversation, asking what keeps you from being heard. A painful limit remains: even this guarantees nothing.
- 09
Analysis of competing hypotheses makes every reasonable explanation compete on the same evidence and keeps the one the evidence refutes least.
Psychology of Intelligence Analysis · Richards J. Heuer Jr. (CIA, Center for the Study of Intelligence) · 1999
The method starts from an awkward observation about how we work: we pick the likely answer by intuition and then look for evidence that supports it. The problem is not that the evidence is false but that most of it fits other explanations just as well. Analysis of competing hypotheses (ACH) has eight steps in Heuer's version, and their core fits into three moves: write down every reasonable hypothesis, ideally with colleagues who think differently; build a matrix with hypotheses across the top and evidence down the side; mark, for each item, which hypotheses it is consistent with and which it is not. The key word is diagnosticity. A fever tells a doctor that the patient is ill, not which illness it is; evidence consistent with every hypothesis does not help you choose and drops out of the calculation. What remains are the few items that discriminate, and those are the ones you re-check first. Then you try to disprove rather than prove, report every hypothesis, and state in advance what would change your mind. The typical trap is confusing an unproven hypothesis with a disproved one. No indication that India will test soon does not mean India will not test — Heuer's example from 1998.
The most probable hypothesis is usually the one with the least evidence against it, not the one with the most evidence for it.Richards J. Heuer Jr., «Psychology of Intelligence Analysis» (CIA Center for the Study of Intelligence, 1999), capitolul 8 «Analysis of Competing Hypotheses», secțiunea «Summary and Conclusion», p. 108
Why it matters A report that counts the evidence for the favoured hypothesis looks solid right up to the moment the same evidence turns out to fit the alternative nobody tested.
In full →Collapse ↑
The ACH matrix has a rarely mentioned side effect: it changes your relationship with your favorite evidence. When you see on paper that your top piece of evidence fits three hypotheses, it stops shining so brightly. Diagnosticity becomes a filter you can apply before collecting, not only after. In practice, before a difficult meeting you can ask: what observation would truly separate my explanations? Then you hunt for exactly that observation. A tenant suspecting a leak can note, before inspecting: if stains grow only when it rains, it's the roof; if they grow even in dry weather, it's the pipe. Writing it down beforehand keeps him from seeing only what confirms his first hunch. Next comes something subtler: what do you do when no available evidence is diagnostic at all?
Sometimes no available evidence separates the hypotheses. Then you have two honest moves. First: create the missing observation. If nobody has checked the accounts, don't wait for a clue; assign someone to check them. Second: treat the silence of evidence as information, not as quiet. Heuer warns that absence of clues is not a clue of absence — in 1998, nobody had signals India would test, and India tested. When even that is impossible, the decision shifts from evidence to consequences: what do you lose if you are wrong in each direction? A homeowner without access to the meter can pay for a small inspection to avoid a large repair. But what happens when your colleagues refuse to play the matrix game?
- 10
Strategic surprise comes not from missing information but from faint signals buried in louder, more plausible noise.
Pearl Harbor: Warning and Decision · Roberta Wohlstetter · 1962
Roberta Wohlstetter's Bancroft Prize-winning study moved the Pearl Harbor question from who was asleep to why it was not seen. Her answer: the United States failed to anticipate the attack not for want of the relevant materials but because of a plethora of irrelevant ones. The signals were there, but they were few and faint, buried in a larger and louder mass of indications pointing elsewhere. In December 1941 there was ample evidence for all the wrong interpretations, and they looked wrong only after the event. The problem survives in the profession's vocabulary as the signal-to-noise ratio. For an analyst the lesson is practical. More collection does not solve the problem and can make it worse, because noise grows with volume. What helps is knowing in advance which signal you are listening for — explicit hypotheses and indicators defined before the crisis — and judging each clue in its own context rather than with hindsight. The typical trap is the retrospective verdict: it was obvious is almost always the illusion of someone who knows the ending. Her most uncomfortable conclusion: strategic warning cannot be counted on, so defences must be designed to work without it.
Signals that are characterized today as absolutely unequivocal warnings of surprise air attack on Pearl Harbor become, on analysis in the context of December 1941, not merely ambiguous but occasionally inconsistent with such an attack.Roberta Wohlstetter, «Pearl Harbor: Warning and Decision» (Stanford University Press, 1962), pp. 387–388 — pasaj reprodus identic în «Nuclear Heuristics: Selected Writings of Albert and Roberta Wohlstetter» (Strategic Studies Institute, 2009), introducerea editorilor
Why it matters A system that collects everything and does not know what it is looking for reproduces 1941: the signal is in the file but never rises above the noise.
In full →Collapse ↑
Wohlstetter adds something subtler than noise: signals are not neutral. The same clue supports several stories at once, depending on context. A fleet on the move could mean attack or just an exercise. Whoever knows the ending automatically picks the right story and believes they saw it all along. The reader in 1941 had no such privilege. It is like a vague message from a friend before a meeting: it might hide a pleasant surprise, or a cancellation. Only after learning the outcome does the message seem to have said one thing. The next question is practical: how do you train yourself to judge clues without knowing how things end?
- 11
Warning is not information you either have or lack but a judgement built from an indicator list prepared in advance.
Anticipating Surprise: Analysis for Strategic Warning · Cynthia M. Grabo (Joint Military Intelligence College) · 2002
Grabo wrote the text in the early 1970s as a classified textbook for warning analysts, drawing on some twenty-five years of experience. The US Defense Intelligence Agency published it in three classified volumes between 1972 and 1974, and the Joint Military Intelligence College later published the declassified, condensed version. Her starting distinction is the one non-specialists most often miss: an indicator is a known or theoretical step the adversary should or may take in preparing for hostilities — something anticipated and put on a watch list. An indication is information that the step is actually being taken. The method works in three moves: build the indicator list in advance for the scenario that worries you (a call-up of reservists, a deployment of forces, a military alert), track what gets ticked off, and judge the pattern rather than isolated pieces. The warning judgement concerns the probability of action, not its imminence, and it can come weeks or months ahead. It pays off wherever surprise is very costly and the adversary's preparations leave traces. The trap is treating warning as a commodity you either have or lack. It is a hypothesis, so it has to be issued before certainty arrives.
Warning is an intangible, an abstraction, a theory, a deduction, a perception, a belief. It is the product of reasoning or of logic, a hypothesis whose validity can be neither confirmed nor refuted until it is too late.Cynthia M. Grabo, «Anticipating Surprise: Analysis for Strategic Warning» (Joint Military Intelligence College, 2002), secțiunea «What Is Warning?» — «Warning Is Not a Commodity», p. 4
Why it matters Without an indicator list written before the crisis, every new clue is read however suits the moment, and the pattern only shows afterwards.
In full →Collapse ↑
An indicator list does more than help you notice; it stops you from interpreting at will. Without it, the same sign means whatever anyone wants: a noise at night is “just the cat” or “an intruder,” depending on your mood. With a list written in advance, each sign is placed beside other signs, not beside your disposition. Grabo implies the real work happens before the crisis, when no one is under pressure. That is when you can calmly spell out what would mean preparations have begun. Which raises a practical question: who writes the list, and when?
- 12
Devil's advocacy tests a single dominant consensus; Team A/Team B sets two equally strong positions against each other.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
Both techniques are contrarian: they do not build an analysis from scratch but attack an existing one. The difference between them is diagnostic, and the primer warns that analysts often confuse them. Devil's advocacy is used when there is ONE dominant consensus: an analyst with doubts, or one designated by a manager, builds the best possible case for an alternative explanation. The steps: summarise the mainline judgement and key assumptions; pick the assumptions most open to challenge; check whether any information is questionable, whether deception is possible, or whether major gaps exist; present the findings and, if needed, draft a contrarian paper clearly labelled as an exercise. Team A/Team B is used when TWO equally strong positions coexist: each side writes out its full argument, with assumptions and data in plain view, and the decision-maker judges which is stronger. It is costly, so it is kept for long-running disputes or far-reaching decisions; analysts are sometimes made to argue the position they do not hold, to expose their own mind-set. The traps: a challenger with no real mandate who merely performs, and an unlabelled contrarian product the reader mistakes for the official view. Reaffirming the original line, with more confidence, is also a valid outcome.
Its primary value is to serve as a check on a dominant mind-set that can develop over time among even the best analysts who have followed an issue and formed strong consensus that there is only one way of looking at their issue.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (US Government, martie 2009), secțiunea «Devil’s Advocacy» — «Value Added», p. 17
Why it matters A consensus years old quietly filters out contrary evidence; without a mandated challenger, nobody checks the assumptions it rests on any more.
In full →Collapse ↑
Choosing between the two techniques starts with a simple question: how many strong explanations actually exist? If everyone believes the same thing, you don't need two equal teams, just one voice trying to break the consensus. If the office is split into two camps, a structured face-off becomes useful. Say your firm has believed for years that a certain type of client leaves only because of price. A devil's advocate would check whether those departures actually track price changes, or something else. But if half the team says "price" and half says "the support relationship," then each camp should write out its full case, with data on the table. The next question is what you do with the result once the exercise is over.
- 13
Red teaming is the independent, structured challenge of your own plan in service of a better decision — not a team playing the enemy.
Red Teaming Handbook (3rd Edition) · UK Ministry of Defence — Development, Concepts and Doctrine Centre (DCDC) · 2021
The British handbook separates four notions that are easily blurred. Red teaming is the activity: the independent application of structured, creative and critical thinking techniques so the decision-maker chooses better. A red team is a team formed specifically to subject an organisation's plans, programmes, ideas and assumptions to rigorous analysis and challenge. A red cell is something else: it adopts the viewpoint or even the persona of an adversary or key stakeholder, estimates what they would do, and can role-play them in wargames. NATO calls a closely related approach alternative analysis: the deliberate application of independent, critical thought and alternative perspective. The third edition adds the red team mindset — quick techniques used routinely by anyone who decides, not only by a separate team. How it is applied: the team works for a specific decision-maker, the end user in the definition; it stays independent of the plan's authors; it picks the techniques that fit the question; it hands over findings the decision-maker can actually use. It pays off before large, hard-to-reverse decisions. The traps: a red team is not immune to groupthink — the handbook devotes a section to it — and a team that wins by demolishing the plan, instead of making it more robust, has missed the point.
Red teaming is defined as: the independent application of a range of structured, creative and critical thinking techniques to assist the end user make a better-informed decision or produce a more robust product.UK Ministry of Defence, DCDC, «Red Teaming Handbook», ediția a 3-a (iunie 2021), capitolul 1 «Introduction», paragraful 1.9
Why it matters A plan's authors cannot see its blind spots on their own; an independent team with explicit techniques finds them before the adversary does.
In full →Collapse ↑
The handbook adds a nuance that is easy to miss: independence does not mean cold distance. The red team works for someone specific, on a specific question, and must deliver something usable. Not a report showing how clever its author is, but an answer the decision-maker can use tomorrow morning. From this follows a practical rule: techniques are chosen to fit the question, not out of routine. If the question is "what are we missing in this plan?", use techniques that surface hidden assumptions. If the question is "how would the other side react?", the angle changes. A household example: before signing a renovation contract, a friend reads the clauses not to wreck your plan, but to show you where it is fragile. The next step is to see what happens when challenge becomes a daily habit, not an event.
The third edition goes one step further: challenge should not wait for the special team. The handbook describes a red team mindset, meaning quick techniques anyone can use daily, at work or at home. It is no longer a ritual for big decisions, but a small reflex: before sending an angry message, you ask how it will land for the reader. The difference from a red team is scale, not nature. Both grow from the same independent, critical thinking. And once the daily reflex becomes habit, a new question appears: who challenges the challengers themselves?
- 14
A premortem asks the team to assume the plan has already failed and explain why, so objections surface while there is still time.
Performing a Project Premortem · Gary Klein (Harvard Business Review) · 2007
The premortem uses a perspective trick that Klein calls prospective hindsight: you imagine the event has already happened. Research from 1989, cited in the article, found that this raises people's ability to correctly identify reasons for future outcomes by 30%. The steps are simple. The team is briefed on the plan. The leader announces that the project has failed spectacularly. For a few minutes, everyone independently writes down every reason they can think of — especially the ones they would ordinarily not mention for fear of being impolitic. Then each person in turn reads one new reason, starting with the project manager, until the lists are exhausted. Finally, the manager reviews them and strengthens the plan. The difference from a conventional risk review is social, not just logical: asking what might go wrong asks someone to oppose a plan that is still alive, whereas asking why it failed only asks them to be clever. The technique also tempers the overconfidence of people too invested in the project. It is worth doing at the start of any important plan. The traps: open discussion before individual writing, where the loudest voices win, and a list of reasons that changes nothing in the plan.
Unlike a typical critiquing session, in which project team members are asked what might go wrong, the premortem operates on the assumption that the “patient” has died, and so asks what did go wrong.Gary Klein, «Performing a Project Premortem», Harvard Business Review, septembrie 2007 (reprint F0709A), partea introductivă a articolului
Why it matters People with reservations stay silent in the planning phase, exactly when it matters most; the premortem turns their objection from an act of opposition into a requested contribution.
In full →Collapse ↑
The card explains how to run a premortem. The next step is what you do with the list. A list of reasons is only raw material; value appears when each reason becomes a decision. Take every risk and ask: do we accept it, reduce it, or change the plan? For instance, if someone notes that a key supplier is often late, the plan gains a backup supplier, not just a shared worry. That is where the real difference shows: without a premortem, the risk stayed unspoken; with a premortem but no action, it stays merely spoken. A session ending in a list and no changes is just a collective confession. The question opening the next step: who decides, on your team, what is worth changing in the plan?
Passage 1 ended with the question of who decides. The answer matters more than it seems. If the manager judges the list alone, many motives die on the way: heard, then waved aside. Klein frames the premortem as an exercise run by the project leader, but its real effect appears when the leader commits to a response in advance. For instance, before the session they can say: any reason named by at least two people gets an owner and a deadline. The list then stops being a stroll through fears and becomes a contract. A premortem without a decision rule announced beforehand stays at the mercy of the boss's mood that day. Which raises another question: what do you do when the reasons voiced contradict the favored plan of the person holding the decision?
- 15
When everyone is sure something will not happen, it is worth writing down how it could happen after all and what you would see first.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · US Government · 2009
These are two related contrarian techniques for the moment when everyone is sure something will not happen. High-Impact/Low-Probability (HILP) analysis asks: if it does happen, what are the consequences, and by what path would we get there? What If? analysis accepts from the outset that the event has occurred and moves straight to how it came about. The primer recalls events once considered unlikely — the fall of the Shah, the collapse of the Soviet Union, the reunification of Germany — and a State Department assessment of 27 November 1941 that gave odds of five to one against the United States and Japan being at war by 15 December. The steps: state the dominant analytic line and the high-impact outcome clearly; assume it has happened; choose plausible triggers, such as the death of a leader, a natural disaster or an economic shock; think backwards, step by step, about what must have occurred; identify one or more pathways; write observable indicators for each and monitor them periodically. It matters most when a judgement rests on limited information or unproven assumptions. The trap: the exercise does not change the probability; it produces signposts to watch and hedging options. Taken out of context, it turns into alarmism.
By shifting the focus from whether an event could occur to how it may happen, analysts allow themselves to suspend judgment about the likelihood of the event and focus more on what developments—even unlikely ones—might enable such an outcome.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (US Government, martie 2009), secțiunea «What If? Analysis» — «Value Added», p. 24
Why it matters Low probability is exactly why nobody prepares the warning signposts; when the event arrives, there is no list that could have announced it.
In full →Collapse ↑
The card shows the techniques and steps. The next step is subtler: asking how something could happen frees the mind differently than asking whether it will. When you judge probability, you look for reasons to dismiss the scenario. When you assume it happened, you look for reasons that make it possible. Think of a friend who says he will never leave town: don't ask whether he intends to, ask what could take him away — a job, a breakup, illness in the family. Each answer becomes a signal you can spot from afar. That's why the exercise works for small decisions too, not just for intelligence services. The next question is how to pick the triggers worth watching.
Choosing triggers is what makes the exercise serious rather than a game of imagination. A good trigger sits in the middle: neither absurd nor certain. A leader's death, an earthquake, a money crisis — all things that have happened before. Pick something impossible, and the signals become useless. Think of an elderly neighbor living alone: a fall in the bathroom is a plausible trigger; a meteorite on the house is not. For each trigger you choose, you reason backward through the steps that must have happened before the event. On that path, the indicators you can actually watch appear. But finding indicators is half the job; what you do with them afterward is another story.
- 16
Scenarios do not guess the future: they separate what is already determined from what is truly uncertain and change the decision-maker's mental model.
Scenarios: Uncharted Waters Ahead · Pierre Wack (Harvard Business Review) · 1985
Wack headed the business environment division of the Royal Dutch/Shell Group's planning department, and his article opens with an uncomfortable observation: forecasts are dangerous precisely because they are usually right. They assume tomorrow will look like today and fail exactly when everything changes. The answer is not a better forecast but accepting uncertainty as a structural feature of the environment. The method separates two things: predetermined elements — what has already happened but whose consequences have not yet unfolded, like monsoon rains in the upper Ganges basin that will certainly show downstream — and critical uncertainties. The former are treated as facts; scenarios are built only on the latter. Today's practice has standardised the form. You pick, by consensus, the two most critical and uncertain forces, turn them into axes, and the four quadrants of the 2×2 matrix become four plausible worlds; the 2009 Tradecraft Primer describes exactly this step. For each world you write the indicators that would show it is becoming reality. The trap, in Wack's terms: first-generation scenarios that merely combine obvious uncertainties, such as one oil price or another, do not help decisions. A good scenario changes the decision-maker's mental model; otherwise it is water on a stone.
Our real target was the microcosms of our decision makers: unless we influenced the mental image, the picture of reality held by critical decision makers, our scenarios would be like water on a stone.Pierre Wack, «Scenarios: Uncharted Waters Ahead», Harvard Business Review, septembrie 1985, secțiunea despre «microcosmosul» decidentului
Why it matters Shell's scenarios did not guess the timing of the 1973 oil crisis, but they prepared management for its possibility — something a single forecast could not do.
In full →Collapse ↑
The next step: a good scenario does not end on paper. It must be rehearsed in the mind, like a dress rehearsal. The decision maker imagines living inside each world and notices where his usual model cracks. The valuable signal is that discomfort: when a plausible world feels absurd, that is exactly where your blind spot sits. A manager picturing his stores closed for a week discovers he has no plan for payments or supplies. That discovery is the point, not the story itself. The practical question becomes: what, in each scenario, would force you to drop a belief you now hold as certain?
Passage 2: Wack insists on a distinction most people skip: not everything uncertain deserves a scenario. Part of the future is already written; only its consequences haven't shown up yet. Children born fifteen years ago will, whatever happens, be teenagers in a few years. A company's decision today to close a factory will hit tomorrow's supply chain regardless of crises. That is the predetermined part: you treat it as fact, not as question. The scenario is built only on what remains genuinely open. The usual mistake is the reverse: doubting settled things while ignoring what nobody can know. A mayor planning schools should treat birth rates as known fact and only internal migration as uncertainty. The next question: how do you actually recognize what is already determined and what isn't?
- 17
Superforecasters were not born, they were found and grown: a tournament scored with the Brier score identified them by performance, and training, elite teams and practice made them better year after year instead of regressing to the mean.
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · Barbara Mellers, Philip E. Tetlock et al. · 2015
Between 2011 and 2015 IARPA, the research agency of the US intelligence community, ran the ACE tournament: hundreds of geopolitical questions with a verifiable answer and a fixed deadline, on which participants gave probabilities they could revise at any time. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, won. The yardstick was the Brier score: the sum of squared differences between the probabilities given and what happened (1 if the event occurred, 0 if not), on a scale from 0 (perfect) to 2; always saying 50% earns 0.5. The 2015 article describes how they were identified and cultivated. At the end of the first year 60 people were selected — the top five from each of the 12 experimental conditions — and placed in five elite teams of 12. The statistical expectation was regression to the mean; it did not happen two years running: the superforecasters became more accurate while the comparison groups regressed. What set them apart: fluid intelligence (the strongest dispositional predictor), political knowledge, actively open-minded thinking and, above all, behaviour — the frequency of updating forecasts was the strongest behavioural predictor of accuracy. Cultivation had three experimentally tested levers: a probability-training module of about 45 minutes, teamwork, and grouping the best together. The method transfers to any team that judges what will happen: questions with a deadline and a criterion, numerical probabilities updated often, a score at resolution, selection by track record rather than rank. The «Superforecasting» course on the same shelf takes it apart step by step.
This worldview predisposes superforecasters to treat their beliefs more as testable hypotheses and less as sacred possessionsMellers, Tetlock et al., «Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions», Perspectives on Psychological Science (2015), p. 273, secțiunea despre stilurile cognitive ale superprognozatorilor
Why it matters Forecasting talent only shows in a score kept over many questions — and an organisation that keeps no score picks its experts by how convincingly they talk.
In full →Collapse ↑
What the 2015 article actually measured (Mellers, Tetlock et al., Perspectives on Psychological Science 10(3)). Table 1 compares, averaged over years 2 and 3, the superforecasters with individuals on the top teams and with the other 1,498 participants who answered at least 25 questions:
SUPERS TOP TEAMS ALL OTHERS Standardised Brier -0.34 -0.14 0.04 (lower = better) Resolution 0.40 0.35 0.32 (higher = better) Calibration (gap) 0.01 0.03 0.04 (lower = better) AUC 96% 84% 75%Effort shows just as clearly: in the first year superforecasters made on average 2.77 forecasts per question against 1.47 for the comparison groups; in years 2–3 they opened 255 stories from the platform's news feed against 55 and 58. They attempted 25% more questions in the first year and about 40% more afterwards. The article offers four explanations that add up: cognitive abilities and style, task-specific skills, motivation and commitment, and an enriched environment (the elite teams). The often-quoted "top 2%" comes from the same group's 2014 article, not from this one.
How to apply this in a newsroom or a forecasting engine. Step 1: every published forecast gets a code, a numerical probability, a deadline and a resolution source — otherwise it cannot be scored. Step 2: at the deadline the outcome is recorded (1 or 0) and the score computed; in the olivLaw engine the resolutions table uses the binary form (p − outcome)², on a 0–1 scale where 50% gives 0.25 — half the score of the tournament's two-outcome form. Step 3: after at least a few dozen resolved questions per author (person, agent or model), compare the scores with a simple benchmark (say, "always the historical frequency") and only then talk about talent. Step 4: the best work together and see each other's forecasts; the rest take the training module. The course "Superforecasting — calibrated forecasting step by step" walks through each step, with worked examples, code and an audioguide.
One last piece remains: how do you decide, at the deadline, whether a prediction came true? The answer lies in how the question is worded. It must be written so that a stranger, with no extra explanation, could say yes or no. "Relations will get tense" fails; "by March, country X will have formally announced its withdrawal from the agreement" works. A couple arguing about "spending too much" can fix a number and a date, and the dispute becomes checkable. The trap is the vague question that can never be lost: without a deadline and a criterion there is no score, and therefore no learning. Once good questions and scores pile up, the natural next question arises: what actually separates the people who score well?
- 18
Before judging a case by its details, ask how similar cases ended: the base rate is the starting point, and the details only adjust it.
Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking · Daniel Kahneman & Dan Lovallo · 1993
Kahneman and Lovallo start from an anecdote. A team writing a school curriculum was estimating when it would finish. Asked how comparable teams had fared, the group's own expert admitted that about 40% had given up and none had finished in under seven years. Everyone had spontaneously taken the inside view: the plan, the obstacles, the scenario. The outside view ignores the details and asks what happened across a class of cases similar in relevant respects. The authors' point is that, applied with equal skill, the outside view is far more likely to yield a realistic estimate. The steps: choose a reference class of similar cases; find the distribution of outcomes, the base rate; place your case within that distribution, asking why it should do better or worse than average; only then adjust, moderately, for the specifics. It applies to any estimate of duration, cost or chance of success — how long coalition talks last, how often sanctions change a regime's behaviour. The trap is a reference class chosen to deliver the answer you wanted, or the claim that ‘our case is unique’, which the authors describe as the almost moral preference for the inside view.
In contrast, the outside view is essentially statistical and comparative, and involves no attempt to divine future history at any level of detail.Kahneman & Lovallo, «Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking», Management Science 39(1), ianuarie 1993, secțiunea «Inside and Outside Views», p. 24
Why it matters Most bad estimates do not come from missing information about the case but from nobody asking how cases of this kind usually end.
In full →Collapse ↑
The hard part of the outside view is that it feels rude. It tells you: your details don't matter, at least not at first. When you plan an apartment renovation, it doesn't ask about your odd wall; it asks how long similar renovations took. That answer seems to ignore everything you know about your situation. But that is exactly its power: it cannot be seduced by the optimism of your own plan. Inside estimates are built from intentions, and intentions rarely match the statistics. The outside view doesn't predict your story; it predicts where stories like yours usually end. Only once you have that starting point can you say what might pull you away from it. The first real decision is not the adjustment, but choosing the comparison class.
Passage 1 stopped at choosing the class. The next step is to look at the whole distribution, not just a comfortable average. In the authors' example, comparable teams did not merely take longer; a large share never finished at all. Your estimate must include that scenario, not just an optimistic deadline stretched a little. When you say you will find an apartment in two months, ask how many similar searches stalled or collapsed, not only how long the successful ones took. The average hides the tail where failures live. And when failure is costly, the distribution matters more than the expected value. But how do you compare your case to the class fairly, without rigging the match?
- 19
Bayes does not ask you to guess the truth, only to make one judgement per item of evidence: how many times more likely it is if the hypothesis is true than if it is false.
Bayes' Theorem for Intelligence Analysis · Jack Zlotnick (CIA, Center for the Study of Intelligence) · 1972
The CIA sponsored in-house research on applying Bayes' theorem to intelligence analysis, and Jack Zlotnick, an analyst who took part, described the method in Studies in Intelligence. It uses the odds form: the revised odds (R) equal the prior odds (P) multiplied by the likelihood ratio (L). Analysts never judged the conclusion directly; they judged only L for each new item — how many times more likely, say, a troop deployment to a border is if war is coming than if it is not. If twice as likely, L = 2 and the odds double. R then becomes the P for the next item. The steps: frame two mutually exclusive hypotheses; fix the starting odds explicitly; for each item of evidence, estimate separately how likely it is under each hypothesis; multiply and move on. It pays off when evidence arrives in a stream and the temptation is to react to the latest item. Zlotnick names a limit: close to the climax, much incoming evidence becomes undiagnostic — equally likely under both hypotheses. A second classic pitfall is treating two reports from the same source as independent; multiplied as if they were, they inflate the conclusion. The typical error remains mistaking evidence that is consistent with a hypothesis for evidence that is diagnostic.
The very best that intelligence can do is to make the most of the evidence without making more of the evidence than it deserves.Jack Zlotnick, «Bayes' Theorem for Intelligence Analysis», Studies in Intelligence vol. 16, nr. 2 (1972), paragrafele introductive
Why it matters Without an explicit updating rule, the latest item of evidence gets the most weight merely because it is the latest, not because it says more.
In full →Collapse ↑
The card shows how the likelihood ratio is computed. The next step is subtler: who sets the starting odds matters enormously, because every later multiplication builds on that starting point. If two analysts begin from different beliefs, the same evidence moves them in different directions, but not necessarily to the same place. A familiar example: two people read the same ambiguous restaurant review. One was already convinced it was worth trying, the other skeptical. Both update correctly, yet they end up with different conclusions. That is why the method demands that initial odds be stated openly, not hidden in one's head. Then you can see where each person started and how far the evidence moved them. A question remains: what do you do when you have no basis at all for the starting point?
When you have no basis for a starting point, the method gives a simple answer: even odds. It does not mean you know something; it means you officially declare that you don't. Then the evidence does all the work. A useful distinction appears here: a piece of evidence can be compatible with your hypothesis and still move you not at all. If its likelihood ratio is one, multiplying leaves the odds untouched. Example: you hear that a neighbor bought bread. That says almost nothing about whether they cook often or order takeout, because both hypotheses make bread equally likely. The disciplined analyst does not ignore the evidence; they measure it and accept the answer, even if it is zero movement. But what do you do when two pieces of evidence seem different and yet say the same thing?
- 20
A close, friendly group is in the greatest danger of no longer thinking critically, and the remedy is a procedure that makes objection a duty rather than a betrayal.
Victims of Groupthink: A Psychological Study of Foreign-Policy Decisions and Fiascoes · Irving L. Janis · 1972
Janis studied foreign-policy fiascoes, starting with the Bay of Pigs, and set them beside better-run decisions such as the Cuban missile crisis and the Marshall Plan. His conclusion, offered in the spirit of Parkinson's laws, is the quote above: the danger grows with cohesion. The symptoms are easy to recognise — an illusion of invulnerability, self-censorship of doubts, an illusion of unanimity, self-appointed mindguards who shield the leader from awkward information. The result is alternatives left unexamined and risks never weighed. Yet Janis does not conclude that cohesion is bad: he notes that groups lacking it can produce even worse fiascoes. The remedies are procedural, not moral. The role of critical evaluator is given to every member, so that objecting is an obligation rather than disloyalty; outside experts are invited to challenge the group's views; after a preliminary decision, a ‘second chance’ meeting lets anyone voice remaining doubts. For an analytic team this means a peer review with an explicit mandate to find weaknesses, and a dissenting note that travels with the product. The typical trap is ritual dissent: a role everyone knows is play-acted, so nobody listens to it any more.
The more amiability and esprit de corps among the members of an in-group of policy-makers, the greater is the danger that independent critical thinking will be replaced by groupthink, which is likely to result in irrational and dehumanizing actions directed at out-groups.Irving L. Janis, «Victims of Groupthink» (Houghton Mifflin, 1972), secțiunea «Are cohesive groups doomed to be victims?», p. 198–199
Why it matters Most analytic teams are small, stable and collegial — exactly the profile Janis describes as most exposed.
In full →Collapse ↑
Passage 1. Janis notices something subtle: the remedies fail if the leader merely announces them. If the boss says "I want critics" but reveals a preference early, the group reads the gesture, not the rules. That is why good procedures shift the weight to timing: the leader stays silent at first, keeps the problem framed neutrally, and speaks only after hearing every position. In a family choosing a vacation spot, a parent who says "I'd love the beach" has already decided, however warmly others are invited. Tone arrives before words. The next question is what happens when the leader truly stays quiet: who fills the gap, and how much power does the first speaker hold?
When the leader stays silent, the first word becomes an anchor. Whoever speaks first offers not just an opinion but sketches the terrain everyone else positions themselves against. Janis shows the illusion of unanimity is born exactly here: each member censors the doubt that would break apparent harmony. The procedural remedy reverses the order: every member writes their position before any discussion, on paper, simultaneously. In a couple choosing an apartment, if each privately writes their top three before any viewing, nothing is contaminated by the other's enthusiasm. Then the lists are compared, and real disagreements surface instead of fake ones. Simultaneous writing turns the leader's silence from a dangerous void into protected space. But a problem remains: what do you do with the member who, after writing honestly, still shifts position upon hearing the majority?
- 21
Effective deception does not change your belief, it confirms it; the most dangerous set of clues is the one that fits perfectly with what you already believed.
Deception Maxims: Fact and Folklore · Central Intelligence Agency, Office of Research and Development · 1980
In 1980 a CIA research paper distilled historical deception cases into maxims for analysts. The first, Magruder's Principle, holds that reinforcing an existing belief is easier than building a new one. The classic case is Normandy: Hitler and most of his senior military advisers expected the landing in the Pas de Calais, the Allies knew this from intercepts, and the cover plan fed exactly that belief. Jones' Lemma, named after the British physicist R. V. Jones, adds that deception grows harder as the victim's channels of information multiply — but, within limits, the more channels are controlled, the more believable the story. Counter-deception follows from both. Write down what you already believe and what the adversary would want you to believe; if they coincide, raise your guard. Check how many genuinely independent channels the confirmation comes from, since controlled channels confirm one another. Look for evidence that should exist if the story is true and is missing. In a hypothesis matrix, carry deception as an explicit hypothesis. The opposite trap is paranoia: a remark attributed to Jones, known as Crabtree's bludgeon, warns that for any set of inconsistent observations someone can devise a coherent explanation, however complicated — so a deception story that explains everything proves nothing on its own.
It is generally easier to induce an opponent to maintain a preexisting belief than to present notional evidence to change that belief.CIA, «Deception Maxims: Fact and Folklore» (aprilie 1980, declasificat în 2015), Maxima 1 «Magruder's Principle — the Exploitation of Preconceptions», p. 5
Why it matters An analyst is easiest to deceive exactly where they feel most certain, because there the adversary need not persuade them of anything new.
In full →Collapse ↑
The card says deception confirms what you already believe. The next step: this makes deception cheap and truth expensive. A liar does not build a new story; he only feeds one already sitting in your head. You do half the work, from memory. That is why deception prefers certainty: there, one clue placed at the right moment is worth a pile of evidence. Think of the phone scammer who says only "I'm calling from your bank" and then goes quiet — you fill in the rest yourself, because you already feared such a call. The question that opens up: if confirmation comes easily, how do you tell how many real sources it comes from?
Passage one asked how many sources the confirmation comes from. The answer is subtler than it seems: the number of sources doesn't matter, their independence does. The CIA study shows that a deceiver can control several channels at once, and controlled channels confirm each other — they seem like many, but they are one voice with several mouths. A captured spy, an intercepted message, a defector: all can be pieces moved by the same director. Think of online reviews: fifteen glowing comments about a restaurant may come from one owner with multiple accounts. The test is not "how many sources say this?" but "how many cannot be controlled by the same hand?". One genuinely independent source is worth more than ten that reinforce each other. And if independent sources are missing, another check remains: what clues should exist if the story were true — and precisely those are missing?
- 22
An adversary is not analysed in a vacuum: first the environment — political, military, economic, social, information and infrastructure systems — then what the adversary can do within it.
Joint Publication 2-01.3: Joint Intelligence Preparation of the Operational Environment · US Joint Chiefs of Staff · 2014
US joint doctrine calls the process JIPOE, joint intelligence preparation of the operational environment. The 21 May 2014 edition of JP 2-01.3, which superseded the 2009 one, sets out four steps: define the operational environment, describe its impact, evaluate the adversary and other relevant actors, and determine their courses of action. The environment covers the physical domains, the information environment (including cyberspace) and the PMESII systems — political, military, economic, social, information and infrastructure. The civilian variant used in business analysis is STEEP: social, technological, economic, environmental, political. To apply it, list the elements, links and pressure points of each system, then ask how they constrain each actor's options — an energy embargo read as an economic event has political and social effects in another country. The companion technique in the 2009 Tradecraft Primer is outside-in thinking: start from the external forces you do not control and only then narrow to your subject, so that no variable is missed. It pays off at the start of a new analysis or when the subject shifts. The trap is the inventory: a complete PMESII list with no conclusion about what matters. The 2014 edition is the latest public one; the current joint doctrine list sits behind a restricted portal.
The JIPOE process provides a disciplined methodology for applying a holistic view of the OE to the analysis of adversary capabilities and intentions.US Joint Chiefs of Staff, JP 2-01.3 «Joint Intelligence Preparation of the Operational Environment» (21 mai 2014), Executive Summary, «The Operational Environment (OE) — A Holistic View»
Why it matters Most surprises come not from the adversary's capabilities but from systems the analysis never looked at — a power grid, a diaspora, a messaging platform.
In full →Collapse ↑
The card says to analyze the environment before the adversary. The next step: systems are not separate; they touch each other. A disturbance in one spreads into the others, often with delay. A cut in trade routes may first hit market prices, then the state budget, then people's trust in government. The analyst asks not only what each system contains, but where the links between them run. A familiar example: a messaging platform looks like an informational system, but it becomes social when groups mobilize through it and economic when local trade moves there. Read it only as technology, and you miss its real effect. The useful question is simple: where does a push in one system come out in another? The next step is to find which of these links are fragile and can be broken.
The previous passage asked where the links run. Now comes selection: not all links weigh the same. The doctrine asks the analyst to find pressure points, meaning nodes where several systems meet in one place. There, a small change produces large effects. A bridge carrying freight, a hospital's supply line and commuters is such a node. If it fails, the effect is not merely technical; it spreads into the economy and social life. In practice, you build a short list: two or three places from which the whole environment can be moved. The rest of the inventory stays background. This gives the analysis a hierarchy, not just a map. The next step is to see who could press on these nodes, and with what intention.
- 23
Open source material becomes evidence only when you can show where it came from, that it has not been altered, and what you did with it from the first search onwards.
Berkeley Protocol on Digital Open Source Investigations · OHCHR & Human Rights Center, UC Berkeley School of Law · 2022
The Berkeley Protocol, published by the UN human rights office (OHCHR) together with the Human Rights Center at Berkeley, sets international standards for digital open source investigations. It grew out of a courtroom problem: smartphone photographs and videos posted online, some compromised or misattributed, cannot serve as evidence if they were collected and kept with unsound methods. The Protocol divides an investigation into six phases: online inquiries, preliminary assessment, collection, preservation, verification and investigative analysis. In practice the steps look like this: log every search (term, platform, date); capture the material in full, with its metadata, and record a cryptographic hash at collection; keep the original untouched and work on copies; verify the source (who posted it, when, from where) separately from the content (geolocation, chronology, shadows, weather); only then interpret. It pays off whenever a finding may end up before a court, a regulator or a hostile journalist. The trap is the reverse order: interpreting first and preserving later — by which time the post has been deleted or edited, and you can no longer show what you saw.
For open source information to be admissible as evidence in court, prosecutors and counsel must typically be able to establish its authenticity and chain of custody.«Berkeley Protocol on Digital Open Source Investigations» (ONU, New York și Geneva, 2022), capitolul I «Introduction»
Why it matters An open source finding that nobody can retrace step by step is an opinion, however good the image.
In full →Collapse ↑
The card tells you what you must be able to show: origin, integrity, the material's path. The next step is less visible: everything is decided in the first minutes, before you know whether the material matters. The moment you click on a post is already the moment you can lose the evidence. If you don't capture the full page right away, with its address, time and context, the source can be edited or deleted before you decide it's relevant. That is why the Protocol treats preservation as a reflex, not as a step you take once you're convinced. A simple example: you see a video of an incident and tell yourself you'll come back to it. By then, the account that posted it is gone. What you remember can no longer be verified by anyone. The practical rule: capture first, judge later. And once you've captured it, the next question appears: how do you prove that copy is identical to the original?
The previous passage ended on a question: how do you prove your copy is identical to the original? The Protocol's answer is the cryptographic fingerprint. At the moment of collection, you compute a hash: a short code generated from the file's content. If the file changes by even one pixel, the code changes completely. You don't need to understand the math; you just need to record it at the same moment as the capture. A hash computed later, from a copy already handled, proves nothing about the original. An everyday example: it's like weighing the package at delivery, not the next day, after the box sat open on your desk. Collection is also when you set aside metadata: time, device, who made the capture. But a good fingerprint says nothing about who posted the material. That is where the next step begins.
- 24
A wargame does not predict, it discovers: it shows what is plausible and where the plan gives way when someone genuinely opposes it.
Wargaming Handbook · UK Ministry of Defence, Development, Concepts and Doctrine Centre (DCDC) · 2017
The 2017 British handbook defines wargaming as a decision-making technique that offers structured but intellectually liberating, safe-to-fail environments — a place to find out what works and what does not, usually at low cost. At its core are the players, the decisions they take, the narrative they create, their shared experiences and the lessons they carry away. The handbook separates analytical, discovery wargames from training, learning ones; for analysis the first kind matters, from planning games to games that inform real executive decisions. The steps: frame the question (what you want to find out, not what you want to prove); choose the players, including a capable opposing side that is free to win; set the rules and an umpire who adjudicates the effects of moves; run the game, then conduct a rigorous post-game analysis. The handbook is explicit that wargames are not predictive and that a single run risks false lessons, which is why several games are played. It pays off when the adversary reacts to what you do — an interaction a static analysis cannot see. The trap is the game staged to confirm the plan, with an opponent who is not allowed to win.
Wargames can illustrate that something is plausible, but will not be able to definitively predict that it is probable.UK MoD, DCDC, «Wargaming Handbook» (august 2017), capitolul 1, paragraful 1.19
Why it matters A plan never contradicted by a free opponent is an untested plan, however good it looks on paper.
In full →Collapse ↑
The handbook brings something rare into planning: permission to lose. In a wargame, failure is cheap and useful, not shameful. When your plan breaks on the game board, you learn something no presentation will tell you. Think of a household budget: better to discover at the table that repairs don't fit than when the car dies on the road. That is why the opposing team must be honest, not polite. A loss in the game is a gift, if it is recorded honestly. The next question is how you pick people brave enough to beat you.
Passage 2: Choosing the adversary is half the game. The handbook insists on a team that thinks like your real opponent and is free to win. If the adversary is weak or restrained, the game only confirms your hopes. It is like boxing sparring: if your partner never hits, you learn nothing about your guard. That is why the umpire matters just as much: he honestly rules on the effects of each move, without shielding the plan. An umpire who softens the blows destroys everything the opposing team was for. And when the adversary truly wins, the hardest step follows: understanding why, not merely who won.
- 25
A product written with the help of a model is still an intelligence product: the same standards and the same question — how do you know — and the answer must lead to real sources, not to the model.
Artificial Intelligence Ethics Framework for the Intelligence Community (v1.0) · Office of the Director of National Intelligence (ODNI) · 2020
In 2020 the US intelligence community published six principles of AI ethics and an implementation framework written as a list of questions. The human-centred principle asks that technological guidance be tempered with human judgement; the framework asks who the accountable human is, what they must know about the model to judge its reliability, and how AI outputs are marked. ICD 505, signed on 17 January 2025, requires data traceability from AI inputs to AI-derived outputs and systems that let analysts meet ICD 203 and ICD 206, the analytic and sourcing standards. The US AI Action Plan of July 2025 also called for an AI assurance standard under ICD 505. Applied in 2026 to a product drafted with a language model, this means: every factual claim leads to a real source that a human has opened and read, not to the model's answer; quotes and figures are checked at the source, because models invent plausible references; probability and confidence remain the analyst's judgement; and whatever the machine produced is marked. The main trap is automation bias: accepting a fluent machine's output without the checks you would demand of a new colleague. The framework says it plainly: machine errors may differ from human errors.
How might you respond to an intelligence consumer asking “How do you know this?” How will you describe the dataset(s) and tools used to make the output?ODNI, «Artificial Intelligence Ethics Framework for the Intelligence Community», v1.0 (iunie 2020), secțiunea «Transparency: Explainability and Interpretability»
Why it matters A model writes fluently and convincingly exactly where it has no sources — and fluency is the signal readers most easily mistake for truth.
In full →Collapse ↑
Fluency is a trap not only for readers, but for authors too. When a model writes a coherent paragraph, your brain reads that coherence as proof the author knows the subject. But coherence is cheap: the model produces it whether or not it has sources. A colleague who writes beautifully but invents a figure gets caught at verification. A model does the same thing, just more often and more confidently. That is why verification must never scale down with how good the text sounds. Example: a report citing an energy-consumption statistic sounds authoritative, yet the number exists only if you find it in a published source. So how will you build a routine where fluency never lowers your verification standard?
Passage 2: Automation bias is subtler than it looks. You don't only accept a false result; you also accept a true one that fits poorly, because the machine said it confidently. The ODNI framework asks for marking what the model produced, precisely so readers know what passed through human judgment and what didn't. Think of a colleague who sends you a polished paragraph but never says what he wrote and what a tool generated. Without the mark, your checking stops where fluency begins. That raises the practical question: where, in your text, should the marking begin?
- 26
Analysis is one step in a cycle; what comes out is at best as good as the question asked at the start and the evidence collected for it — which is why the intelligence requirement is written before any search.
Joint Publication 2-0, Joint Intelligence · US Joint Chiefs of Staff · 2013
Doctrine describes the cycle as six overlapping categories of operations: planning and direction (what we must learn and for which decision), collection (from which sources), processing and exploitation (from raw signal into something readable — translation, decryption, extraction), analysis and production (the judgement), dissemination and integration (it reaches the decision-maker, in time) and evaluation and feedback (was it any use?). JP 2-0 insists it is not a conveyor belt: the steps run in parallel, get skipped (an image already collected answers directly), and new requirements emerge from analysis. The piece analysts most often treat as "given" is the requirement. A priority intelligence requirement (PIR) is a question the commander needs answered for a specific decision — not "what is X doing", but "will X do Y before Z, so that we know whether to activate plan W". A question phrased that way dictates what is collected, which indicators are watched (lesson 11), and when the answer "we do not know yet" is itself useful. For anyone outside the military system the cycle holds just as well: a firm that asks "what is happening in the market" gets noise; one that asks "will competitor X enter segment Y by Q2, which would change our pricing decision" gets a list of sources, indicators and a deadline. The trap: skipping planning because "we know what we are looking for". Feedback closes the loop — without it, the same requirement keeps being collected long after the decision was made.
The six categories of intelligence operations are: planning and direction; collection; processing and exploitation; analysis and production; dissemination and integration; and evaluation and feedback.US Joint Chiefs of Staff, JP 2-0 «Joint Intelligence» (22 octombrie 2013), capitolul I, secțiunea 3 «The Joint Intelligence Process»
Why it matters A brilliant analysis of a question nobody asked for a decision is literature, not intelligence.
In full →Collapse ↑
A worked example: from "what is X doing" to a collectable requirement. The fictional firm Alfa sells invoicing software in Romania and hears that a competitor, Beta, "is moving". The information question is useless; the requirement is built backwards from the decision:
DECISION Alfa decides in Q1 2027 whether to cut the price of its SME bundle by 15%. REQUIREMENT Will Beta launch an SME offer in Romania before 31 March 2027? WHY If yes, the price cut is made pre-emptively; if no, it is postponed a year. DEADLINE Answer useful until 15 January 2027; after 31 March it no longer matters. INDICATORS hiring notices (RO sales, Romanian-language support) · trademark / .ro domain registration · partnerships with accountants · test prices on the site · business-press mentions · a Romanian-language demo at an event SOURCES Beta's careers page, OSIM/EUIPO, the .ro registry, the business press, two accounting conferences in Q4 2026 STATUS "we do not know yet" = useful until 15 January; after that, decide with what we haveEvery line above decides something concrete about collection: the indicators say what is watched, the sources say where, the deadline says when to stop. Without the first line, the rest cannot be written.
The six categories applied to the same requirement. Planning and direction: the PIR above, with an owner and a review date. Collection: someone subscribes to the careers pages and the OSIM bulletin, someone attends the two conferences. Processing and exploitation: the notices are turned into a table with date, role, city; the conference photos are transcribed. Analysis and production: on 10 January, three sales hires in Bucharest plus a trademark registered in November lead to the judgement "probable (55–70%) that Beta launches by Q2, less likely before 31 March". Dissemination and integration: the judgement goes into the material for the pricing meeting, with the indicators that would change it. Evaluation and feedback: on 1 April, what happened is recorded, and whether the judgement helped the decision — that is where the next one is calibrated (lesson 17).
What gets skipped: if Beta announces the launch publicly on 20 December, processing and analysis have nothing left to do; the answer goes straight to the decision-maker. Doctrine describes exactly this short-circuit as normal, not as an exception.
- 27
New ideas do not come from free-flowing talk but from a divergent phase with rules (nothing is censored, an "outsider" at the table, an hour until the commonplaces run out) followed by a convergent one; and a written chronology is the cheapest way to see what is missing and what does not fit.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · Guvernul SUA (CIA, Center for the Study of Intelligence) · 2009
The primer puts brainstorming ahead of all the other techniques, as an aid to forming hypotheses (lesson 9) or indicators (lesson 11). Its rules run against intuition: never censor an idea, however unconventional; ask what prompted it, because an unstated assumption may sit there; allow time — usually an hour until the rules are set, the group relaxes and the conventional wisdom runs out, and only then do the truly new ideas appear; bring at least one "outsider", someone with a different training or culture but familiar with the topic. The divergent phase produces; the convergent phase groups the ideas around key concepts and makes them usable. The primer describes a two-phase, twelve-step process, but the essence is the separation: do not evaluate while you generate. The chronology is the sister technique, less spectacular and more often useful. You put the events in the order they happened, with the source of each, and look at three things: gaps (what should have happened between two points and appears nowhere), coincidences (two things in the same week that no source connects) and order (the effect is reported before the cause — either the source is wrong or the cause is something else). A well-kept chronology is also the antidote to selective memory: what contradicts the current hypothesis stays on the list, with its date. Both are start-of-project techniques. The most frequent mistake is doing them after the conclusion has formed, as decoration.
Paradoxically, brainstorming should be a very structured process to be most productive.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (Guvernul SUA, martie 2009), secțiunea «Brainstorming», paragraful «The Method»
Why it matters A group that evaluates ideas as it produces them stops at the first that sounds reasonable — usually the one it already had.
In full →Collapse ↑
A structured brainstorm, in the primer's form, for the question from lesson 26 ("will Beta launch an SME offer?"). Divergent phase: six people, one of them from accounting (the outsider), each writing on sticky notes, no discussion, for 15 minutes, every hypothesis and indicator that comes to mind; the moderator puts them on the wall without comment. Convergent phase: the group moves the notes into groups and names them.
GROUP "Beta enters directly" RO hires · trademark · Romanian demo · test prices GROUP "Beta enters via partner" deal with an accountants' network · integration with a local ERP GROUP "Beta buys" talks with a local player · a recent funding round GROUP "Beta does not enter" focus on another country · exits from small markets · restructuring GROUP "False signal" hires for multilingual support, not RO · a defensive trademarkThe last two groups exist only because nobody censored: "Beta does not enter" and "the signal is false" are the hypotheses a group convinced that "Beta is moving" would never have written. They become the columns of the ACH matrix (lesson 9), and the indicators from the first three groups go into the warning list (lesson 11).
The chronology, kept as a table with the source on every row. The rule: one row per event, the most precise date available, the source, and a "fits the hypothesis?" column filled in only at the end.
DATE EVENT SOURCE HYPOTHESIS 2026-09-02 Beta closes a funding round (€12M) financial press all 2026-10-14 trademark "Beta Contab" filed at EUIPO EUIPO register enters directly / defensive 2026-10-20 notice: 2 sales positions, Bucharest careers page enters directly 2026-11-05 Beta presents at the accountants' conference attendee (note) enters directly / partner 2026-11-18 article: Beta "withdraws from Hungary" regional press does not enter? ← does not fit 2026-12-01 a third sales position, Cluj careers page enters directlyThree things surface only once the rows are in order. The gap: between the funding (September) and the trademark (October) there is no product hire — a direct entry would have required one. The coincidence: the Hungarian withdrawal comes two weeks after the Romanian conference; no source links them, but together they suggest reallocation, not retreat. The order: the trademark is filed before the first hires — consistent with "enters directly", inconsistent with "defensive trademark" (which would not be followed by positions). The arrowed row stays on the list precisely because it does not fit.
- 28
Before you analyse the actors you follow, list the forces none of them controls — social, technological, economic, environmental, political — and ask how each could upend the problem; otherwise you find the missing variable after you have coded the whole file.
A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis · Guvernul SUA (CIA, Center for the Study of Intelligence) · 2009
The technique is designed for the start of an analytic project, when the goal is to identify all the critical external factors that could influence how a situation develops. The primer's method has four steps: describe the problem generically; list all the key forces (social, technological, economic, environmental, political — the STEEP grid) that could have an impact but over which you can exert almost no influence (globalisation, social stress, the Internet, the global economy); then move to the factors over which an actor or policymaker can exert some influence (market size, customers, competition, partners; in the government domain, policy actions or the behaviour of allies and adversaries); assess how each force could affect the problem and check, on the evidence, whether it actually does. The value added is twofold. The first is preventing a mundane and expensive mistake: the primer gives the example of a database built with too few categories, where analysts realise too late that fields are missing and have to recode every file — outside-in, with modest effort, reduces the risk of missing important variables early. The second is getting out of the "inbox": the analyst who looks from the inside sees what he controls and what his actors control; the external changes that can profoundly affect their field stay invisible until they become a crisis. Lesson 22 (PMESII) answers "what is the environment"; outside-in answers "what from outside the environment can change it". They are used together, but the questions differ.
Most analysts spend their time concentrating on familiar factors within their field or analytic issue. That is, they think from the “inside”—namely, what they control—out to the broader world.«A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis» (Guvernul SUA, martie 2009), secțiunea «Outside-In Thinking», paragraful «Value Added»
Why it matters The variable missing from the model appears in none of the sources you follow, precisely because you did not know to follow it.
In full →Collapse ↑
The STEEP grid applied to the same question about Beta, before any collection. Each row is a force neither Alfa nor Beta controls; the right-hand column is the question that could upend the problem.
FORCE (nobody controls it) HOW IT COULD UPEND THE PROBLEM S mandatory e-invoicing for all firms SME demand jumps → everyone enters, not just Beta T invoices generated directly by AI models the invoicing product becomes a feature, not a market E recession / expensive credit SMEs postpone any new subscription; so does Beta E EU data regulation (local hosting) entry requires in-country infrastructure → higher cost, delay P change of government, new tax thresholds the definition of "SME" shifts → Alfa's segment movesOnly below this list come the factors the actors can influence: Alfa's price, Beta's partners, the sales channel. Note that row T questions not the answer but the question: if invoicing becomes a feature of an AI assistant, "will Beta enter the segment" is no longer the requirement that matters.
How the list is checked, so it does not remain an exercise in imagination. Step four of the primer demands evidence that the force actually affects the problem: for each row, a source and an indicator.
FORCE EVIDENCE IT IS ACTING NOW INDICATOR TO WATCH e-invoicing the official rollout calendar for all firms the SME effective date AI in invoicing two competing products launched "invoices from chat" Beta announces a similar feature credit the policy rate, SME association surveys new subscriptions per quarter local data the regulation's text, compliance deadlines Beta rents hosting in RO tax thresholds the bills under debate a change to the VAT / micro thresholdThe practical result of the technique is the third column: five indicators that did not appear in the list from lesson 26, because that list looked only at Beta. If one of them lights up, the requirement is rewritten — not the answer.
- 29
A network is read with three measures, not one: who has the most ties (degree), who sits on the path between groups (betweenness) and who reaches everyone quickly (closeness) — and Krebs's lesson is that the most visible person is not necessarily the one the information flows through.
Mapping Networks of Terrorist Cells · Valdis E. Krebs · 2002
Krebs reconstructed from public sources the network of the 19 hijackers of 11 September and measured it. Mohamed Atta had the highest degree (0.417) and the highest closeness (0.571), but not the highest betweenness: there Nawaf Alhazmi led (0.334 against 0.318). His conclusion is cautious and instructive: the metrics "do not necessarily confirm" leader status, because "we are obviously missing nodes and ties". A covert network is built to look different from what it is — sparse ties, cells that touch through a single person, members who do not know each other. Low density is a security choice, not an organisational weakness. The three measures answer different questions. Degree: who is the most active in contacts — good for finding hubs, bad for finding who controls the flow. Betweenness: who sits on the most shortest paths between others — the person whose removal breaks the network into pieces; often a discreet one. Closeness: who reaches everyone in the fewest steps — who learns first. An analysis that reports a single ranking hides the structure. The traps, in order of frequency: confusing co-appearance in the same documents with a relationship (two names in one article are not a tie); forgetting that a missing tie in the data is usually missing collection, not a missing relationship; treating centrality as guilt. Krebs writes explicitly that his map is incomplete and that mapping is a stage, not a conclusion. For olivLaw the rule is the same as in the lesson on actors: an affiliation is asserted only with a source, and co-occurrence is called co-occurrence.
Analyzing networks after an event is fairly easy for prosecution purposes. Mapping covert networks to prevent criminal activity is much more difficult.Valdis E. Krebs, «Mapping Networks of Terrorist Cells», Connections 24(3), 2002, rezumatul articolului
Why it matters The person with the most ties is the one you see; the person the information flows through is the one you must look for.
In full →Collapse ↑
The three measures, computed on a small network with free tools. Seven fictional people, ties from sources (in practice each edge would have a chronology row with its source). `networkx` is a standard Python library for this:
import networkx as nx G = nx.Graph() G.add_edges_from([ ("Ana", "Bogdan"), ("Ana", "Cristina"), ("Ana", "Dan"), # Ana: many ties inside cell 1 ("Bogdan", "Cristina"), ("Dan", "Elena"), # Dan: the only bridge to cell 2 ("Elena", "Florin"), ("Elena", "Gabriel"), ("Florin", "Gabriel"), ]) for name, fn in (("degree", nx.degree_centrality), ("betweenness", nx.betweenness_centrality), ("closeness", nx.closeness_centrality)): top = sorted(fn(G).items(), key=lambda kv: -kv[1])[:3] print(f"{name:12s}", " ".join(f"{n}={v:.2f}" for n, v in top))degree Ana=0.50 Elena=0.50 Bogdan=0.33 betweenness Dan=0.60 Ana=0.47 Elena=0.47 closeness Dan=0.60 Ana=0.55 Elena=0.55Ana and Elena have the most ties; Dan has only two, but sits on every path between the two cells — betweenness puts him first. Exactly Krebs's pattern: the visible one (Ana) and the one everything passes through (Dan) are different people.
How to read the map without inventing. Three hygiene rules, each with its shape in the table:
RULE WHAT IT LOOKS LIKE IN THE DATA one edge = one source a "source" column on every tie; without a source, the edge is dotted co-occurrence ≠ relationship the tie type is explicit: "signed with", "seen with", "quoted together" absence is missing collection a cell with no outside ties = "unknown", not "isolated"And a check Krebs makes implicitly: recompute the measures after removing a node. If removing Dan leaves two separate components, his betweenness was not an artefact of the data but the structure of the network. If instead the network stays connected through someone you did not have on the map, you have just found the missing collection.
The link to the rest of the course: the nodes come from actor extraction (the lesson on the actor network in analyses), the typed and sourced edges come from the chronology (lesson 27), and high centrality is a hypothesis to test with ACH (lesson 9), not a conclusion.
- 30
An intrusion is not an event but a progression in seven phases; it is a "chain" because any broken link stops the whole — so a defender who recognises a campaign's pattern can stop it early, with intelligence, rather than just clean up after it.
Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains · Eric M. Hutchins, Michael J. Cloppert, Rohan M. Amin (Lockheed Martin) · 2011
The 2011 paper borrowed from military targeting doctrine the idea of a chain — find, fix, track, target, engage, assess — and rewrote it for intrusions: reconnaissance (researching targets: websites, mailing lists, technologies), weaponization (building the payload into a document or file), delivery (email, website, USB), exploitation (the code runs), installation (persistence), command and control (the channel to the outside) and actions on objectives (exfiltration, lateral movement). The essence, the authors say, is that the intruder "must develop a payload to breach a trusted boundary, establish a presence and only from there take actions" — and a single failure along the way interrupts the entire process. The paradigm shift is for defence. Until then defence was reactive: detect the code, delete it. The chain says every phase leaves indicators (a sender, a hash, a domain, a persistence technique) and that a campaign's indicators repeat from one intrusion to the next, because the adversary reuses infrastructure and tools. Whoever maps campaigns — not incidents — can "move detection up the chain" and block delivery or exploitation before installation. The limits, voiced by critics too: the model is linear and malware-centred; an attack with stolen credentials or an insider abuse has no "weaponization"; the post-compromise phases are compressed into one, which is why MITRE ATT&CK (lesson 31) detailed exactly that part. It nevertheless remains the frame in which the right question is asked: not "what was this file", but "in which phase of whose campaign are we".
The intrusion kill chain is defined as reconnaissance, weaponization, delivery, exploitation, installation, command and control (C2), and actions on objectives.Hutchins, Cloppert, Amin, «Intelligence-Driven Computer Network Defense…» (Lockheed Martin, 2011; Proceedings of the 6th International Conference on Information Warfare and Security), secțiunea 3.2 «Intrusion Kill Chain»
Why it matters Whoever investigates incidents cleans up forever; whoever investigates campaigns stops the next intrusion at delivery.
In full →Collapse ↑
The paper's course-of-action matrix: for each phase, what you can do — detect, deny, disrupt, degrade, deceive, destroy. An example filled in for a document-phishing campaign, of the kind that reaches Romanian firms weekly:
PHASE DETECT DENY DISRUPT reconnaissance web logs (crawlers) — — weaponization — — — delivery the email filter block sender / domain quarantine exploitation endpoint sensor patches, macros disabled — installation EDR (persistence) — kill the process command/control proxy, DNS block the C2 domain cut the connection actions file-access audit — —The weaponization row is empty not by accident: it happens at the adversary, outside the defender's visibility. But its artefacts — a document built with the same tool, the same metadata — show up at delivery, and those are precisely what tie today's intrusion to the one two months ago.
Campaign analysis, in a table: two intrusions, the same indicators on different phases. This is the paper's "campaign mapping", reduced to its essence:
PHASE INTRUSION A (March) INTRUSION B (May) SHARED? reconnaissance — — — weaponization PDF with exploit, tool X PDF with exploit, tool X yes delivery sender "hr@…", IP 203.0.113.7 sender "hr@…", IP 203.0.113.9 partly (same /24) exploitation CVE-2026-1234 CVE-2026-1234 yes installation service "svchost_update" service "svchost_update" yes C2 dom-a.example, port 443 dom-b.example, port 443 partly (pattern) actions archiving accounting documents — (stopped at C2) —The analyst's conclusion: not "two incidents" but "one campaign, the same weaponization tool and the same persistence, rotated infrastructure". The defensive rules that follow are written on the shared indicators (the tool, the service, the IP block), not on the unique ones (today's C2 domain). Intrusion B was stopped at C2 precisely because A's indicators were already in the filters — that is "moving detection up the chain" in practice.
For the olivLaw reader: Cyber Daily applies exactly this grid every day to public threat reports; lesson 31 shows how ATT&CK gives a standard name to every technique in the columns above.
- 31
An intrusion event has four corners — adversary, capability, infrastructure, victim — linked by edges the analyst pivots along: from a domain to all its victims, from a tool to the adversary who uses it; ATT&CK then gives every behaviour a common name, so two teams can talk about the same technique.
The Diamond Model of Intrusion Analysis · Sergio Caltagirone, Andrew Pendergast, Christopher Betz · 2013
The Diamond Model formalises what intrusion analysts do intuitively: "an adversary deploys a capability over some infrastructure against a victim". The four core features are the vertices; the meta-features (time, phase, result, direction, methodology, resources) order events into activity threads. The authors say explicitly that at discovery most features are unknown and get filled in as facts emerge. The value is in pivoting: you know a C2 domain (infrastructure) → you look for which other victims talked to it → you find the shared capability → you group the events under one adversary. Every edge is a hypothesis to test, not a fact; the model forces you to say which edge you relied on. ATT&CK, published by MITRE since 2015 and described in the 2020 methodology paper, is complementary: "a curated knowledge base and model for cyber adversary behavior", organised into tactics (the short-term goal: initial access, persistence, lateral movement, exfiltration), techniques (how the goal is achieved) and procedures (a group's concrete implementation). It is explicitly "not an exhaustive enumeration of attack vectors" — it is a vocabulary. When a report says "T1053 — scheduled task", two organisations know they are talking about the same thing and can compare detection coverage. How the three frames connect: the chain (lesson 30) says which phase you are in, Diamond says who-with-what-through-what-against-whom, ATT&CK names the technique. The shared trap: attribution. An "adversary" corner filled in on the strength of a single reused piece of infrastructure is a low-confidence hypothesis, however convincing the group's name sounds.
For every intrusion event there exists an adversary taking a step towards an intended goal by using a capability over infrastructure against a victim to produce a result.Caltagirone, Pendergast, Betz, «The Diamond Model of Intrusion Analysis» (Center for Cyber Intelligence Analysis and Threat Research, 2013), secțiunea 4 «Diamond Event», Axioma 1
Why it matters A group name attached to an event on the strength of one edge is a story; on three verified edges, a hypothesis; only with repeated activity threads does it become an assessment.
In full →Collapse ↑
A Diamond event written as a card, then the pivots. The convention: every corner has a value, a source and a confidence; what is unknown stays "unknown", it is not guessed.
EVENT E1 (2026-05-12, phase: C2) adversary unknown capability tool X (hash abc…) source: EDR confidence: high infrastructure dom-b.example → 198.51.100.4 source: proxy confidence: high victim the accounting department of Alfa source: inventory confidence: high meta direction: victim → infrastructure · result: success · methodology: T1071 (web C2) PIVOT 1 infrastructure → victims: who else talked to 198.51.100.4? → Gama (2026-03) PIVOT 2 capability → events: where else did hash abc appear? → intrusion A (March) PIVOT 3 victim → pattern: what do Alfa and Gama have in common? → the same accounting vendorThree pivots turned one event into an activity thread with a victimology hypothesis (the shared vendor) — which is, in turn, a collection requirement, not a conclusion.
The same facts, named with ATT&CK so they can be compared with someone else's. Each row is a technique with its identifier; the right-hand column is what you have (or lack) as detection.
TACTIC TECHNIQUE (ID) SEEN IN A / B DETECTION initial access T1566.001 spearphishing attachment A, B email filter: yes execution T1204.002 user opens the file A, B — persistence T1543.003 Windows service A, B EDR: yes command and control T1071.001 web protocols A, B proxy: partial collection T1560 archive collected data A file audit: no exfiltration T1041 exfiltration over the C2 channel A —What the table says without naming a group: two detection gaps (collection, exfiltration) on techniques already used against you. That is the point of the vocabulary — coverage is measured on techniques, not on "groups". Attribution, if it comes, comes from the convergence of the Diamond corners (the same capability, the same victimology, the same procedures), and is written with the probability term from lesson 3.
- 32
An influence operation is analysed like an intrusion: not "is it false or true", but which tactic it pursues, with which techniques, through which accounts and channels — DISARM borrows ATT&CK's form so defenders name the same behaviour by the same name and can share it.
DISARM Disinformation TTP Framework · DISARM Foundation · 2024
The framework began in 2019 (under the name AMITT) from the observation that defenders against disinformation had no shared vocabulary, while information security had had one for years. DISARM is explicitly "part of work on adapting information security practices to help track and counter disinformation" and is built "in the style of the MITRE ATT&CK framework": a "red" framework (the disinformation creator's tactics and techniques, by phase — planning, preparation, execution, assessment) and a "blue" one (countermeasures, listed by the earliest phase where they help). There are STIX templates too, so incidents travel between organisations on the same channels as threat indicators. What that changes in analysis: the question is no longer whether a message is false, but which behaviour carries it. "Creating inauthentic accounts", "coordinated amplification", "hijacking a hashtag", "laundering a narrative through a seemingly independent site" are techniques with identifiers; a campaign is a chain of such techniques, and the olivLaw detector (the lesson on narrative coordination) measures exactly one of them — coordination — and separates it from syndication. François's assessment grid (actors, behaviour, content) points the same way: behaviour is easier to prove than intent, and content alone is the weakest signal. The framework also carries a warning rare in technical documents: the set of "blue" countermeasures was derived in 2019 by volunteers and includes tactics seen in countries with different values (censorship, for instance), and the foundation urges caution and publishes a companion guide. For an analyst that is a lesson in method: a catalogue describes what is done, not what is legitimate to do.
DISARM is a framework designed for describing and understanding disinformation incidents.DISARM Foundation, «DISARM Disinformation TTP (Tactics, Techniques and Procedures) Framework», README-ul depozitului oficial (DISARMFoundation/DISARMframeworks), primul paragraf
Why it matters Whoever judges only the content loses: the message can be true, the account fake, and the amplification bought.
In full →Collapse ↑
A fictional incident decomposed along the "red" framework's phases. The writing rule: one row per observed technique, with the evidence; what is suspected, not observed, is marked as a hypothesis.
PHASE TECHNIQUE (DISARM description) EVIDENCE STATUS planning target audience: the communes of one county — hypothesis preparation accounts created in a batch (12, same week) creation dates, profile photos observed preparation a "news" site with no newsroom, 3-month domain the domain registry, no imprint observed execution simultaneous posting of the same text (±4 min) shared 4-grams > 0.45 across 9 accounts observed execution pickup by two regional publications citing the site as the "source" observed assessment a second wave, reworded text, same accounts the same 12 accounts, 6 days later observedThe "status" column is what keeps the analysis honest: the target audience is inferred from geography, not documented; everything from "preparation" down is checkable by anyone with the same data. The "blue" countermeasures align to the earliest phase where they help: here, at preparation (flagging the batch of accounts), not at execution.
The ABC grid applied to the same incident, as a stress test of the conclusion. Three questions, in order of probative strength:
DIMENSION QUESTION ANSWER IN OUR CASE Actors who are they, and do they present as what they are? 12 new accounts, no history, generic names → inauthentic (probable) Behaviour is it coordinated, automated, paid? simultaneous posting + a second wave → coordinated (highly probable) Content is it false, misleading, manipulated? the text is an interpretation, not a checkable falsehood → inconclusiveThe correct conclusion is written on behaviour: "a coordinated amplification operation, carried by probably inauthentic accounts, around content that contains no checkable false claims". Not "disinformation" — the word would promise something the third column does not support. The probabilities are written with the terms from lesson 3, and what is missing (who paid, who wrote) becomes a collection requirement (lesson 26), not speculation. The technique holds for a legitimate campaign too: an NGO mobilising its members in a coordinated way will come out "coordinated" on the second row and "authentic" on the first — the grid is built to distinguish, not to condemn.
- 33
An analytic product is written for someone with three minutes and a decision: the judgement in the first sentence, with its probability term and confidence level, then the evidence with each item's source, then what would change the judgement — the reverse order from how the analyst thought.
Joint Publication 2-0, Joint Intelligence · US Joint Chiefs of Staff · 2013
Doctrine lists the attributes of a good product — anticipatory, timely, accurate, usable, complete, relevant, objective, available — and draws a rule of form from them: the decision-maker has no time to reconstruct the reasoning, so the conclusion goes first. The format is called BLUF (bottom line up front) and has a fixed structure: the first sentence is the judgement, with the probability term from lesson 3 and the confidence from lesson 4 ("we assess it is probable (55–70%) that X…; moderate confidence"); then the key evidence, each item with its source and quality (lessons 5–6); then the alternative hypotheses rejected and why; finally, the indicators that would change the judgement (lesson 11) and the collection gaps. The US community's sourcing standard, ICD 206, requires that every statement of fact be traceable to its source — not a bibliography at the end, but a link on every sentence that carries a figure or a quotation. olivLaw applies the same rule through the gate that refuses to publish a number without a source. The difference between a report and an essay is that in a report the reader can check any sentence without asking the author. Three recurring mistakes. The judgement buried at the end, after "context" — the reader gives up first. Mixed probability words ("possible, but probable") — lesson 3 forbids them. Confidence confused with probability — "high confidence that it is unlikely" is a correct sentence and must be writable. A simple test: if someone reads only the first sentence, do they make the same decision as after the whole report? If not, the first sentence is wrong.
They must be able to quickly apply intelligence to the task, and may not have sufficient time to analyze complex intelligence reports. Therefore the “bottom line” must be up front and understandableUS Joint Chiefs of Staff, JP 2-0 «Joint Intelligence» (22 octombrie 2013), capitolul II, secțiunea despre atributele produsului de intelligence (II-7)
Why it matters The report the decision-maker did not read as far as the judgement was written for the analyst, not for the decision.
In full →Collapse ↑
A complete BLUF report on the requirement from lesson 26, as it would look on one page. Every figure has its source in brackets; the probability terms are the calibrated ones.
JUDGEMENT We assess it is unlikely (20–35%) that Beta launches an SME offer in Romania before 31 March 2027, and probable (55–70%) that it does so by 30 June. Moderate confidence: consistent evidence, but from open sources, with no access to Beta's plans. EVIDENCE · trademark "Beta Contab" filed at EUIPO on 14 Oct 2026 (EUIPO register, A1) · 3 RO sales positions posted between 20 Oct and 1 Dec (Beta careers page, A1) · withdrawal from Hungary announced on 18 Nov (regional press, B2) — suggests reallocation · no product/support position in RO as of 10 Jan (same channel, observed absence) ALTERNATIVES "enters via partner": no evidence of a deal (4 accountants' networks checked) — provisionally rejected "false signal / defensive trademark": refuted by the positions that followed the trademark WHAT WOULD a Romanian-language support position or an RO demo → we move to "probable" before 31 March CHANGE IT an announced exit from small markets including RO → we move to "unlikely" for 30 June too GAPS Beta's internal plans; the state of integration with a local ERP (open collection requirement)The test: someone who reads only the first two lines takes the decision "we postpone the price cut, review on 15 January" — the same one they would take after the whole page.
What olivLaw's publish gate does with such a text, and why it is the same rule as ICD 206. Every number in the body must be found in the linked sources (absolute tolerance 0.051 or relative 0.6%), every citation must exist, and the probability vocabulary is closed. An excerpt from the checklist, as it runs before "publish":
CHECK PASSES? WHAT HAPPENS IF NOT every figure appears in a linked source yes publication is refused (Q7) every sourced statement carries its link per sentence yes — the probability term is from the calibrated list yes the term is recomputed in code, not chosen by the model "there is no evidence that…" only if sources are silent yes gate Q10 refuses a falsely declared absence the conclusion is in the first sentence ("In brief") yes gate Q6 requires the headerThe link to doctrine is direct: "usable" and "objective" from the JP 2-0 list are not adjectives but checks. A report that passes them can be contradicted by a reader on a specific sentence — and that is exactly the definition of an analytic product, as opposed to an opinion.