“This worldview predisposes superforecasters to treat their beliefs more as testable hypotheses and less as sacred possessions”Barbara Mellers, Philip E. Tetlock et al. · Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015 · Mellers, Tetlock et al., «Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions», Perspectives on Psychological Science (2015), p. 273, secțiunea despre stilurile cognitive ale superprognozatorilor
Superforecasters were found by their score and grown through training, teams and practice, and the best behavioural predictor was how often they updated their forecasts.
Between 2011 and 2015 IARPA, the research agency of the US intelligence community, ran the ACE tournament: hundreds of geopolitical questions with a verifiable answer and a fixed deadline, on which participants gave probabilities they could revise at any time. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, won. The yardstick was the Brier score: the sum of squared differences between the probabilities given and what happened (1 if the event occurred, 0 if not), on a scale from 0 (perfect) to 2; always saying 50% earns 0.5. The 2015 article describes how they were identified and cultivated. At the end of the first year 60 people were selected — the top five from each of the 12 experimental conditions — and placed in five elite teams of 12. The statistical expectation was regression to the mean; it did not happen two years running: the superforecasters became more accurate while the comparison groups regressed. What set them apart: fluid intelligence (the strongest dispositional predictor), political knowledge, actively open-minded thinking and, above all, behaviour — the frequency of updating forecasts was the strongest behavioural predictor of accuracy. Cultivation had three experimentally tested levers: a probability-training module of about 45 minutes, teamwork, and grouping the best together. The method transfers to any team that judges what will happen: questions with a deadline and a criterion, numerical probabilities updated often, a score at resolution, selection by track record rather than rank. The «Superforecasting» course on the same shelf takes it apart step by step.
What the 2015 article actually measured (Mellers, Tetlock et al., Perspectives on Psychological Science 10(3)). Table 1 compares, averaged over years 2 and 3, the superforecasters with individuals on the top teams and with the other 1,498 participants who answered at least 25 questions:
SUPERS TOP TEAMS ALL OTHERS
Standardised Brier -0.34 -0.14 0.04 (lower = better)
Resolution 0.40 0.35 0.32 (higher = better)
Calibration (gap) 0.01 0.03 0.04 (lower = better)
AUC 96% 84% 75%Effort shows just as clearly: in the first year superforecasters made on average 2.77 forecasts per question against 1.47 for the comparison groups; in years 2–3 they opened 255 stories from the platform's news feed against 55 and 58. They attempted 25% more questions in the first year and about 40% more afterwards. The article offers four explanations that add up: cognitive abilities and style, task-specific skills, motivation and commitment, and an enriched environment (the elite teams). The often-quoted "top 2%" comes from the same group's 2014 article, not from this one.
How to apply this in a newsroom or a forecasting engine. Step 1: every published forecast gets a code, a numerical probability, a deadline and a resolution source — otherwise it cannot be scored. Step 2: at the deadline the outcome is recorded (1 or 0) and the score computed; in the olivLaw engine the resolutions table uses the binary form (p − outcome)², on a 0–1 scale where 50% gives 0.25 — half the score of the tournament's two-outcome form. Step 3: after at least a few dozen resolved questions per author (person, agent or model), compare the scores with a simple benchmark (say, "always the historical frequency") and only then talk about talent. Step 4: the best work together and see each other's forecasts; the rest take the training module. The course "Superforecasting — calibrated forecasting step by step" walks through each step, with worked examples, code and an audioguide.
One last piece remains: how do you decide, at the deadline, whether a prediction came true? The answer lies in how the question is worded. It must be written so that a stranger, with no extra explanation, could say yes or no. "Relations will get tense" fails; "by March, country X will have formally announced its withdrawal from the agreement" works. A couple arguing about "spending too much" can fix a number and a date, and the dispute becomes checkable. The trap is the vague question that can never be lost: without a deadline and a criterion there is no score, and therefore no learning. Once good questions and scores pile up, the natural next question arises: what actually separates the people who score well?
Why it matters Forecasting talent only shows in a score kept over many questions — and an organisation that keeps no score picks its experts by how convincingly they talk.