‹ Shelves / Intelligence analysis ■ Superforecasting — calibrated forecasting step by step (2026)From the question that can be lost to the Brier score, the outside view, Bayes, identifying and cultivating superforecasters, aggregation, calibrated intervals and AI models — with worked examples, implementation in a forecasting engine, a quiz and an audioguide in every lesson.
25 lessons · 61 min
The reading track 25 lessons All With quiz · 25 All authors Philip E. Tetlock · 2 Glenn W. Brier · 2 Barbara Mellers, Lyle Ungar, Jonathan Baron, Philip E. Tetlock et al. · 2 Barbara Mellers, Eric Stone, Philip E. Tetlock et al. · 2
01 A forecast can be measured only if its question can be lost: a precise event, a deadline, a resolution source and a criterion a stranger can apply without asking what you meant. Expert Political Judgment: How Good Is It? How Can We Know? · Philip E. Tetlock · 2005 ✓ Quiz · 3 questions › 02 A word like ‘likely’ means very different things to different people; write the number, and let the word be computed from the number, never the other way round. Words of Estimative Probability · Sherman Kent · 1964 ✓ Quiz · 3 questions › 03 The Brier score is the mean of the squared gaps between probability and outcome: 0 is perfect, and ‘always 50%’ scores 0.5 in the original two-sided form and 0.25 in the one-sided form olivLaw uses. Verification of Forecasts Expressed in Terms of Probability · Glenn W. Brier · 1950 ✓ Quiz · 3 questions › 04 The Brier score splits into three parts — reliability, resolution and uncertainty — and being calibrated is not enough: whoever always states the average frequency is perfectly calibrated and still says nothing. A New Vector Partition of the Probability Score · Allan H. Murphy · 1973 ✓ Quiz · 3 questions › 05 A strictly proper scoring rule makes stating exactly what you believe the only optimal strategy; Brier and the log score are such rules, hit rate and absolute error are not. Strictly Proper Scoring Rules, Prediction, and Estimation · Tilmann Gneiting & Adrian E. Raftery · 2007 ✓ Quiz · 3 questions › 06 Rounding probabilities to a few words or bands costs measurable accuracy, and the cost grows the better the forecaster is; keep the number and score the number. The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament · Jeffrey A. Friedman, Joshua D. Baker, Barbara A. Mellers, Philip E. Tetlock & Richard Zeckhauser · 2018 ✓ Quiz · 3 questions › 07 Start every forecast with the question ‘how often does this happen in situations of this kind?’: the base rate of a reference class is the starting point, and the details of the case only adjust it. Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking · Daniel Kahneman & Dan Lovallo · 1993 ✓ Quiz · 3 questions › 08 A question that looks impossible becomes estimable once you break it into steps you can approximate; but if every step must happen, their probabilities multiply, and the total is lower than the weakest step. Judgment under Uncertainty: Heuristics and Biases · Amos Tversky & Daniel Kahneman · 1974 ✓ Quiz · 3 questions › 09 Each piece of evidence multiplies the forecast's odds by its likelihood ratio; superforecasters do this often and in small steps, not rarely and in leaps. Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · Barbara Mellers, Philip E. Tetlock et al. · 2015 ✓ Quiz · 3 questions › 10 What an expert believes matters less than how they think: ‘foxes’, who combine several explanations and treat their beliefs as hypotheses, forecast better than the ‘hedgehogs’ of one big theory. Expert Political Judgment: How Good Is It? How Can We Know? · Philip E. Tetlock · 2005 ✓ Quiz · 3 questions › 11 A forecast's error has three sources — bias, insufficient information and noise —, and in the IARPA tournaments training, teaming and elite selection gained accuracy mostly by cutting noise. Bias, Information, Noise: The BIN Model of Forecasting · Ville A. Satopää, Marat Salikhov, Philip E. Tetlock & Barbara Mellers · 2021 ✓ Quiz · 3 questions › 12 Tetlock and Gardner's ten commandments are, nearly all of them, balances between two opposing errors; turned into required fields, they become a checklist that a person and an AI model can both follow. Superforecasting: The Art and Science of Prediction · Philip E. Tetlock & Dan Gardner · 2015 ✓ Quiz · 3 questions › 13 Superforecasters were not discovered by a guru but by an experiment: IARPA had five university teams answer the same questions under the same score, and the Good Judgment Project won by combining training, teams, grouping the best together and statistical aggregation. Psychological Strategies for Winning a Geopolitical Forecasting Tournament · Barbara Mellers, Lyle Ungar, Jonathan Baron, Philip E. Tetlock et al. · 2014 ✓ Quiz · 3 questions › 14 Superforecasters were picked on score alone, and the real test came afterwards: instead of regressing to the mean, as the lucky do, they got even better — a sign that their accuracy is partly discovered (intelligence, knowledge) and partly built (effort, frequent updating, teams). Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · Barbara Mellers, Eric Stone, Philip E. Tetlock et al. · 2015 ✓ Quiz · 3 questions › 15 One hour of probability training — comparison classes, averaging several estimates, simple models, predictable biases — cut the Brier score by 6–11% against a control group, and the effect lasted for months; the figure is credible precisely because it was measured by random assignment. Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments · Welton Chang, Eva Chen, Barbara Mellers, Philip E. Tetlock · 2016 ✓ Quiz · 3 questions › 16 GJP teams beat forecasters working alone, and the elite teams were a "steroid injection": not because they agreed quickly, but because they asked each other questions, answered them and shared evidence until disagreement melted away on the strength of information. Psychological Strategies for Winning a Geopolitical Forecasting Tournament · Barbara Mellers, Lyle Ungar, Jonathan Baron, Philip E. Tetlock et al. · 2014 ✓ Quiz · 3 questions › 17 An in-house superforecasting programme has five parts: a pipeline of scorable questions, a tournament with fixed rules, a leaderboard with a confidence interval, a minimum number of questions before any selection, and a feedback loop — and each part must be defended against people who learn to play the score instead of reality. Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · Barbara Mellers, Eric Stone, Philip E. Tetlock et al. · 2015 ✓ Quiz · 3 questions › 18 AI models and agents are selected exactly like people: on future questions with no known answer, with a scored track record per source, compared with a naive baseline through the Brier skill score — and whoever fails to beat the baseline on a sufficient record leaves the ensemble, however sophisticated. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities · Ezra Karger, Philip E. Tetlock et al. · 2025 ✓ Quiz · 3 questions › 19 A crowd of independent estimates, combined by a simple rule, usually beats its typical member; the mean, the median and the track-record-weighted mean are three different rules, and choosing between them is a decision, not a detail. Vox Populi · Francis Galton · 1907 ✓ Quiz · 3 questions › 20 The average of calibrated forecasts comes out too close to 50 percent; extremizing pushes the aggregate towards 0 or 1 in log-odds, but it is justified only when forecasters hold different information, and its strength is learned from resolved questions. Modeling Probability Forecasts via Information Diversity · Ville A. Satopää, Robin Pemantle, Lyle H. Ungar · 2014 ✓ Quiz · 3 questions › 21 A market aggregates through price, a prediction poll through an algorithm; in the ACE tournament the market beat the simple mean of the polls, but team polls aggregated with temporal decay, track-record weights and recalibration beat the market. Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls · Pavel Atanasov, Phillip Rescober, Eric Stone, Samuel Swift, Emile Servan-Schreiber, Philip Tetlock, Lyle Ungar, Barbara Mellers · 2017 ✓ Quiz · 3 questions › 22 A quantitative forecast is a distribution, not a number: first it must be calibrated — the 80 percent interval catches reality 80 percent of the time — and only then as narrow as possible. Probabilistic forecasts, calibration and sharpness · Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery · 2007 ✓ Quiz · 3 questions › 23 Scenarios must be mutually exclusive and cover everything, tails must be modelled as tails rather than as bigger Gaussian noise, and any probability a model produces is recalibrated on data held out in chronological order. A Tutorial on Conformal Prediction · Glenn Shafer, Vladimir Vovk · 2008 ✓ Quiz · 3 questions › 24 In 2026 the best AI forecasting systems, with task-built scaffolding, reached a statistical tie with the superforecaster median on ForecastBench, but models used out of the box remain clearly behind; the best result comes from combining people with models and from scoring each only on questions created after its knowledge cutoff. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy · Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, Philip E. Tetlock · 2024 ✓ Quiz · 3 questions › 25 A forecasting engine improves only if the loop closes: scorable questions, numerical probabilities with a source, resolution, score, and the score flowing back into weights, recalibration and the retirement of weak models; the order matters, because without measurement no other step can be checked. Verification of forecasts expressed in terms of probability · Glenn W. Brier · 1950 ✓ Quiz · 3 questions ›