Author
Barbara Mellers, Philip E. Tetlock et al.
1 reading card from 1 book · 2015.
1 card
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015
A forecast with no deadline and no resolution criterion cannot be lost, so it cannot teach you anything.
Between 2011 and 2015 IARPA, the research agency of the US intelligence community, ran geopolitical forecasting tournaments: hundreds of questions with a verifiable answer and a fixed deadline, on which participants gave probabilities they could revise at any time. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, won, and the book Superforecasting (Tetlock and Gardner, 2015) tells how. The yardstick was the Brier score: the sum of squared differences between the probability given and what happened (1 if the event occurred, 0 if not). Zero is perfect; always saying 50% earns 0.5 on a two-outcome question. The method has four steps. Phrase the question so that a stranger could decide at the deadline whether it happened; give a number, not a maybe; update often, in small steps, as evidence arrives; at resolution, record the score and check calibration — of the claims made at 70%, did roughly 70% come true? It pays off for any judgement that recurs. The trap is choosing vague questions that can never be lost: with no deadline and no criterion there is no score, and so no learning.
“This worldview predisposes superforecasters to treat their beliefs more as testable hypotheses and less as sacred possessions”