Book
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
by Barbara Mellers, Philip E. Tetlock et al. · 2015 · 2 reading cards
2 cards
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015
Superforecasters were found by their score and grown through training, teams and practice, and the best behavioural predictor was how often they updated their forecasts.
Between 2011 and 2015 IARPA, the research agency of the US intelligence community, ran the ACE tournament: hundreds of geopolitical questions with a verifiable answer and a fixed deadline, on which participants gave probabilities they could revise at any time. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, won. The yardstick was the Brier score: the sum of squared differences between the probabilities given and what happened (1 if the event occurred, 0 if not), on a scale from 0 (perfect) to 2; always saying 50% earns 0.5. The 2015 article describes how they were identified and cultivated. At the end of the first year 60 people were selected — the top five from each of the 12 experimental conditions — and placed in five elite teams of 12. The statistical expectation was regression to the mean; it did not happen two years running: the superforecasters became more accurate while the comparison groups regressed. What set them apart: fluid intelligence (the strongest dispositional predictor), political knowledge, actively open-minded thinking and, above all, behaviour — the frequency of updating forecasts was the strongest behavioural predictor of accuracy. Cultivation had three experimentally tested levers: a probability-training module of about 45 minutes, teamwork, and grouping the best together. The method transfers to any team that judges what will happen: questions with a deadline and a criterion, numerical probabilities updated often, a score at resolution, selection by track record rather than rank. The «Superforecasting» course on the same shelf takes it apart step by step.
“This worldview predisposes superforecasters to treat their beliefs more as testable hypotheses and less as sacred possessions”
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015
The new odds are the old odds multiplied by the evidence's likelihood ratio.
Bayes' theorem has a form you can do in your head: the odds after the evidence equal the odds before, multiplied by the likelihood ratio. Odds are p / (1 − p): 0.32 becomes 0.47 to 1. The likelihood ratio answers a single question — how many times more likely am I to see this evidence if the hypothesis is true than if it is false? A ratio of 2 doubles the odds, 0.5 halves them, and 1 moves nothing, however dramatic the news. In logarithms, evidence adds up: every ratio of 2 adds 0.69 to the log-odds. Mellers and colleagues (2015, p. 277) found that frequency of updating was the strongest behavioural predictor of accuracy. In the tournament's first year, superforecasters made 2.77 forecasts per question on average, against 1.47 in the comparison groups. A later study by Atanasov and colleagues (2020) showed, on the same data, that people who make small, frequent updates are more accurate than those who make rare, large revisions. Two traps. The first: the same information counted twice, because it appeared in three outlets that cite each other. The second: confusing how surprising a piece of news is with how diagnostic it is. The analysis of competing hypotheses in the olivLaw engine sums scores from −2 to +2 and normalises them; it has neither a prior nor likelihood ratios, so it is not Bayesian.
“Frequency of belief updating was important; it turned out to be the strongest single behavioral predictor of accuracy”