Book
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
by Barbara Mellers, Eric Stone, Philip E. Tetlock et al. · 2015 · 2 reading cards
2 cards
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015
Do not select anyone until the confidence interval of the score difference excludes zero, and that usually takes dozens of common questions.
The GJP offers a recipe to copy. Questions have a deadline and a clear criterion and are chosen by someone other than the players. The score is Brier, computed daily on the standing forecast and averaged over days and questions (Mellers et al. 2015), so that a good forecast made early is worth more than a correct one made the day before the deadline. Because everyone chose their own questions, scores were standardized per question. The comparison groups included only people with at least 25 questions per tournament, and the leaderboard showed the top 10% in each condition. The reward for accuracy was status, not money. Luck or skill? The difference between two analysts over 30 questions may be noise. If per-question score differences have a standard deviation of 0.08 (one-sided form), the standard error of the mean is 0.08 / √30 ≈ 0.015, so a 0.02 advantage does not escape the 95% interval. To tell it apart you need about 60 common questions. Selection comes after the second season, not the first. Goodhart's law bites here: a measure that becomes a target stops measuring. Someone scored on hit rate will say 51% instead of 90%; someone who picks their questions takes the easy ones; someone who sees the team average copies it. The proper rule, standardized scores, a mandatory question set and separately written forecasts are the defences.
“Although governments invest massive sums to predict the actions of political actors and the consequences of those actions, they invest astonishingly little to evaluate the effectiveness of their favorite methods of generating predictions.”
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015
Luck regresses to the mean, skill does not; the test of a superforecaster is the second season, not the first.
At the end of year one the GJP picked 60 forecasters — the top 5 from each of the 12 conditions — and put them in 5 elite teams of 12; after year two it picked another 60, on accuracy alone (Mellers et al. 2015). Mellers et al. 2014 and Friedman et al. 2018 describe the cut as "the top 2%". Had their success been luck, they should have regressed to the mean. They did not: in years 2 and 3 their standardized Brier score was −0.34, against −0.14 for the next-best group and 0.04 for the other 1,498 (lower is better), and the area under the ROC curve was 96%, 84% and 75%. The comparison groups regressed; the superforecasters improved. What set them apart? The authors offer four explanations. Abilities: at least one standard deviation above the general population in fluid intelligence, likewise in crystallized intelligence and even more in political knowledge. Style: more actively open-minded thinking and need for cognition, though these effects were "markedly weaker". Effort: in year one, 2.77 forecasts per question against 1.47; updating frequency was the strongest behavioural predictor. Environment: elite teams that shared their news. Their conclusion: superforecasters are "partly discovered and partly created". Selection without the environment would have wasted the talent.
“Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics.”