Author

Welton Chang, Eva Chen, Barbara Mellers, Philip E. Tetlock

1 reading card from 1 book · 2016.

ShelvesSuperforecasting — calibrated forecasting step by step (2026)BooksDeveloping expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments 2016

1 card

01
One hour of probability training — comparison classes, averaging several estimates, simple models, predictable biases — cut the Brier score by 6–11% against a control group, and the effect lasted for months; the figure is credible precisely because it was measured by random assignment.
Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments · 2016
One hour of probability training cut the Brier score by 6–11% against control, and that number exists only because someone kept a control group.
In year one the GJP tested two modules of about 45 minutes: probability training and scenario training. Both beat the untrained group, and probability training beat scenario training (Mellers et al. 2014); the benefit lasted two seasons, each 8–10 months long. The probability module was summed up by the acronym CHAMP: comparison classes, historical trends, averaging several estimates, mathematical models and predictable biases. From year three it became CHAMPS KNOW: S for selecting questions and effort, plus four political-reasoning rules — who the players are and what they want, institutional norms, other perspectives, wildcards (Chang et al. 2016). How much did it matter? The abstract of Chang et al. gives a 6–11% better Brier score than control; year by year, the text reports 10–11% in the first, 12% in the second, 6% in the third and 7% in the fourth. Cognitive ability and practice contributed separately, with no significant interactions, and the authors consider the effects a lower bound of what a longer training would give. The part most often forgotten is the design: participants were randomly assigned to training or control, on the same questions. Without that, "we forecast better after the course" may only mean the year was easier.
“Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition.”
Open the card →

One hour of probability training — comparison classes, averaging several estimates, simple models, predictable biases — cut the Brier score by 6–11% against a control group, and the effect lasted for months; the figure is credible precisely because it was measured by random assignment.