Author
Barbara Mellers, Lyle Ungar, Jonathan Baron, Philip E. Tetlock et al.
2 reading cards from 1 book · 2014.
2 cards
Psychological Strategies for Winning a Geopolitical Forecasting Tournament · 2014
A good team does not avoid disagreement; it turns it into precise questions until evidence, not pressure, brings consensus.
Theory said independent forecasters were safer: their uncorrelated errors cancel out in the average, while teams risk groupthink, social loafing and information cascades. The GJP data contradicted it. In year one, teams were more accurate than those who only saw the crowd's distribution, who in turn beat those working alone; a member's number of comments correlated with accuracy, r = −0.19 and −0.22 in the first two years (Mellers et al. 2014; negative means a lower, so better, Brier score). Teams also received instruction: high standards of proof, explaining forecasts, constructive critique. In the elite teams the effect grew. Superforecasters wrote about 5 times as many question comments as the next-best group, posted 36 and 44 forum messages against 2 and 5, and shared more than 10 times as many links in year 2 (Mellers et al. 2015). They also asked more questions — one question mark per 213 words, against one per 526 — and got replies to 23% of them, against 4%. The spread of their forecasts narrowed over a question's life, while for the others it widened. Tetlock and Gardner (2015) name three skills: reconstructing the other side's argument, precision questioning and constructive confrontation — disagreement without hostility.
“Team forecasters received additional training in how to help the group be more accurate by maintaining high standards of evidence and proof. They were taught strategies for explaining their forecasts to others, offering constructive critiques, and building an effective team.”
Psychological Strategies for Winning a Geopolitical Forecasting Tournament · 2014
A forecasting tournament turns a dispute about methods into a measurement: the same questions, the same deadline, the same score.
From 2011, IARPA, the research agency of the US intelligence community, ran the ACE programme: five university-based research groups recruited forecasters, elicited their probabilities and aggregated them (Mellers et al. 2014). Every day each group submitted its aggregate forecasts, all scored with the same rule, the Brier score in its original two-sided form, from 0 (perfect) to 2. The questions were chosen by IARPA, not by the researchers: just over 100 per season in the first two years, about 150 in the third, open for 102 days on average (Mellers et al. 2015). The Good Judgment Project (GJP) treated the tournament as an experiment. In year one, participants were randomly assigned to 3 kinds of training × 4 ways of working (alone, alone but seeing the crowd's distribution, in a team, in a prediction market) — 12 conditions. It won with four levers: probability training, teams, grouping the best forecasters into elite teams, and an algorithm that weighted forecasters by track record, gave less weight to old forecasts and "extremized" the average. The authors themselves warn that a strategy that "won" is not necessarily optimal. The figure that circulates — superforecasters "about 30 percent better" than analysts with access to intercepts — comes from a David Ignatius column (Washington Post, 2013): a report of a comparison never published in peer-reviewed form. The safe lesson is another: a tournament makes methods comparable.
“Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament.”