Book
Modeling Probability Forecasts via Information Diversity
by Ville A. Satopää, Robin Pemantle, Lyle H. Ungar · 2014 · 1 reading card
1 card
Modeling Probability Forecasts via Information Diversity · 2014
Extremize in log-odds only across sources with different information, and learn the exponent from resolved questions.
Baron, Mellers, Tetlock, Stone and Ungar (Decision Analysis, 2014) give two reasons why an aggregate should be made more extreme. First: the probability scale is compressed at its ends, so random noise pushes the average towards 0.5. Second: each forecaster knows they see only part of the total information and stays cautious; put them together and the group knows more than any of them, but the average keeps each one's caution. The general form used in the Good Judgment literature: turn each probability into log-odds, ln(p/(1−p)), take the mean, multiply it by an exponent a and map back to a probability. With a = 1 you get the geometric mean of the odds; with a above 1 the aggregate moves away from 0.5. Satopää and colleagues (International Journal of Forecasting, 2014) derived a one-parameter logit aggregator of this kind; the course quotes no fitted value of a, because we did not see it in the primary text. The value is learned on your own data. When NOT to extremize: when forecasters share the same information. Satopää, Pemantle and Ungar show that the optimal aggregator moves smoothly from averaging, when information overlaps completely, to voting, when it does not overlap at all; in their example the optimal aggregate rises from 2/3 to 4/5 only because the overlap drops. Five runs of the same model on the same news bundle are not five independent sources.
“In fact, the averaging aggregators are uncalibrated and under-confident, i.e., too close to 1/2, even if the individual forecasts are calibrated (Ranjan and Gneiting, 2010).”