Bias, Information, Noise: The BIN Model of Forecasting

1 card

1. 01

   A forecast's error has three sources — bias, insufficient information and noise —, and in the IARPA tournaments training, teaming and elite selection gained accuracy mostly by cutting noise.

   Bias, Information, Noise: The BIN Model of Forecasting · 2021

   A number that never moves is not free of noise but free of information.

   Kahneman, Sibony and Sunstein (Noise, 2021) separate two kinds of error. Bias is systematic error: everyone is wrong in the same direction. Noise is unwanted variability: judgments that should be identical come out different, from one analyst to the next or from one day to the next. The BIN model of Satopää, Salikhov, Tetlock and Mellers (Management Science, 2021) applies the distinction to forecasts and adds a third component, information: how much of the useful signal available the forecaster actually uses. The published abstract says that noise reduction plays a surprisingly consistent role across all three interventions of the Good Judgment Project: training, teaming and tracking of talent. The paper's working version also gives the order of magnitude: eliminating noise would have cut the control group's Brier score by roughly 50%, eliminating bias by roughly 25%, and information would account for the rest. Even training designed against biases worked mostly by reducing noise. For a single forecaster there is a useful identity: the mean Brier score of a set of forecasts equals the score of their average plus their variance. Noise is paid for directly, and the average of several independent judgments cuts it. Beware, though, of the reverse confusion: a number that never moves is not free of noise but free of information. The Seldon score in the olivLaw engine sat between 46.5 and 47.0 across 496 runs, for four months.

   “The BIN model reveals that noise reduction plays a surprisingly consistent role across all three methods of enhancing performance.”

   Open the card →