Jeffrey A. Friedman, Joshua D. Baker, Barbara A. Mellers, Philip E. Tetlock & Richard Zeckhauser

1 card

1. 01

   Rounding probabilities to a few words or bands costs measurable accuracy, and the cost grows the better the forecaster is; keep the number and score the number.

   The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament · 2018

   Score the number, not the word, because rounding costs the best forecasters the most.

   Does the difference between 70% and 75% matter? Friedman, Baker, Mellers, Tetlock and Zeckhauser tested the question on 888,328 forecasts across 380 questions from the Good Judgment Project tournament. They took the numerical probabilities, rounded them artificially to ever coarser scales and recomputed the Brier score. The conclusion in their abstract: coarsening estimates in the manner of qualitative expressions, including those recommended to analysts, consistently sacrifices accuracy. The size of the cost depends on the forecaster. For all participants, rounding to seven equal bins of estimative words raised the mean Brier score by 1.9%; for superforecasters, by 40.4%. With three confidence levels the increases were 11.8% and 236.1%, according to table 1 of the 2017 preprint. The more finely you know, the more you lose when forced to speak coarsely. The practical rule for a forecasting engine has three parts. Store the point probability, not the label. Score the point probability, not the band's midpoint. Show the word only as a translation, next to the number (lesson 2). The point_probability field in the forecast_resolution table does exactly that; the 26 old forecasts that carried only words must not be ‘repaired’ by assigning them the midpoint of a band.

   “We find that coarsening numeric probability assessments in a manner consistent with common qualitative expressions—including expressions currently recommended for use by intelligence analysts—consistently sacrifices predictive accuracy.”

   Open the card →