Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery

1 card

1. 01

   A quantitative forecast is a distribution, not a number: first it must be calibrated — the 80 percent interval catches reality 80 percent of the time — and only then as narrow as possible.

   Probabilistic forecasts, calibration and sharpness · 2007

   Calibration first, then sharpness: count how often your intervals catch reality.

   For a continuous quantity, such as the EUR/RON rate in three months, an honest forecast gives an interval or a distribution. Gneiting, Balabdaoui and Raftery (2007) set the rule: maximise sharpness (narrow intervals) subject to calibration. A narrow interval that misses often is worse than a wide one that tells the truth about uncertainty. Three tools. Coverage: of all the 80 percent intervals issued, what fraction caught the realised value? The test uses the binomial distribution, not eyeballing. The PIT histogram: for each forecast, compute at which quantile of the forecast distribution the outcome fell; if the forecasts are ideal the histogram is flat, and a U shape says "intervals too narrow". The authors warn that a uniform PIT is necessary but not sufficient. The scores: the pinball loss for one quantile, the interval score of Gneiting and Raftery (JASA 2007, equation 43) — width plus a penalty of 2/α times the size of the miss — and the CRPS, the integral of Brier scores over all thresholds. In the olivLaw engine, the honest path for the five headline assets stores 80 and 95 percent intervals in `market_forecast_point`, but nothing ever checks how often reality falls inside them. The ensemble in forecast_service averages the models' interval bounds instead of combining their distributions, which understates uncertainty exactly when the models disagree.

   “We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration.”

   Open the card →