A New Vector Partition of the Probability Score

1 card

1. 01

   The Brier score splits into three parts — reliability, resolution and uncertainty — and being calibrated is not enough: whoever always states the average frequency is perfectly calibrated and still says nothing.

   A New Vector Partition of the Probability Score · 1973

   A perfectly calibrated forecaster can be useless if they always state the average; the value comes from resolution.

   In 1973 Allan Murphy showed that the Brier score can be split into three terms: Brier = reliability − resolution + uncertainty. Uncertainty belongs to the world, not the forecaster: if events happen at rate ō, it equals ō(1 − ō) and cannot be changed. Reliability (calibration) measures how far your probabilities are from the observed frequencies: of the things you said at 70%, did about 70% happen? It is added to the score, so smaller is better. Resolution measures how far the frequencies of your groups move away from the overall average: can you separate the cases that will happen from those that will not? It is subtracted, so larger is better. The classic example is the ‘climatological’ forecaster. If it rains on 30% of days and you say 30% every day, you are perfectly calibrated — reliability is zero — but resolution is zero too, and your score is exactly the uncertainty. Brier was already using this benchmark in 1950. A good forecaster must be calibrated and have resolution. In table 1 of Mellers and colleagues (2015), superforecasters had a resolution of 0.40 against 0.32 for everyone else, and a calibration of 0.01 against 0.04. The visual tool is the reliability diagram: the forecast probability on the horizontal axis, the observed frequency on the vertical, in bands. Points on the diagonal mean calibration. In olivLaw, the public forecasts page draws five bands of 20 points each and shows a band's frequency only from five outcomes upwards (BAND_MIN_N = 5).

   “The new partition consists of three terms: 1) a measure of the uncertainty inherent in the events, or states, on the occasions of concern (namely, the PS for the sample relative frequencies); 2) a measure of the reliability of the forecasts; and 3) a new measure of the resolution of the forecasts.”

   Open the card →