Verification of Forecasts Expressed in Terms of Probability

2 cards

1. 01

   A forecasting engine improves only if the loop closes: scorable questions, numerical probabilities with a source, resolution, score, and the score flowing back into weights, recalibration and the retirement of weak models; the order matters, because without measurement no other step can be checked.

   Verification of forecasts expressed in terms of probability · 1950

   Measure first, improve second, and always feed the score back into the engine.

   The course comes down to one idea: a forecast that does not come back as a score teaches nobody anything. In 1950 Brier asked that the verification scheme not push the forecaster in any undesirable direction; strictly proper scores (lesson 5) do exactly that, and the rest of the course is about what you do with the score. The olivLaw engine already has good parts: the resolution table with a one-sided Brier, the public score page, the honest market path that chose a drifting random walk after Prophet lost to the naive baseline (17.6 percent MAPE against 3.0 percent on gold), the Platt-calibrated I&W model. What is missing is the links. 26 of 56 old forecasts have no probability and cannot be scored. The Seldon score sat between 46.5 and 47.0 over 496 runs, and the regime-shift detector has never written a row. Nobody knows which source is better, because resolution does not know who forecast. The roadmap has eight steps, in the order in which each makes the next one checkable: scorable questions, a source on every forecast, the base rate, the Seldon threads in the resolution table, ACH on likelihood ratios, learned aggregation, interval coverage and, last, the recalibration loop. Each step has a probe in system_health and a success measure: the Brier skill score against climatology, the coverage of 80 percent intervals, the number of regime-shift events caught.

   “one essential criterion for satisfactory verification is that the verification scheme should influence the forecaster in no undesirable way.”

   Open the card →

2. 02

   The Brier score is the mean of the squared gaps between probability and outcome: 0 is perfect, and ‘always 50%’ scores 0.5 in the original two-sided form and 0.25 in the one-sided form olivLaw uses.

   Verification of Forecasts Expressed in Terms of Probability · 1950

   Before comparing two Brier scores, find out whether they are on the 0-to-2 scale or the 0-to-1 scale.

   In 1950 the meteorologist Glenn Brier proposed a simple rule for grading forecasts expressed as probabilities. For each forecast and each possible outcome, subtract from the probability given 1 if that outcome happened and 0 if it did not, square the difference and add up; then average over all forecasts. Brier says it himself: the minimum is zero, for a perfect forecast, the maximum is 2. The Good Judgment Project tournaments used the same 0-to-2 scale. On a two-outcome question, Brier's form counts the error twice, once for ‘yes’ and once for ‘no’. That is why many systems use the one-sided form, (p − o)², with values between 0 and 1. On binary questions the two differ by exactly a factor of 2. Someone who always says 50% scores 0.5 in the two-sided form and 0.25 in the one-sided form. When you read a Brier score, the first question is: which form? Brier's form also works for more than two outcomes. If you give 20%, 70% and 10% to ‘falls’, ‘holds’, ‘rises’, and the outcome is ‘holds’, the score is 0.2² + 0.3² + 0.1² = 0.14. In olivLaw, the forecast_resolution table stores the one-sided form, and the public forecasts page reports its mean.

   “it is obvious that the score P has a minimum value of zero for perfect forecasting and a maximum value of 2 for the worst possible forecasting.”

   Open the card →