Strictly Proper Scoring Rules, Prediction, and Estimation

1 card

1. 01

   A strictly proper scoring rule makes stating exactly what you believe the only optimal strategy; Brier and the log score are such rules, hit rate and absolute error are not.

   Strictly Proper Scoring Rules, Prediction, and Estimation · 2007

   A strictly proper rule is one under which lying never scores better than the truth.

   A scoring rule is a contract: you give a probability, the world gives an outcome, the rule gives you points. Gneiting and Raftery's question (2007) is whether that contract encourages you to tell the truth. A rule is proper if your expected score is best when you report exactly what you believe; it is strictly proper if that optimum is unique, so any deviation costs you. Brier himself demanded in 1950 that the verification scheme ‘should influence the forecaster in no undesirable way’. The Brier score and the log score, −ln(p) for the outcome that happened, are strictly proper. They differ in how harshly they punish misplaced confidence: Brier is bounded (at most 1 in the one-sided form), while the log score grows to infinity when you gave 0% to something that happened. The log score suits settings where rare events matter; Brier is more forgiving and easier to explain. Hit rate — ‘I was right if I said over 50% and it happened’ — is not proper. If you believe 70%, you get the same expected score at 51% and at 99%, so the rule does not reward honesty and invites rounding to the extremes. Absolute error, |p − o|, is worse: if you believe 70%, its expected score is best at 100%, so it pays you to lie. Mind the orientation: Gneiting and Raftery write scores positively, higher is better; here, as in olivLaw, lower is better.

   “A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ≠ F. It is strictly proper if the maximum is unique.”

   Open the card →