Book
A Tutorial on Conformal Prediction
by Glenn Shafer, Vladimir Vovk · 2008 · 1 reading card
1 card
A Tutorial on Conformal Prediction · 2008
Scenarios sum to 1, tails have tails, and validation only moves forward in time.
A set of scenarios is a probability distribution in narrative clothing. To be scorable it must be MECE — the scenarios exclude one another and together cover every possibility — and their probabilities must sum to 1. The engine's Seldon threads start from 20, 35, 25, 15 and 5 percent, summing to 100, so the form is right; the problem is another: they stayed nearly constant for months. Tails need something other than more noise. The scenario generator in simulation_service calls "black swan" the variant whose Gaussian perturbation has a standard deviation 2.5 times larger. For a normal with variance 1, the chance of a move beyond 4 standard deviations is 0.0063 percent; for a Student distribution with 3 degrees of freedom and the same variance it is 0.62 percent — almost a hundred times larger. A wider normal is still a normal: no jumps, no correlations that tighten in a crisis. Recalibration repairs a model's probabilities without retraining it. Platt scaling fits a logistic regression on the model's scores, isotonic regression a monotone step function, and conformal prediction (Shafer and Vovk, 2008) gives intervals with guaranteed coverage 1 − ε, but only if the examples are exchangeable; time series break the assumption. The most rigorous component in olivLaw, the I&W model (`train_iw_scenario`), uses Platt correctly but validates with randomly shuffled folds, so the future leaks into training.
“Given an error probability ε, together with a method that makes a prediction ŷ of a label y, it produces a set of labels, typically containing ŷ, that also contains y with probability 1 − ε.”