Vox Populi

1 card

1. 01

   A crowd of independent estimates, combined by a simple rule, usually beats its typical member; the mean, the median and the track-record-weighted mean are three different rules, and choosing between them is a decision, not a detail.

   Vox Populi · 1907

   Combine independent estimates and pick the aggregation rule from past scores, not from taste.

   At the Plymouth fat-stock show, 787 valid tickets guessed the weight of an ox after slaughter. Galton reported in Nature that the middlemost estimate, the median, was 1207 lb against a true 1198. Kenneth Wallis redid the arithmetic from Galton's worksheets in 2014: the correct median was 1208, the true weight 1197, and the mean of all tickets, which Galton gave only in a later letter, was 1197 — zero error. The lesson has two layers: the crowd was right, and the published measurement carried transcription errors. Check famous numbers too. Why does it work? Under squared loss, and therefore under the Brier score, the error of the mean is always at most the mean of the individual errors — a consequence of convexity, not luck. The gain is large when errors are diverse and cancel out; it vanishes when everyone read the same source. The median resists outliers but throws information away. The track-record-weighted mean gives more say to those who scored better, and in the Good Judgment Project aggregation with weights, temporal decay and extremizing beat the simple mean (Mellers et al. 2014). In the olivLaw engine, the war-gaming agent combines three sources with fixed weights: 0.5 for the model's estimate, 0.3 for the analysis prior and 0.2 for the panel. The weights were not learned from any score. They are an opinion written into code.

   “Now the middlemost estimate is 1207 lb., and the weight of the dressed ox proved to be 1198 lb.; so the vox populi was in this case 9 lb., or 0·8 per cent. of the whole weight too high.”

   Open the card →