Author
Ezra Karger, Philip E. Tetlock et al.
1 reading card from 1 book · 2025.
1 card
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities · 2025
Every model earns its place in the ensemble through its skill score on future questions, not through reputation or tests on already-resolved questions.
The rules of lessons 13–17 apply unchanged to AI models and agents. Each source — model, agent, person — has its own record on resolved questions. The comparison is with a naive baseline: for binary questions, climatology (always stating the base rate), for series, the random walk. The Brier skill score is 1 minus the ratio of the source's score to the baseline's: positive means it beats the baseline, negative means you would do better always stating the base rate. And selection needs a minimum and a confidence interval, as with people. The trap specific to models is leakage: a model tested on already-resolved questions may "know" the answer from its training data. That is why ForecastBench uses only questions with no known answer at submission time. In the 2025 paper, human experts beat the best model (p below 0.001). On the 30 September 2026 leaderboard, the best system with its own scaffolding has a Brier Index of 69.1 and the superforecaster median 68.8 (higher is better), a non-significant difference (p = 0.63); without scaffolding, the best model scores 62.0 and the superforecasters 67.7. Metaculus wrote in June 2026 that claims of bots matching the best humans rest on tests against already-resolved questions. In the olivLaw engine, Prophet lost its backtest against the naive baseline on all five headline assets — on gold, a mean absolute percentage error of 17.6% against 3.0% — and yet it stays in the ensemble for BVB stocks, with a fixed weight.
“To avoid any possibility of data leakage, ForecastBench is comprised solely of questions about future events that have no known answer at the time of submission.”