Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy

1 card

1. 01

   In 2026 the best AI forecasting systems, with task-built scaffolding, reached a statistical tie with the superforecaster median on ForecastBench, but models used out of the box remain clearly behind; the best result comes from combining people with models and from scoring each only on questions created after its knowledge cutoff.

   Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy · 2024

   Score models only on questions created after their knowledge cutoff, and combine them with people.

   The Schoenegger et al. (2024) study had 12 models forecast 31 questions from the same tournament as 925 people: the models' average could not be statistically told apart from the people's, and the average of people and models came out better than either. ForecastBench, built by Karger, Tetlock and colleagues, uses only questions that resolve in the future, precisely to rule out leakage. On 29 January 2026 the Forecasting Research Institute reported superforecasters in first place, 0.017 Brier points ahead, and extrapolated parity for November 2026, with a 95 percent interval between January 2026 and November 2027. The official leaderboard of 30 September 2026 shows, on the tournament Brier index (higher is better), the best AI entry at 69.1 and the superforecaster median at 68.8, with overlapping intervals and p = 0.63: a statistical tie, not a win. On the leaderboard of models used out of the box, without scaffolding, superforecasters score 67.7 and the best model 62.0, a significant difference. On Metaculus's FutureEval in spring 2026, 10 professionals beat the team of the top 10 bots by 1.25 points per question over 99 questions, but not significantly (p = 0.247). The main trap is leakage: a model scored on questions resolved before its knowledge cutoff remembers the answer. In the olivLaw engine, the Monte Carlo recalibration agent asks the model to report its Brier "if history exists", but never gives it that history.

   “we use an ensemble of 12 LLMs to make probabilistic predictions about 31 binary questions, comparing them with those made by 925 human forecasters in a 3-month tournament. We show that the LLM crowd outperforms a no-information benchmark and is statistically indistinguishable from the human crowd.”

   Open the card →