An in-house superforecasting programme has five parts: a pipeline of scorable questions, a tournament with fixed rules, a leaderboard with a confidence interval, a minimum number of questions before any selection, and a feedback loop — and each part must be defended against people who learn to play the score instead of reality.

“Although governments invest massive sums to predict the actions of political actors and the consequences of those actions, they invest astonishingly little to evaluate the effectiveness of their favorite methods of generating predictions.”Barbara Mellers, Eric Stone, Philip E. Tetlock et al. · Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions · 2015 · Mellers, Stone, Murray et al., «Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions», Perspectives on Psychological Science 10(3), 2015, secțiunea «Forecasting Tournaments», p. 268

Do not select anyone until the confidence interval of the score difference excludes zero, and that usually takes dozens of common questions.

The GJP offers a recipe to copy. Questions have a deadline and a clear criterion and are chosen by someone other than the players. The score is Brier, computed daily on the standing forecast and averaged over days and questions (Mellers et al. 2015), so that a good forecast made early is worth more than a correct one made the day before the deadline. Because everyone chose their own questions, scores were standardized per question. The comparison groups included only people with at least 25 questions per tournament, and the leaderboard showed the top 10% in each condition. The reward for accuracy was status, not money. Luck or skill? The difference between two analysts over 30 questions may be noise. If per-question score differences have a standard deviation of 0.08 (one-sided form), the standard error of the mean is 0.08 / √30 ≈ 0.015, so a 0.02 advantage does not escape the 95% interval. To tell it apart you need about 60 common questions. Selection comes after the second season, not the first. Goodhart's law bites here: a measure that becomes a target stops measuring. Someone scored on hit rate will say 51% instead of 90%; someone who picks their questions takes the easy ones; someone who sees the team average copies it. The proper rule, standardized scores, a mandatory question set and separately written forecasts are the defences.

Worked example: Ana starts a programme on the desk. She has 12 analysts and six months. Her plan, by week:

week 0        the rules: one-sided Brier score, computed daily on the standing forecast;
              questions written by a colleague who does not play; each has a deadline,
              a criterion and a resolution source
weeks 1–26    10 new questions a month: 5 mandatory for everyone (the common set),
              5 optional; forecasts are written separately, discussion comes after
monthly       a calibration letter per analyst: probability bands, how many came true
              in each, the three largest misses and their post-mortem
month 6       leaderboard only for those with at least 25 questions, with a bootstrap
              confidence interval; nobody is "super" yet
month 12      second season: if the leaders stay on top, they join the elite team

Now the arithmetic at month 6. Analyst A has, on the common set of 30 questions, a mean Brier of 0.150; analyst B, 0.170. The per-question differences have a standard deviation of 0.08. Standard error: 0.08 / √30 ≈ 0.0146. The 95% interval: 0.020 ± 1.96 × 0.0146 = 0.020 ± 0.029, i.e. from −0.009 to 0.049. Zero is inside, so Ana cannot yet say who is better. How many questions would she need for a 0.02 difference to equal the interval's half-width? (1.96 × 0.08 / 0.02)² ≈ 61.5 — about 62. At 10 questions a month that is a year, one more reason to wait for the second season.

Defences against gaming: the mandatory common set (nobody can dodge hard questions), the daily score (you cannot wait for the last day), Brier rather than hit rate (you cannot say 51% to "hit"), separately written forecasts (you cannot copy the average), and the reward is a mention, not a bonus — as in the GJP, where accuracy brought status, not money.

In the olivLaw engine. The parts of a programme half exist. The forecast_resolution table holds the probability, the deadline, the outcome and the one-sided Brier score; the cockpit has the resolution queue; the `build` function of forecast_score computes the mean Brier, a hit rate (p ≥ 0.5) and a reliability diagram in bands of 0.2, shown only when a band has at least five rows (BAND_MIN_N = 5); the forecast.unresolved_past_due probe in system_health flags forecasts past their deadline and unresolved. Three things are missing: a field saying who made the forecast (resolved_by says who resolved it, not who forecast it), a confidence interval around the mean, and a threshold below which no comparison is published. And 26 of the 56 old forecasts have no probability, so they cannot be scored at all.

The proposal (it does not exist yet): an internal leaderboard that reports each forecaster's mean and bootstrap interval, and refuses to compare anyone below 25 questions.

import random
from statistics import mean

MIN_QUESTIONS = 25


def bootstrap_ci(scores: list[float], n: int = 5000, seed: int = 1) -> tuple[float, float]:
    rng = random.Random(seed)
    k = len(scores)
    means = sorted(mean(rng.choices(scores, k=k)) for _ in range(n))
    return means[int(0.025 * n)], means[int(0.975 * n) - 1]


def paired_difference(a: dict[str, float], b: dict[str, float]):
    common = sorted(set(a) & set(b))            # common questions only
    if len(common) < MIN_QUESTIONS:
        return None                             # too few: no comparison
    diffs = [b[q] - a[q] for q in common]       # positive = A better
    lo, hi = bootstrap_ci(diffs)
    return round(mean(diffs), 4), round(lo, 4), round(hi, 4), len(common)


rng = random.Random(3)
a = {f"q{i}": max(0.0, rng.gauss(0.15, 0.08)) for i in range(30)}
b = {q: max(0.0, s + rng.gauss(0.02, 0.08)) for q, s in a.items()}
print(paired_difference(a, b))   # if the interval includes 0: not known yet

The paired comparison (same question, two forecasters) removes question difficulty from the difference, just as the GJP's standardization did. A new probe in system_health could count forecasts published without a point_probability in the last 7 days: the target is zero, because a forecast without a number enters no leaderboard.

Why it matters A leaderboard with no confidence interval and no minimum number of questions promotes luck and teaches everyone to play the score.