“In fact, the averaging aggregators are uncalibrated and under-confident, i.e., too close to 1/2, even if the individual forecasts are calibrated (Ranjan and Gneiting, 2010).”Ville A. Satopää, Robin Pemantle, Lyle H. Ungar · Modeling Probability Forecasts via Information Diversity · 2014 · V. A. Satopää, R. Pemantle, L. H. Ungar, «Modeling Probability Forecasts via Information Diversity», preprint arXiv:1406.2148, secțiunea 2.2, p. 8
Extremize in log-odds only across sources with different information, and learn the exponent from resolved questions.
Baron, Mellers, Tetlock, Stone and Ungar (Decision Analysis, 2014) give two reasons why an aggregate should be made more extreme. First: the probability scale is compressed at its ends, so random noise pushes the average towards 0.5. Second: each forecaster knows they see only part of the total information and stays cautious; put them together and the group knows more than any of them, but the average keeps each one's caution. The general form used in the Good Judgment literature: turn each probability into log-odds, ln(p/(1−p)), take the mean, multiply it by an exponent a and map back to a probability. With a = 1 you get the geometric mean of the odds; with a above 1 the aggregate moves away from 0.5. Satopää and colleagues (International Journal of Forecasting, 2014) derived a one-parameter logit aggregator of this kind; the course quotes no fitted value of a, because we did not see it in the primary text. The value is learned on your own data. When NOT to extremize: when forecasters share the same information. Satopää, Pemantle and Ungar show that the optimal aggregator moves smoothly from averaging, when information overlaps completely, to voting, when it does not overlap at all; in their example the optimal aggregate rises from 2/3 to 4/5 only because the overlap drops. Five runs of the same model on the same news bundle are not five independent sources.
Worked example. Three of Ana's colleagues, each with a different source — one reads the business press, one watches the forward market, one talks to the treasurers of two banks — give 0.6, 0.7 and 0.8 on a binary question. Step by step:
p p/(1-p) ln(p/(1-p))
0.6 1.500 0.4055
0.7 2.333 0.8473
0.8 4.000 1.3863
mean log-odds 0.8797
a = 1 1/(1+e^-0.8797) = 0.707 (geometric mean of the odds)
a = 1.5 1/(1+e^-1.3195) = 0.789
a = 2 1/(1+e^-1.7594) = 0.853
a = 2.5 1/(1+e^-2.1992) = 0.900The linear mean would be 0.70; the log-odds aggregate with a = 1 gives 0.707, almost the same. Extremizing only appears once a exceeds 1.
What you buy and what you risk, in one-sided Brier (p − o)²:
a aggregate p Brier if YES Brier if NO
1.0 0.707 0.086 0.500
1.5 0.789 0.045 0.623
2.0 0.853 0.022 0.728If the event happens, a = 2 cuts your score by a factor of four; if it does not, it raises it by almost half. Extremizing is therefore a bet that the three sources gather different information which, put together, justifies more confidence than any one of them has. If all three had read the same report, the bet is lost from the start.
Ana does not pick a from her gut. She takes the resolved questions, computes the aggregate's Brier for every a from 0.5 to 3.0 and keeps the best — then checks on held-out questions. On a small set, a jumps around: in the sketch below, on six questions, the a chosen without one question at a time ranges from 1.0 to 3.0. That is the signal that more data is needed before extremizing.
In the olivLaw engine. Today there is no extremizing anywhere in the engine. The war-gaming agent (`WarGamingAgent._calibrate`) runs a linear pool with fixed weights, and the simulation consensus (`_compute_consensus`) is a plurality vote plus min, max and mean across runs. The engine-specific trap: many of the sources are NOT independent. The simulation runs, the model's estimate and the analysis prior all feed on the same news bundle and often the same model; information overlap is high, so theory says a near 1. Extremizing them blindly would count the same news story three times.
The proposal (does not exist yet): a log-odds aggregation function with a learned monthly from the resolution table, separately per family of sources, with a fixed at 1 until at least 50 resolved questions, and with the leave-one-out choice of a reported so that instability is visible.
from math import exp, log
def logit(p: float, eps: float = 1e-4) -> float:
p = min(max(p, eps), 1 - eps)
return log(p / (1 - p))
def extremize(ps: list[float], a: float = 1.0) -> float:
# mean log-odds times a; a = 1 = geometric mean of the odds
m = sum(logit(p) for p in ps) / len(ps)
return 1 / (1 + exp(-a * m))
def brier(rows, a: float) -> float:
return sum((extremize(ps, a) - o) ** 2 for ps, o in rows) / len(rows)
def fit_a(rows, grid=None) -> float:
grid = grid or [x / 10 for x in range(5, 31)]
return min(grid, key=lambda a: brier(rows, a))
def loo_a(rows) -> list[float]:
# a chosen without question i: what a new forecast sees
return [fit_a(rows[:i] + rows[i + 1:]) for i in range(len(rows))]
rows = [([0.6, 0.7, 0.8], 1), ([0.3, 0.2, 0.25], 0), ([0.65, 0.7, 0.6], 1),
([0.4, 0.55, 0.5], 0), ([0.85, 0.8, 0.9], 1), ([0.3, 0.35, 0.2], 1)]
print(round(extremize([0.6, 0.7, 0.8], 2.0), 3), fit_a(rows), loo_a(rows))Run, the sketch gives 0.853, a = 1.3 on the full set and leave-one-out values between 1.0 and 3.0. The proposed probe in system_health would track the Brier of the extremized aggregate against the non-extremized one over the last 90 days; if the difference turns positive, a reverts to 1 automatically.
Why it matters An engine that only averages good forecasts loses resolution; one that extremizes copies of the same source becomes overconfident — the difference is information diversity, not the number of votes.