Author
Philip E. Tetlock
2 reading cards from 1 book · 2005.
2 cards
Expert Political Judgment: How Good Is It? How Can We Know? · 2005
What experts think matters far less than how they think.
Tetlock spent twenty years collecting and scoring the forecasts of political experts and borrowed two portraits from Isaiah Berlin. The hedgehog ‘knows one big thing’: it works within one tradition and applies the same formula to every problem. The fox ‘knows many little things’: it borrows from several traditions and accepts ambiguity. The conclusion of Expert Political Judgment (2005, p. 21): the foxes consistently edge out the hedgehogs, and their most decisive victories come in long-term forecasts inside their own domain. The chapter's key sentence: what experts think matters far less than how they think. The IARPA tournament measured the modern version of the fox, actively open-minded thinking: deliberately seeking evidence that contradicts you and being willing to change your mind. Mellers and colleagues (2015) found that superforecasters score higher on this trait and on need for cognition, but warn that cognitive-style effects are markedly weaker than those of fluid intelligence and of behavioural commitment, such as updating frequency. Style helps; it does not replace the work. In practice, the fox writes down in advance the signs that would change its mind and keeps several models in competition. In the olivLaw engine, the financial forecast pipeline runs five models every day regardless of their track record, and the ensemble gives them fixed weights; Prophet carries the largest weight, even though it lost the backtest to the naive model on all five headline assets.
“we are better off turning to experts who embody the intellectual traits of Isaiah Berlin’s prototypical fox—those who “know many little things,” draw from an eclectic array of traditions, and accept ambiguity and contradiction as inevitable features of life”
Expert Political Judgment: How Good Is It? How Can We Know? · 2005
If a clairvoyant cannot tell without clarification whether you were right, you have not made a forecast but a statement.
In Expert Political Judgment (2005), Philip Tetlock set one condition before any measurement: the clairvoyance test. If you handed your forecast to someone who already knows the future, they must be able to say, without asking for clarification, whether you were right. ‘The leu will weaken’ fails the test: by how much, against what, by when, according to whom? A question that cannot be lost cannot be won either, so it teaches you nothing. A scorable question has four parts. The event, phrased as an observable fact rather than a trend. The deadline, a calendar date. The resolution source, meaning who publishes the fact. The criterion, meaning the exact threshold and the edge cases: ties, data revisions, the source disappearing. On top comes the probability, a number between 0 and 1. If any part is missing, the judgement may be very clever, but it cannot be graded. In the olivLaw engine the verdict lives in the forecast_resolution table: the point probability, the deadline, the outcome, the score, the source of the verdict and the status ‘resolved’ or ‘void’ (void when the criterion turned out to be unresolvable). The cost of bad questions has been measured: of the first 56 forecasts in the analyses, 26 carried no point probability and could not be scored at all.
“The most important of these was the clairvoyance test: our measures had to define possible futures so clearly that, if we handed experts’ predictions to a true clairvoyant, she could tell us, with no need for clarifications (“What did you mean by a Polish Peron or . . . ?”), who got what right.”