Review of Expert Political Judgement (Philip E. Tetlock)


Very interesting book!


Game Theorists. The rivalry between Sherlock Holmes and the evil

genius Professor Moriarty illustrates how indeterminacy can arise as a

natural by-product of rational agents second-guessing each other. When

the two first met, Moriarty was eager, too eager, to display his capacity

for interactive thinking by announcing: “All I have to say has already

crossed your mind.” Holmes replied: “Then possibly my answer has

crossed yours.” As the plot unfolds, Holmes uses his superior “interac-

tive knowledge” to outmaneuver Moriarty by unexpectedly getting off

the train at Canterbury, thwarting Moriarty who had calculated that Paris

was Holmes’s rational destination. Convoluted though it is, Moriarty

failed to recognize that Holmes had already recognized that Moriarty

would deduce what a rational Holmes would do under the circum-

stances, and the odds now favored Holmes getting off the train earlier

than once planned.23


Indeterminacy problems of this sort are the bread and butter of behav-

ioral game theory. In the “guess the number” game, for example, con-

testants pick a number between 0 and 100, with the goal of making their

guess come as close as possible to two-thirds of the average guess of all

the contestants.

24 In a world of only rational players—who base their

guesses on the maximum number of levels of deduction—the equilib-

rium is 0. However, in a contest run at Richard Thaler’s prompting by

the Financial Times,

25 the most popular guesses were 33 (the right guess

if everyone else chooses a number at random, producing an average guess

of 50) and 22 (the right guess if everyone thinks through the preceding

argument and picks 33). Dwindling numbers of respondents carried the

deductive logic to the third stage (picking two-thirds of 22) or higher,

with a tiny hypereducated group recognizing the logically correct answer

to be 0. The average guess was 18.91 and the winning guess, 13, which

suggests that, for this newspaper’s readership, a third order of sophisti-

cation was roughly optimal.





Our reluctance to acknowledge unpredictability keeps us looking for

predictive cues well beyond the point of diminishing returns. 39 I witnessed

a demonstration thirty years ago that pitted the predictive abilities of a

classroom of Yale undergraduates against those of a single Norwegian

rat. The task was predicting on which side of a T-maze food would ap-

pear, with appearances determined—unbeknownst to both the humans

and the rat—by a random binomial process (60 percent left and 40 per-

cent right). The demonstration replicated the classic studies by Edwards

and by Estes: the rat went for the more frequently rewarded side (getting

it right roughly 60 percent of the time), whereas the humans looked hard

for patterns and wound up choosing the left or the right side in roughly

the proportion they were rewarded (getting it right roughly 52 percent of

the time). Human performance suffers because we are, deep down, de-

terministic thinkers with an aversion to probabilistic strategies that ac-

cept the inevitability of error. We insist on looking for order in random

sequences. Confronted by the T-maze, we look for subtle patterns like

“food appears in alternating two left/one right sequences, except after

the third cycle when food pops up on the right.” This determination to

ferret out order from chaos has served our species well. We are all bene-

ficiaries of our great collective successes in the pursuit of deterministic reg-

ularities in messy phenomena: agriculture, antibiotics, and countless other

inventions that make our comfortable lives possible. But there are occa-

sions when the refusal to accept the inevitability of error—to acknowledge

that some phenomena are irreducibly probabilistic—can be harmful.


indeed, but generally it is wise to not accept the unpredictability hypothesis about some fenomena. many things that were thought unpredictable for centures turned out to be predictable after all, or at least to some degree. i have confidence we will see the same for earthquakes, weather systems and the like in the future as well.


predictability (and the related determinism) hypothesis are good working hypotheses, even if they turn out to be wrong some times.


this is what i wrote about years ago on my danish blog here. basically, its a 2×2 table:


What we think/what is true Determinism Indeterminism
Determinism We keep looking for explanations for fenomena and in over time, we find regularities and explanations. We waste time looking for patterns that arent there.
Indeterminism We dont spend time looking for patterns, but there actually are patterns we that cud use to predict the future, and hence we lose out on possible advances in science. We dont waste time looking for patterns that arent there.


The above is assuming that indeterminism implies total unpredictability. This isnt true, but in the simplified case where were dealing with completely random fenomena and completely predictable fenomena, this is a reasonable way of looking at it. IMO, it is much better to waste time looking for explanations for things that are not orderly (after all), than risk not spotting real patterns in nature.


Finally, regardless of whether it is rash to abandon the meliorist search

for the Holy Grail of good judgment, most of us feel it is. When we weigh

the perils of Type I errors (seeking correlates of good judgment that will

prove ephemeral) against those of Type II errors (failing to discover

durable correlates with lasting value), it does not feel like a close call. We

would rather risk anointing lucky fools over ignoring wise counsel. Radi-

cal skepticism is too bitter a doctrinal pill for most of us to swallow.





But betting is one thing, paying up another. Focusing just on reactions

to losing reputational bets, figure 4.1 shows that neither hedgehogs nor

foxes changed their minds as much as Reverend Bayes says they should

have. But foxes move more in the Bayesian direction than do hybrids and

hedgehogs. And this greater movement is all the more impressive in light

of the fact that the Bayesian updating formula demanded less movement

from foxes than from other groups. Foxes move 59 percent of the pre-

scribed amount, whereas hedgehogs move only 19 percent of the pre-

scribed amount. Indeed, in two regional forecasting exercises, hedgehogs

move their opinions in the opposite direction to that prescribed by Bayes’s

theorem, and nudged up their confidence in their prior point of view after

the unexpected happens. This latter pattern is not just contra-Bayesian; it

is incompatible with all normative theories of belief adjustment.8