Superforecasting on Epistemic Bait and Switch
A great discussion of the mistaken Iraq forecasts of the intelligence community (IC) from Superforecasting:
[F]ew things are harder than mental time travel. Even for historians, putting yourself in the position of someone at the time – and not being swayed by your knowledge of what happened later – is a struggle. So the question “Was the IC’s judgment reasonable” is challenging. But it’s a snap to answer “Was the IC’s judgment correct?” As I noted in chapter 2, a situation like that tempts us with a bait and switch: replace the tough question with the easy one, answer it, and then sincerely believe that we have answered the tough question.
This particular bait and switch – replacing “Was it a good decision” with “Did it have a good outcome?” – is both popular and pernicious. Savvy poker players see this mistake as a beginner’s blunder… Good poker players, investors, and executives all understand this…
So it’s not oxymoronic to conclude, as Robert Jervis did, that the intelligence community’s conclusion was both reasonable and wrong. But – and this is the key – Jervis did not let the intelligence community off the hook. “There were not only errors but correctable ones,” he wrote about the IC’s analysis… Would that have made a difference? In a sense, no. “The result would have been to make the intelligence assessments less certain rather than to reach a fundamentally different conclusion.”… This may sound like a gentle criticism. In fact, it’s devastating, because a less-confident conclusion from the IC may have made a huge difference: If some in Congress had set the bar at “beyond a reasonable doubt” for supporting the invasion, then a 60% or 70% estimate that Saddam was churning out weapons of mass destruction would not have satisfied them…
But NIE 2002-16HC didn’t say 60% or 70%. It said, “It has…” “Baghdad has…” Statements like these admit no possibility of surprise. They are the equivalent of “The sun rises in the east and sets in the west.” At a White House briefing on December 12, 2002, the CIA director, George Tenet, used the phrase “slam dunk.” He later protested that the quote had been taken out of context, but that doesn’t matter here because “slam dunk” did indeed sum up the IC’s attitude.
As far as I know, Tetlock is not a pacifist, but this is still intriguingly consistent with my common-sense case for pacifism. I’ll know more Monday, when I finally get to meet him in person. If you see me there, please say hi.