I’ve long been skeptical of the statistics for AIDS in Africa. The whole story had a quasi-Soviet flavor to it. The main difference: Soviet growth statistics were too good to be true, while African AIDS statistics were too bad to be true. Reflecting on the incentives cemented my skepticism: Just as the Soviet Union had a strong incentive to exaggerate its growth numbers in order to get the world’s respect, researchers and advocates had a strong incentive to exaggerate their AIDS number in order to get the world’s money.

However, whenever I shared my doubts with others, they almost always told me I was way off-base. Biased statistics about Soviet growth were hard to expose; biased statistics about AIDS would not be.

Now the Washington Post is backing me up:

The new data suggest the rate never reached the 30 percent estimated by some early researchers, nor the nearly 13 percent given by the United Nations in 1998.

The study and similar ones in 15 other countries have shed new light on the disease across Africa. Relying on the latest measurement tools, they portray an epidemic that is more female and more urban than previously believed, one that has begun to ebb in much of East Africa and has failed to take off as predicted in most of West Africa.

Why were the numbers exaggerated for so long?

Such disparities, independent researchers say, skewed years of policy judgments and decisions on where to spend precious health-care dollars.

“From a research point of view, they’ve done a pathetic job,” said Paul Bennell, a British economist whose studies of the impact of AIDS on African school systems have shown mortality far below what UNAIDS had predicted. “They were not predisposed, let’s put it that way, to weigh the counterevidence. They were looking to generate big bucks.”

[…]

“It’s pure advocacy, really,” said Jim Chin, a former U.N. official who made some of the first global HIV prevalence estimates while working for WHO in the late 1980s and early 1990s. “Once you get a high number, it’s really hard once the data comes in to say, ‘Whoops! It’s not 100,000. It’s 60,000.’ ”

Chin, speaking from Stockton, Calif., added, “They keep cranking out numbers that, when I look at them, you can’t defend them.”

Still, Peter Ghys, a UNAIDS epidemiologist who now acknowledges that earlier numbers were too high, implausibly chalks the the mistakes up to random error:

Ghys said he never sensed pressure to inflate HIV estimates. “I can’t imagine why UNAIDS or WHO would want to do that,” he said. “If we did that, it would just affect our credibility.”

Can’t imagine it? Here’s a simple model: People face a trade-off between getting the results they want and protecting their credibility. Sometimes they decide that risking their credibility is worth it – and this was one of those times.