Steven Randy Waldman elaborates on some ideas that I set out in response to his initial post.

service providers in these industries are themselves uncertain of the value they are able to provide. Yet providers work hard to hide and downplay their uncertainty. Politicians pushing new programs offer authoritative projections of brilliant outcomes, although many initiatives fail once the lights of the bill-signing fade. Healthcare, finance, and education are built around credentials and prestige, despite questionable correlations between these tokens and value provided. Healthcare, finance, and educational institutions market themselves hard, portraying themselves as professional, competent, and above all, effective. These claims are not certain to be lies: High competence might sit within the wide confidence intervals that would surround a fair evaluation. But successful institutions do, and must, misrepresent those confidence intervals (to others, and sometimes to themselves).

There is more worth reading. In effect, Waldman is that these are professions that select for hubris and an activist mindset. If you are cautious and self-doubting, you have to act otherwise if you want people to believe you are a good school administrator, doctor, political leader, or finance professional. (See also Rescued from the Comments, which provides some hard evidence for what we are talking about.)

I would say that it is not the fault of the people in these professions that they select for hubris. The problem is ultimately that consumers prefer confidence. In some sense, what consumers are paying for is hope. That late-stage medical procedure represents hope. For overcoming disadvantages with which people are born, education represents hope. For getting rich without having to work hard or save obsessively, money managers offer hope. And do I even need to say that people look to politicians to supply hope?

Waldman concludes,

Forecasts that they will dominate, or prescriptions that we should specialize in these sectors to exploit alleged comparative advantage, should be greeted unenthusiastically

Phrases like “Hope springs eternal” or “the triumph of hope over experience” come to mind.