I recommend Bloom and van Reenen’s 2010 JEP piece on “Why Do Management Practices Differ Across Firms and Countries?” The paper is mis-titled, but excellent nonetheless. Its main mission is to measure management practices, then describe how they vary – and they get a lot further than you’d expect.
Measurement. Bloom and van Reenen ran a massive “double-blind” survey of thousands of firms in 17 countries. As they explain:
One part of this double-blind technique is that managers are not told they are being scored or shown the scoring grid. They are only told they are being “interviewed about management practices.” To do this, we used open questions in the survey. For example, on the first monitoring dimension, we start by asking the open question “tell me how you monitor your production process,” rather than closed questions such as “do you monitor your production daily [yes/no].” We continue with open questions focusing on actual practices and examples until the interviewer can make an accurate assessment of the firm’s practices…
The other side of our “double-blind” approach is that our interviewers are not told in advance anything about the firm’s performance… We randomly sample medium-sized firms, employing between 100 to 5,000 workers. These firms are large enough that the type of systematic management practices chosen are likely to matter. However, these firms are generally small enough that they are not usually reported in the business press, so the interviewers generally have not heard of these firms before and so should have no preconceptions.
They explain their 18 criteria in detail in the article, and highlight evidence of their external validity: “[F]irms with ‘better’ management practices tend to have better performance on a wide range of dimensions: they are larger, more productive, grow faster, and have higher survival rates.” Not bad.
Results. Besides their evidence on external validity, Bloom and van Reenen have nine additional big findings. The most interesting to me:
1. The richer the countries, the better the management.
2. Key exception: Multi-nationals are managed at First World levels almost everywhere.
3. Overall, corporations’ separation of ownership and control looks like a
non-problem, or even a plus. Firms run by founders – and family firms
run by family CEOs (typically by primogeniture) – are well below average.
4. Government firms are the worst-managed of all.
Survey skeptics will presumably dismiss this entire paper. Since their dogmatic objections are mere words, however, I suggest we follow the skeptics’ own advice and ignore them. There’s a lot more work on management left to do, but I for one salute Bloom and van Reenen for getting as far as they have.
HT: Robin
READER COMMENTS
MernaMoose
Mar 15 2010 at 5:15am
It’s not that we don’t believe all surveys. Just the many that are so obviously biased. For example the ones that imply I’d be so much happier if I was a good progressive liberal. Or a western European. Or a God fearing Christian. Or a etc etc. You know, the kinds of surveys that give “survey” a bad name.
This survey though, might actually be worth something.
david
Mar 15 2010 at 6:06am
MernaMoose’s comment inadvertently shows exactly why surveys are distrusted – their acceptance or rejection is in large part driven by the ideology of the reader rather than any methodological objection to the survey. It doesn’t matter what the survey is or does to be discarded offhand, just whether their conclusions are “obviously biased”! Quantitative evidence is at least not so easily dismissed.
Pushing for survey acceptance means – shock and horror! – accepting surveys on the opposite side of the political fence.
Current
Mar 15 2010 at 10:24am
[Comment removed for supplying false email address. Email the webmaster@econlib.org to request restoring your comment privileges. A valid email address is required to post comments on EconLog.–Econlib Ed.]
Paul
Mar 15 2010 at 11:28am
Bloom and van Reenan have been doing some fascinating work on this issue. I can’t remember this particular paper’s title off the top of my head, but it discussed an experiment where some Indian textile firms were given free access to management advice and consultation, and compared to a control group which did not. The differences at the end of the experiment were impressive!
I suppose one take-home point is that management matters, even if it’s hard to measure when you’re close to the frontier. Rather like pornography, “you know it when you see it”.
OneEyedMan
Mar 15 2010 at 11:46am
I distrust surveys to the extent that I believe self-deception, lying, and ignorance distort the true information that the surveyors sought to gather.
A double blind survey goes a long way to address motives for lying and some for self-deception but much of the ignorance problems remain. If the consequences for managerial ignorance are different (lower perhaps) in less developed economies then this will influence the results. My experience is that managers know their business in all sorts of countries and industries, so I doubt that is a problem here. Though, managers of government run firms have the weakest incentives to know their businesses, so maybe I should be more skeptical.
John Thacker
Mar 15 2010 at 1:34pm
Since Bryan is a “survey believer,” why does he care that it was a double-blind survey? Or that the survey didn’t He believes that we can trust surveys regardless of methodology, right? Or have we had enough straw men for now?
Les
Mar 15 2010 at 2:02pm
It seems to me that surveys are inherently unreliable because they are merely opinion, as opposed to fact.
I believe that survey results can be validated if they agree with empirical fact-based studies. But, if one can perform empirical fact-based studies, then why would one ever bother to conduct a survey?
jc
Mar 15 2010 at 7:43pm
@ Paul
Thanks for the tip. Here’s the link if anyone else is interested:
http://www.stanford.edu/~nbloom/MM.pdf
(24% increase in annual profits for the firms receiving management advice and consultation.)
Thomas Sewell
Mar 16 2010 at 12:45am
There’s a difference between a self-reporting survey with biased questions and answers and an interview designed to elicit unbiased factual information.
If you ask someone to fill out a survey that shows a red square and asks, “Is this color Red?”, something as simple as that will bias the responses. If you ask, “What color is this square?” and leave it open-ended, you will get a much better and more nuanced response.
The problem is that the first kind of survey is more likely to get a researcher the results they’re looking for and providing answers (either y/n, t/f, like/dislike, scale of 1-5, or multiple choice) is much easier to tabulate automatically, giving an illusion of accuracy.
Well-designed surveys don’t have a built-in bias.
Political surveys drive me crazy for this. Even when they aren’t actually push-polling, they include all sorts of assumptions into their questions. Thinks like do you consider yourself “very liberal, liberal, moderate, conservative, very conservative” as if everyone must fall into that spectrum.
Even worse is when they ask about what issues are most important to you, but give you no way to express your desired solution to the issue. So if you respond “education”, they assume that you want the government to get more involved in education as the “solution”, when you may feel the issue is very important, but want the exact opposite solution.
This particular survey seems much better designed.
Tony Eaves
Mar 16 2010 at 8:48am
I believe that the survey described in the article could be extremely beneficial to businesses. By keeping the survey “blind” so that the people you are surveying are not feeding you the answers you want to hear, you are in turn receiving more truthful answers than what you would receive on a non-blind survey. Also by asking open-ended questions the survey taker is more likely to answer the question truthfully than picking out the correct answer in a closed-ended question.
I found the results that Bloom and van Reenen found very interesting. Basically the results show that people who are elected or inherit ownership or management positions are the worse managed. I can see that happening, for example, if a company was passed down in the family from father to son. It doesn’t mean that just because it is staying in the family it is best for the company, in truth the son may no little or nothing about proper management skills and the company would be better in a more experienced person management. The same could be said for elected management due to the fact more politics can be played than actually taking their management skills into consideration.
Brian
Mar 16 2010 at 10:48am
I agree with MernaMoos, It is not that I distrust all survey’s, it just depends on the subject, population, method, and how the questions are asked. Surveys for marketing items I tend to trust if done correctly. Why? they tend to be specific, cover know buying decisions (which are know based on a long history of feedback), and the current emotional state that affect the answers tend to also effect buying. However, introspective questions on how I live my life or act are extremely unreliable (in most areas). You can get in a very rough ballpark but many people are either not honest or have not deceived themselves in seeing themselves differently than how they really act or would act.
Then you have political surveys which has its own sets of problems. Meaning of words is one I think surveyors can do a better job . One needs to be very careful of the specific words used verse being careful about the meaning of the words used.
For example, I am a Christian and I was in a small group and I brought up that we all make Judgment calls every day, therefore we judge on what is right and wrong and how society and individuals should act. Yet a half a dozen people Insisted it was wrong to judge others that the Bible said it was wrong? I argued with them for ages, I got a dictionary and to prove I was using the word correctly and I did not mean condemn or judge as if I were a little God. I even got a Greek lexicon to prove I was using the word in the right biblical context. It was not that they did not hold that view (they where not clueless).
It did not matter what I showed them or what angle I took the prove I was using the word correctly, they had totally redefined the meaning of the word to fit their belief system (what they had been taught). I learned a major lesson, that I have seen numerous times since. Specific cultures have very strong feelings and believes associated to specific words or sayings. So in this case it would not matter if a surveyor was very careful in defining the word or how it was used. The use of that specific word would of automatically triggered a specific response for what I guess would be a very large population of people in certain circles of the Christian culture.
With all that being said I think the mentioned study does tend to be more credible than most (due to the double blind method ) and presents good information that I think needs to be explored. However, it does have shortfalls specifically in some of the questions (Targets are stretching, and Performance dialogue) and the skill (experience) of the MBA students to correctly evaluate (wading though the BS and understand what the actual behavior was verses how the manager was wording the behavior) the responses by managers to the questions. Now if these MBA students had a few years work experience in the industry they were critiquing, I would say this survey is very valid. Throw out a few of the criteria that are important but extremely difficult to evaluate in some industries with only a little time and you have some very good useful research in management practices.
Noah Yetter
Mar 16 2010 at 1:56pm
What constitutes “‘better’ management practices”? I can’t really think of a way to define them that isn’t circular w.r.t. firm performance.
Brittany
Mar 17 2010 at 6:30pm
I feel the type of survey discussed in the article could be valuable for a business. By keeping the survey open ended it gives you more truthful answers instead of answers you really want to hear. I believe by giving an open ended question you will receive much more variety in answers and in turn be able to improve your firm or business more. Bloom and van Reenen’s results were very interesting. I didn’t think government managed firms would have the worst management overall but I thought wrong. If I thought it out in deep concentration it would make sense that government managed firms weren’t managed very well because more politics would enter the firm than management skills. Family run firms also have well below average management ratings. They also found that the richer the country the better the management, this makes sense. If a country is poor of course they aren’t going to have the materials or education to manage as well as a richer country. I had never really thought about it but I think an open ended question survey is the best type of survey to be given, if you want the most truthful answers, because not everyone fits into certain “stereotypes” or issues that a closed ended survey may contain.
Comments are closed.