Teacher Evaluations and Superstition
Megan McArdle has a long post on the issue of measuring teacher quality. Meanwhile, The New York Times profiles James Heckman, whose careful research suggests that by the time a child reaches school age it is too late to make much difference.
If the best evidence is that it is almost impossible to make a long-term difference in education, then the statistical evidence on teacher quality is bound to be highly unreliable. What appears to be teacher quality is likely to be random variation. The low rate of replication of statistical teacher evaluations that Megan discusses is consistent with that.
There is a term that Daniel Klein alerted me to called “white hat bias.” What it means is that findings that favor a popular political viewpoint will be published, while those that contradict that viewpoint will tend to be discarded. So many people have a vested interest in believing that teachers make a difference that one has to be very wary of white hat bias in studies that purport to show such differences.
Along these lines, I am afraid that I am skeptical of Rick Hanushek’s claim that the best teachers are really effective and the worst are really ineffective. If that were true, then I think we would observe private schools dramatically outperforming public schools, holding student characteristics constant, and I do not think that is what the data say. Instead, when we see differences, those differences typically do not persist over time.
In education research, intensive efforts are made to find differences caused by teachers or other inputs. This is a worthwhile effort, but whenever studies are published showing such differences, they need to be discounted heavily for the biases induced by various filters in the research and publication process. The likelihood of any strong difference holding up in repeated study is quite low.
Dec 27 2010 at 8:57pm
Not without the profit motive.
Dec 27 2010 at 9:35pm
I fully agree. There is also a sorting problem at the school level. There is some indication that school administrators have favorite teachers and those teachers get more of the students that have better self control, are more eager to learn and have parents who are committed to the child’s school success. These perceived student traits are not documented and are not used in research of teacher effectiveness. Almost all teachers do a better job of teaching eager, well-behaved students with involved parents. So the school level sorting creates the perception of a value-added teacher. Even in so called random student classroom placement, there is anecdotal and subtle statistical indications that the sorting occurs as part of the initial placement or later after classes start. If this presorting does really occur, it would mean that we have the causation backwards. Good students cause the value added teacher instead of the reverse.
NIH recently funded a study that found a mother’s literacy level very important to a child’s education success. Improving a mother’s literacy may be a more worthwhile effort than focusing on teacher effectiveness.
“…programs to boost the academic achievement of children from low income neighborhoods might be more successful if they also provided adult literacy education to parents.
The researchers based this conclusion on their finding that a mother’s reading skill is the greatest determinant of her children’s future academic success, outweighing other factors, such as neighborhood and family income.”
Dec 27 2010 at 10:15pm
This comes up a lot on econlog, but I admit it confuses me. Are we talking about grade school teachers or what? Surely you could put the same student through two different multivariable calc classes at the college level and get different results. So what is this all about?
Dec 27 2010 at 11:16pm
I think Arnold is making a huge mistake in interpreting the teacher effectiveness literature. The take home messages are:
1. There really are individual teachers who are measurably better than others.
2. Easily measured attributes (e.g., credentials, gender, age, seniority, etc) usually do not correlate with effective teaching (i.e., test score improvement).
In other words, Dr. Kling may be a good teacher, but that can’t be explained by his econ PhD or # of years in the classroom. It’s “Kling specific” (or a “fixed effect” in statistics jargon).
It’s easy for people to read conclusion #2 and claim that teaching quality is bogus. My interpretation is simpler and makes more sense given that there appear to be some really good teachers.
Teaching is mainly about coaching and connecting with students so that they will endure the material. Teachers also need some internal self-discipline so they will stick to the methods that are known to work (e.g., repetition in arithmetic, or phonics in reading). These two characteristics are really about personality more than credentials or other easily observed characteristics. So that’s why you get the weird results from teacher research – yes for teacher fixed effects, but no effect for covariates.
Finally, Arnold is responding to the finding that your school or teacher doesn’t change your entire life. Teacher effects are modest, but educational attainment is a life long process. A good teacher will rarely change the remedial student to college bound material, but it can matter, at least in the short term.
Dec 28 2010 at 9:58am
There’s another kind of publication bias that is relevant here. That is, results that are unexpected (by potential readers) are more likely to be published than results that boringly confirm what people already commonly accept (and are much more likely to be published than results that are inconclusive).
Right or wrong, Heckman’s research has been published and widely discussed in part because it is surprising in its conclusions. How many others have studied the same question and found the opposite? Is their work not being cited here simply because no one is familiar with it because it’s boring to read that something we all assume to be true (that children continue to be influenced by parents, peers, and teachers well into young adulthood) actually is true?
Dec 28 2010 at 11:42am
“Along these lines, I am afraid that I am skeptical of Rick Hanushek’s claim that the best teachers are really effective and the worst are really ineffective.”
I have to disagree with him as well, but I suspect for different reasons.
From my own experience, the best teachers are really effective, and the worst teachers are worse than ineffective; they’re actively counter-productive.
I’m sure this is going to be different for different people, since not everyone learns the same way, but the bad teachers I had didn’t just not teach, they got in the way of learning.
Ineffective teachers could easily have been replaced, for me, by letting me loose in the library instead. The bad teachers, on the other hand, actively worked to hinder my education to the extent that I would have been better off simply being locked in a sensory deprivation tank for the period of the class.
This obviously wasn’t what they were trying to do (at least not all of them; some of them I’m not so sure about). My “advanced math” teacher who couldn’t perform basic subtraction quite obviously cared and wanted to teach. She was simply incapable of doing so. Others were simply marking time: the syllabus says do this now, so we do this now, no deviations, sit-down-and-shut-up.
And just because I can’t resist throwing out some quotes:
“‘Educational’ refers to the process, not the object. Although, come to think of it, some of my teachers could easily have been replaced by a cheeseburger.” — Terry Pratchett
“I can prove at any time that my education tried to make another person out of me than the one I became. It is for the harm, therefore, that my educators could have done me in accordance with their intentions that I reproach them; I demand from their hands the person I now am, and since they cannot give him to me, I make of my reproach and laughter a drumbeat sounding in the world beyond.” — Kafka
“Men are born ignorant, not stupid; they are made stupid by education.” — Bertrand Russell
“My schooling not only failed to teach me what it professed to be teaching, but prevented me from being educated to an extent which infuriates me when I think of all I might have learned at home by myself.” –George Bernard Shaw
“There are only two places in the world where time takes precedence over the job to be done. School and prison.” — William Glasser
“School-days, I believe, are the unhappiest in the whole span of human existence. They are full of dull, unintelligible tasks, new and unpleasant ordinances, brutal violations of common sense and common decency. It doesn’t take a reasonably bright boy long to discover that most of what is rammed into him is nonsense, and that no one really cares very much whether he learns it or not.” — H. L. Mencken
Dec 28 2010 at 2:20pm
You might find some of the recent studies of “value-added” teacher models interesting, especially since they are popular among pay-for-performance advocates. See, e.g.,
Dec 28 2010 at 3:13pm
AMEN! from Fabio Rojas above. Also good is his point that we are measuring things because they are easy to measure and not because of any correlation with learning outcomes.
I guess what this current melange of conflicting data says to me is that something is just different about the U.S. that allows us to defy the test score results.
Dec 28 2010 at 5:14pm
One reason it would be hard to get people to think that teachers don’t matter is that we all went to school, and we had some horrible teachers and some great teachers.
Dec 28 2010 at 8:58pm
This could be seen as an extension of Judith Rich Harris’s argument in The Nurture Assumption. In the same way that parents have limited impact on the way children turn out (beyond the genetic contribution), why would another set of adults be expected to have a greater effect?
Dec 29 2010 at 10:11am
Public schools pay better than private schools. It is harder to get a public school job than a private school job. Why would you expect private schools to have better teachers than public schools and thus ouperform? That makes no sense.
Dec 31 2010 at 8:15pm
From middle up through high school, there have been, are, and will always be those students who, for whatever reasons, simply do not want to learn – or have not yet matured enough to see the point in learning – and so will not allow themselves to be guided, coached, cajoled, praised or bribed into trying. They’ve long ago seen ever trick in the teacher’s standard issue bag of encouraging tricks and rejected them all. They’re nominally bright, usually; they just have no use for schooling. Either they’ll wake up some day and TRY or they won’t. But no adult in their lives will ever have so much of an influence on that decision as to be held responsible for it. Yet teachers today are increasingly held to be responsible for exactly that.
My point is, why is this most important factor – the child’s willingness to cooperate with instruction – always seem to be overlooked in these conversations?
Find THE BEST teacher you can, measured by whatever standards you like. He or she will be made to look less than the best by such students because past a certain point there’s nothing more to be tried, nothing more to be done except let the student fail. Who usually gets the anal exam first to find out why the student failed? Whose curricula, lesson plans, pedagogy, portfolios and lunchroom scuttlebutt are fine tooth combed to find the reasons for the failing? Who often takes the fall for (as everyone in-house usually well knows) a student’s utter lack of concern?
Holding good teachers responsible for the behavior choices of certain students is so unfair it’s absurd. If it’s ever been seriously discussed, I’d greatly appreciate the appropriate links.
Jan 5 2011 at 11:37am
There are a number of ways that it doesn’t pass the smell test that all teachers perform the same. Let me try a few:
1. Some public school teachers almost criminally neglect the education of their classrooms. It’s only almost, because teacher merit is explicitly and aggressively not part something that impacts a teacher’s career path. Some of them have zoos of classrooms, and others teach the students random junk that the teacher is interested in but is not useful to know and is not on the official curriculum. If you ask teachers at any public school, they can usually name a few such slackers. These are the ones that would be better off replaced by cheeseburgers.
2. In other classrooms, students leave the class tremendously smarter than when they started. If you give a trial AP test to students at the beginning of the year, and then at the end of the year, in some classes they do a lot better at the end of the year. Such teachers are doing a much better job than a cheeseburger.
3. Different public school teachers have highly variable AP test results. If you ask around the teachers in a school, they usually know how the others are doing. It’s too big a claim to swallow that all these teachers are wrong, that they’ve been mistaken, that they didn’t have enough controls in their study. I’m not talking about a small effect. You don’t need to compute p-values when one teacher has one 2 and twenty 4-5’s while another teacher has five 2’s, ten 3’s, and one 5. Spreads like this happen all the time.
4. If you ask around a skilled workspace, you’ll often learn that you need a certain amount of education to be qualified for the job. If you ask the workers, most of them can name a few classes where they learned crucial skills in a classroom, while for many others it was a waste of time. Are we to really believe that these workers are mistaken, that they really didn’t learn anything in the classes they think they did?
5. If you ask people who work in a skilled profession of some kind, they always say that some of their classes were a waste of time, and that others they actually learned something from. The useful ones must surely have had teachers that are better than cheeseburgers.
I’m with the other posters in thinking we are making this too complicated. Some teachers do better than others, and this is easily measured if you look at the test scores.
Comments are closed.