Michael Rizzo does New York. He finds no difference in school spending between schools “above the line” (high pass rates relative to their poverty index) and those below the line. However, the regression line appears to do a very poor job of explaining the pass rates, so that being above the line or below the line is almost entirely a matter of being above average or below average in terms of pass rates. I suggested in a comment on his blog that he should group the data by deciles in order to try to extract a better signal.
Reader Jonathan Klick had his undergraduate thesis published by The American Economist. In it, he found no signficant relationship between school spending and test scores in Pennsylvania, controlling for the percent of the student body coming from low-income households.
This type of research is not publishable (The American Economist serves to encourage undergraduate economics research–it’s not going to affect anyone’s tenure decision). And yet it’s interesting. I think society would be better off if more economists were doing simple, interesting work than doing the fancier stuff that gets published.
UPDATE: In an email, Rizzo writes,
n case you were interested, by putting the math performance and the poverty data into deciles and rerunning a simple regression, you get the following result:
Schools that perform better than expected spend $13,843 per student.
Schools that perform worse than expected spend $14,195 per student.
READER COMMENTS
Dr. Steven J. Balassi
Aug 1 2008 at 1:48pm
Very interesting and relevant! This may shed some light on the situation in California. CA’s education system is consistently ranked near the bottom yet we spend more than average. This research leads me to conclude there are other negative factors to our poor performance, not just spending.
I also agree that we economists need to focus some of our research on areas which are more relevant. Helping companies, countries, and citizens in everyday endeavors would be very beneficial.
AC
Aug 1 2008 at 2:50pm
Why isn’t it publishable? Is it a matter of taste — journals like fancy stuff rather than simple stuff — or is it a lack of rigor?
Foundational Reading Program
Aug 1 2008 at 3:04pm
Grouping the data would be a much more efficient way to display the statistics.
agent00yak
Aug 1 2008 at 3:08pm
One of my good friends did his undergrad thesis on the benefits of technology spending in California schools (The tech spending grants were awarded via a semi-lottery system). He found that the spending had negligible effects on student achievement even though the grants were used for their intended purpose.
Joe Weir
Aug 1 2008 at 3:51pm
Maybe I’m missing something here, but shouldn’t these analyses simply covary for (partial out) the effects the FARMS then look at the relationship between spending per pupil and pass rate?
Hopefully Anonymous
Aug 2 2008 at 6:04am
The hordes of economists working for the government in pretty much every alphabet soup “administration” don’t do simple, interesting work?
This isn’t a derisive question, I’m genuinely curious.
Troy Camplin, Ph.D.
Aug 2 2008 at 10:06am
I made these same observations not long ago on a comment I made here on this blog by simply looking at spending rates per pupil per state and comparing it to test scores in those states. No correlation. Actually, there appeared to be a subtle bias in favor of low-spending schools.
spencer
Aug 2 2008 at 12:12pm
Interesting results.
But I would like to see this analysis done for three groups.
1. Urban.
2. suburban
3. rural.
I suspect that what you will find is a negative
relation between pass rates and spending in urban schools, a very strong positive relationship in suburban school and a more or less neutral ranking in rural schools.
conchis
Aug 3 2008 at 6:01pm
Presumably the key issue with all of this sort of thing is to try to find exogenous variation in spending, such that you can rule out the possibility that schools that do worse receive more money because they’re doing worse rather than the other way around.
I would assume Hanushek, Hoxby and co have done stuff on this?
Tracy W
Aug 4 2008 at 6:34am
Actually, there appeared to be a subtle bias in favor of low-spending schools.
It’s possible that badly-performing schools tend to attract more government funds to try to improve their performance.
Or, alternatively, more money means more things are purchased, and the principal winds up spending all their time doing administrative things like sorting out the Deputy Careers Counsellor’s penion payments and whether to purchase a new character development programme, rather than looking at instructional quality.
Boonton
Aug 4 2008 at 9:19am
Very interesting and relevant! This may shed some light on the situation in California. CA’s education system is consistently ranked near the bottom yet we spend more than average. This research leads me to conclude there are other negative factors to our poor performance, not just spending.
Indeed, but keep in mind correlation does not equal causation. CA may spend more because they have more problems. If they reduced spending to match a lower spending state they may find even worse performance.
spencer
But I would like to see this analysis done for three groups.
I suspect the issue isn’t urban so much as cultural barriers. Language is probably the most important issue. If you have to worry about improving English skills first that leaves less time for math and other subjects.
Suppose you took out these two factors:
1. Students who don’t speak English well.
2. Students whose native language is neither English or Spanish. (There are plenty of people who are fluent in English and Spanish, if a school is confronted by many kids who speak a diverse array of other languages it is harder to address the problem by just hiring some bi-lingual teachers).
I suspect you would then notice a positive correlation between spending and results.
I understand why conservatives like the idea that school spending doesn’t produce better results but why this is the case is pretty troubling. It can’t be about competition. School spending usually happens at the most local level in the US. Real estate agents will tell you the quality of a school district is one of the drivers of home prices. Also, unlike local police and fire most people will come into direct contact with the local public school system in some format. A poorly run public school is a lot more noticeable to a lot more voters than, say, a poorly run zoning dept. Likewise better private schools also tend to cost more than lower quality ones (don’t jump to bring up Catholic schools here, the question is how much do they cost not how much do parents pay. Catholic schools tend to charge parents less than their cost using donations from their congregation as well as indirect gov’t subsidy to make up the difference).
Let me turn around and play economics professor to the economics professor. You’ve earned a C by showing there’s no simple relationship between spending more money and getting better performing schools. Now to get a B+ or better tell us why?
It is almost unheard of for something with better quality not to cost more. Assuming that high quality teachers are less common than average ones, wouldn’t the market model insist that the price of a high quality teacher be more than a low quality one? Isn’t there something that smells like a free lunch in this conservative meme against public schools? “We can get more by spending less (or at least not more)!” Why doesn’t that sound a bit like the mythical liberal social program that ‘pays for itself’ by boosting income, lowering unemployment and crime?
Chuck
Aug 4 2008 at 11:41am
The last time we visited this, it was pointed out that we are looking at very small spending difference in the two groups. In this case, $352 is 2.5% of $14,000.
It would seem to me that an equally valid conclusion is that the spending per pupil is not meaningfully different or that whatever difference you are seeing here is more about how much these schools spend on football or class room repairs.
That said, I think one can get a very good education at all education levels spending half as much as we do. We ‘waste’ a lot of money on extra curriculars that make you well rounded person without improving your math score.
Boonton
Aug 4 2008 at 12:16pm
We come back to how do you measure a positive result. I think a math score is a bit like measuring the quality of food by calories. If you’re talking about a survival situation, that’s a good metric. If you’re the food critic for the NY Times that’s a horrible metric.
In the last thread I suggested that residents measure schools by a similiar set of relative metrics. If the district is doing horribly, they may focus on the crude but easy to track metrics like what % of students pass the state math test. Once you get to a certain level (say 80% passing), though, the focus shifts to more subtle metrics like the quality of extra-curricular activities.
This seems perfectly rational to me. If the school is already doing a good job of the most basic of basics (reading, writing and arithmatic) wouldn’t consumers then demand the next level of service…namley making one more ‘well rounded’. How do you become more well rounded by cutting back on activities?
Many people would say a school that had no extra-curricular activities, no sports teams, no debate teams, no clubs…is less of an education than one that has those things even if the math scores are exactly the same. The market seems to as colleges (private and public) and employers seem to give such things credit.
Troy Camplin, Ph.D.
Aug 4 2008 at 9:31pm
In a free market economy, things of better quality do in fact cost more. But when you have a true monopoly, price goes up as quality goes down. This is basically what we have. The poor go to public schools more than other groups. The government thus has a monopoly over the education of the poor — thus, those schools cost more and have worse quality. With middle-income and wealthy people, the schools have to compete with home schooling and various private schools. Thus, they have to provide better quality. It just so happens that you can in fact provide high quality education for not too much money, so those public schools don’t have to spend as much as the schools in poorer areas, where the administrators use the poor quality of education they are giving to argue for more money for themselves. And then there is the issue of corruption in the poorer districts. My wife used to work for a poor Dallas school, and the AC was down all the time. She said people would come in, look around, do nothing at all, and presumably get paid for it. The AC would remain unfixed for weeks, before they came back and did it again. You can’t tell me that whatever company they hired wasn’t connected to someone in the district office. This is what makes education cost more for the poor, the fact that their public schools are being charged for work not done — and nobody cares about it. And the people who are most corrupt are the ones begging for more and more money to “help” the poor.
Boonton
Aug 5 2008 at 11:13am
In a free market economy, things of better quality do in fact cost more. But when you have a true monopoly, price goes up as quality goes down. This is basically what we have. The poor go to public schools more than other groups. The government thus has a monopoly over the education of the poor
Even monopolies must face competition. Microsoft may have 90% of the desktop operating systems but that doesn’t mean there are no alternatives.
First of all, public schools grab most students because they are free. No they aren’t free to the taxpayer but they are free to the parent. Simply because a good is free, though, doesn’t mean it has an automatic lock on the market. If that was the case we’d all be working in offices that run OpenOffice.
Second, public schools are probably the most intensely local of all gov’t services. It’s a lot easier to ‘shop’ for a good school by moving around than it is to ‘shop’ for a good state tax system or regulatory regime. People do make the quality of the schools one of their prime considerations in choosing where to live and the fact that schools are locally different means you can often choose while keeping the same job and still living in the same general area.
Third, local schools are the one feature of local gov’t that many voters come into direct contact with. IMO it’s a lot easier for a local gov’t to get away with having a totally inept building inspector or zoning board than a local school. What % of voters interact with those two things in a year? 2%? The local school, though, impacts a majority of voters at any given time directly or indirectly. On the local level there’s simply a lot more scrutiny of the schools than anything else. I’m not saying that’s a perfect substitute for competition but it usually should be a good thing.
Not surprisingly, I suspect most of the time when you have a really bad local school you also have a really bad local gov’t which leads me to suspect the problem is a dysfunctional system rather than just the schools. Washington DC, for example, has a rep for bad schools but also has a rep for lots of other bad things from police who can’t solve most of the murders to even snow plowing and garbage pickup.
With middle-income and wealthy people, the schools have to compete with home schooling and various private schools. Thus, they have to provide better quality
True, having more money means more options which means more competition. Someone who can spend up to $50K a year on vacation will have more travel agents beating on his door with a greater array of options than someone who needs to stretch to spend $2K. For this to become a major factor in public schools, though, you need to have a lot of segregation between rich and poor.
For the most part, I don’t think you often see that. Yes there are districts of deep poverty as well as districts with great wealth but in many parts of the US you’ll find mixes of both.
It just so happens that you can in fact provide high quality education for not too much money, so those public schools don’t have to spend as much as the schools in poorer areas, where the administrators use the poor quality of education they are giving to argue for more money for themselves. And then there is the issue of corruption in the poorer districts. My wife used to work for a poor Dallas school, and the AC was down all the time. She said people would come in, look around, do nothing at all, and presumably get paid for it.
I’m not at all sure “it just so happens” that you can provide high quality education for “not too much money” all the time. This is the problem that often erupts in these types of discussions, we fall into ancedotes. Yes I believe there are schools that are money pits who could see dramatic improvements paid for by simply going after waste and corruption. Likewise I’m sure there’s some middle class family who could double their consumption by simply shopping more wisely and clipping coupons. On average, though, for a middle class family to consume twice as much will require twice as much income. Is that true for schools?
Well look at what Michael Rizzo found:
Schools that perform better than expected spend $13,843 per student.
Schools that perform worse than expected spend $14,195 per student.
Let’s say poor schools are run by corrupt people who rack up the spending without improving results. What would the results look like? Schools that perform worse than expected would be spending a lot more per student (after all, the money would be going to kickbacks for the administrator’s ‘cousin’ who owns the AC repair company etc.).
But on average that doesn’t seem to be the case. Yes worse than expected schools do seem to spend slightly more (and maybe the very corrupt schools are the outliers driving the average up) but the relationship doesn’t seem to be there.
My hypothesis is that some schools just happen to benefit from minor random variations that allow them to do slightly better than expected. The example I gave previously was to imagine two poor districts. One district just happens to have a Church that sets up a volunteer tutoring program while the other doesn’t. Both have the same style of school administration. The school with the nearby Church, though, will indirectly benefit. If the school without it wants to match its sister school, it would have to spend money to create its own in-house tutoring program. The numbers, though, would make it seem like the first school is being run by smarter people who get better results with less money. The reality, though, is that both schools are being run equally well….one is just being benefitted by statistical ‘noise’ that you don’t see by looking at a crude figure like a poverty rate.
Troy Camplin, Ph.D.
Aug 5 2008 at 11:49am
As to your last comment, that would be averaged out statistically. Also, you don’t directly address my observation that there appears to be a positive correlation between high levels of corruption and impoverished areas. Certainly that needs to be addressed not only in relation to schools, but to the issue of poverty in general. I don’t know about where you live, but everywhere I’ve lived, the poor, middle-incomes, and the rich don’t exactly live in the same areas. Finally, we spend a lot of money on schools for things that have proven to be completely unnecessary for education. There is an entire education industry that survives on government kickbacks and convincing ignorant bureaucrats that the latest fashion they invented to sell their products will transform education, when it in fact has at best no, and too often negative, results. This is aside from the fact that public education still accepts behaviorism, which has been proven to be almost entirely incorrect as a model of human psychology.
Tracy W
Aug 5 2008 at 11:51am
I think a math score is a bit like measuring the quality of food by calories. If you’re talking about a survival situation, that’s a good metric. If you’re the food critic for the NY Times that’s a horrible metric.
Too many calories are harmful. How is too much maths harmful?
If the district is doing horribly, they may focus on the crude but easy to track metrics like what % of students pass the state math test. Once you get to a certain level (say 80% passing), though, the focus shifts to more subtle metrics like the quality of extra-curricular activities.
Quite possibly. The point of the exercise though was to show that increased funding did not improve test passing.
Many people would say a school that had no extra-curricular activities, no sports teams, no debate teams, no clubs…is less of an education than one that has those things even if the math scores are exactly the same.
Perhaps. That was not the question the post was seeking to answer though.
Boonton
Aug 5 2008 at 3:01pm
Too many calories are harmful. How is too much maths harmful?
Don’t read too much into an analogy. If you’re starving to death it makes a lot of sense to rate a hamburger better than a well prepared dish on the grounds that you need calories and need them now. Once you’re beyond starving, though, you can rate food on other factors like taste, freshness, balance etc.
I suspect when 80-90% of the kids are passing the math test parents suspect the 10-20% left represent more individual problems than problems with the school itself. They start using different metrics to measure the quality of their schools. That’s all.
Finally, we spend a lot of money on schools for things that have proven to be completely unnecessary for education.
Possibly but there’s a difference between test passing and education. A full music program that includes band, choir and a yearly musical…for example…will probably have only a marginal improvement on test scores yet would be an indication of a better school.
The point of the exercise though was to show that increased funding did not improve test passing
True but that’s not quite the same as saying test passing cannot be increased without increased funding.
Remember the counter question I posed is why wouldn’t increased funding produce better results? The “poor schools are corrupt and would graft away more money” answer has been disproven by the data provided here.
Boonton
Aug 5 2008 at 3:17pm
Another point about the metric….if people stop seeing higher pass rates as a metric for school quality they will fund other things…..
In other words, imagine a middle class school gets from 50% to 85% pass rate by funding a tutoring program. Now that the parents and voters see the 85% rate they decide additional funding will go into music and art programs rather than trying to make the pass rate 95%.
What’s going to happen to the data is you’re going to get schools that are spending more money but have little or no improvement in the math score. It would be an error, though, to conclude that higher math scores don’t require more money. On the contrary they do, but because some schools spend money on things other than math scores the regression’s results show no relationship between money.
To do this right….and it may not be possible to do it right….you should try to measure how much money is spent on math education and compare that to test scores.
Troy Camplin, Ph.D.
Aug 5 2008 at 7:41pm
I don’t see how the claim that poor schools are more likely to be corrupt has been disproven. Demonstrate how that has been disproven. It’s a fact here in Dallas.
Also, it’s been proven that learning music improves math scores, so your example is a poor one. But it does raise the question that, since we know for a fact that knowing music improves math scores, why all students aren’t taught music, if we want higher math scores.
Boonton
Aug 6 2008 at 8:18am
I didn’t say the data proved poor schools don’t suffer from more corruption than rich schools, I said that if poor schools were all corrupt (or on average corrupt) then the data would show that poor schools performing worse than expected would be spending a lot more than poor schools performing better than expected. The difference between them, though, is not statistically important. The bad poor schools spend $352 more than the good ones, or about 2.5% more.
Spending wise the difference between a poor school getting good results and one getting bad results is less than a hair.
Also, it’s been proven that learning music improves math scores,
Keep in mind marginal cost here. Music very well might help math but it probably takes a lot of spending on music education to boost math scores by 10 points as opposed to direct tutoring in math.
The point is that ‘quality’ is a very broad term. In a deeply troubled school it is easy to measure improvements in quality by very basic and crude metrics like pass rates on a standardized math test. As a school undertakes an effort to improve, it probably would target those easy metrics first because the payoff is quick and easy to see. At a certain point, marginal costs increase. Going from a 60% to a 85% pass rate is probably something that can be accomplished with some concentrated effort. Once you’re at 85%, though, the remaining 15% who fail probably reflect individual problems or issues rather than issues with the whole school. Going from just 85% to 90% might take as much effort or even more than the original improvement of 60%-85%.
That doesn’t mean all the ‘easy fixes’ have been exhausted, though. At this point the effort to improve quality might turn to other things like adding to music and art programs or sports or school clubs. IF the school gets additional funds it might put it into those things rather than pouring all resourses into a quest to get the pass rate to 100%.
What this means for the data analysis we are trying to do here is that the relationship between spending and math scores will get muffled. Some spending is being focused on things other than better math scores. This doesn’t mean that you can get better math scores without spending, although it may appear so at first glance.
Remember regression analysis is probably the most dangerous thing social science has ever developed. It’s really easy to set Excel to churning on data but anything it tells you has to be mulled over for ages. That’s why statistics text books maybe give you one chapter on how to do regression (if that, sometimes they just say let Excel do it) and then half a dozen chapter of hurdles you have to make your results jump through before youc an begin to say anything useful.
Troy Camplin, Ph.D.
Aug 6 2008 at 4:17pm
Actually, high levels of corruption could mask the true cost. It may in fact be true that more money improves education — if the money actually gets to the students — but so much money is lost through corruption that the benefits disappear.
Also, music education improves math learning in addition to whatever benefits there are to tutoring, so there is still an argument for including music education. Further, the evidence shows significant improvement. There are so-called music teachers in elementary school, but none of them actually teach music, but rather have their students listen to C.D.s (records, when I went).
Boonton
Aug 7 2008 at 10:35am
Troy,
Assume a corrupt school will have bad results. If that is the case then the schools with bad results would show up in the regression as spending a lot more than the schools with good results. The real results were that the schools with bad results spent only a statistically tiny amount more than the schools with good results.
This means either that a corrupt school that wastes money rather than spending it wisely will do just as good on test scores as a pure one or the number of corrupt schools is not statistically significant at any particular income level.
Troy Camplin, Ph.D.
Aug 7 2008 at 1:00pm
YOu didn’t actually read what I said. I said that it may be that spending does affect cores, meaning if a school is corrupt, then the money is not going to the students. It won’t be the small percentage above what the good schools are getting, but that and more money, brining the actual spending per student down considerably. Suppose we have two schools, school A and school B. Both schools spend the same amount on AC maintenance, but school A have a working AC, and school B does not. You are saying that there’s no significant corruption at school B because school A and school B have both spent the same amount of money on AC maintenance. You are talking about statistically insignificant differences, but you’re not taking into consideration how the money is being spent at all. If you can’t seem to understand this, it’s no wonder that so many schools are so corrupt and the poor don’t get educated. In a corrupt school system, the money simply isn’t getting to the students like it is in a non-corrupt school. This is true even if the apparent spending is the same.
Boonton
Aug 8 2008 at 7:42am
If I read you correctly, you’re saying that the corrupt school is wasting its money so it performs worse than expected. It does, however, limit its corruption so its per pupil spending is not too far above average….perhaps to avoid drawing too much attention.
I suppose that could be a possibility. It does essentially support my main assertion that the data does not disprove a relationship between education and money (i.e. to get more of one you need more of the other). It shows that the regression might hide this truth if money is spent on things other than the dependent variable you are testing for.
One way to address this might be to come up with several measures of educational quality. Perhaps math scores, reading scores, a metric of culture like how many books were read in a year and so on. Then take a random sample of schools and try to divide their budgets….’math centered spending’, ‘reading spending’ and so on lumping everything else into ‘other’ or ‘admin’. Then test to see if there’s a relationship between spending and the metrics. Does a poorer school that puts more $ in the library produce more books read per year while another poor school that does supplemental math tutoring scores better on math?
Troy Camplin, Ph.D.
Aug 8 2008 at 10:25am
You might need to do something a little closer to the source of the problem to figure out of corruption is a systematic problem. How is the money being spent? Does it actually get spent on what they claim it is being spent on? When it is claimed that something is done, is it actually done?
Comments are closed.