K DeRosa draws the chart that I proposed.
Here is the chart for 497 school districts in Pennsylvania for 2005. On the X axis I have the percentage of students in the district that do not receive free and reduced meals. On the Y-axis I have the pass rate for PA’s PSSA 11th grade exam (math and reading).
…The overperforming schools [the ones above the regression line] actually spent less on average. Go figure.
I’m supposed to get the Maryland data from a friend today.
READER COMMENTS
liberty
Aug 1 2008 at 8:58am
Cause and Effect. I think you may have the order backward.
wintercow20
Aug 1 2008 at 9:36am
Arnold,
Here is my quick stab at a chart for New York State for 2005.
-Mike
Chuck
Aug 1 2008 at 9:45am
It would be nice to know the median and range of expenditure per pupil in addition to the average. Or at least a max and min from each group to get a feel for if there are flyers in the data skewing the averages.
I think this analysis needs to also consider things like population density.
My brother in law lives in South Dakota, and a big deal with education funding there is how you get kids in rural areas to school – do your drive them 100 miles to a single facility to have low overhead with high transportation costs, or do you have many very small schools with high student-teacher ratios, etc.
To account for this kind of issue, I propose comparing the school districts total population and total areas? (Or maybe the medians. Ideally, one would make a box-and-whiskers plot of the spending per pupil data for the two groups…)
kderosa
Aug 1 2008 at 9:53am
Instructional expenditures show the same relationship as total expenditures.
Boonton
Aug 1 2008 at 10:45am
From the site:
For school districts falling below the regression line the average total expenditures was $11,417.
For school districts falling above the regression line the average total expenditures was $11,214.
The $200 less in spending above average schools do is probably not statistically significant. That would seem to cut against the conservative idea that bad schools are just money hogs where more spent equals worse results. But is this telling us anything real? I’m not sure?
It seems to be saying all the schools are spending about the same. If you took a set of schools and ramped up their spending to $22K would you push them above the regression line? Would the push be enough to justify such a radical increase in spending? We can’t really tell from this data.
What does seem to be, though, is that more poverty makes it harder to run a school where a lot of kids pass the math test. Given a certain level of poverty, the above average schools seem to be able to spend no more than the below average ones.
To really learn what’s going on here you’d need to start doing some serious comparisions of what goes on in the above average versus below average. One possibility might be that the above average ones really do cost more but they are getting a price cut. For example, imagine a community that has a lot of professionals that volunteer to tutor kids for free at a nearby church or clinic. The school district would benefit from those additional resources even though you’d never locate them in their budget. A school district doing the exact same thing in a different community that was exactly the same except for the volunteers would see a lower success rate. If that school wanted to achieve what its sister school did it would have to spend more because its community (for whatever reason) lacks the volunteer tutors.
Boonton
Aug 1 2008 at 11:09am
Another thing that makes this tricky, how good is passing the math test as a measure of school quality?
Imagine we were talking about food. If we were talking about a country deep in famine, calories consumed per day would be a good metric. But for a country like ours, that’s a horrible metric. It would imply that people who hit McDonalds every day are getting better food than those who dine at four star restruants!
Now try to imagine a good school in a good district. If you get 90% of the kids to pass the test perhaps the test stops being a useful metric. The people of that district no longer care about increasing the pass rate on the test and start caring about other things like does the school have rich culture resources (a well done school play, foreign exchange programs, good sports programs etc.). The type of quality improvement they do is no longer related to passing the test.
To go back to the food example…. McDonald’s may improve the quality of its burger by adding two more meat patties making it double or triple in calories. The fine French Rest., though, may try to improve its quality by using fresher ingredients. Calories may go up only a little or even drop a bit but the cost will go up and the critics won’t complain that they aren’t getting more for their money!
Tom
Aug 3 2008 at 11:02pm
I’m a little confused here. Isn’t it obvious that, once you’ve controlled for the relative wealth of a school district, the unexplained variance in test scores is going to be due to factors besides spending per student? Although spending and wealth may not be exactly the same, they’re correlated enough that including one factor in your regression should make the residuals uncorrelated with the other.
I’m having trouble understanding how this result proves that spending doesn’t matter.
Tracy W
Aug 4 2008 at 6:48am
Isn’t it obvious that, once you’ve controlled for the relative wealth of a school district, the unexplained variance in test scores is going to be due to factors besides spending per student?
But no one controlled for the relative wealth of the school district. The control in KdeRosa’s work was percentage of students that receive free and reduced meals (for presentational reasons he showed it the other way round).
So what this test shows is that, controlling for the proportion of poor students (as measured by those receiving free and reduced meals), the unexplained variance in student performance was not due to variance in spending per students.
I’m having trouble understanding how this result proves that spending doesn’t matter.
Okay, in steps.
1. We want to see if school funding matters for educational achievement.
2. But the difficulty is that schools that have lots of poor kids are handicapped in passing tests, as the kids tend to start school ill-prepared. This will be true regardless of how much money you chuck at those schools. And we do find statistically that schools with lots of poor kids keep showing up with low passing rates. This confuses any interpretation
3. So we control for the number of poor kids, using some proxy, such as percentage receiving free and reduced meals.
4. Positive signs are easier to interpret, so we invert the percentage of kids receiving free and reduced meals to the perecentage not receiving free and reduced meals.
5. So we can say that, holding the percentage of kids recieving free and reduced meals constant, more funding is not correlated with more passing the test.
Now, if school spending did increase the percentage of kids passing the test, independently of the number of disadvantaged kids at the school, then we would expect to see the opposite effect.
So the lack of correlation is an indication that school spending, at least at levels in America, doesn’t matter.
Tracy W
Aug 4 2008 at 6:54am
Another thing that makes this tricky, how good is passing the math test as a measure of school quality?
Imagine we were talking about food. If we were talking about a country deep in famine, calories consumed per day would be a good metric. But for a country like ours, that’s a horrible metric. It would imply that people who hit McDonalds every day are getting better food than those who dine at four star restruants!
I’m not sure of the point of this. Are you arguing that too much knowledge of maths is like an excessive number of calories? Classically, people who are better educated tend to be healthier, richer and generally better off. Meanwhile, people who are obese tend to be worse off.
http://www.npc.umich.edu/news/events/healtheffects_agenda/cutler.pdf
I find your hypothesis hard to believe.
Tracy W
Aug 4 2008 at 7:09am
To really learn what’s going on here you’d need to start doing some serious comparisions of what goes on in the above average versus below average.
This has been done over the years. Here’s a synthesis from 1995. http://www.nwrel.org/scpd/esp/esp95toc.html
Typically, what goes on at high-performing schools (once you’ve controlled for student’s socio-economic status) is things like:
– Teachers Provide Clear and Focused Instruction.
– Administrators and Teachers Base Curriculum Planning on Clear Goals and Objectives
– Administrators and Teachers Assure That School Time is Use for Learning
– Administrators and Other Leaders Provide Incentives, Recognition, and Rewards to Build Strong Staff Motivation
– District Leaders and Staff Monitor Student Progress Regularly
(These are only examples from a long list).
You look at a list like that, and it makes sense that there’s no particular relationship between school spending and school performance once you are above a minimum level. More money will not make a school organise its day to use time effectively or make teachers use clear examples.
Comments are closed.