
On a recent post, a commenter suggested something that struck me as a perfect example of what I will call Grey’s Law. Grey’s Law comes from an offhanded comment made by the YouTuber CGP Grey in one of his videos, where he said:
The specific comment that brought Grey’s Law to mind was remarking on the education system, where a commenter said “if the goal was better education we would just do what Massachusetts and New Jersey do and avoid what Oklahoma does.” And this seems to make sense at first! After all, if you want to make any system better, wouldn’t you look at examples of systems that are performing well, find out what they are doing, and then simply use their system everywhere else? Certainly, that’s the first thing you’d think of, and it looks sensible, and it would be easy to implement so…oh wait, right, Grey’s Law. So, what’s wrong with this seemingly sensible solution?
At the highest level, it falls into one of the major pitfalls of High Modernist thought. I’ll summarize this pitfall by cribbing from Scott Alexander’s review of Seeing Like a State, where Alexander describes one of the tenants of High Modernism as believing “the solution is the solution. It is universal. The rational design for Moscow is the same as the rational design for Paris is the same as the rational design for Chandigarh, India.” Or, as this commenter would contend, the optimal education system for students in Massachusetts or New Jersey is the optimal education system for all students, everywhere. All we need to do is find out what they’re doing in Massachusetts and just do that everywhere.
But why on earth would we assume this is true? Students aren’t a Standardized Product Unit who all respond to a given education system in the same way – nor are groups of students in different states, towns, or districts. A system that works extremely well in New Jersey might be only moderately successful in Pennsylvania, and completely ineffective for students in the Appalachian region. Even a system that works well in one particular school district might be terribly suited for students in the next district over, or even from one classroom to the next in the very same school. Simply assuming that “we” (whatever “we” is supposed to mean) can just decide what the “right” system is and implement it everywhere handwaves away the enormous variety and complexity of circumstances that exist in different areas and among different students.
As an aside, I see exactly this sort of thinking a lot in my work as a healthcare analytics consultant. Most doctors I’ve worked with (thankfully, not all) often invoke the term “Best Practices” in a manner that almost makes you expect it to followed by the sign of the cross. They think that “we” just need to Determine Best Practices, often by looking at a particular institution that’s having strong success in a given area. From there, we need only Implement Best Practices at their own institution, and they’ll get the same results. And it never, ever works out that way. I wish it did – it would make my work so much easier! It would mean for any given issue for which a hospital needs help, my team and I would just need to find an effective solution once, and then for every future job, we can just implement the proper tool or system at the new institution and get equally good results.
Alas, reality is not that simple or that simplistic. Every institution has different constraints, different patient populations, different resources – even the personalities of the medical staff can make a huge difference on how effective solutions are from place to place. It takes a good deal of work getting familiar with all the local circumstances to work out an effective solution – and that solution, sadly, won’t be applicable on the next job.
I also saw this iteration of Grey’s Law play out frequently in my time in the military. Most of the time, any given servicemember is at a given unit for about three years before getting orders to the next unit. During your time at a given unit, you’d almost certainly see the commanding officer replaced, along with the Sergeant Major and other lower-level officers and enlisted leaders. You always hoped that the new bigwigs would be one of the good ones. And one of the most reliable signals we learned for predicting which ones would be good or bad was their attitude on how their experience at their previous unit should inform what they do at the current unit. A universally bad sign was when they said something to the effect of “Back at my last unit, we did things in such and such a way, and everything worked great there. So going forward, we’re going to do it in such and such a way here too.” Commanders with that attitude were, in practice, terrible, ineffective commanders whose methods, once implemented, were a drag on unit performance.
Luckily, in practice, there was a way around such people, to prevent them from hampering mission effectiveness too much – a way described by James C. Scott in his book Two Cheers for Anarchism:
In the same way, when we had a new commander who was drunk on the idea that what worked well at his last unit must also work well here, the result, in practice, was “a practiced and judicious disregard of many regulations” set out by that commander in favor of “the effective informal understandings and improvisations outside those rules.” This was not done flagrantly of course – it was a subtle understanding. But there was a sort of satisfied amusement among us to see a planner utterly convinced their plan was working, and equally unaware that the plan only appeared to be working to them because people knew better than to actually follow the plan.
READER COMMENTS
Richard W Fulmer
Oct 31 2023 at 3:52pm
There is also the question of whether schools in Massachusetts and New Jersey really are better than those in Oklahoma. The answer may depend on what researchers chose to measure. For instance, people often claim that – based on standardized test scores – Texas schools are worse than those in Minnesota. However, breaking the test scores down by race, we find that we’ve been fooled by Simpson’ Paradox. White, Hispanic, and Black students in Texas all do better than their Minnesotan counterparts. The “confounding variable” is that Texas has far more minority students than Minnesota.
David Seltzer
Oct 31 2023 at 4:51pm
Kevin, good stuff. Especially about the military. I served in VP40 1962 1963. South China Sea. The Leading Chief was a 25 year vet. He was older than most of the pilots who were really pretty cool with their enlisted flight crews. Chief Steele understood the tenor of things and was as much a mentor as an official. I was first mech in my flight crew. We knew the informal rules and did our jobs well. The squadron showed great trust in our abilities even though many of us were in our early twenties. Chief Steele retired and was replace by a an E8 who came from years aboard ship. He was rigid and by the book. He viewed the Airdale Navy as undisciplined and sloppy. It just didn’t work. While the officers couldn’t say much, privately they commiserated with us. The man single handedly undermined moral.
steve
Oct 31 2023 at 5:23pm
Having been both enlisted and an officer i have been on both sides of that. Had good and bad officers, good and bad enlisted. Helps you to learn creative problem solving. At my current place we talk a lot about culture. We work pretty hard to maintain it, even firing high RVU producing docs when they dont adapt. We have a good system of mentors and pretty open communication among leadership so that if I have an issue with a new doc or new head nurse somewhere I can talk with their leadership and someone will mentor them so that work within the norms of behavior we expect. Works pretty well though If I am honest I would say that if someone is especially clinically competent they get more latitude.
Steve
David Seltzer
Oct 31 2023 at 5:55pm
Good points HM Steve. (I’m assuming you were a Corpsman in the Navy). VP40 was a Patrol Squadron hunting sub activity in the South China Sea; dropped sonobuoys in the water. Our anti-sub tech (AX1) was something of a savant. He often referred to our PPC, patrol plane commander, by his first name which, by the way, was Steve.
steve
Oct 31 2023 at 5:04pm
I was trying to be brief as I was running late for an appointment, but would this not also apply to school choice/vouchers? As I am sure you know there are lots of studies on this issue and the results are mixed. More recent studies seem to be favoring choice/vouchers by a smallish margin. Does that mean that they worked in those areas but might not in others? Anyway, when I write about this in more detail what I note is that Massachusetts has had much better results and for a long time. Oklahoma has had poor results and for a long time ie both of these states have established track records. Does this mean we can or want to import what they do in total? No, but it does mean, I think, that you at least want to look at what they do and see if you can learn something that might help.
Best practices? Not a phrase I hear much where I work. Evidence Based Medicine is the code phrase. I think that we maybe once when that was an in style concept tried to import whole a practice from another institution. Fortunately, my boss helped to get it aborted as it was kind of awful for us. However, we still steal lots of ideas from other institutions. I have lead groups or gone on my own to other places and gained a lot of valuable insights, sometimes just learning what not to do. So in my managerial/leadership experience I would say you are foolishly arrogant if you dont think you can learn from others and pretty stupid if you want to try to copy other programs without figuring out how to adapt it to your own circumstances (and also get buy in).
Every year I hire new docs. I hire new midlevels every month or two. They all want to turn our place into the place where they used to work. It’s what they know and are comfortable with. What I tell them all is that unless they think they see something awful I dont want to hear anything about where they came from for at least 6 months. After that if they think they see something we could do better they can tell me but when they do I want a brief written plan about how they would do it, especially how it would affect other staff as changing what we do would affect others. As a result I never hear about the trivial stuff. I get serious suggestions and we adopt or use them to modify what we do maybe 1/3 of the time. Oh, I also tell them they have to be willing to help work to institute their preferred changes.
Steve
robc
Nov 1 2023 at 1:07pm
Absolutely. And it also supports school choice/vouchers.
I think part of the reason that most studies of charter/vouchers shows marginal, at best, improvement is that parents are bad at choosing the best schools for their kids. I include myself in that. I chose a charter Montessori school for my daughter. Is Montessori the best choice for her, or would another school be better? I don’t know, I don’t have identical sextuplets that I could send to 6 different schools, see which works best and then send all of the next set of coincidentally identical sextuplets (identical to the first set even!) to the best result.
And the best choice for the plurality of kids may or may not be the best choice for my kid. I could only look at what is out there, think about my daughter’s particular situation, and choose what I think is best.
And school choice is the only way for that to happen.
steve
Nov 1 2023 at 4:00pm
“And school choice is the only way for that to happen.”
So carried out at scale what we see is some parents choosing better schools and some choosing worse and on an average we dont see much change. However, if your goal was to improve your child’s education why not improve the quality of all the schools? Look at systems that are performing well. Are there aspects you could integrate into your system? Look at schools doing poorly. How much of what they do is copied by your schools? Remember er that the studies you look at generally cover a small number of schools for a shortish period. You can look at states to get much larger populations over much longer time periods so you are less likely to be looking at a random effect.
Coincidentally, a nice review article on the portability of medications/therapies. The people who think it impossible to import ideas from other states or countries think we can just import drugs without any issues. Not always true.
https://www.healthcare-economist.com/2023/10/31/transportability-of-comparative-effectiveness-evidence-across-countries/
Steve
robc
Nov 1 2023 at 5:57pm
Because I am not the education czar. And I am not sure I know how to do that. But I think a lot of schools getting to experiment and a lot of parents making choices they think are best will improve things IN THE LONG RUN. We don’t have enough school choice yet. Charter/voucher everywhere and give it a few generations to work out the kinks and I think the average quality of education will be noticeably improved.
No, because, once again, I am not education czar. But also yes, I plan on homeschooling in 7th and 8th grade, so I am sure I will integrate lots of things that I think work well. But that only effects one student and is lost in the noise.
steve
Nov 2 2023 at 9:39am
You are not the education czar but you support school choice/vouchers. Why not just support schools looking at what is done elsewhere to try to improve? You dont have to be czar to support better education. What you are supporting is school choice for the sake of having choice, knowing it has little effect on education quality. That’s your choice to make but I think we need to accept that better education is not the primary goal of choice.
Steve
robc
Nov 2 2023 at 10:32am
I think better education, as the parents define, for the individual child, is the primary goal of choice.
The primary problem with measuring the results, is that “as the parents define” part. The seondary problem is the “individual child” part. Aggregations lose a lot of detail.
And yes, part of my support of school choice is choice for choices sake. But I really think it will lead to better education over the course of multiple generations. Well, I think it leads to better education ON THE INDIVIDUAL LEVEL for some individuals immediately, but in the aggregate I think it will take a long time to get the separation that will show significant improvement.
robc
Nov 2 2023 at 10:38am
Let me give an example of the parental measure problem.
A hypothetical parent with a voucher is choosing a school for their kid. They say, “I don’t really care that much about educational attainment, I just want a strong christian/moslem/jewish/etc indoctrination.”
I have yet to see a study of vouchers that attempts to measure that. If they did, they may find that educational attainment was the same, or slightly diminished, but what the parents are actually wanting increased significantly (it should, as the public schools shouldnt be doing that at all, of course).
nobody.really
Oct 31 2023 at 5:20pm
“Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.” H.L. Menken, “The Divine Afflatus,” in the New York Evening Mail (November 16, 1917)
Floccina
Nov 1 2023 at 6:23pm
Yes, they should all attract a lot of highly educated, intelligent people to their states and drive away the less educated, less intelligent people with building restrictions.
Of course my state, Florida, has the best schools when broken down by race and Hispanic identity to avoid Simpson’s paradox. See here. How did Florida do it, by having more immigrants come from the cream of the crop of Latin America.
The above of course is tongue in cheek.
Knut P. Heen
Nov 2 2023 at 8:45am
Should a poor person buy a helicopter because rich people buy helicopters?
The big problem with education is that people don’t understand that they have to educate themselves. Teachers, professors, etc. can only give you ideas about where you can find various material to study. It is like serving food. The cook cannot eat the food for you. It is idiotic to measure the performance of the cook by the BMI of the clientele.
Comments are closed.