Another Frustrating Education Study
By Arnold Kling
From Richard Arum and others. They find that employment outcomes of college graduates are positively related to student performance on the Collegiate Learning Assessment (CLA).
Who cares?. As Bryan points out, this sort of study confounds ability with learning, which makes it uninteresting. What would be more compelling would be a finding that students whose CLA scores rose the most during college had the best employment experience. That would come closer to saying that the students learned something, and that what they learned affected employment outcomes.
What adds to the frustration is that the authors had the data. That is, they had the data on growth in CLA. However, they stuffed it into another variable:
Academic engagement/growth is a summary measure including taking courses with reading and writing requirements, hours studying and demonstrated growth on the CLA.
So instead of a variable that comes closer to separating learning from ability, they combined it with other variables that easily could be correlated with ability. Frustrating.
I should add that, in general, creating an index out of variables instead of entering the variables separately is bad practice. You are starting from a situation in which the dependent variable, Y, might be determined by X and Z (assuming linearity) as
Y = aX + bZ
where a and b are unknown coefficients. When you create a “summary measure” that combines X and Z, you are imposing a ratio a/b that is based not on the data but instead on your arbitrary assumptions. Unless you have some strong theoretical or empirical reason to impose a specific ratio (which is very unlikely to be the case here), doing so produces statistically biased results.