Biases in Evaluation Scales
By Arnold Kling
Elaborating on a point I raised <a href = "http://www.econlib.org/"here, I am going to make the following conjecture:
In an evaluation scale (e.g, rate this professor on a scale of 1 to 5), the mean evaluation is biased toward the middle.
Is this conjecture false? Is it true, but widely known? If it is true but not published, then someone should formally prove it and submit it to a journal.When someone is asked to respond to a survey with an answer based on a subjective point scale, the response may not be the person’s true best response. The respondent may be reacting to a particular mood, or using a particular interpretation of the question, or making the wrong mark with a pencil.
The average measurement error on points scales is not zero, because the scales are truncated at the endpoints. On a scale of 1 to 5, a person whose true evaluation is 1 can only make an error in one direction–toward a higher number. Similarly, a person whose true evaluation is 5 can only make an error in the other direction–toward a lower number.
If the true group mean is 4.6 out of five, then the measured group mean is likely to be lower, because low-side errors will be included but high-side errors will be truncated. Conversely, if the true group mean is 0.4 out of five, the measured group mean is likely to be higher.
If the scaled evaluation is used as a dependent variable in a regression, the slope of the line will be biased toward zero. For example, suppose that in truth there is a positive relationship between X and Y. If Y’s values are biased toward the middle because of measurement error, there will be fewer observations with high-X, high-Y or low-X, low-Y than there would be the case if Y were measured without error.