Statistical vs. Material Significance
By Arnold Kling
I appreciate Bryan’s pointer to an article by Alan Gerber and others on personality and ideology. However, the article illustrates what I consider to be a methodological error. That error is to use statistical significance as a metric.
Statistical significance is not a measure of the importance of a relationship. Statistical significance is a measure of the unlikelihood that you got your results solely due to chance sampling error.
In measuring the importance of personality traits in predicting political ideology, I would ask how much of political ideology can be explained by personality. The answer, if you look at the R2 in table 6 is, “not much.” For economic conservatism, in fact, the R2 ranges from .07 to .15. Because the sample sizes are large, this is statistically significant. But from a practical point of view, one would have to describe variation in economic ideology as almost entirely unexplained by these variables.
In statistical research, I believe strongly that you should focus on the practical significance of results. Statistical significance is not practical significance.
For example, suppose that you found that the elasticity of demand for good X was -3.0, but the t-ratio was 1.8, while the elasticity of demand for good Y was -.06 with a t-ratio of 8. The practical significance is that X is elastic while Y is inelastic. But if you were to use the t-ratio as a metric, you would conclude that the demand for Y is more elastic than the demand for X. That would be incorrect.