Ranking Public School Quality (Badly), Part 2:
Does per pupil spending predict average SAT scores?
8 Jul 2007 in Information Quality
Following up on yesterday's post, in which we pointed out numerous information quality errors in the high school ranking by Christina Settimi and published by Forbes, we decided to address this question using her data.
We were curious about whether there was a relationship between school spending and school performance. Her conclusion was that school spending didn't predict school-average SAT scores, and we objected that her data, model and analysis didn't support her conclusion.
But that does not mean her conclusion is false. It just means you cannot get there her way.
Using Settimi's data, we performed a simple regression analysis attempting to predict school-average SAT scores with per pupil spending. the results are shown in the chart below. There is a slight upward trend but the effect is not statistically significant. The p-value for the independent variable is 0.16. Even if there is a real relationship here, it is extremely weak. Trebling per pupil spending from $5,000 to $15,000 is predicted to increase the average SAT score from 1029 to 1064. .gif)
School-average SAT scores should be normally distributed by design, but per pupil spending is not so residuals in this regression are heteroskedastic. (Note that there are fewer data points on the right side.) That makes our estimate biased. Correcting this can be complicated and we did not want to spend a lot of time beating this dead horse. So we did the next best thing: We experimented with several logarithmic transformations of the dependent variable to obtain something that "looked" normal. It turns out that a base-10 logarithmic transformation is about as good as any. (Base-10 logs are vertically aligned with their dollar-denominated equivalents in the chart above.)
.gif)
But it doesn't make any difference. If there is any simple association between per pupil spending and school-average SAT scores, the relationship is extremely weak and Christina Settimi's ranking exercise wasn't needed to show it.
And there still might be a genuine relationship that can't be detected, either because Settimi's data and methods are wrong (our point from yesterday), or because other variables we haven't controlled for cause the relationship to be hidden. In any case, if average SAT scores are higher in school districts that spend more money, there are other factors involved. Spending more money almost certainly will not raise school-average SAT scores.
From Andrew Berger on 12 July 2007, 20:30
Interesting. I am curious as to why you talk about p-values but then use a r-squared value in your graph. They are different things. Which is why I am curious as to why you just used SAT scores, as the article doesn't say anything about "per pupil spending predicting SAT scores". It just says more spending doesn't mean better schools. Then you say you can reach the same conclusion as her, but not in the same way she does. I couldn't find where she stated how she did it for you to make that comparison. Right now it just seems like you did the same thing as her except you threw in more statistical terms to explain your method and in the end just came to her same conclusion. So after I read this, I felt like I was watching that Staples commercial...the one where everyone is around an office meeting and one guy suggests something and is met with stares and silence. But then the boss says the exact same thing and everyone says it is a great idea. The only difference was he just waved his hands when he said it.
From Richard Belzer on 18 July 2007, 09:30
Andrew,
Thank you for your comment!
P-values are used to summarize the statistical significance of regression coefficients. R^2 is a measure of what proportion of variance in the relationship is explained by the variables included. In this case, per-pupil spending is not statistically significant (the conventional but arbitrary threshold is p<.05), and spending explains 2% of the variance in school-average SAT scores.
Christina Settimi's thesis is that money doesn't buy school quality. Per-pupil expenditure was her sole variable for "money," and school-average SAT scores and graduation rates were her two variables for "school quality." Both "quality" variables are problematic. She acknowledged that graduation rates were not calculated consistently, and she alleged (but did not provide supporting evidence) that guidance counselors steer marginal students toward the "weaker" ACT test. Both would be significant sources of bias. I identified another important bias: School-wide average SAT scores are are pushed down as more marginal students take the exam.
Each bias interferes with the use of graduation rates and school-average SAT scores as measures of quality, even if it could be argued that they are in principle meaningful quality measures. (I explained in my first post why I believe they are not.)
In my second post, I showed that there is no relationship between per-pupil spending and school-average SAT scores. Correlation does not prove causation. Nevertheless, correlation is essential for the purpose to which Settimi used the data. She ranked about 100 schools using graduation rates and school-wide SAT scores as the two quality indicators in her index, and per-pupil expenditures as the sole measure of financial resource inputs.
The regression shows that there is no relationship between per-pupil expenditures and one of her school quality indicators. That means her quality ranking has a significant random component. Is school quality random?
--Richard Belzer