Login

Home > Blog

Ranking Public School Quality (Badly):
Forbes' has problems with information quality

7 Jul 2007 in Information Quality

School quality is something about which everyone seems to have an opinion, and a compulsion to rank. For example, every year there is a kerfuffle about the college rankings published by US News and World Report. This year the ranks of college administrators refusing to provide data to US News has grown because opponents have become more organized.

Elementary and secondary school rankings are the latest trend, and the number of rankings can be expected to grow as more statistics are made available. What do these rankings actually mean?

Forbes' Christina Settimi reviewed 97 public school districts to rate "the best and worst school districts for the buck." But her methodology has too many fatal flaws to make the results interpretable. It offers an excellent case study in how not to provide statistical information. A federal agency armed with the same data could not have disseminated the Forbes ranking without violating the federal Information Quality Act and associated implementing guidelines:

• The Forbes ranking has no value for its intended purpose (that is, it lacks "utility").
• The Forbes ranking is a biased comparison (that is, it is not substantively or presentationally "objective").
• The Forbes ranking is neither transparent nor reproducible (that is, it was not produced in a procedurally objective manner).

Each one of these errors is fatal for information dissemination by the federal government. Contrary to the adage, the Forbes ranking is not good enough for government work.

WHAT IS THE SAMPLE FRAME?

There are many more than 97 school districts in the US. Why did Settimi choose these 97?

Using research provided by the Tax Foundation, a nonpartisan tax research group based in Washington, D.C., Forbes began with a list of the 775 counties in the country with populations greater than 65,000 that had the highest average property taxes. From this list we isolated the 97 counties where more than 50% of per-pupil spending contributions comes from property taxes.

School districts funded primarily through means other than property taxes are excluded. The reason for this exclusion is nowhere made clear. Any "bang for the buck" ranking that excludes thousands of school districts for arbitrary reasons has no value. When a sample frame is unrepresentative of the population, results from the sample cannot be applied to the population.

So when Settimi says "Marin County, Calif., provides the best bang for the buck," the best that can be said is that Marin ranked highest of the 97 school districts Settimi considered. As we show below, even that statement has to be presented with huge caveats. Settimi ranked it tops on criteria that she chose but did not reveal.

WHAT IS MEASURED?

Settimi's ranking purports to link two outputs (college entrance examination scores and high school graduation rates) with one input (per pupil spending) and one intermediate snapshot (proportion of high school students taking college entrance examination tests). These outputs are not the same as outcomes; an objective measure of school quality must control for the academic quality of inputs. A school that begins with 90th percentile students and raises them to the 91st percentile achieves very little, whereas a school that begins with 50th percentile students and raises them to the 80th percentile achieves a lot. The latter school is properly judged to have produced high quality, whereas the former school merely had high quality inputs and did no harm.

Which high schools are included in Settimi's calculations? For Alexandria VA (rank = 97), there is only one high school. But for Marin County CA (rank = 1), this cannot be determined. There are four high school districts in Marin County, with seven major high schools and several specialty and continuation high schools. These four districts are very different. For example, date available from the U.S. Department of Education's National Center for Educational Statistics shows that less than 5% of pupils in the Tamalpais High School district are eligible for free school lunches -- a commonly used proxy for economic disadvantage. But in neighboring San Rafael City High School District, 27% of pupils are eligible for free lunches. Which of these districts Is Settimi using for her comparison? She does not say.

High schools in Marin County vary. Whites and Hispanics each comprise 41% and 45% , respectively, of the students at San Rafael HS. At the other five major high schools, however, whites are 83% (Redwood), 72% (Tamalpais), 67% (Novato), 78% (San Marin), and 65% (Tomales). Hispanics are 4% (Redwood), 6% (Tamalpais), 20% (Novato), 12% (San Marin), and 31% (Tomales). Black students are a curiosity; the highest percentage at any Marin County high school is 7% (Tamalpais).

T.C. Williams HS in Alexandria VA is very different. It is 26% white, 43% black, and 24% Hispanic. 23% of its students are eligible for free school lunches. Of 7.051 students who attended the seven  major Marin County high schools, only 1,228 (17%) attended a school with a similar proportion of economically disadvantaged kids. Of these seven schools in Marin County, T.C. Williams is demographically most like San Rafael HS and its per-pupil spending is similar ($15,029 vs. $14,698). Whereas San Rafael is evenly balanced between white and Hispanic students, T.C. Williams has about twice as many black students as whites and Hispanics. It is culturally a completely different place. Yet, on some other conventional  measures of  "school quality," T.C. Williams fares better. Its pupil-teacher ratio (15.6) is a lot lower than that of San Rafael HS (21.4).

When the data for T.C. Williams HS are disaggregated, racial differences in educational performance become obvious:

• 2005-06 English proficiency: 93% (white), 71% (black), 71% (Hispanic)
• 2005-06 mathematics proficiency: 89% (white), 67% (black), 75% (Hispanic)
• 2005-06 science proficiency: 95% (white), 64% (black), 64% (Hispanic)

Analogous racial differences in student performance are evident in data from San Rafael HS found on California Standardized Testing and Reporting (STAR):

• 2006 English proficiency in 11th grade: 74% (white), 16% (Hispanic)
• 2006 mathematics proficiency
  
  • Grades 9-11, measured in 12th grade: 61% (white), (Hispanic)
  • Algebra I, measured in 9th and 12th grades: 22% and 17% (white), NA% and 3% (Hispanic)
  • Geometry, measured in 9th and 12th grades: 73% and 39% (white), NA% and 13% (Hispanic)
  • Algebra II, measured in 10th and 12th grades: 51% and 35% (white), NA% and 12% (Hispanic)
• 2006 science proficiency
  
  • 10th grade life sciences: 63% (white), 12% (Hispanic)
  • Biological/life sciences, measured in 10th and 12th grades: 63% and 52% (white), 10% and 7% (Hispanic)
  • Chemistry: 26% (white), (Hispanic), 1%
  • Physics: 48% (white), NA%(Hispanic)

[NA% means too few students were tested to report summary percent passing without revealing student identities.)

More detailed examination likely would reveal that high-performing students of any race or ethnic heritage are enrolled in advance placement classes where they encounter more demanding standards, similarly high-performing students, and possibly different teachers. In short, they may go to the same physical facility for school but their school experience is fundamentally different.

ARE THE DATA VALID?

Settimi suggests that they are not. Regarding high school graduation rates, she says they are not measured uniformly and she did not even attempt to validate the data she used:

Graduation rates can be inflated by choosing not to include some students in the calculation. SAT and ACT scores are biased measures of average student performance because not all students take these exams. The more marginally qualified students take them, the lower will be the school's average. So one way to increase a school's average is to discourage marginal students from taking them. Interestingly, the proportion of students taking the SAT was higher in 97th-ranked Alexandria (65%) than in 1st-ranked Marin County (60%).

More representative data on student performance are available from NCES' National Assessment of Educational Progress, California's STAR system, and Virginia's School Report Cards. Settimi doesn't reveal why she used school district-wide SAT scores, which are clearly subject to selection bias.

WHAT ABOUT SCHOOL EXPENDITURE DATA?

Settimi's data on school spending are similarly problematic. She says Alexandria VA spent $15,029 i("Fiscal Year 2004, adjusted for the cost of living in the county's associated Metropolitan Statisical [sic] Area") -- the same figure reported by NCES. But she also says Marin County spent just $6,579 per pupil. NCES says the four high school districts spent $8,311 (Novato Unified), $14,698 (San Rafael City), $19,417 (Shoreline Unified), and $21,254 (Tamalpais Union). Not only do none of these figures match, they vary by more than 250%.

District-wide figures also disguise variations in funding within schools belonging to the same district. Access to within-district data is surely very difficult, and hampered by accounting systems that are not transparent, logically to prevent such differences from being detected.   

Finally,  these figures capture only school expenditures that are "on the books." Schools often raise considerable sums from other sources, notably parents, to fund special programs. Yet the fact that parents are willing to donate to fund special programs means that district-wide average funding is a poor proxy for actual financial inputs -- and that's even if  the public figures are accurate.

ARE THE CONCLUSIONS VALID?

Notwithstanding all of these problems, Settimi is not deterred from drawing sweeping conclusions:

The caveats to our methodology notwithstanding, our study shows that there are big differences in the quality of education relative to spending among counties and is further proof that money is not the only--or perhaps even the most important--factor when it comes to the quality of education.

But her study shows no such thing. She did not directly or indirectly measure school quality. She included just one input -- per pupil spending -- and apparently did that badly. She ignored many other important inputs, such as IQ, teacher qualifications and training, the availability of college prep courses, parental involvement, and socioeconomic status. She tried to link this single input measure to a pair of outputs -- graduation rates and SAT/ACT scores -- without regard for whether the data were any good. (She acknowledged that graduation rates probably were biased but did nothing to validate them, and she ignored the dominant source of bias in average SAT/ACT scores: the number of marginally qualified students who take them.)

The result is that the rankings for Marin County and Alexandria could be reversed even using Settimi's dubious methodology, just if the underlying data were corrected. Marin County fares especially well when per pupil expenditures are underestimated by, say, a factor of 2.

FATAL DEFECTS IN INFORMATION QUALITY

If for some reason the federal Department of Education wanted to rank school districts on cost-effectiveness, it could not legally disseminate a product as deeply flawed as the one produced by Christina Settimi and published by Forbes. The reason is that this ranking violates information quality standards that apply to EdD and which Forbes ought to consider applying to its own work products.

• Utility. Statistical information intended to influence decision-making must be useful for the purpose. Settimi's ranking is intended to influence perceptions of the relative quality of high school districts as a function of how much they spend. It has no value for this purpose because it uses severely limited data, some of which can be shown to be wrong, and relies on a ranking procedure that utilizes arbitrary weights and not statistical analysis.
• Integrity. Settimi knew her data on graduation rates was suspect but made no effort to validate it.
• Objectivity. Biased data prevent the analyst from achieving objectivity except by chance. Settimi herself suspects that her graduation rate data are biased, but she is oblivious to the dominant source of bias in average SAT/ACT scores. It's not the subtle effect of school counselors guiding students toward the "easier" ACT, but the fact that any school-wide average is reduced when a large number of marginal students take either test.
• Transparency and reproducibility. Forbes prints a table of data but does not make Settimi's methodology transparent. Her sources are listed as "Tax Foundation, Economy.com, counties, school district officials, high school administrators, SchoolMatters". The inability of a qualified independent analyst to reproduce the work means that it is presumptively substandard.

A federal agency cannot legally disseminate a nominally statistical report that fails all of these information quality standards. Reports that fail these standards should  not be taken seriously. Unfortunately, that's the only standard that Christina Settimi's study actually meets.

 

---

 

Selected Educational Statistics on Marin County (CA) and Alexandria (VA) High Schools
High School (County) [District]	Pupils (Pupil/ Teacher Ratio)	Number (%) Free Lunch Eligible	Demographic Data for School Reported by NCES			2000 Census Demographic Data for School District: Numbers and % Reported by NCES			Per Pupil Expend. Reported by NCES	Per Pupil Expend Reported by Forbes
Redwood (Marin) [Tamalpais Union]	1,467 (18.1)	27 (2%)	White Black Hispanic	1,211  29 60	83% 2% 4%	*Total White Black Hispanic	22,737 19,469 611 1,253	100% 86% 3% 1%	$21,254	$6,579
Tamalpais (Marin) [Tamalpais Union]	1,132 (18.1)	58 (5%)	White Black Hispanic	820 78 73	72% 7% 6%
San Andreas (Marin) [Tamalpais Union]	135 (16.9)	6 (4%)	White Black Hispanic	106 11 9	79% 8% 7%
San Rafael HS (Marin) [San Rafael]	1,008 (21.4)	268 (27%)	White Black Hispanic	417 33 450	41%  3% 45%	*Total White Black Hispanic	13,924 9,664 412 4,016	100% 69% 3% 29%	$14,698
Madrone HS (Marin) [San Rafael]	47 (15.7)	0 (0%)	White Black Hispanic	10 6 27	21% 13% 57%
Terra Linda (Marin) [San Rafael]	1,027 (19.2)	70 (7%)	White Black Hispanic	742 31 158	72% 3% 15%
Juvenile Hall (Marin) [Marin Co Ofc of Ed]	177 (44.2)	40 (23%)	White Black Hispanic	81 24 63	46% 14% 36%	NA			NA
Phoenix Academy (Marin) [Marin Co Ofc of Ed]	30 (1)	9 (30%)	White Black Hispanic	18 1 8	60% 3% 27%
Marine County Special Education (Marin) [Marin Co Ofc of Ed]	249 (5.9)	51 20%)	White Black Hispanic	149 26 44	60% 10% 18%
Marin Oaks (Marin) [Novato Unified]	65 (15.9)	6 (9%)	White Black Hispanic	50 4 9	77% 6% 14%	*Total White Black Hispanic	12,540 9,814 266 2,119	100% 78% 2% 17%	$8,311
Marin School of Arts And Technology (Marin) [Novato Unified]	178 (17.8)	4 (2%)	White Black Hispanic	132 3 25	74% 2% 14%
Nova Education Center (Marin) [Novato Unified]	97 (22.0)	1 (1%)	White Black Hispanic	62 0 8	64% 0% 8%
Novato (Marin) [Novato Unified]	1,146 (24.8)	141 (12%)	White Black Hispanic	769 46 230	67% 4% 20%
San Marin (Marin) [Novato Unified]	1,051 (24.8)	74 (7%)	White Black Hispanic	820 27 121	78% 3% 12%
Shoreline HS (Marin) [Shoreline Unified]	7 (-1)**	2 (29%)	White Black Hispanic	5 0 2	71% 0% 29%	*Total White Black Hispanic	1,344 1,016 13 402	100% 76% 0% 30%	$19,417
Shoreline Independent Study (Marin) [Shoreline Unified]	3 (-1.0)*	NA	White Black Hispanic	3 0 0	100% 0% 0%
Tomales(Marin) [Shoreline Unified]	220 (12.4)	56 (25%)	White Black Hispanic	144 0 69	65% 0% 31%
TC Williams (Alexandria) [Alexandria]	2,859 (15.6)	659 (23%)	White Black Hispanic	740 1,219 692	26% 43% 24%	*Total White Black Hispanic	21,537 9,363 7,006 5,031	100% 43% 33% 23%	$15,029	$15,029
Source: NCES except where noted. Notes: * 2000 Census Data: Total population 18 years of age or younger ** As reported.

 

---

 

|
"Best And Worst School Districts For The Buck"
(According to Forbes.com)

Contact | © 2008 Neutral Source | About