The Owl Newsletter

Responses to Newspaper Accounts of the Statistical Uncertainty of the API Call Reporters to Task for Sloppy Stats

ISSUE 51 | SEPTEMBER 10, 2002

[This free e-mail newsletter about school information, accountability and the public is provided by School Wise Press. To add a colleague's name to the distribution, please send us their names and e-mail addresses to: stever@schoolwisepress.com. If you'd rather not receive this, simply notify us by phone at (800) 247-8443, or by e-mail, including the word "unsubscribe" in the subject line of your message.]

In a previous newsletter, I reported on the ORANGE COUNTY REGISTER's challenge to the API and the award program it's based upon. Since then, I've received comments from readers (some of them policy leaders) that are well worth sharing. Some of the REGISTER's findings have been slammed, particularly their claim that as much as one-third of API awards have been handed out in error.

In brief, I believe the REGISTER's story was a mish-mash of right and wrong conclusions. This mess is well worth sorting out.

First, Kerry Mazzoni, the governor's secretary of education, wrote to the ORANGE COUNTY REGISTER in defense of the API's reliability on August 12.

Like every system of measurement ever devised, the API has a margin of error. But Stanford Prof. Ed Haertel, who chaired the committee that devised the API, notes that the API, with an accuracy of 98 percent, is the most accurate state assessment system in the nation. He comments that, Because of the mass of data collected, the API is, in fact, much more reliable than the Scholastic Aptitude Test (SAT).

Second, Stanford professor Ed Haertel, who was also co-chair of the technical design group of the API, rebutted the ORANGE COUNTY REGISTER's claims of broad uncertainty in awarding bonus funds. In an August 12 letter-to-the-editor, he wrote:

The bald assertion in the lead paragraph of your Sunday article, that the state has no idea whether one-third of the schools receiving cash awards actually earned the money, is flat wrong. For each and every school in the state, the best possible decisions are made using available data in the manner prescribed by law, following uniform, carefully developed procedures. There is uncertainty attached to ANY test score. All information is imperfect. We use tests to improve the quality of decisions, because we know better decisions can be made with them than without them.

Third, Stanford professor David Rogosa wrote two research briefs for the CDE, affirming the REGISTER's identification of error margins in the API, but disagreeing with the size and consequences of those error margins. He gets into the heart of the statistical flaws in the REGISTER's story, and argues that they exaggerated the extent of the APIs error margins by a factor of ten. You can best read his comments in full at the two links below.

For good measure, I also attended a conference at Stanford last Thursday, hosted by the Education Writers Association, at which Prof. Rogosa and the reporting team from the REGISTER presented their respective findings. Here's the heart of their disagreement.

1. They disagree about the size of the error margin. The OCR reporters believe it is ten times the size that Prof. Rogosa does.

2. They disagree about the consequences of this in determining which schools are rewarded or sanctioned.

But more important is their area of agreement. Indeed, its clear that measurement error is unavoidable. The SAT-9 and the CST both capture this error margin. The API, as an index based upon both tests, captures it as well.

Both agree that the CDE should have footnoted all test score and API releases with statements of their statistical assumptions, including error margins and confidence intervals. Political polls do this all the time. The CDE should have done the same.

OWL ARCHIVE | BACK TO NEWSLETTER REGISTRATION PAGE