The API is Not as Precise as Needed to Defend Awards and Sanctions

ISSUE 50 | AUGUST 21, 2002

[This free e-mail newsletter about school information, accountability and the public is provided by School Wise Press. To add a colleague's name to the distribution, please send us their names and e-mail addresses to: stever@schoolwisepress.com. If you'd rather not receive this, simply notify us by phone at (415) 432-7800, or by e-mail, including the word "unsubscribe" in the subject line of your message.]

The API data has hit the fan, and the mess has splattered all over the schoolhouse walls. What had been a serviceable indicator of schools' test score results has fallen from grace largely because its creators lacked candor.

Because the API was the basis upon which the governor's accountability system compared schools, people expected it to be rock solid, unerring, and exact. Both its fans and its critics shared this assumption. So when the Orange County Register broke a story on August 11 about a 20-point error margin in the API, the myth of its precision began to unravel.

As any college sophomore in an introductory stats class learns, statistical measures have limits. They contain unavoidable flaws: measurement errors, sampling bias, the standard error which accounts for uncertainty, and more. Political polls and surveys, for example, usually declare these factors in a footnote that might read, Results are accurate within plus or minus 3 percentage points, at a 95 percent confidence interval.

In fact, Harcourt's SAT-9 student report which is mailed home to parents includes a visual treatment of this error margin for added emphasis. Scores are graphed as bands rather than as points. This shows that a student who scores at the 65th national percentile rank might score between the 57th NPR and the 73rd NPR if the student took the test again. (See detail).

But the API's equivalent declaration has been buried deep within the technical notes. Only in July did the CDE release the report of statistician David Rogosa of Stanford University titled, "Plan and Preview for API Accuracy Reports."

Here Mr. Rogosa examines the chances that the API will result in a false positive (awarding API bonus funds to a school that did not make statistically significant progress) or a false negative. With obvious concern, he notes that for a school with three subgroups, and an API score in the middle deciles, that a false positive or negative is possible 10 percent of the time.

Where was this footnote when the API scores were initially released in 1999? Why wasn't Mr. Rogosa asked to clarify the limits of the API's reliability in November 2000 when his first interpretive notes were published by the CDE? As a scholar, Mr. Rogosa knew that the API was not a metal ruler. The CDE staff also must have known this. But rather than trust that the public would understand and accept a shade of imprecision in the API, the CDE's communications staff omitted the footnotes, and left the public with the impression that the API was flawless.

The consequences of the imprecision of the API are indeed startling. They include:

• 35 percent of schools receiving bonus funds showed year to year gains that could have been due to chance
• Schools that might have deserved bonus funds didn't get them
• Many schools are on the state's list of underperforming schools and the federal governments watch list unfairly
• Diverse schools with more subgroups have a harder time meeting their growth targets than less diverse schools
• The smaller the school, the more likely it was to meet its growth targets.

The four-part series appeared in the Orange County Register, and was picked up by the Associated Press. The links to each portion of the series appear below.

Test scores unreliable (August 11)

Students exclusion brings frustration (August 11)

Rules hurt diverse schools (August 12)

Awards ignore key factors (August 13)

The Owl will follow-up with more coverage exploring the consequences of this series of revelations. Legislators are already preparing to introduce a bill to modify the API. More to come.

WHAT YOU CAN DO IN THE MEANTIME

1. With the SAT-9 and California Standards Test release only days away, I recommend you make the most of this time to prepare a statistical primer for the press, the public, and your staff. Now is the time to cover the ABCs of interpreting test scores: (a) their statistical reliability at the student level; (b) reliability of grade-level results; (c) reliability of school-level summaries.

2. Understand the real meaning of your schools' API results. Get some help from your math department's leading faculty, and determine whether your schools' year-to-year progress is statistically significant. (The 20-point error margin is only an average. Actual error margins will vary based on each school's enrollment.)

3. Prepare to defend the API by retreating to a defensible boundary. The API does, indeed, have value. But just as it is well suited for some things, it is ill suited for others. Take a look at its other limitations.

OWL ARCHIVE | BACK TO NEWSLETTER REGISTRATION PAGE