The Owl Newsletter

ISSUE 37 | JULY 26, 2001

COMMON ERRORS IN REPORTING TEST SCORE TRENDS

This owl's been reviewing the ways that educators report test score results over time. What I've seen deserves some hooting and hollering.

The bad news: it's often reported in ways that wouldn't pass muster in a high school statistics class.

The good news: there's lots of room for improvement.

With SAT-9 results now landing on your desks, reporting measures of progress is no doubt a timely subject. And with school accountability report cards now requiring a detailed reporting of grade level progress over three years, this challenge sits today on someone's desk in your district.

You are probably used to seeing a table that arrays years and grade levels in a fashion similar to this:

The problem with this traditional approach is that the very thing you are analyzing — one grade level at a time — is made up of entirely different students each year. If ever there was a case of comparing apples with oranges, this is it.

This owl thinks it's time to raise a ruckus over this. So does the American Statistics Association, which awarded educators their booby prize for committing the worst abuses of statistical methods in recent years. Another wise critic has recently spoken up on this subject, the empathetic and talented education reporter, Jay Mathews, of the WASHINGTON POST. He wrote on July 17:

"The new standards and testing movement has produced a national obsession with these ups and downs. We celebrate a five percent jump in the local elementary school's test scores and lament a similar drop at our middle school. Yet those are measures of different children. We are comparing the apples of the class of 2000 to the oranges of 2001."

Read the full version of this worthwhile essay.

You have every reason to end this unfortunate tradition of misreporting test score trend data. The school accountability report cards you publish are the ideal opportunity to do so. Just take the class of students who graduated in 2000 and 2001, and those to graduate in 2002, and present their SAT-9 results over time. If your students have a fairly low churn or mobility rate, and if you use a simple cut-line metric like percent of students scoring above average, you're getting closer to measuring what counts.

True, this method is not ideal. Norm group problems, particularly in ninth and tenth grades, are unavoidable. Yes, the test itself doesn't measure what many California schools teach. Yes, value-added measures of progress are far superior. All true. But why cling like a barnacle to a pier piling, adhering to bad methods? It's time to show your public you're paying attention. Measure what counts, and you will be rewarded for raising the level of public discussion of how schools are doing.

OWL ARCHIVE | BACK TO NEWSLETTER REGISTRATION PAGE