The Owl Newsletter

ISSUE 38 | AUGUST 24, 2001

HOW TO UNDO THE DAMAGE OF MISREPORTING OF SAT-9 RESULTS ... FINDING OPPORTUNITY AMID CONFUSION

The Owl has been watching the festival of confusion over SAT-9 test score reporting in the daily papers. What folly. What passion. What a mess.

The flawed reporting is only part of the problem. Flawed presentation of results by educators compounds the problem. And Harcourt's reports for parents (both versions) do little to clarify matters.

FLAWED REPORTING

There are so many flaws to share. I'll contain myself to the most common two. If you have favorites of your own, however, please share them.

1. Unrepresentative small samples. Last week's newspapers included tables of data, page after page, presenting results by grade level and curricular area. In the battle to be first, many papers like the San Francisco Chronicle reported one grade level's results in each school. In a K-5 elementary school with 540 students, the scores of 90 students in fourth grade could hardly be considered representative of the whole. Since the CDE doesn't release school-level summary results, newspapers often just publish what they're handed. In this case, it's grade-level and curricular area national percentile ranks.

2. Comparing apples with oranges. Newspapers trying to measure progress committed the same statistical sin that educators have been committing for decades. They compare the results of grade levels in consecutive years. So last year's fifth-graders are compared with the fifth-graders of the year prior to that. What's the point in comparing entirely different cohorts of students? It would be like comparing the size of the fish I caught last weekend with the size of the fish I caught a year ago — not useful, and terribly misleading.

WHAT YOU CAN DO

There are opportunities galore for you to improve the quality of your SAT-9 reporting. Consider this only the first installment on this subject.

1. Speak plainly. If you can explain results without relying on education jargon, you'll be a step ahead of the crowd. Avoid as much as possible the statistical vocabulary of percentile ranks, norm groups, and the like. If you want a sober assessment of how much damage ed-speak can wreak, give this terrific article from the Los Angeles Times a look.

2. Speak honestly. When a parent wants to know how his kid's school is doing, hand him the results summarized at the school level. This roll-up is most easily done by looking at the cut-line measures (i.e., percent of students scoring above average). The concept works. Most parents will get it. While it may disguise other factors, here are three ways to show results using cut-line measures.

These sample reports are for Santa Clara County high schools' math scores.

3. Speak up in person. Your back-to-school nights are the first "live" setting where your principals and your parents can discuss results. Reporting results in person, face-to-face, is the opportunity best suited for a subject as complex as SAT-9 results. What you want are questions. The dialogue between principals and parents is one way you can find those questions, and make the data mean something.

VIDEO RESOURCES TO HELP PRINCIPALS STAND AND DELIVER, AND TO PARENTS "GET IT"

We have recently reviewed the videotapes of Jim Cox, a veteran teacher of educators whose strengths include making sense of assessments of all kinds. He's produced one set of videotapes for parents, and another for educators. Click on the link to find out more.

Or you can call Leanne to request a brochure and pricing information at: (909) 245-0828. Her email address is: leanne@sanfordsys.com.

OWL ARCHIVE | BACK TO NEWSLETTER REGISTRATION PAGE