Judging Assessments

When testing data lands on reporters’ desks, it typically winds up in the paper broken down by who is doing well and who isn’t. But at a panel on Friday, two testing experts asked reporters to begin testing the tests themselves.

Before the next battery of test results comes out, there are a handful of questions Tom Van Essen (Education Testing Service) and Stanley Rabinowitz (WestEd researcher) would like reporters to ask:

What is being measured?
Why is it being measured?
How is what’s being measured measured?
How are the results reported?
What are the intended or unintended consequences? What behaviors are these tests encouraging in our classrooms?

Tests are not cheap to make or tailor to a specific set of standards. As a result, states and districts will sometimes pay for one test and use it to measure things it wasn’t designed to measure. A test meant to gauge whether students can pass a certain proficiency bar will also end up being used to measure how well teachers teach. Rabinowitz called this “mission creep” and said it’s important for reporters to determine whether a state is trying to answer too many questions with one test as a way of saving money.

Another thing to look out for is whether a state purchased an “off the shelf” test and didn’t pay for any alterations that would make it higher quality, but also more expensive. Also, once the new testing program is in place, its normal to see a drop in students’ scores at least in the first year. If there’s no dip, reporters need to find out why that is.

Asked which test they would get rid of if they could choose, Van Essen and Rabinowitz both picked interim benchmark assessments — tests given several times a year to judge how well students will do when the final summative test rolls around. The problem (with some of them, not all) is sometimes the tests are essentially copies of the state exam so teachers could gleam the same information just by looking at last year’s test results. The tests can end up pushing teachers to focus on details, like whether students understand alliteration, for weeks at a time and at the expense of other material.

Rabinowitz divulged what he called one of his greatest professional discoveries: “Any teacher who could use the very sophisticated tools that are being developed for formative diagnostic testing doesn’t need it because she’s already doing it in the classroom. If the teacher needs these complex tools, she probably can’t use them without a lot of professional development.”