Testing Advisory Committee Raises Concerns Over ISTAR Test

A testing advisory committee raised concerns about an assessment that tests students with severe cognitive disabilities. (Photo Credit: Robbie/Flickr)

A state testing committee recommended the State Board of Education not include test scores from an assessment for students with cognitive disabilities into school A-F grades. The advisory committee cited issues with that assessment’s validity.

The Indiana Standards Tools for Alternate Reporting, or ISTAR, is the assessment in question. It’s an alternative to ISTEP+ for students with significant cognitive abilities. This is the first year students took the test, and the state’s Technical Advisory Committee (TAC), a group that provides guidance on testing issues, says it might not be valid.

The ISTAR costs the state $2.5 million each year, and tests around 7,000 students in grades three through eights as well as tenth grade.

The way this assessment works: All students taking it take a version in the fall that places them on one of three tiers, levels one through three. Level one is the least challenging of the test versions and level three is the most challenging. Then, three times throughout the year, ISTAR students take the assessment at the level they tested into.

The problem occurs with the scoring of this assessment, because each level is scored the same way. So a student taking the more challenging level three version could score less than a student who took the level one, even though his or her test was harder.

The TAC found, after the test was implemented for the 2015-2016 school year, that this scoring system isn’t fair to all of the students.

At the State Board of Education’s work session Tuesday, members of the TAC told the board they would recommend not using these scores in 2016 A-F school scores.

ISTAR’s test vendor, Questar, tried to put the different levels of the test on the same plane by putting common questions in all the tests. But the number of these questions worries TAC member Karla Egan.

“There weren’t enough to feel secure,” said Egan.

Fellow TAC member and testing consultant Ed Roeber echoed the concerns, saying the score on ISTAR doesn’t give the state a good picture of how these students performed.

“If you can’t compare the performance on tier one, two or three items, you may have a situation where fewer students are passing the third tier,” Roeber said. “You might conclude they didn’t do as well on the assessment.”

Roeber suggested the TAC should be involved in test development from the earliest stages, rather than reviewing it once it’s created.

Many board members raised concerns about this process, saying this issue should have been caught earlier.

“I’m just curious,” asked board member Vince Bertram. “At what point are those type of issues discussed, rather than administering the test and coming to this conclusion.”

But Michelle Walker, Director of Assessment for the Department of Education, says it was challenging to develop the test in only six months. This is because the previous test used for these students was part of a national consortium, which the state had to pull out of after the legislature said Indiana couldn’t be part of testing consortiums.

The TAC is meeting Thursday, and the test vendor, Questar, will present possible solutions to address this issue.