Review: On Average, Online Disruptions Didn’t Hurt Indiana’s ISTEP+ Scores

July 29, 2013 | 1:00 PM

By Kyle Stokes

• 5 Comments
• Tweet

Despite frozen computer screens, failing logins and overloading servers on this spring’s online ISTEP+ tests, a state-appointed testing expert has concluded “the vast majority of [Indiana] students scored as well as they would have had the interruptions never happened.”

The Center for Assessment’s Richard Hill released that finding Monday, capping a six-week review of more than 495,000 tests from across Indiana and likely clearing the way for state education officials to move forward with calculating A-F performance ratings for schools.

“However, this is not the same as saying no student in the state was affected,” Hill says, and state officials add the next phase of the study might identify specific students whose test scores shouldn’t count when those letter grades are calculated.

Problems with testing company CTB/McGraw Hill’s servers ground testing to a halt across the state on April 29 and April 30, directly disrupting the exams of more than 79,000 students — roughly one-sixth of all tested students. The company has since apologized and faces possible financial penalties of at least $613,000.

State superintendent Glenda Ritz hired Hill on June 10 to determine whether education officials could consider the exam results as valid. CTB paid the $53,000 cost for his services. Company officials are conducting their own validity study, too.

As with the test itself, the stakes attached to Hill’s review are high for schools. The letter grade ratings of elementary and middle schools hang almost entirely on ISTEP+ data.

But in the months following the exams, Indiana teachers and school administrators have expressed their doubts that the ISTEP+ results could ever be meaningful. A teacher who also serves on the State Board of Education said in May, “If this were data that I were taking in my classroom to inform my instruction, I’d throw it out.”

The state’s largest district, Fort Wayne Community Schools, even issued an ultimatum, saying it would not accept the test results without a third party review.

“This was a very disruptive situation. It was not conducive to students doing their best on the test,” said Chris Himsel, superintendent of Northwest Allen Community Schools, told StateImpact in June.

However, if anything, “students who were interrupted had somewhat larger gains across years than those who were not interrupted,” Hill concluded — a finding that mirrors the case of Wyoming’s botched online tests in 2010 that we wrote about earlier this month.

Overall, Hill adds the data show “students who were interrupted had gains that were as high as the students who were not interrupted.”

State lawmakers have asked Hill and Ritz to testify at a Monday meeting of the General Assembly’s Commission on Education to discuss the ISTEP+ exam. Ahead of that 1 p.m. meeting, the state superintendent issued a written statement:

Although Dr. Hill’s report found that the statewide average score was not affected by the interruptions, there is no doubt that thousands of Hoosier students were affected. As Dr. Hill stated in his report, ‘We cannot know definitively how students would have scored this spring if the interruptions had not happened.’ Because of this, I have given local schools the flexibility they need to minimize the effect these tests have on various matters, such as teacher evaluation and compensation. I have also instructed CTB McGraw-Hill to conduct enhanced stress and load testing to ensure that their servers are fully prepared for next year’s test and ensure that this never happens again.

“It is a heavy burden on the state of Indiana to have to deal with this issue,” Ritz told reporters on July 19. “It’s unacceptable for teachers to be evaluated in a large part on [ISTEP+ scores].”

Hill added his study only offers a big picture view of the test results, but doesn’t necessarily offer a complete picture. He writes:

However, this is not the same as saying no student in the state was affected. It certainly is possible that some students were affected; if so, those occurrences were overshadowed by the lack of impact on the vast majority of students. The interruptions data from CTB would permit a study of specific interruption patterns that might indeed permit one to identify students who likely were impacted by the interruptions. Indeed, CTB has proposed some patterns in the data that will be pursued during the next phase of this study, and it is possible that some students will then be identified as having been affected by the interruptions. If so, that will be important information to take into account during reporting.

Along with the results of Hill’s validity study, state education officials created this interactive map showing “how many students in each school district experienced computer interruptions during ISTEP testing in Spring 2013.”

(Just to be clear: State officials used the same Google mapping tools we often use at StateImpact to visualize data, but we’re just embedding this map. We didn’t create it.)