Indiana

Education, From The Capitol To The Classroom

Review: On Average, Online Disruptions Didn’t Hurt Indiana’s ISTEP+ Scores

Bloomington fifth grader Georgie Stevens plays a computer game. On Monday, April 29, Georgie's class was among the first in the Indiana to feel the impacts of the server errors that ground testing to a halt across the state.

Kyle Stokes / StateImpact Indiana

Bloomington fifth grader Georgie Stevens plays a computer game earlier this summer. On Monday, April 29, Georgie's class was among the first in the Indiana to feel the impacts of the server errors that ground online ISTEP+ testing to a halt across the state.

Despite frozen computer screens, failing logins and overloading servers on this spring’s online ISTEP+ tests, a state-appointed testing expert has concluded “the vast majority of [Indiana] students scored as well as they would have had the interruptions never happened.”

The Center for Assessment’s Richard Hill released that finding Monday, capping a six-week review of more than 495,000 tests from across Indiana and likely clearing the way for state education officials to move forward with calculating A-F performance ratings for schools.

“However, this is not the same as saying no student in the state was affected,” Hill says, and state officials add the next phase of the study might identify specific students whose test scores shouldn’t count when those letter grades are calculated.

Problems with testing company CTB/McGraw Hill’s servers ground testing to a halt across the state on April 29 and April 30, directly disrupting the exams of more than 79,000 students — roughly one-sixth of all tested students. The company has since apologized and faces possible financial penalties of at least $613,000.

State superintendent Glenda Ritz hired Hill on June 10 to determine whether education officials could consider the exam results as valid. CTB paid the $53,000 cost for his services. Company officials are conducting their own validity study, too.

As with the test itself, the stakes attached to Hill’s review are high for schools. The letter grade ratings of elementary and middle schools hang almost entirely on ISTEP+ data.

But in the months following the exams, Indiana teachers and school administrators have expressed their doubts that the ISTEP+ results could ever be meaningful. A teacher who also serves on the State Board of Education said in May, “If this were data that I were taking in my classroom to inform my instruction, I’d throw it out.”

The state’s largest district, Fort Wayne Community Schools, even issued an ultimatum, saying it would not accept the test results without a third party review.

“This was a very disruptive situation. It was not conducive to students doing their best on the test,” said Chris Himsel, superintendent of Northwest Allen Community Schools, told StateImpact in June.

However, if anything, “students who were interrupted had somewhat larger gains across years than those who were not interrupted,” Hill concluded — a finding that mirrors the case of Wyoming’s botched online tests in 2010 that we wrote about earlier this month.

Overall, Hill adds the data show “students who were interrupted had gains that were as high as the students who were not interrupted.”

State lawmakers have asked Hill and Ritz to testify at a Monday meeting of the General Assembly’s Commission on Education to discuss the ISTEP+ exam. Ahead of that 1 p.m. meeting, the state superintendent issued a written statement:

Although Dr. Hill’s report found that the statewide average score was not affected by the interruptions, there is no doubt that thousands of Hoosier students were affected. As Dr. Hill stated in his report, ‘We cannot know definitively how students would have scored this spring if the interruptions had not happened.’ Because of this, I have given local schools the flexibility they need to minimize the effect these tests have on various matters, such as teacher evaluation and compensation. I have also instructed CTB McGraw-Hill to conduct enhanced stress and load testing to ensure that their servers are fully prepared for next year’s test and ensure that this never happens again.

“It is a heavy burden on the state of Indiana to have to deal with this issue,” Ritz told reporters on July 19. “It’s unacceptable for teachers to be evaluated in a large part on [ISTEP+ scores].”

Hill added his study only offers a big picture view of the test results, but doesn’t necessarily offer a complete picture. He writes:

However, this is not the same as saying no student in the state was affected. It certainly is possible that some students were affected; if so, those occurrences were overshadowed by the lack of impact on the vast majority of students. The interruptions data from CTB would permit a study of specific interruption patterns that might indeed permit one to identify students who likely were impacted by the interruptions. Indeed, CTB has proposed some patterns in the data that will be pursued during the next phase of this study, and it is possible that some students will then be identified as having been affected by the interruptions. If so, that will be important information to take into account during reporting.

Along with the results of Hill’s validity study, state education officials created this interactive map showing “how many students in each school district experienced computer interruptions during ISTEP testing in Spring 2013.”

Indiana Department of Education

State education officials released this interactive map showing the percentage of students in each district who experienced an interruption during the April-May 2013 administration of the online ISTEP+ exam. Note, state officials write, “some students were interrupted in more than one content area, so adding up the content area interruptions may exceed the total percentage of students interrupted.”

(Just to be clear: State officials used the same Google mapping tools we often use at StateImpact to visualize data, but we’re just embedding this map. We didn’t create it.)

Comments

  • Mouse Rat

    When I try to find Richard Hill’s background on Google using “dr. richard hill center for assessment bio”, the third result is a link to McGraw-Hill Education. What is his background? Why the Google result linking him to McGraw-Hill? Why is there no biographical information for this man anywhere on the web?

    • kystokes

      That result does not link him to McGraw-Hill education. It simply is a return that may be coming up from associations between his name and CTB/McGraw-Hill from the coverage of his review. As I type this, I know I have a paper copy of his CV somewhere, but not on me.

      • Mouse Rat

        Well, let me tell you… I know the presentation of “students interrupted” — definitely in the case of one school district in particular — was grossly under reported!

        On top of that, the focus on “interrupted” does not take into account the cost of the delays to schools and families of students, nor does it count the days where no testing could take place.

        In light of the Tony Bennett A-F system change for his big donor and to save face with GOP leadership, I don’t believe the results of this report for a second.

  • Cynthia Kingma

    So when are they going to release the scores to the schools? School is starting very, very soon and it would be really good for teachers and administrators to have the scores!

    • kystokes

      Scores should be released to parents and students by the end of August. Not 100% sure whether that means schools will have access as well, but I’d guess they would.

About StateImpact

StateImpact seeks to inform and engage local communities with broadcast and online news focused on how state government decisions affect your lives.
Learn More »

Economy
Education