Inside A Disrupted ISTEP+ Exam: What One Student Says About Her Score's Validity
Kyle Stokes / StateImpact Indiana
Bloomington fifth grader Georgie Stevens plays a computer game. On Monday, April 29, Georgie's class was likely among the first in Indiana to feel the impacts of the computer problems that ground testing to a halt across the state.
Almost as quickly as Georgie Stevens signed into her online ISTEP+ exam, the testing website kicked her out.
“It said, ‘Get off and get back on,’ there’s something wrong with it,” the Bloomington fifth grader remembers. “So I did.”
Then the site booted her out again. And again.
She asked for help, wondering at first whether her teacher still wanted her to finish the exam — but Georgie realized all of her classmates’ hands had shot up in the air too.
It would’ve been a normal testing day if it weren’t April 29, 2013, when memory issues on testing company CTB/McGraw-Hill’s computer servers began kicking students out of their online exams between 8 and 9 a.m.
Georgie logged on at 9:15 a.m. that morning, meaning her class was likely among the first in Indiana to have problems with their tests.
- What Was It Like In A Disrupted ISTEP+ Session?StateImpact Indiana’s Kyle Stokes explains why widespread disruptions won’t necessarily invalidate the results.Download
“I was thinking, ‘Oh, this isn’t working, she’ll be fine, we’ll all do it another day.’ But I was like, ‘I don’t want to do it another day when I could be doing something else,'” Georgie says.
— Glenda Ritz, state superintendent
Georgie tried to answer as many questions as she could. But the site kept kicking her out of her test — sometimes before she could even answer the test — and she says she only completed, maybe, a half-dozen questions.
She says the experience was irritating — her mom remembers Georgie coming home rattled that day — and Georgie doesn’t think the state should use the test results in teacher evaluations or in issuing A-F letter grade ratings to schools.
But she also says she got time in a later testing session to go over the questions she hurried through and doesn’t think the interruptions impacted her score.
“I’m a good student. I don’t think my score’s going to be affected by that,” she says.
Widespread Disruptions
But how typical is Georgie’s experience? Leslie Rutkowski, a testing expert at the Indiana University School of Education, says students’ experiences on the exam don’t necessarily demonstrate the full impact of a disruption.
Related
Posts
- Were Online ISTEP+ Testing Disruptions More Widespread Than We’re Hearing?
- With ISTEP+ Issues & New Model Coming, How Indiana Schools Will Get Their A-F Grades This Year
- After Online Testing Debacle, Ritz Appoints Third Party To Review ISTEP Results
- AP: State Officials Will Seek Outside Review Of ISTEP+ Results
- ISTEP+ Test Results Coming In July, Six Weeks Behind Schedule
- After Disrupted Exams, Indiana’s Largest District Says It Won’t Accept ISTEP+ Results
Combine data from CTB with local schools’ reports and, as we’ve pointed out, 145,000 of all completed tests have been flagged as “disrupted” in one way or another.
But state superintendent Glenda Ritz says “disrupted” can mean many things.
“Some students might’ve been interrupted for a few seconds. Some may have been interrupted for 30 seconds. Some may have been interrupted for longer periods of time,” Ritz said last week.
And some students’ testing problems — unlike Georgie Stevens — had nothing to do with logging in. Laura Cain, Director of Strategic Initiatives at the Fort Wayne Community Schools, says there’s a reason testing protocols call for creating a calm testing environment.
“If I’m a student that logged on and didn’t have any interruptions,” Cain says, “you don’t know what two students sat next to me and cried the entire time I took the test because they had irregularities. And I hurried up and finished because I didn’t want to cry too.”
A Data Puzzle
State education officials have hired the the Center for Assessment’s Richard Hill to study the numbers before deciding whether to use the test scores in evaluating teachers or issuing letter grades to schools.
Kyle Stokes / StateImpact Indiana
State superintendent Glenda Ritz holds a press conference in her office.
Speaking generally, Rutkowski says there are several ways to analyze the impact of a disruption on a test result. For instance, she says, reviewers can zero in on a particular test question and compare the rate of correct answers on disrupted exams with the rate on uninterrupted exams.
But Rutkowski says the international assessments she studies have thrown out entire countries’ results because of much smaller problems — such as a printing error in the exam booklet.
“I think the threshold for throwing out data or throwing out test scores should be much lower in a high stakes situation because the stakes are high,” Rutkowski says.
Ritz says she wants to help districts minimize the test results’ role in teacher evaluations, but declined to speculate further on how the Center for Assessment’s report might impact how state officials determine schools’ letter grades.
But many local school administrators are resigned to the likelihood the state may have no choice but to use the results. As Monroe County Community Schools superintendent Judy DeMuth told StateImpact last month:
In a perfect world, I would like to see that the state of Indiana invalidates all the scores and they say to CTB, ‘You owe us another year on our contract. This has gone awry, it didn’t work, and we’re going to invalidate this.’ The realism of it is that I believe that the scores will be used in the state of Indiana, and we owe it to our students to do the very best we can to get the most realistic picture of those scores. Although I’ll be lobbying our state superintendent to say all children were affected and we really should invalidate the entire state, she has guidelines, laws, policies she has to live by.
Podcast: Play in new window | Download