Indiana

Education, From The Capitol To The Classroom

As State Officials Ponder Fines, Testing Company Execs Explain ISTEP+ Disruptions

Apologetic executives from the testing company whose servers overloaded twice — interrupting April’s online ISTEP+ exams twice — told the combined membership of the Indiana House and Senate Education Committees that they took responsibility for the errors.

“I have family here in Indiana,” said Ellen Haley, president of CTB/McGraw-Hill. “It was very disappointing to have this happen and have this impact schoolchildren in Indiana. Please accept my sincere apologies and regret that anything at all happened.”

Haley’s statements came on the same day Department of Education officials announced they would be seeking at least $613,600 in damages from CTB — though a Department of Education spokesperson later stressed that the penalties “could reasonably go into the millions” — in part to pay for an outside review of the disrupted exams that’s already underway.

Citing that forthcoming review, which will offer recommendations on whether the state should throw out this year’s ISTEP+ results, co-chairs of the General Assembly’s Commission on Education — Rep. Bob Behning, R-Indianapolis, and Sen. Dennis Kruse, R-Auburn — urged lawmakers to focus their questions on the causes and immediate impact of the disruptions.

What — Exactly — Happened

Haley confirmed several of the details state superintendent Glenda Ritz sketched out in May. For instance, Haley says the most widespread disruptions, on April 29 and April 30, were both caused by two distinct memory issues with the company’s servers.

On those days, those problems rendered students and teachers powerless, facing what many of them called “the globes” or “the globe screen” — the term that describes the “Please Wait” screen test-takers saw. During her testimony, Haley explained these pauses happen by design:

The system captures every answer choice as soon as the answer choice is selected. We capture it and we transfer it from the students response to our servers — we have many servers, virtual servers, we have a database. We’re capturing every time they make an answer choice. We capture it, we transfer it. If there’s any reason why the system feels like it cannot transfer the answer, it will not let the student go on.

Thus, the globe screens.

Haley says company engineers performed 50 stress tests on its back-end servers prior to testing and engineers addressed one memory issue on Monday evening by adding more memory to the servers.

“The funnel had filled up,” explained CTB’s Chief Digital Officer Stephen Laster, so the engineers effectively added a bigger funnel.

But the first memory issue — with the server itself — had obscured the second software-related memory issue that caused problems to persist into Tuesday:

Tuesday morning seemed fine. By the middle of the morning, the late morning, schools started calling us again. We had engineers monitoring the system. We had people in the state, in the Department of Ed office, watching — we have a dashboard you can see how many students are testing, how many are being interrupted, how far along each student is in the test that they’re taking. We could see at about the same time that something was happening again. It was actually also happening on Monday, April 29, but we couldn’t see it because the main problem on Monday was masking the problem on Tuesday.

Haley pointed out, however, that after technicians addressed Tuesday’s issues, all students eventually completed their exams.

How Lawmakers Reacted

Rep. Kreg Battles, D-Vincennes, though, said memory problems with CTB’s servers didn’t explain the extent of the problems. Other local school officials have described problems with the online testing sites that go beyond the testing website kicking students out of their tests.

“On some of the testing, the full answer bank would not load,” said Battles, who’s also a teacher. “The kid literally had a question where there was no correct answer.”

After the hearing, Behning — the House Education Committee chair — told StateImpact he thought CTB officials were forthcoming, but says he still is looking for answers to questions such as those Battles raised.

“They probably brought up maybe more questions than some resolutions at this point in time,” Behning said. “We’re going to have to wait for when the validity studies are done.”

Though lawmakers do not have a direct hand in the fines — that’s the executive branch’s purview — Sen. Jean Leising, R-Oldenburg, asked Haley if she were in their shoes, what would she do about the company’s contract? Here’s Haley’s response:

I think you have to look at the whole contract and all the services we provide, to be honest — the test that’s developed, all the standard-setting that’s done, all the scoring, all the reporting, the building of the online system. Certainly, if I were a legislator, I would ask for answers about those two days and what language we have in the contract and what fines we should pay for that. And things that the state would perhaps want us to do. For example, the validity study — we’ve agreed to pay for the validity study. There are probably other things outside the exact letter of the contract to help make things right in a state we’ve had a long-term relationship with. But I think the contract, as it’s written, with a penalty section, and penalties, in contemplation of all the other things that have gone well, I wouldn’t say… I really, honestly, don’t see this as rising to the level of throwing the contract out, or the vendor. We’re dealing with technology, it needs to go right, it’s students in schools, we need to make it right, but I think the contract as it’s written gives us the opportunity to do that.

The Indiana Commission on Education will meet again on June 29.

Comments

  • Karynb9

    One of the (many) problems is that even if the validity studies determine that individual student’s overall results are valid (the test says they passed…and the validity study confirms that to be a valid result for that student), I can’t imagine the validity studies are going to be specific enough to allow anyone to reasonably use this year’s scores for purposes of determining student growth either this year or next year. The growth model is based on much more specific, to-the-exact-points-scored information that can’t possibly be confirmed by these validity studies. One or two questions may not impact the overall “pass” status for most students…but they can absolutely put that student into a “low growth” range this year with their teacher (if the growth model is a part of teacher evaluations in that district like it is with RISE) and school unfairly suffering the consequences. I would like to see more questions asked about that when the validity study is complete — are those who complete the study confident in not only the DNP/Pass/Pass+ status of tests that were reviewed and “cleared,” but also confident enough in the exact score obtained that it should be used as a part of a growth model with very high-stakes results for teachers and schools.

    I would also like for those “very apologetic” “most students were just simply ‘interrupted’ and allowed to continue within a few seconds” executives come and proctor a computer lab of 25 fourth graders taking ISTEP and ensure that every student received exactly 38 minutes of time to take his or her test (per the standardized, this-test-isn’t-valid-otherwise, instructions). Of course, not all 25 of the students had problems. However, maybe you had ten students who had problems. All of the problems came at different times…lasted for different amounts of time…and while you were helping Student A with his problem, Students B and C had THEIR hands up waiting for assistance from you, with you unable to know exactly how long Students B and C were HAVING problems since you were busy with Student A. Again, the tests are NOT timed by the computer — they are human-timed with the proctor being the person to tell the students when their time is up.

    Actually, since we can no longer simulate the exact testing environment, just take those executives and Indiana legislators to Chuck E. Cheese. Bring them over to the Whac-A-Mole game. Have them play the game…but not only do they need to be whacking the moles at the appropriate time when they pop up, they also need to be timing exactly how long each mole stays down inside of the machine throughout the duration of the game before popping up and being a “problem.” Once they have determined that each mole spent exactly 38 minutes being “down” inside of the machine, they can be finished with the game. They need to do this on at least three machines at the exact same time, of course (each Whac-A-Mole machine only has five moles). They will also need to be interrupted during this process by someone playing the role of a principal/testing coordinator/technology director who enters the room for a conversation or calls to find out what’s going on and how many students are affected. Pretty much sounds like the “standardized testing environment” that many Indiana teachers had to monitor during the first few days of ISTEP.

About StateImpact

StateImpact seeks to inform and engage local communities with broadcast and online news focused on how state government decisions affect your lives.
Learn More »

Economy
Education