Education, From The Capitol To The Classroom

Why The Online ISTEP+ Servers Couldn't Handle The Testing Load

Photo Illustration by Kyle Stokes / StateImpact Indiana

It’s not that J.D. Ferries-Rowe doesn’t buy the explanation testing company representatives gave the Indiana General Assembly last week for the disruptions of April’s ISTEP+ exams — he does.

But Ferries-Rowe, the tech director at Brebeuf Jesuit Preparatory School in Indianapolis, wanted lawmakers to press CTB/McGraw-Hill executives further about the root cause of the problems: a lack of memory on the company’s testing servers in New Jersey.

“I wanted the follow up question, ‘Why didn’t you have enough memory on those computers?’” Ferries-Rowe says. “Was it just a cost-cutting measure, were you trying to save money? Did you grossly underestimate? Did you forget to carry the one? What happened?”

Though the state’s $95 million contract with CTB specifies the company’s online systems must be capable of testing up to 70,000 Indiana students at once, it turns out Ferries-Rowe’s question — why couldn’t the server handle the load? — is not easy to answer.

“That sounds maybe simpler than it is. If all 70,000 students were taking the exact same test at the exact same time with the exact same pages, that might be a math problem,” says Dennis Cromwell, who helps run OnCourse, the online system at Indiana University that students use to check grades, register for classes and take online quizzes.

Overseeing a computer system with similar capacity needs to the ISTEP+ servers —OnCourse has to handle as many as 80,000 users at once during peak times — Cromwell says it’s important to remember that users are doing things at different times. He tells StateImpact:

Think about the fact that there are different ISTEP tests for different grade levels. All of the students are doing it at different times — so one might be processing one piece of the test versus another. Does that make a difference how far a student is into that test as comparison to one that’s just beginning…

I can’t say until I really know the application in some kind of detail. But you’d have to look at it. Is the load on one component of the test the same as the other?

That load is often uneven, Cromwell says, so basically, server technicians have to strategically balance that load so the system can handle it. To do that, they allocate memory. To know when parts of the system will feel stress, Cromwell says server technicians perform “load tests” to map out how the system responds.

CTB Chief Digital Officer Stephen Laster told lawmakers Friday company technicians performed more than 50 stress tests of its servers, including 30 load tests.

Cromwell says he’s sat in the same hot seat before when things have gone wrong. After launching a new version of the OnCourse site in 2007 and 2008, Cromwell says Indiana University’s servers had the same load problems.

A fault in a tiny subcomponent caused a huge problem with the system — a problem big enough that university faculty called Cromwell before their committees to explain what went wrong.

“This kind of serious problem — if it were something obvious, it would’ve gotten fixed before McGraw-Hill delivered. There’s something subtle about the delivery of the ISTEP+ tests to the Indiana students that caused the problem,” Cromwell says.

While he empathizes with CTB staffers, Cromwell says the bottom line is that the company didn’t deliver on its promises.

Despite the scope of the disruptions — the computer problems disrupted at least 78,000 students’ exams; districts say the impacts are even broader than that — Ferries-Rowe says there’s an even bigger unanswered question. As he told StateImpact:

You have the mechanical problems of making sure that this many tests can be carried out with this many students. We’re two years away from the PARCC test coming down the pipe and we’ve never had a non-glitch e-test. It’s three years in a row now we’ve had something go wrong. So we’re not there yet, but we’re told we will be there in two years? That makes me nervous.

In some ways, this was an interesting hearing because the focus was on the mechanics of the testing process and not whether or not this test was giving us information that would be useful for students, for parents, for teachers, for student improvement. That’s a bigger discussion that would be nice to be having, but we’re not going to have that if we’re focused on, ‘Did we put enough memory in the servers?’


  • Jimmy Brown

    “The pressure to publish is so great,” thought Kyle Stokes to himself. “I guess I will type up this interview and post it even though there is nothing new in it. I’ll even make the headline a question, even though my story makes a very feeble attempt to answer it with only one source who wouldn’t know the answer anyway. I’m so clever.” Stokes then turned his attention back to loving men.

  • Nancy Papas

    Thank you for this article. Test interruptions for 3 years in a row raise large questions about the provider. Do profits come before performance?

    High-stakes standardized tests cost tens of millions and most are so unnecessary. Some schools interrupt 140 of the 180 instructional days with standardized tests in various grades. At even 1 hour per day, that accrues to more than 5 weeks of instructional time lost to testing. Under the old A+ program, Indiana added 5 days to the school year for testing, NOT 5 weeks.

    Not all the tests are state tests, but I-STEP pass-fail tests don’t specify what instruction is needed the next year to correct student deficiencies. And if results don’t accurately reflect what students know and can do, the only accepted proof of that is other standardized testing.

    Testing companies are laughing all the way to the bank. Students hate the high pressure “drill and kill” routine that kills a love of learning; and teachers hate all the time needed to enter, chart, trend, analyze, and/or report the avalanche of data created by all the tests rather than preparing lessons, grading homework, helping students, communicating with parents, staffing after-school obligations, and more. Teachers need but don’t have full-time clerical assistance to do the entry and charting of data for them. They’re doing at least two jobs and not getting paid that well for one.

    No one is suggesting that we do away with all testing, but for years, schools did an excellent job with parts of 4-5 days spent on standardized Iowa or California tests. Except for SAT or ACT tests, all the other tests were teacher designed, based on what was actually taught, and with quicker feedback to students and parents than today’s computerized tests.

    It’s time to push the reset button to prioritize teaching and learning over testing.

About StateImpact

StateImpact seeks to inform and engage local communities with broadcast and online news focused on how state government decisions affect your lives.
Learn More »