The Surprising Story Of Wyoming's Troubled Online Tests — And What Indiana Can Learn From It

• AUDIO

July 18, 2013 | 9:10 AM

By Kyle Stokes

• 2 Comments
• Tweet

The story of Wyoming’s first foray into online standardized testing is part cautionary tale, part hope-filled parable.

In 2010, the first year Wyoming gave its entire standardized test online, students across the state reported the testing website had rejected their login or crashed mid-test — much like with this year’s ill-fated ISTEP+ exam.

But as an outside review of the scores later showed, none of the interruptions impacted scores on the Proficiency for Wyoming Students, or “PAWS,” exam.

In fact, Wyoming’s test scores went up across the board that year — despite the fears of state education officials, who asked the federal government months before getting the results to throw out the 2010 data.

It’s a possible best-case-scenario for Indiana educators, who fear their own evaluations or their schools’ letter grades hang on the results of a validity study of this year’s disrupted ISTEP+ exam that’s due out sometime soon.

• The Curious Case Of Wyoming's 2010 Online TestsStateImpact Indiana‘s Kyle Stokes tells the surprising story of what happened with Wyoming’s first fully-online PAWS exam.Download

Yet when compared with this year’s ISTEP+, Wyoming’s 2010 PAWS experience raises many of the same questions about the future of online standardized testing — in part, because the problems students experienced were the same.

The PAWS Test

Launching the PAWS test — and getting it ready for statewide administration — was one of the tasks that landed on Jim McBride’s desk when he assumed the role of interim state superintendent in July 2005.

McBride, a former science-teacher-turned-Air-Force-commander, remembers his directive on the PAWS this way: “Get this train moving. It’s on the track, get things moving forward.”

After Wyoming voters elected him to a permanent four-year term in 2006, McBride tells StateImpact his intention in developing an online PAWS exam was to “get out of the assessment business”:

It drains resources and is very, very time consuming in terms of classroom. What we were hoping to do was, ‘Yes, let’s develop a… good-working, simple-to-administer online assessment that our kids could log into whenever they were ready.’ For example, I have an eighth grade granddaughter who reads at above a high school level. She could take the reading assessment for high school any time she wanted and do fine. Why don’t we let her take it and bank it?

McBride lost a re-election bid later in 2010 — though not the only reason for his loss, McBride considers the PAWS debacle that year a “contributing factor” — and now works in Wyoming’s Department of Health.

The Problems

From the students’ perspective, the problems on Wyoming’s 2010 PAWS test looked a lot like the disruptions Indiana students experienced on this year’s ISTEP+ exam.

Screens froze, went blank or were covered with “Java script jibberish.” Sometimes testing items took a long time to load. Sometimes it wasn’t clear whether the testing website had saved answers.

Patrice Riley is the curriculum director for Big Horn County District 1, a rural district about 70 miles from Yellowstone National Park and not far from the Wyoming-Montana border. She remembers the writing portion in particular gave students a lot of trouble in 2010.

“A kid would type in their entire response and the screen would go blank. We weren’t sure if it got submitted or disappeared into cyberspace or what happened,” Riley says.

Marc LaHiff, an administrator in Wyoming’s largest school district in Cheyenne, remembers one high school student who submitted his answer on the writing test three times.

“I was in on several conference calls with the state superintendent and representatives of the districts,” LaHiff says, “and I can tell you, districts were angry.”

They were angry — or worried — because they felt their ratings under the No Child Left Behind Act could suffer based on tainted data.

The Vendor

McBride remembers glitches every year a district piloted an online PAWS test. But testing vendor NCS Pearson assured him they were making improvements.

In hindsight, McBride says Wyoming felt like the guinea pig for testing systems Pearson was building.

“They blew a whole lot of smoke at us about their capability and what they had planned to do,” McBride says. “I think they were using us as sort of a laboratory. A lot of vendors come to Wyoming for that reason because you’ve got just 500,000 people [living in the state] and, well, we don’t even have 100,000 students.”

Pearson officials apologized to Wyoming students and teachers in the summer of 2010, saying the company was “dismayed to have let you down.”

Last week, company spokeswoman Stacy Skelly told StateImpact in a written statement the company’s “number one priority is always the accuracy and validity of our tests.”

“When mistakes do happen, we take immediate action to address them and analyze what occurred so it doesn’t happen again,” Skelly wrote, adding, “As part of our process to guard against unexpected online test delivery outages, we project anticipated testing volumes far in advance and routinely do test runs.”

The Results

In the spring of 2010, Wyoming education officials asked the U.S. Department of Education to grant the state a waiver from that year’s NCLB ratings. (Schools that don’t make “Adequate Yearly Progress” or “AYP” under that law can face serious consequences.)

As part of that waiver, state education officials asked an outside testing expert to review the results of the PAWS test. The expert they picked is the same one conducting Indiana’s validity study this year: the Center for Assessment’s Richard Hill.

This is where the surprises begin.

In Hill’s report, first issued on July 30, 2010, he pinpointed a group of 400 students who he could say with certainty experienced disruptions on their PAWS tests. He found this group’s final exam scores were no worse than expected — if anything, they were better — and recommended the state consider the exam scores valid.

Later that summer, newspaper reporter Jackie Borchardt, who was covered education for the Casper Star-Tribune at the time, remembers statewide scores on the test came out.

“Generally speaking, they went up across the board,” Borchardt remembers — despite all of the disruptions.

“Everybody released their breath that they’ve been holding and said, ‘Oh, okay, it’s not as bad as we thought’ — at least in our district,” Patrice Riley says.

But here’s the irony:  despite Hill’s recommendation, the federal government opted to grant Wyoming its waiver. Those great test scores wouldn’t count.

“I talked with principals and superintendents who felt maybe we should give the waiver back because these scores are great and our school’s been struggling for a couple of years and this score would’ve turned us around,” Borchardt says.

The Aftermath

Soured on online testing after the 2010 exam, Wyoming has since reverted to giving tests with paper and pencil.

Patrice Riley says she’s ambivalent about that move. On one hand, she says teachers don’t miss the hassles of online testing.

But on the other hand, “If I can just get my car and put the key into the starter and it starts, why would I want to go back to the old crank on the front of the engine?” Riley says.

The question in Indiana: Can Wyoming’s history repeat itself? Is it possible Indiana could post test score gains despite all of the problems?

“Could we be lucky here in Indiana — have that happen again? I suppose so, but most folks wouldn’t speculate or expect that to be the outcome,” says Terry Spradlin of IU’s Center for Evaluation and Education Policy.

While it’s possible, he says, the stories of crying kids and stressed teachers during this year’s ISTEP+ exam suggest students’ scores might be more adversely impacted. But, he says, he’ll defer to the Hill’s recommendation.

“Looking at historical data for schools and individual students, there should be enough information to extrapolate whether the scores are indeed valid,” Spradlin says.

Podcast: Play in new window | Download