Independence Of ISTEP Validity Study Called Into Question

The SBOE commissioned a validity study of the 2015 ISTEP+, but the executive director of the SBOE made comments suggesting changes to the report. Democrats say this was an effort to cover criticisms of the test in the report. (photo credit: David Hartman (Flickr)

State Board of Education Executive Director John Snethen revised the executive summary of an independent report that studied the validity of the 2015 ISTEP+. Critics say Snethen’s revisions change the tone of the report.

The report was commissioned by the State Board of Education and authored by two nationally recognized testing experts, Edward Roeber and Derek Briggs. The executive summary, showing Snethen’s revisions is now available to the public.

For example, an earlier version of the report stated: “It is safe to say that the 2016 ISTEP+ program is a work in progress, put in place quickly and without the usual procedures (e.g. field testing) used with most new assessment programs.

Snethen highlights the italicized section above, and commented in the document: “this is an example of a statement that could raise concern without additional context. Rather than saying ISTEP was put in place quickly, maybe we could just state what the 2015 ISTEP implementation timeline was, whether that was a reasonable timeline, and if not, what different timeline is recommended.”

Many of these types of changes were in the executive summary of the report, which outlines the findings of seven more technical studies. State Board of Education spokesperson Marc Lotter says Snethen was only trying to make the report’s findings clearer. He says the report includes both positive and negative findings.

“The experts found that the test is valid, they also found issues with the test,” says Lotter. “We wanted to make sure that if you’re going to have the executive summary that the general public is going to understand, it gets beyond some of the in the weeds testing issues and items that the experts uncovered. We also make sure we are presenting the information about the validity study and the fact that it is valid.”

Lotter says Snethen’s comments were suggestions to Roeber and Briggs, who still had control over the final version of the executive summary. He says the language they did adopt from Snethen’s comments “captured the spirit” of their original writing in the report but makes it easier for the general public to understand.

One example of this is when Roeber and Briggs wrote they found “several shortcomings in the design and development of the 2015 ISTEP+ program.” Snethen changed that line to we “recommend improvements in the design and development of the 2015 ISTEP+ program.”

In terms of assessing children, Roeber and Briggs did find the test is a valid tool by assessment standards. Roeber and Briggs were also hired to advice the state on how to shorten the ISTEP+ last year. They consult for many states and have also worked as Indiana’s testing consultants.

Governor Mike Pence appointed Snethen as the SBOE Executive Director in 2015 and spoke about the controversy in a press conference.

“I haven’t seen the memorandum, I haven’t seen the recommended changes, but I have every confidence in the integrity of our team, the integrity of the members and the staff of the State Board of Education,” said Pence. “We’ll just rely on those experts and we’ll accept the findings of the validity report over the ISTEP exam.”

But Democrats have criticized the situation, saying the Pence administration and his appointees tried to doctor a report that showed the switch to new standards and a new test in a negative light.

“This is politics at its worst,” said Democratic Party Chair John Zody in a statement. “Governor Pence knew the state wasn’t ready for the new ISTEP test, which he pushed for to fit his own political agenda. So what does Mike Pence do to cover his tracks? He directs his appointees to manipulate a review that was originally meant to be done independently from the state.”

The SBOE commissioned this report after to make sure the test is valid after Pence pulled Indiana out of the Common Core standards and the DOE wrote new standards in a matter of months. This led to the ISTEP+ being significantly longer last year, delays in scoring from CTB and then lower scores across the state.