VT parents upset over lack of access to standardized test scores and data suppression

Parents looking for the 2021 standardized test scores for Union Elementary School in Montpelier would find "***". The three asterisks on the online assessment dashboard indicate the data has been suppressed by the Vermont Agency of Education.

Data suppression is commonly used by the education agency for datasets with fewer than 11 students so an individual student's performance can't be determined. However, the agency is also suppressing data for the average-sized Montpelier elementary where 422 students were enrolled and at least 70 in each grade level took the Smarter Balanced Assessment (or SBAC) test that spring.

Across Montpelier Roxbury Public Schools − which includes two elementaries, a middle school and a high school − 16 out of 21 standardized test results have been suppressed. Those numbers track how students in third through ninth grade performed on statewide English language arts and mathematics tests as well as the Vermont Science Assessment (VTSA) for grades 5, 8 and 11. The science scores were not suppressed, however.

When the data are released, school and district-level results can take a year before the public can see them.

Parents vote on school budgets, provide direction for how federal COVID relief funds are allocated, make choices about where their student goes to school and if their child needs additional learning support. They say school and district comparison data is vital for these decisions, particularly as schools and families respond to COVID's impacts on education, and not providing the information at all or not doing so in a timely manner is unacceptable.

And while the issue was raised in Montpelier, the concern extends beyond Vermont's capital city.

"As tax-payers we have a right to know how our state and district are performing - and to evaluate their administrative and policy decisions," said Lori Duff, a parent to 10- and 11-year-old students in Montpelier Roxbury schools. "Keeping data embargoed does not allow the community a voice in the process and allows leaders to make decisions unchecked."

The Agency of Education says their hands are tied by federal privacy laws and a system that relies on a lengthy data collection and cleaning process. Furthermore, the agency says their obligation to release school and district data is not to parents and the public but is, instead, for policy and administrative use.

How performance data is collected and published in VT

Vermont students take standardized tests in the spring − which have been the SBAC and VTSA, though this year the state switches to a new assessment, Cognia. Here's the general timeline from the past two years with 2022 providing an example:

• March to June 2022- students take state standardized tests.

• June 2022- families are mailed individual test scores (the embargo for individual test scores was lifted June 14 in 2021 yet Montpelier families didn't receive scores until November). Superintendents receive preliminary data of how their schools and district performed. This data, however, is embargoed and cannot be released to families or the public until the Agency of Education updates the Annual Snapshot and Assessment dashboard on their website.

• November 2022 - the Agency of Education releases data about how the state performed as a whole.

• Spring 2023 - the Agency of Education completes the Annual Snapshot and Assessment dashboard, releasing school and district performance data that can be used to compare schools and districts to one another and against the state results. Students take the next round of standardized tests.

Accurate enrollment data is needed to calculate performance averages across schools and districts. Ted Fisher, director of communications for the Agency of Education, said Vermont has always been organized in such a way that districts report their data up to the agency, but that process can take time.

"Sometimes it takes well into winter for some districts to submit clean, complete enrollment data," he said. He mentioned other states may have a centralized data system and can pull that information with the push of a button.

How data suppression works

Data suppression logic is a mathematical function that is applied in the Annual Snapshot tool, which is an online dashboard maintained by the education agency. Fisher said it performs as expected given the complexity of the task even if it looks like it doesn't make sense.

Suppression hides values with fewer than 11 students, Fisher said, but it also takes into account every way the data has been reported and divided so that there is no way to mathematically figure out an individual student's performance.

Fisher gave the example that 3% of students may be among a historically marginalized class and make up less than 11 students. In that case, scores for the 97% of students not in a historically marginalized class may not be released because by deduction, individual students in the 3% could be singled out.

"All grades" can be prioritized in the system instead of a single grade if there is a conflict. This explains why there is at times district-level data available but no grade level data by school. Fisher said in Montpelier Roxbury's case "all grades" had been prioritized, which it was meant to be, and triggered a cascade of suppression.

When individual student score sheets are released, they are compared against school and district averages.

Fisher said that parents' feelings that the agency is not living up to its obligation to release complete and timely assessment results for districts and schools mis-characterizes the role of statewide assessments and data.

State-level data is best used for administrative and policy purposes, he said. He explained it is used for evaluating curriculum and professional development or informing strategic planning. Statewide assessments are not specific enough or timely enough to address real time student performance which is at the center of the parents' concerns, Fisher said.

He said formative assessments − which are internal tests given by the schools − are much better at diagnosing current needs and targeting instruction.

How data became a sticking point in Montpelier schools

A group of parents in Montpelier Roxbury Public Schools began digging into district-wide data after Superintendent Libby Bonesteel said in communications with families and presentations to the public and school board that her district didn't suffer a measurable decrease in learning during COVID. This assertion is contrary to what other districts and even the state reported to have experienced, which, as a result, have prioritized academic recovery.

Parents say the lack of learning loss was used as justification for millions in federal relief funds being allocated to non-academic needs. When they tried to test the superintendent's assertions against objective, statewide assessment data from 2021 and 2022, parents said the information was largely unavailable.

The Agency of Education cited FERPA, a federal law that protects student privacy, and an lengthy embargo on data being released to the public as the reasons information couldn't be released and directed parents to policies regarding data suppression.

Lori Duff watched her first and third graders (at the time) struggle with remote learning and said she couldn't fathom there was no learning loss. She has a dyslexic child and was hopeful federal funds would be put toward summer programming to help students make academic gains, as was done in other districts. But, in Montpelier the initial plan's focus was to use federal funds for the replacement of old windows. Ultimately no summer educational program was offered.

"Without any publicly available data for 'learning loss' to justify it, there was no supplemental remedial educational programming offered to help kids catch up after the pandemic in our district. We ended up hiring a tutor," she said.

Parents in Montpelier Roxbury schools say they have other reasons to question assessment data. Some disagree with the district's approach to early literacy which they say isn't based upon data-driven best practices. Parents say internal assessments have shown students performing well while outside testing indicated a lack of grade-level achievement, including an indication for special education services or a learning disability not caught by the school.

Grace Pazdan has two children in the district but began having concerns about her fourth grade son's ability to read. "In our experience, our son scored at or above grade level when assessed by classroom teachers, but much poorer when assessed by a trained literacy interventionist."

She paid for an evaluation when he was in third grade which determined he was dyslexic. She continues to have concerns about literacy assessments in the classroom and would like to depend on statewide results as another measure. "I believe it provides the most objective information about students' academic proficiency," she said.

For the past two years, most Vermont families received their students' individual SBAC and VTSA scores within a couple months of taking the test − usually over the summer − while in Montpelier parents didn't see their child's scores until November. In fall of 2021, Bonesteel told parents the data wasn't being released because the Agency of Education had embargoed the data. The agency embargoes school and district data until completion of the assessment dashboard, but not individual data.

Parents went to the agency to challenge their district's action of withholding the results and the agency responded saying the individual results were federally required to be released in a timely manner. Soon after the district mailed families their child's scores, according to Duff.

Bonesteel reiterated to the Free Press that school assessments did not indicate learning loss.

"In the fall, our initial screening data in regards to academic areas showed very little year to year 'loss' that was any different from any other year," she wrote. She puts little emphasis on the statewide SBAC and VTSA assessment results which she says haven't been consistent since 2019, saying they were not valid or reliable because of the pandemic. She said in her district they use smaller, more consistent and easily accessible internal data to make classroom and individual decisions. She welcomed any parent wanting to know how their student was performing in classroom formative assessments, screeners or diagnostic assessments to contact the district.

An internal winter assessment showed an increase in learning disabilities in some grade levels that could have been impacted by the pandemic, she said, and added that the district hired several academic intervention positions. She said for COVID recovery, the district has prioritized social emotional learning where the greatest need was indicated.

Why parents say standardized test data is crucial

The four Montpelier Roxbury parents the Free Press talked to said they feel hamstrung by a lack of data to inform how their children are doing academically. They have concerns classroom level data may not be reliable or objective and state level data regarding district performance that could provide quantifiable evidence is non existent. Their children are caught in between and continue to be impacted.

Hannah Reid, who has a fourth grader and second grader at Union Elementary School, said she recognizes standardized test scores are not an ideal measure of student success, but they are one of the few quantifiable measures she has to work with. Students' needs can't be effectively addressed without data that indicates what those needs are, she said.

"This is a public school system, which means we, as a community of students, teachers, caregivers, administrators, taxpayers share the responsibility and the privilege of providing the best possible education to our young people," Reid said. "In order to successfully deliver on this promise, we need to work together and we need all the information we can get our hands on to guide our collective decision-making."

Lisa Burns has been raising the alarm of the lack of data to the school board and creating her own data charts based on the limited data she's been able to glean from school presentations. Her child is a senior and has had much of their high school education altered by COVID. When her student could have been receiving interventions, Burns feels the school has instead based decisions upon out-of-date data.

Burns said this school year is coming to a close and the budget is set already for the 2023/24 school year yet the most recent available statewide data comes from the 2018/19 school year. "All those decisions about curriculum, professional development, spending and priorities have been based on pre-COVID data. So that lack of data informs 5 school years in our district - that is 5 of 12 years of a students' educational career," she said.

Duff and Burns point to problems with how the state releases data and its reasoning for data suppression. Duff said other districts of similar size did not have information suppressed and there was no explanation what made their district require suppression. She felt the refusal to report current information and sit on it for a year while it had already been released to superintendents inexplicable.

"The law does not specify that the data is to be used solely for administrative and policy uses," Duff said refuting the statement from the Agency of Education and pointing toward the Every Student Succeeds Act, saying the law ensures information is provided to 'educators, families, students and communities'. "It seems that the state has adopted this rationale because of its own timeline."

"The AOE maintains a supremely un-user friendly and archaic website," Burns said and says the agency has never answered her phone calls or replied to her voice messages. Burns and Duff found the agency unwilling to provide proficiency data based upon whole grade level numbers which, in all cases had more than 11 students, independent of what was available in the dashboard. The dashboards are where the numbers were subdivided into categories small enough to trigger data suppression. The refusal stood even when parents submitted public records requests under Vermont law for the data.

"The lack of publicly available data absolves the state and our school leadership of responsibility for providing the best education to every student in a timely fashion. It ensures that our schools will forever be chasing their tails," Burns said.

Contact reporter April Barton at abarton@freepressmedia.com or 802-660-1854. Follow her on Twitter @aprildbarton.

This article originally appeared on Burlington Free Press: Vermont parents upset over access to state assessment scores