The Pandemic has Changed the Accountability Landscape
What a National Survey Reveals about State ESSA Accountability Systems in 2022
Readers of this blog know that my Center colleagues and I have been thinking a lot about the impact of COVID-19 on assessment and accountability systems (see D’Brot, Marion). Our interest is both in understanding the impact (see Betebenner, Marion, Dadey) and in helping states navigate through the disruptions of the last two years (see Domaleski, D’Brot). With this blog post, I’m sharing results from a national survey designed to understand the scale of the impact of COVID and resulting changes to federally mandated school accountability, and how the survey related to our work with the Council of Chief State School Officers (CCSSO) to support a handful of states as they grappled with the challenges of standing up their federal accountability systems for 2021-22.
While the field learns more about the extent and impact of COVID disruptions, those disruptions are certainly impacting federal accountability in the near and long term and in varied ways across states. Given this variation, states need custom approaches to supporting their accountability system work, which is why, working closely with CCSSO, we developed a plan to match interested states with their peers and selected experts in a mini-TAC (technical advisory committee) structure that we called “critical friends sessions”. We used those sessions, which took place in May, to dive into state-specific topics on implementing ESSA accountability in 2022.
It was clear from these discussions that state leaders are deeply interested in maintaining the integrity of their accountability systems despite the complications of the year. Everyone – including those in the invited-expert role – felt the challenge of working with limited or compromised information. They craved information from other states while doing their best with what information they had.
And the stakes are high. State leaders have had to make decisions that impact which and how many schools will be identified for different types of support and intervention, and whether already-identified schools may exit that designation. These decisions must anticipate the best direction for carefully designed accountability systems, sometimes without key information, and other times in a hurry, when the necessary information is finally available. Accountability systems face such challenges in any year, but accountability for 2021-22 is uniquely tough because states’ understanding of COVID’s impact on their educational systems has been complicated by compromised data. And we all lacked a complete picture of state plans to adjust for the data complications of the last two school years.
Filling in the State Accountability Picture for 2022
Eager to flesh out that incomplete picture, we distributed a survey to 58 entities with accountability systems: all 50 states, the District of Columbia, the Bureau of Indian Education, the Department of Defense Education Activity, and five U.S. extra-state jurisdictions. The survey was fielded in April and May of 2022, and asked respondents to share recent test participation, graduation rate, and attendance rate data; information about their plans to make changes to their ESSA accountability systems for 2021-22; and how they were addressing reporting and communication challenges associated with data limitations resulting from the pandemic. For simplicity, I will refer to all respondents as states in the remainder of this post.
We received 52 responses. The most important takeaways were:
- Most aspects of states’ accountability systems were changing in 2022.
- Most states are coping with incomplete or uneven data.
- Outcomes and identifications of schools for additional support had different meanings in 2022.
- We should all interpret accountability results with caution.
Accountability Systems – and What They Mean – Are Changing
States are shifting their accountability systems in two ways. They’re: 1) changing the indicators themselves, and 2) changing identification and exit criteria and timelines. These changes are for 2021-22 accountability, that is, the reports state education agencies are releasing this fall.
Most respondents planned to change their academic achievement and/or growth indicators for 2021-22. More than a third planned to change academic achievement, and more than half were changing growth calculations. Interestingly, states’ planned approaches varied greatly.
Of the 18 states reporting plans to change their achievement calculations, 14 said they’d change which and/or how many years of data they use. A few reported plans to change performance expectations or how they weight the achievement indicator. Some states planned to apply multiple changes.
Of the 27 states reporting plans to change their growth calculations, 10 said they would shift to a “skip-year” approach. Seven states planned to change the number of years they factor into their calculations. Some planned to use different growth targets or replace growth with a non-growth indicator. Two said they’d base growth calculations on a different prior score. As with the achievement indicator, some states are applying multiple changes.
Additionally, most states reported that they’re changing school identification timelines, exit criteria, or both. Fifteen states had – as of May – elected not to exit schools in 2022 (for 2021-22 accountability).
The scope of these states’ plans was far-reaching. Our survey showed that 70 percent of our respondents (33 states) planned to submit an addendum – the legal provision that permits accountability changes for 1 to 2 years – to the U.S. Department of Education. Fifty-five percent of respondents (27 states) planned to submit amendments (often in addition to addenda) for permanent changes to their accountability systems.
States’ plans were not just talk. According to the U.S. Department of Education, 27 states have approved addendums and 10 states have approved amendments as of October 2022.
Unprecedented Data Challenges Caused by the Pandemic
Both test participation and attendance rates were lower in 2020-21 than they were the last time statewide assessments were administered, in 2018-19. Statewide test participation rates were below 80% in 2020-21 in more than a quarter of the states. Test participation also varied across districts and by student group within states. These findings were reinforced in critical friends conversations. States with irregular test participation have had to grapple with how best to determine identifications with less reliable school- and student group-level outcome data.
In normal times, attendance-based measures are generally the most stable component of accountability systems. But the last few years have not been normal times. Compared to 2018-19, 35% of respondents reported lower statewide attendance rates in 2019-20 and 44% reported lower rates in 2020-21. States were also concerned about the quality of attendance data, though they’re a little less concerned in 2021-22 than they were the previous year. Four in 10 states reported a “substantial observed pandemic impact” on attendance data completeness and/or quality in 2019-20 and 2020-21. That number dropped to 13% for 2021-22.
Graduation rates seem less affected by COVID, but it’s worth noting that states have adjusted graduation requirements in ways that may have cushioned those rates from the negative impacts of disrupted opportunity-to-learn.
What Does This All Mean for 2022 Accountability?
Considering what we heard in critical friends sessions along with survey findings, we must start with an important acknowledgment: Data from 2021-22 are not only required by law but critical to expanding our understanding of the pandemic’s impact and examining potential patterns of recovery. Still, and not surprisingly, caution is necessary.
- School or student-group level comparisons will be problematic without taking into account information about student coverage (participation, data completeness) and possible shifts in the population (enrollment changes).
- Comparisons across years cannot be made because of changing definitions and indicator calculations without clear documentation and a way to account for population changes.
Put simply, context matters, especially if making claims about supposed patterns of school or student group performance. Those responsible for implementing accountability determinations for 2021-22 have an unenviable task, but they should remember that ultimately, their work is just one part of the accountability and school-improvement picture. Everyone who interprets and uses the results of the system is part of the picture, too.
To reduce the risk of misinterpretation and misuse and protect system integrity over these years of change, it’s important to be clear upfront about the vision and purpose of the system and to provide as much detail as possible when reporting the results. These findings – which are also summarized in this slide deck – certainly impact accountability for 2023 and beyond. These posts by my colleagues Chris Domaleski and Juan D’Brot, about principled approaches to accountability, are a fitting starting point for states as they consider the appropriate next steps for their accountability systems.