School Disruption Due to COVID-19

A High-Level Overview of Likely Implications and Options for Assessment and Accountability

Like many of you, I’m closely following the news of the growing threat associated with the COVID-19 virus. Obviously, it is a difficult time for many and my primary thoughts are with those whose health or welfare are directly affected.  

During this time, education leaders are grappling with tough decisions regarding whether to close schools or suspend activities. These decisions have many wide-ranging implications, and one such implication we’re focused on at the Center is the short- and long-term impact to assessment and accountability systems. 

I outline below what I believe are primary implications of closing/suspending schools and some of the options available to address disruptions to assessment and accountability programs. I focus on the initial and primary questions I believe will be raised by a partial or full disruption of the elements that contribute to a state’s assessment and accountability system. Decisions such as closing schools, suspending testing, or reducing the testing window are beyond the scope of this post and are influenced by factors more significant than administrative or technical concerns. I also do not address implications with respect to federal or state law. However, it’s worth noting that the United States Department of Education has already signaled an openness to targeted one-year waivers for selected assessment and accountability provisions of the Every Student Succeeds Act.  

To be clear, this post is a “first-take” intended to quickly provide some ideas and give shape to the discussion. In the coming weeks, the Center will be working closely with states and partner organizations to offer more detailed guidance and support.    

What Is the Potential Impact for Assessment?  

There are many potential impacts on assessment results, but they generally fall into one of the following conditions given the types of actions being considered in response to the threats posed by COVID-19:

Naturally, each of these conditions raises important questions regarding whether the results should be reported or used for school accountability. 

Producing assessment reports may be appropriate and defensible for the above conditions 3 and 4 at the student and/or summary level if the state can affirm the integrity of the conditions for instruction and assessment. Even so, the state may have a policy interest in withholding scores if their release is thought to lead to misconceptions or promote ill-advised practices. 

For example, providing summary reports based on a non-representative subset of participants may give rise to confusion about performance trends.   

It is always advisable to withhold individual and summary reports when there are credible threats to meaningful interpretation as in condition 2 above. 

What is the Potential Impact for Accountability? 

Accountability issues are even more complex. Missing or incomplete assessment results raise questions about comparability of accountability determinations within and/or across years. For example, some schools may have the same subset of data used in the state’s accountability system, such that comparison of performance among these schools in the same year is technically feasible and may even seem sensible. But these results will not support the same meaning and interpretation as those from the full system in previous years. Accordingly, it is not advisable to proceed with accountability reporting and/or consequences without making appropriate adjustments to existing model features such as weights, performance standards, or reporting categories.   

Condition 4 above describes the situation where there may be some schools with full data in the current year and previous years. In this case, claims of longitudinal comparability may be supported for those schools with complete data. However, comparability within a year to schools with partial data would not be supported.  Given that many states apply normative criteria to establish performance categories (e.g. the lowest 5% of Title 1 schools are designated for Comprehensive Support and Improvement (CSI), this condition also presents threats to reporting accountability outcomes that can be interpreted and used as intended.

Since results from condition 2 above should not be reported, it stands to reason they should not be used for accountability. Therefore, this scenario is a subset of the issues described for conditions 1, 3, and 4 and ultimately leads to the same conclusion:  reporting meaningful accountability results is threatened. 

Additionally, if complete and comparable assessment results are missing, it is likely that other accountability elements are compromised as well, such as data related to course completion or attendance. In sum, disruptions pose multiple and serious threats to statewide school accountability systems.  

What are Some Options for Accountability?

Given that the threats to producing meaningful accountability results with missing data are multi-faceted and substantial, what are the options with respect to accountability consequences? 

Most states will likely conclude that some version of a ‘hold harmless’ policy is the most reasonable course of action. A common method for applying this policy is to have schools maintain the same accountability designation from the previous year with no cumulative impact for cases where multiple years with the same designation could lead to a more severe consequence. 

Alternately, states may determine new criteria and standards for an accountability designation based on a subset of the same indicators or different set of indicators. From a ‘hold harmless’ perspective, such an approach may be applied to provide an opportunity for schools to demonstrate evidence of improvement only (i.e., the school receives the more favorable of the prior year or current year designation).  However, the comparability questions raised in the previous section may lead to legitimate concerns of fairness if some but not all schools have an opportunity to demonstrate progress or the outcomes are based on different indicators.    

It is certainly possible to apply statistical tools to address the impact of missing data to a limited degree. For example, if some student data are missing from a school or district, methods are available to impute values or estimate a range of projected performance to approximate the plausible population values. However, there are substantial limitations to these approaches from both a technical and policy perspective, and they should be avoided unless certain key strong assumptions are met. 

Looking Ahead

Unfortunately, the impact of disruptions in assessment on accountability systems are not likely to be restricted to a single year. For example, nearly all state accountability systems incorporate measures of academic growth that require one or more prior measures to produce an outcome. Additionally, the common practice of banking test scores and multi-year averaging for small n-sizes are other examples of disruptions that span multiple years. None of these issues alone is necessarily insurmountable, but together they create some non-trivial challenges.  Consequently, waivers and/or flexibility may be necessary beyond 2020.  

Final Reflections 

I’ll close by reiterating that the foremost concern as the threat of COVID-19 grows is obviously the health and welfare of our communities. The issues and implications for assessment and accountability are far less important. Even so, I know the compassionate and thoughtful professionals I work with in educational assessment and accountability are motivated to develop effective solutions to address the full range of challenges ahead. I hope this post gives some momentum to that conversation. The Center for Assessment will continue to put out technical and policy briefs to help support our hard-working state and district colleagues.