Understanding Pandemic Impact and Addressing Recovery 

District Leaders Describe the Importance of An Integrated Perspective

At the Center for Assessment, we often work with state departments of education. However, most of the action regarding pandemic recovery efforts takes place at the district and school levels. Therefore, one of the sessions at the Center’s recent Reidy Interactive Lecture Series (RILS) was designed to learn from the work of high-performing districts to understand and recover from the full pandemic impact.

Colleagues from Gwinnett County in Georgia, a large district of approximately 181,000 students across 142 schools, and Shelby County in Kentucky, a smaller district of approximately 7,000 students across 13 schools, joined us for this session. They shared their experiences of the pandemic and recovery efforts. 

One of the objectives of this session was to remind everyone of the importance of the language and framing that are used to think about, discuss, and approach the issue of “pandemic impact” and “pandemic recovery”. We thus began this session by reminding participants that: 

• pandemic impact is a complex, multidimensional phenomenon with many interdependencies that cut across school, family, and community contexts,

• discussions of impact should be informed by appropriate qualitative and quantitative data with stories about possible root causes crafted carefully, and

• addressing the pandemic impact requires multifaceted recovery efforts that need to be fine-tuned in local contexts to be most effective. 

In this post, I briefly review some of the major themes of this session; you can find all presentations on the Resources page of the Center’s website.

The Predominant Perspective on Pandemic Impact

Typical framings of pandemic impact focus mostly on the academic perspective, which is perhaps not too surprising. This perspective involves trying to quantify and visualize the degree to which learners’ trajectories in “core” subject matters have been affected, both globally as well as for meaningfully definable subgroups. In other words, this is a perspective with a predominant focus on math, early reading / ELA, and science. 

General trends in these areas have been concerning across all grade levels and are often broadly consistent across different assessment systems such as educational surveys, state assessments, and interim assessments. As one might expect, however, once one looks a bit more closely, patterns are somewhat complex with the magnitude and direction varying across grade bands and content areas for instance; see this recent literature review and the RILS session materials by Damian Betebenner and colleagues for illustration. 

One thorny issue is the distinction between “recovery” and “acceleration”, which can prove both helpful and problematic. Specifically, when pandemic recovery is discussed in terms of “acceleration of learning” – to make up for the reduced learning as a result of the pandemic – one immediately envisions a “race” or “competition” and the additional pressures that are being put on students. Even though such terminology is technically accurate and can sometimes be helpful for managing expectations about achievement gains in upcoming years, this phrasing can also be dangerous if not carefully contextualized. 

Similarly, as one participant pointed out during the meeting, we are also prone to using fallible language that may put us at risk of constructing flawed explanations. In pandemic contexts, we are de facto comparing an impoverished version of remote learning under crisis conditions – when many organizations had to build out remote capacities on the fly – to more polished and robust versions of remote learning such as those instantiated in online schools. 

In short, academic perspectives should essentially be treated as the entry-point into conversations and strategic efforts around recovery efforts. As one district member noted, when dealing with deep trauma and social-emotional impacts, academic performance is not the primary concern, at least not for a while. Moreover, the pandemic has exacerbated a wide variety of systemic inequities across various student groups (e.g., students with disabilities, students identifying as LGBTQ+, English language learners, students of color, and ethnic minorities) as documented in a recent report by the Office of Civil Rights. These inequities cannot be productively discussed without considering various conditions of learning.

Conditions of Learning and the Impact on Social-emotional Well-Being

Literature reviews have also started to appear in the area of social-emotional learning. In the session we discussed some of the important effects that the pandemic had on the conditions of learning, especially for traditionally underserved and disadvantaged student groups:

• Increased teacher shortages and inequitable access to qualified teachers
• Limited technology access in school and at home by caregivers
• Limited general supports for learning at school and at home from caregivers
• Stronger negative effects on mental and physical health
• Weaker sense of belonging in school and communities
• Fewer opportunities for employment and increased uncertainty about career pathways 
• Acute trauma due to severe illnesses and deaths of family members, friends, and teachers

Our district colleagues reiterated several of these themes and underscored that in-person instruction generally had positive effects on learning. They also discussed the comprehensive, multifaceted perspectives that they manage within their learning ecosystems such as providing additional counseling support, offering refocused after-school programs with foci on behavioral and mental health, reimagining teacher professional development efforts, and so on. Importantly, schools played outsized roles during the pandemic in providing basic nutrition and childcare services for families.

The size of the district plays an important role in how these issues play out, of course. For example, our Shelby County colleague discussed in depth the emotional trauma and ongoing impacts of having lost a nationally known, beloved teacher with close-knit community ties due to COVID. At the same time, because of  its smaller size, the district was able to innovate rather flexibly to implement their alternative school design with minimal administrative overhead, which produced targeted solutions that served their community best. Our colleagues in Gwinnett Country discussed their management problems from a predominantly structural perspective. They foregrounded systemic plans for data analysis, school initiatives, and strategic planning, which led to similar kinds of innovations at local levels. Both district teams emphasized the importance of multi-year strategic planning.

Tackling Data Challenges in Pandemic Recovery Efforts

There are myriad methodological challenges in addressing issues of pandemic recovery, not the least of which is that reliable and meaningfully interpretable longitudinal data are difficult to come by for both academic and social-emotional outcomes. The problem of available data was mentioned by both districts in some way. 

This issue is a national, even international, issue, of course. For academic outcomes, beyond educational surveys such as PISA, TIMSS, PIRLS, and NAEP, the most extensive longitudinal data in a state typically exist for interim assessment providers. These providers often have hundreds of thousands, if not millions, of data points across multiple states from similar student groups that can be leveraged for norm- and criterion-referenced analyses. These analyses are made possible through extensive data engineering, science, and visualization infrastructures and, for some providers, through a strong commitment to effective virtual learning at scale. The lesson learned here is clear: having a systematic data strategy and associated best practices are critical for state education agencies and districts as well.

As a result of these internal capabilities, these providers have released a variety of reports on recovery / unfinished learning in 2021 and 2022. It is worth noting that the kinds of tasks that appear on different assessments are not necessarily closely aligned with those on state assessments, which means that different aspects of particular competencies, as well as different mixtures of competencies, may be assessed across assessments. When it exists, this kind of undesirable misalignment creates an overall incoherence issue that makes acting upon multiple data sources challenging for states and districts. 

Unfortunately, states themselves had inconsistent rates of participation for the recent years of their state assessments, and these rates varied further by grade band, content area, alternate assessments, and subgroups. Inconsistent participation created situations wherein student cohorts across years are not equally representative, which has made systematic inferences complicated to say the least. For social-emotional outcomes, state data are even more limited across the board as systematic attention to these capacities has begun relatively recently, at least at scale and with targeted data collection and reporting.

Closing Note

Dealing with a pandemic on the ground is a very complex business and requires concerted, multipronged efforts that can be implemented systematically at different scales for all districts. Doing so requires a holistic, integrated perspective of student learning and development that cuts across school, family, and community contexts. As my colleague Chris Domaleski pointed out in his closing remarks for RILS, systematic descriptions of performance trends informed by the best available assessment and accountability information to quantify “pandemic impact” are indeed necessary to inform policy conversations. However, developing more nuanced explanatory narratives coupled with a deeper understanding of the root causes of observed effects to develop targeted strategies is much more challenging. 

How are you thinking about the complexities of pandemic impact and recovery in your district or state?