Issues and Considerations that the COVID-19 Pandemic Presents for Measuring Student Growth

Discussing the Implications of Calculating Student Growth Scores for Accountability and Using Growth as a Tool to Analyze the Impact of School Closures on Student Learning

The COVID-19 (novel coronavirus) global pandemic is having far-reaching effects in all facets of our lives. The impact on student education has been discussed in general terms in the media relative to school closures, and we at the Center for Assessment have been adding to the conversation by giving our thoughts on how changes to statewide assessment administrations may impact state efforts toward measuring student growth for the current and subsequent academic years. 

The following lays out many of the scenarios confronting states and presents options states may want to consider as they plan for a resumption of testing in 2021.

Possible Assessment Administration Scenarios

Depending upon the state, there are several ways in which the pandemic has impacted student assessment in Spring 2020. Here are some of the feasible assessment administration scenarios that we can foresee (or have already seen) states taking in the face of COVID-19 outbreaks across the nation.

• No testing
• Full (normal) testing (e.g., WIDA-ACCESS English Language Proficiency Testing administered in the winter before school closures)
• Partial testing (test forms administered to a portion of students before school closures)
• Abbreviated testing (shortened test forms administered)
• Delayed “normal” testing (administered at the end of the delayed school year or beginning of next school year)

States administer several assessments each year, so it is possible for several of the above scenarios to play out in a single state. Within a single assessment, most state education agencies will likely mandate a uniform response for all schools in the state regardless of differing regional impacts of COVID-19, meaning each state or district will likely face only one of the possible scenarios for their primary assessment program. However, given that states have additional assessment programs (e.g. English Language Proficiency assessments) with differing administration windows, multiple scenarios may be relevant.

General Considerations for the calculation of student growth

In more than a decade of work advising states on the calculation of student academic growth, the Center for Assessment has encountered numerous assessment situations presenting less than ideal circumstances for the calculation of student growth:

• Several states participating in the New England Common Assessment Program (NECAP) took summative assessments in the fall of 2013, and then as members of PARCC took tests in spring, 2015 – a year-and-a-half gap for the calculation of student growth. 
• Many states have transitioned between assessments and have pursued measuring student growth using assessments consisting of different formats and sometimes built toward different content standards.
• States administering end-of-course assessments often encounter situations where students do not take exams in consecutive years leading to the calculation of growth spanning two or more years.

Because of the multitude of ways that states have administered assessments over the past decade, in discussions with states/organizations about measuring student growth, it is often useful to separate discussions of growth into: 

• an analytic component (can a particular growth quantity actually be calculated), 
• and a use component (is it valid to use a growth quantity for a given purpose). Fulfilling the first part is often easier than the second.

No Testing in 2019-2020

At the time of publication, 47 states have canceled their general state summative tests. The cancellation prevents states from the calculation of most of the academic measures used for accountability including the calculation of annual student academic growth between 2018-2019 and 2019-2020. Looking forward, annual student academic growth for 2020-2021 will be impeded as well because of no prior score from 2019-2020.

Because academic growth is central to many state’s accountability systems, many states are actively investigating the calculation of student academic growth spanning two years: 2018-2019 to 2020-2021. In general, this span doesn’t present any difficulty in the calculation of student growth quantities like SGPs. However, whether they can be used in a manner similar to one-year growth measures requires justification.

Full Testing in 2019-2020

In other testing circumstances, states were able to complete statewide testing (e.g., English Language Proficiency testing), enabling the state to go forward with a business-as-usual approach in 2020. However, looking forward one year, criterion-referenced growth analyses may be impacted due to canceled school issues from the previous spring.

Partial Testing in 2019-2020

If partial testing was performed, depending upon the proportion of students tested in the state, norm-referenced analyses (e.g., SGP, value-added, percentile ranked residuals) might mask issues related to selection bias (the students for whom the growth norm is calculated isn’t representative of the entire state). One way around this issue is for the state to use growth norms calculated from prior years as a substitute for those that would be derived in the current year.

Comparing Short- and Full-Form Testing

In cases where a state implements abbreviated testing (e.g., administering a short form), the calculation of growth is once against straightforward. However, comparing growth results based upon an abbreviated form to those from previous years where a full form was used is contingent upon the equivalence of the two forms. If the forms differ substantially, then the growth results will not be comparable. This situation has come up repeatedly in recent years within assessment consortia providing abbreviated forms as well as offering dual-mode testing (online and paper). 

Testing Delays

Lastly, at least one state is considering whether to delay some high school state summative testing (e.g., PSAT and SAT) until the fall. Student academic growth could be calculated from spring 2018-2019 to fall 2020-2021. The use of those growth scores in accountability would require justification as well. 

The variety of ways in which state testing is impacted will require a multi-faceted approach.

General Considerations for the Use of Student Growth for Accountability

When the usual test data is either not available or fundamentally different in some way from what has traditionally been used, states should thoroughly investigate whether the uses of such data, especially for high-stakes decisions, are appropriate.

States should investigate and inform such decisions by performing the proposed analyses on existing historical data to investigate the impact of using such analyses (e.g., using two-year academic student growth) instead of what was used (e.g., one-year academic student growth). Disparate results from the different techniques would provide strong evidence that the proposed approach is inconsistent with the state’s standard approach. 

Even in situations where partial testing occurred, using historical data to see the impact of how similar partial testing would affect accountability results would help inform the use of such results even for non-accountability reporting purposes.

Beyond Accountability: Using Student Growth for Research Purposes to Inform Policy

Severe disruption to regular school activities will likely have a negative impact on student learning, which will almost certainly be reflected in student test scores. These impacts may not be consistent across a state. For example:

• Impacts may be geographically concentrated and students in sub/urban schools may see more widespread and long-lasting school closures than rural areas.
• Impacts may be concentrated due to digital access such that students with limited access to remote learning opportunities may be affected more than other students in their own school or other schools who are better able to access those resources. 
• In states that do pursue testing in some form, we may also see that schools/teachers/students in highly-impacted areas may also put far less emphasis or effort into those tests, resulting in lower-than-expected test performance this year.

Investigations of (differential) impact on student achievement and growth will be critical to help ameliorate impacts of the disruption to education of the COVID-19 pandemic. Student growth percentile analyses, in particular, are ideally suited to quickly investigate the impact of disruptions to student learning.

• Using historical results associated with two-year growth rates of student groups, compare two-year results (e.g., mean SGP) for student groups between 2017 and 2019 with those for 2019 to 2021. Disparate results between groups indicate that equity disparities should be investigated thoroughly.
• Investigate the impact of the pandemic on student learning between 2019 and 2021 by examining 2019 to 2021 two-year growth alongside two-year growth from 2016 to 2018 and 2017 to 2019. Combining the two or more cohorts of data allows the state to compare growth rates across years and identify the impact of the COVID-19 pandemic by calculating the mean SGP for students who are part of the 2021 cohort.
• If fall 2020 data is available for a given assessment (e.g. an interim assessment test at the beginning of the year), use that data to try and determine whether school disruption and student learning loss is more extreme for some groups over others. Results such as these are important for informing interventions in the fall.
• Even though several states completed 2019-2020 English Language Proficiency testing, there may be lasting impacts on language acquisition for English language learners due to school closures even into spring 2021. Investigation of 2019-2020 versus 2020-2021 growth would be warranted.

Depending upon the circumstances present in a state, many other analyses can be pursued using growth data that many states calculate as part of their annual summative assessments. These analyses have the potential to help states understand specific impacts on student learning, which can be critical in overcoming the negative ramifications of this pandemic.

Recommendations

The effects of the COVID-19 pandemic on state assessment and accountability are likely to be significant and long-standing. Even if school returns to normal in the near future, the cancellation/interruption of student testing in almost all states will lead to ripple effects within accountability systems for several years, especially relative to student growth, which takes account of prior years. 

As states move to grapple with these changes, we recommend they quickly begin empirical investigations of potential 2020-2021 data analysis options using historical data assessment data. If alternative analytic strategies are pursued (e.g., using student academic growth from 2018-2019 to 2020-2021), then it will be necessary at a minimum to have validated those approaches using existing data.

More importantly, conducting growth analyses can inform policy and practice beyond just accountability. Educational equity will likely be a major concern going forward and growth analyses can be of significant assistance in identifying the differential educational impact of the COVID-19 pandemic so that policies can be tailored to ameliorate these problems. Though there will be overlap in any analytic approaches states apply, there will likely be idiosyncrasies across states requiring unique approaches and analyses.

Adam Van Iwaarden is the co-author/co-developer of the SGP package. For a decade, Adam has led SGP analyses in more than a dozen states and has co-developed innovative analytics extending the base SGP functionality.