Addressing the Accountability Challenge of Missing Data With A Performance Profile

Consider a Different Way to Put The Pieces Together

In a previous post I discussed the challenges of rebuilding accountability systems developed for the Every Student Succeeds Act (ESSA) in 2021-2022. Prominent among these challenges is missing data due to multiple years of pandemic-related disruptions. What are the best alternatives to solve this problem? In this post I’ll suggest a tool, the performance profile, that may help address the issue of partially missing data.   

The missing data problem is particularly pernicious when it comes to computing academic growth. Obviously, growth requires at least two measures:

1. A prior test, which is usually the state test a student takes in the previous academic year, and 
2. A post test, which is the state test from the current year.

Many growth models use more than one prior test score when possible. Even if nearly all students test in 2022, some will be missing a prior test score from 2021 and no students will have a test score from 2020.  

What is an Accountability Index Score? 

Before discussing the performance profile, it’s useful to discuss how state accountability systems typically combine indicators to create an overall measure of school performance. The most common method is the weighted composite or index score. There are a few ways to implement it, but it is similar to the method classroom teachers often use to calculate course grades for students. Various assignments are weighted, and at the end of the course the teacher calculates the final grade by aggregating performance across all assignments using the weights assigned to each. Substitute accountability indicators for assignments and an accountability index might look something like the following simplified example:     

Calculating an Accountability Index When an Indicator is Missing 

Suppose growth scores cannot be calculated for some schools in 2022 due to missing data in 2021. One might be tempted to simply remove Growth as an indicator, redistribute the weights in proportion to the remaining indicators, and calculate a new accountability index score. Many states routinely use this approach for schools that do not enroll a sufficient number of English learners to compute a score on the Progress in English Language Proficiency indicator. 

Using the illustration above, the accountability index results without the Growth Indicator might look like this: 

Note that the performance on the remaining indicators remained constant, but the total scores changed substantially. School C, which achieved the highest score of the three schools due to strong growth scores, now has the lowest score of the three. By simply dropping growth and redistributing the weights, the accountability model is measuring something very different; it’s much more heavily focused on achievement or status.  

That information will be of limited utility in helping education leaders identify the schools that are most urgently in need of support. Consider, too, that school B looks relatively favorable without growth, but much less favorable when their low growth rate is considered. Without growth, the accountability system fails to detect school B’s downward trend, which may signal a need for support. 

One might argue for redistributing weights differently (e.g. assign more weight to attendance) but that, too, fundamentally changes the meaning and interpretation of the accountability score.  

Consider Using a Performance Profile Instead of an Accountability Index

There’s another way to combine indicators, which is commonly termed a performance profile. Unlike an index, where higher performance on one indicator offsets lower performance on another, this method doesn’t treat all indicators alike. Rather, the method explicitly defines which configurations of performance across indicators, or performance profiles, are valued.  

There is nothing in ESSA that prohibits aggregating indicators in this manner, and although not widely used, some states began using performance profiles prior to the pandemic.

As one simple example, assume designers want to determine rules for entry into or exit from a support classification based on a set of indicators, each of which is assigned a rating of Below Standard, Satisfactory, or Exemplary. Using this approach, designers may create rules to define the profiles that are acceptable, such as: 

• Any school that is Below Standard in both Achievement and Growth must be prioritized for support, regardless of their performance on other indicators.  
• To exit support, a school must be Exemplary in either Achievement or Growth and no lower than Satisfactory on all other indicators. 

Some example profiles based on these rules might look like the following:

What Does This Have To Do With Missing Data? 

A performance profile is better suited than an accountability index to force an answer to the question: is there sufficient evidence to support a decision about school classification? If data are missing, a performance profile reveals that insufficient evidence is available, which is an important distinction in contrast to simply redistributing weights in an index system. Instead of producing an alternative index score with only the appearance of comparability, a performance profile approach acknowledges there may be too much uncertainty to make a consequential decision without additional steps, as with school C in the previous example.  

What should education leaders do in this case? One approach is to retain the current accountability condition under the assumption that assigning a new category or exiting an existing category is not supported by evidence. Another approach is to evaluate that school with respect to a different source of available evidence appropriate to the decision. For example, the school may be evaluated based on an alternate measure of academic growth, survey results, a qualitative review, and/or other factors. The decision threshold for alternate evidence should be appropriate to the consequences as discussed in a prior post.     

Conclusion

It is important to be honest about the uncertainty associated with missing data. Simply recalculating an index score with a subset of components only masks the problem and risks communicating information that cannot be meaningfully interpreted. It is preferable to adopt an approach that is clear about the sufficiency of the evidence required to inform decisions about school support and the impact of missing data. Establishing performance profiles is one way to accomplish this and will create a system that is more credible and useful.