Improving Equity: What Makes Accountability Indicators Meaningful?
Ensure You’re Receiving Valuable Information Necessary to Support Student Equity and Improvement
This is the third in a series of CenterLine posts by our 2019 summer interns and their Center mentors based on their project and the assessment and accountability issues they addressed this summer; and it is the second post by Nikole Gregg. Nikole is from James Madison University and worked with Brian Gong on how states have attempted to promote equity through the design of their ESSA accountability systems.
In my previous post, I described ways in which states differ in their processes for identifying schools that need mandatory support from the state, which in part contributes to the differences in identification rates across states. However, the identification process by itself is insufficient to improve equitable outcomes in terms of student learning.
Creating Meaningful Change with a School Improvement System
There is an entire improvement system that needs to be in place to meaningfully make changes to the equity within state education systems. This improvement system includes:
- identification from accountability scores,
- support from the state,
- purposeful evidence-based intervention at the local level, and
- a post-intervention evaluation of accountability scores.
Furthermore, there must be established goals for improvement. Thus, the identification process is just the start to a larger improvement system to improve equity.
An uncommon consideration within this improvement system is how the accountability measure is connected with the intervention. If accountability or indicator scores do not provide context of student ability, then it is difficult for schools and districts to take appropriate action from accountability scores. Thus, the accountability and indicator scores should be easily and appropriately interpretable for use, while balancing the reliability of the indicators and accountability system.
There are great resources regarding the reliability of the accountability system. Specifically, this work by Kane and Staiger pertains to the likelihood of misclassification of schools into identification categories. However, when considering how the accountability system is being used, we are speaking to the fidelity, credibility, and utility of the accountability system. One way to support the utility of the accountability system is to improve the interpretability of the indicator measures. For example, if a measure does not produce a score that is interpretable and usable by a district or school, a state may consider adjustments to this measure so that schools and districts can interpret accountability scores to inform equity interventions.
How Can we Use Indicators to Improve the Interpretability of Equity Information?
Equity is defined as the improvement of underperforming student groups. Thus, this proposed indicator intends to evaluate who and where the lowest performing student groups are, and to evaluate the change in student group performance over time. Ideally, this change over time is reflected in improvement of student group performance; however, there is substantial evidence that lower-performing groups of students also have lower growth rates over time, meaning that it is almost impossible for students to catch up. In order for lower-performing students to catch up to their higher-performing peers, interventions must be well-informed, well-implemented, evidence-based, and occur early in the student’s educational experience (i.e. before third grade).
One way to help inform interventions is by having an interpretable and useful equity measure. As an example, the Center has recently been thinking about how to recalculate a state’s proposed equity measure to best inform possible equity interventions.
Currently, this equity measure is based on comparisons of subgroup performance at the school and the state-level.
- For each subgroup, the state accountability score for that subgroup (e.g. students with disabilities) is subtracted from the accountability school score for that subgroup.
Subgroup Score = School Subgroup Score – State Subgroup Score
- The equity score for each school is the unweighted average of the subgroup scores for each of its subgroups.
Equity Score = (Subgroup1 Score + Subgroup2 Score + … + Subgroupn Score )/Number of Subgroups
- A negative score means that aggregated across all subgroups, the school subgroups are performing lower than the state average. A positive score means that across all subgroups, the school is outperforming the state average.
A key point is that the equity measure does not give subgroup-specific information to the school or district. Consider this school example (School 2 in the table at the end of this post), where the average equity score across all subgroups is 5.435, indicating that the school’s subgroups are outperforming the state average. However, the Hispanic students within this group have an equity score of -25.04, indicating that this specific subgroup is performing well below the average accountability score of the Hispanic students in the state. This information is canceled out with the averaged equity score currently used by this state. Therefore, this school would not be flagged, although their Hispanic students are underperforming; consequently, the school and public might not know that intervention is necessary to improve the performance of its Hispanic students, and improve equity.
In other words, this information of Hispanic student underperformance is hidden in the current calculation of the averaged equity score. It is important to note that within the state dataset, this is not the only occurrence of this canceling-out phenomenon (See Table 1 for more examples). This phenomenon occurs for about 10 percent of the schools within this state.
Are All Goals Equal in Promoting Student Equity?
Furthermore, consider again the definition of equity: the improvement of underperforming students.
Currently, this state equity measure compares student subgroup performance to the average performance of that particular subgroup. What if statewide this particular subgroup is underperforming compared to all students? Is it working toward equity if the goal is to improve those subgroups only to the still-underperforming subgroup state average?
It may be worth considering a new reference point for the equity measure. For example, should subgroup performance be compared to overall student performance? Or, should subgroup performance be compared to a set criterion that reflects the state goals? Either way, these may better reflect the genuine perspective of equity – the improvement of underperforming students – not just comparing underperforming subgroups to a standard that is still underperforming.
In this post, I mentioned two suggestions to help schools better interpret this equity measure:
- The state may consider providing subgroup information, and not just an average score.
- It may be beneficial to re-consider the comparison being made between school subgroup scores and a reference point. This reference point can better represent equity, and subsequently improve the interpretability of the equity score.
Both of these suggestions involve the increased interpretability, and hopefully use, of the equity scores.
These considerations, and other possible considerations of improving the interpretability and usability of accountability scores, will involve increased collaboration. In order to increase interpretability of accountability scores for school and student improvement, accountability systems must include score users in the development process. Teachers, policy-makers, parents, and education agencies that implement and create interventions must be involved in this utility process.
And lastly, for the outside education agencies to interpret accountability scores appropriately, they must be more knowledgeable and involved in the pursuits of accountability systems. There will always be tradeoffs, prioritizations, and compromises in the development of accountability systems, but the system will be more useful when all parties are involved in and understand the system.
In sum, the accountability system can help inform possible interventions to improve equity. However, there must be increased communication and connection between the accountability scores and intervention. There also must be an increase in communication with accountability specialists and education intervention specialists. Then, valued, measurable outcomes by the accountability system can best inform education agencies to improve performance of the most underperforming subgroups.
Addendum
The table below provides additional examples demonstrating how the average score across subgroups can mask both strong and weak performance by particular subgroups of students.
