Subtraction in Assessment and Accountability (Re)Design
The Overlooked Power of Less
In design and problem-solving, subtraction often takes a back seat to addition. As Leidy Klotz explains in his book Subtract: The Untapped Science of Less, the impulse to add—to increase complexity, introduce features or layer systems—is deeply ingrained in our biology, culture and economic systems.
Klotz details how humans often add—in ventures ranging from bike design to bridge-building—instead of subtracting in our search for good solutions. He then describes the ways that subtraction can bring us to better solutions.
After reading Klotz’s book, I couldn’t stop thinking about the implications of this idea for assessment and accountability. In this post, I’ll explore subtraction’s relevance and appeal when designing and redesigning these systems. After I discuss subtraction, I provide three scenarios where subtraction can lead to better assessment and accountability systems. I conclude with implications for those who work to improve these systems.
Subtraction as Action
Subtraction is an action. Less is an end state. Sometimes less results from subtraction; other times, less results from not doing anything. There is a world of difference between the two types of less, and it is only by subtraction that we can get to the much rarer and more rewarding type. (Klotz, p.16)
The concept of subtraction as an action, not an endpoint, is critical to understanding what it entails. When we consider what we can remove from an existing structure or idea to achieve some goal(s), we are in subtraction territory.
A teacher who removes a redundant project from her lesson planning is engaged in subtraction. So is a district leader who stops requiring a test with no evidence of instructional utility. Ditto for the technical advisory committee that encourages a state with an unwieldy number of indicators to consider limiting to a handful those for identifying schools in need of support.
It’s easy to agree with Klotz that subtraction and addition are complementary design tools; the reason to attend to the first is that we are especially prone to “subtraction neglect,” as Klotz writes. Thus, we need to deliberately bring subtraction to the forefront.
Here are three scenarios illustrating how subtraction could—or did—benefit states’ work on testing and accountability.
1. Streamlining Legislative Requirements for Accountability Tests
It is a tall order to require a test to (a) help identify the schools most in need of support and (b) provide parents, teachers, and school leaders individual feedback on the specific academic needs of students as indicated by their performance on specific items. And yet, this is what the federal Every Student Succeeds Act (ESSA) requires.
That second requirement has helped spur the production of subscores, despite research showing that for essentially all state and interim tests, such scores do not add information beyond what is already conveyed in the total score. Reporting subscores that do not provide the added value conjures an illusion of utility, feeding demand for them.
For this and related reasons, prominent assessment experts have called for removing the second requirement (information about students’ academic needs derived from their performance on specific items) from any future reauthorization of federal education law.
2. Aligning Accountability Systems With State Priorities
The economics Nobel laureate Herbert Simon coined the term “satisficing” in 1956. Klotz adopts this combination of “satisfying” and “sufficing” to describe a decision-making or problem-solving strategy in which one searches through available alternatives until an acceptability threshold is met. This might describe the process some states used to develop their comprehensive state plans for ESSA.
They developed solutions that blended federal requirements with state priorities. The result was a single system that satisfied two needs (state and federal). This strategy sets up a potential problem: The single system has more layers than it needs for its primary purpose (federal accountability), and revising it requires federal approval, even for features that were never federal requirements. Subtraction offers a way out of this conundrum.
My colleague Brian Gong argued that “[s]tates and their partners can operate two accountability systems in parallel: The one required by the federal ESSA law, and another one they design to reflect their goals and values more fully.”
Admittedly, there is an obvious element of addition here (two systems rather than one). But it took subtraction to get to the resulting, leaner approach Brian describes: accountability mechanisms not required by ESSA are moved out of the state’s federally required accountability system and placed in a more flexible domain where the state can freely innovate.
This gets us to a more optimal state of “post-satisfied less,” where the requirements of ESSA are minimally met, and the state can pursue its priorities. (For a review of different models for balancing federal requirements and state priorities, see Chris Domaleski’s five ways to “right-size” ESSA accountability.)
Along these lines, the Center recently advised a state that wanted to remove long-term goals and progress measures from its calculations of the index it used to identify schools for support and improvement. Including these goals and measures explicitly in the index had complicated the state’s efforts to shed light on school improvement.
Because the state shifted targets annually (as a direct consequence of including the progress measures in the index computation), a school could make real progress on its indicators yet receive a decreased score on the overall index, undermining any meaningful year-to-year interpretations of change for a school. The state can introduce greater coherence in its accountability system with a subtractive move: keeping the measures but removing them from the computations.
3. Incorporating Student Outcomes in Educator Evaluation Systems
Some states require schools to include student outcomes in the formal evaluation of educators. My Center colleagues and I are helping one such state improve its process for doing this. During the early phases of the project, the state’s task force participants defined five quality criteria for teachers’ student-improvement goals and engaged in group exercises to describe how proposed goals meet those criteria.
In a small-scale roll-out, however, we learned this process wouldn’t work in practice. We planned to ask teachers to justify their student-improvement goals against the five quality criteria and then ask their supervisors to evaluate those justifications. However, we realized that time demands on teachers and supervisors would likely lead to bare-minimum compliance.
As in the other cases, subtraction was our friend here. We removed that documentation step and asked educators to review the quality criteria and say why they chose their goals. We also encouraged educators to use existing high-quality measures rather than feel they needed to design their own, and we reduced the materials educators would need to review for the goal-setting process from 40-plus pages to three five-minute videos.
Foregrounding Subtraction in Assessment and Accountability (Re)Design
As an antidote to our built-in subtraction neglect, I propose three areas and associated questions for those of us engaged in assessment and accountability (re)design.
- Confront subtraction neglect. Am I neglecting subtraction? Can it positively affect whatever problem or challenge I’m considering? Candidate areas for subtraction include components, processes, steps, expectations, regulations, barriers, disincentives, staff time and information load.
- Address barriers to subtraction. Klotz points out that loss aversion and the negative valence of subtraction terminology are powerful psychological barriers working against us. How can we overcome these barriers?
- Put addition back on the table. How can addition and subtraction complement each other in my particular (re)design setting to enable our systems to approach the ideal of post-satisfied less?