Assessment Consequences Matter: A Look Back at RILS 2024

Exploring the Impact of Tests on Individuals, Systems

Consequential uses of assessment are everywhere; they factor into grade promotion and high school graduation, to cite just two examples. Meaningful intended consequences can open important doors, while negative consequences can shut them, sometimes temporarily, sometimes much longer. 

For example, students depend on accurate results from educational assessments for appropriate accommodations and learning supports. Additionally, strong performance on college readiness assessments provides opportunities for scholarships, financial aid, and entry into students’ preferred institutions. 

Consequential uses of assessment can have important direct effects on individuals, but they can also have long-lasting systemic effects. Consider, for example, the way assessment data shapes school ratings in the current approach to school accountability, which has historically produced mixed results for school improvement. 

We thought it was time to have important conversations about assessment consequences at our annual conference, the Reidy Interactive Learning Series (RILS), last month in Portsmouth, N.H. We had the privilege of spending a day and a half thinking about how to understand, anticipate, and address the consequences of assessment for individuals and systems. And what an engaging event it was! 

In the true spirit of a field where technical expertise meets creative thinking, participants used many clever references to illustrate their points. We recall allusions to time travel, international cities, mixing boards, cake, spaghetti—even werewolves! In this blog we will briefly reminisce and reflect on what we took away from the conference.

Capturing Consequences

We started with a few very simple premises:

• Consequences usually cannot be meaningfully characterized through simple classification schemes. One needs to carefully consider various layers of context and diverse use cases that reflect direct and indirect effects.
• Consequential uses of assessment refer to situations in which the use of assessments or assessment practices has long-lasting effects on individuals or systems.
• Individuals experience effects either directly, through immediate impacts of the assessment information use, or indirectly, through longer-term effects within their systems of use.

We kept these guiding questions in mind throughout all our sessions:

–     Who is affected by our assessment system choices?

–     In what ways are they affected?

–     How do we know this?

–     How can we amplify positive, intended consequences?

–     How can we address or eliminate negative, unintended consequences?

–     How do we know what works?

–    How do we build constituent trust in proposed solutions?

–    How do we shift dominant (or prevailing) narratives to make a difference?

–    How do we include all the constituents in the work?

During the conference, we heard from several colleagues in the field and at the Center who shared their perspectives and experiences in delightful ways; next we discuss a few key takeaways. 

Consequential Takeaways

There is not enough space in this blog to recap all the nuanced discussions that took place. Nevertheless, inspired by the contributions of our participants, we offer some themes that surfaced repeatedly for us during the conference:

• Assessment and accountability are human endeavors. When working on technical issues, we must keep the impact of our assessments on individuals and groups at the forefront of our minds, particularly their implications for systemic implementation. 
• The technical quality of assessments and assessment systems is a necessary but insufficient condition for acceptable use. To fully explore consequences, we need to include a range of research methods and perspectives from other disciplines. 
• Beware of oversimplifications and embrace contextual and systemic complexities when developing thoughtful solutions, even if simpler stories are useful in other moments to engage in strategic conversations. Rich work happens at intersections where issues are intertwined, interconnected, and interdependent.
• Acceptability of consequences can be related to false alarms and missed detections. As with type-I and type-II errors in statistics, the relative acceptability of each will shape implementation and responses.
• Unintended consequences are often reasonably predictable when the right voices are heard early enough in the process; not all blind spots are necessary.
• There is rarely a single, clear, uniform, sustainable and scalable path forward that points to “the one correct” approach, so intentional experimentation, monitoring, and reflection are key parts of the work.
• Building trust is at the heart of assessment and accountability reimagining work, so creating networks and collaboration structures with inclusion, empathy, and reciprocity is critical. A key part of this is for leaders to show up consistently to community, family, and leadership meetings with clear messages and an open mind.
• Consequences get shaped in important ways through the responses that we provide for people with different outcomes and the surrounding cultures we co-create, which can be especially powerful as we move away from a shame-and-blame mentality to an asset-based support mentality. 
• Policies at various levels of the system—federal, state, district, and school—can serve as powerful levers for incentivizing change. But it is often important to seek buy-in from the key constituents that are affected by the policies.
• National momentum to make learning experiences more student-centered and personalized has been growing and, along with it, the desire to reconfigure the roles of assessment, to create measures that signal school quality in a holistic manner, and to support how state agencies and local districts work together better. 
• The relative lack of federal action at the moment provides a great opportunity for local innovation. District and state leaders can signal the importance and impact of their innovations up the policy chain to make the case for particular federal changes in the future. 
• Research-based guidance exists for understanding the impacts of different educational policies and the creation of sustainable, systemic changes in schools with the intended effects around assessment use; our participants shared numerous valuable resources.

In the end, these ideas are not rocket science from a conceptual perspective—as one participant put it, “rocket science is easier”—but are notoriously challenging to put into practice in thoughtful, intentional, and rigorous ways. We saw this illustrated in the various use cases we heard about at the conference and know from our own work. 

Resources For All

We left the conference with a renewed sense of admiration for the hard work that our participants do on a daily basis in service of our children, their families, and their communities. The conference also reaffirmed our respect for the complexities of assessment and accountability work and the importance of carefully blending deep technical work with knowledge of local context.  

If you missed this year’s conference, you nevertheless have access to the main presentations in the RILS section of our library. But if you want to get the full interactive experience, please consider joining us next year in person!