Creating a Framework for Assessment Literacy for Policymakers

How We Can Support Education Policymakers in Making Important Assessment Decisions

This is the first in a series of CenterLine posts by our 2019 summer interns and their Center mentors based on their project and the assessment and accountability issues they addressed this summer. Brittney Hernandez from the University of Connecticut worked with Scott Marion on assessment literacy for policymakers.

Improving assessment literacy has been an important and challenging issue in educational reform and assessment circles for a long time. The focus, however, has generally centered on describing what teachers need to know about designing, selecting, interpreting, and using assessments. 

With few exceptions, the assessment literacy needs of policymakers have been largely ignored. 

And honing in on what policymakers need to know is not as simple as taking what teachers need to know and tweaking it a little. Policymakers engage with fundamentally different aspects of assessment. Not only are statewide assessments different than the assessments teachers use in the classroom, but policymakers must make different decisions and use assessment evidence in very different ways than classroom teachers. 

Scott Marion’s recent post discusses frequent changes to statewide assessments as one way this assessment literacy gap among policymakers has manifested. In this post, I will describe our efforts, as part of my summer internship at the Center for Assessment, to conceptualize policymaker assessment literacy and design approaches for improving policymakers’ assessment knowledge and skills.

What’s Important for Education Policymakers to Know About Assessment? 

If you asked an assessment expert what policymakers need to know about assessment, it would not be surprising to hear, “Well, everything.” 

It’s difficult for those intimately ingrained in technical nuance and contextual uncertainty to parse out and prioritize assessment topics, and it’s difficult for them to fully understand the weight of the assessment decisions being made on a policy level. The challenge in identifying the most important assessment topics for policy use, coupled with a number of contextual factors, has facilitated a gap in knowledge between experts and policymakers. We have been working to bridge this gap by developing a series of assessment topics that are for policy, and, therefore, tailored to the types of decisions that policymakers make.

Supporting Assessment Literacy for Education Policymakers

We considered both breadth and depth in what it means for a policymaker to be “assessment literate”. 

Breadth is concerned with the scope of assessment literacy topics. We’ve identified five general domains (i.e., topics) that are important for policymakers: 

• General Understanding: Key knowledge important for all policymakers responsible for making assessment-related decisions that serves as a foundation for more complex assessment topics,
• Design and Development: An understanding of the options available for creating and selecting an assessment (i.e., procuring a custom test, a consortium test, or a “shelf” test), how to appropriately allocate resources, and how the state assessment can support a balanced local assessment system,
• Technical Quality: An understanding of the concepts related to the quality of an assessment (i.e., validity, alignment, reliability, fairness),
• Quality Assurance: An understanding of processes and procedures necessary to produce trustworthy individual and aggregate test scores,
• Communication: An understanding of reporting methods, communication to support assessment credibility in the eyes of the public, as well as how to appropriately communicate the limitations of assessment claims.

These domains are not hierarchical except that the general understanding domain is necessary in order to effectively comprehend any of the other domains. 

Depth, on the other hand, considers the amount of information necessary in each domain. This concept is partially captured in the structure of the domains, which posits that policymakers should have a comprehensive understanding of the topics in the “general understanding” domain. Beyond that, the depth of information needed to be assessment literate really hinges on the decisions that the specific policymaker is making based on the assessment results.

How Do We Make Assessment for Policy?

Once we grounded what assessment literacy actually means for policymakers, we identified resources that were available and had been developed through state education departments, independent consultants, vendors, and assessment consortia. Some resources were buried in large resource banks and only discoverable upon intentional searches. This situation revealed a need for a platform to host curated materials to purposefully organize and direct policymakers to the different currently-available resources. Such a platform should allow various users the ability to tailor their experience specifically to the topic(s) they need. The resources must be organized in a way that puts policymaker’s time at the forefront, allowing for purposeful and efficient interactions with the platform. 

Since this platform will include curated resources available across the assessment literacy domains, it will provide the opportunity to identify new resource development needs. Some domains are more comprehensively supported by existing resources than others, and identifying where new resources are needed is important for continuing assessment literacy efforts. We are planning, for example, to develop a new module to address the specific content and skills articulated in the general domain to ensure we provide targeted and engaging resources to address this critical domain. 

Developing this platform and new professional learning modules will be crucial first steps to making strides necessary to bridge the gap between assessment and policy by 

(1) providing a unified platform for the larger assessment community to potentially contribute to and directly impact policy, and 

(2) providing policymakers with an easily-navigable platform to effectively consume and use assessment information.

Ensuring policymakers are “assessment literate” begins by understanding their specific roles, the ways that they interact with educational assessment and what information and resources they need to make meaningful change for student learning. Only by capturing a full picture of this level of need can we provide the most impactful solution for supporting education policymakers.