Could Two Through Year Assessment Designs Provide Both Summative and Instructional Information?

An Exploration of Feasibility for a Through Year Assessment System

In a past post, I showed the logical conflict that exists for an assessment to claim to provide both end-of-the-year summative evidence and instructionally-useful evidence intended to improve student learning within the year. The challenge in making both a summative claim about what a student can do at the end of the year and the use of assessment for instructional purposes throughout the year led me to conclude that a through year assessment system could focus on providing end-of-year summative or instructional information, but not both at the same time with the same assessments.

Readers of my previous post with an eye for detail may have already noticed how exactly the summative claim was formulated around what a student knows and can do at the end of the year, and wondered, “Couldn’t summative and instructional purposes both be served if the claims were adjusted?”

In this post, I consider two current through year assessment designs that do adjust the claims in asserting to provide both summative and instructional assessment information from the same through year assessment. While both are clever designs, my conclusion is neither manages to truly provide both end-of-year summative and instructional assessment information from the same assessment.

The Grade Book Assessment Model

The “grade book model”  is quite similar to many teachers’ grading models, in which a summative grade is given for each weekly quiz or biweekly test or each final exam. Then those grades are combined to produce an overall summative grade for the year. The summative claim is not only about the student at the end of the year, but also the student at the times the through year assessments were administered as well as at the end of the year.

For simplicity, my example uses a through year design with four assessments, one in each quarter; although there could be different numbers of assessments at different times of the year. The assessments are intended to be instructional; that is, provide information that can be used to help the students improve their learning after receiving the assessment. The assessments are also counted as contributing toward an end-of-year summative score. If contributing to an end of unit or other grade, the assessments may also count locally as summative at the time of administration.

In our hypothetical example, the scores of the student over the course of the year are given in Figure 1.

Figure 1: Example of a through year “grade book” approach.

One thing the through year assessment provides in terms of summative information is how the student performed during the year. In particular, the through year assessments provide information on how “on track” the student was to the end of the year, at the points in time the assessments were administered.

A “Grade book” Approach to Through Year Assessment

What does a grade reflect? Not what the student “knows and can do at the end of the year,” as the example above shows. A cumulative grade over through year assessments largely marks how well the student did on assessments of intermediate content at times of the teacher’s choosing. In many ways, a grade book approach supports the claim or interpretation: “This student learned the content at the pacing set by the teacher, district, or curriculum.”

Advocates for a grade book approach will need to show how this claim is relevant, especially for students who do learn, but not at the posited pacing schedule.

A “grade book” through year model also creates the need to specify how the evidence gathered during the year (e.g., the four test scores) will be combined to arrive at a summative determination. Is certain assessment evidence more important and therefore should count more than others? Should more recent evidence count more than evidence gathered earlier in the year? What to do about missing assessments? How should a common scale be constructed to combine the different evidence? These are well-known issues for teachers when constructing grades, and also hold true when creating a grade book model for through year assessment. Valid interpretations of scores based on a grade book approach would most likely require an understanding of what content was assessed when, the weights for each through year assessment, and other important variables.

There may be many good reasons for a grading approach to summative assessment, including providing motivation, accountability, and general feedback during the year. However, as I argued in my previous post, a through year assessment can provide evidence suitable for an interpretation about what the student knows and can do at the end of the year as long as the student does not change before the end of the year. It should be clear that the claims about student performance provided by a through year grade book approach should not be deemed comparable to an end of year summative determination, especially when the instructional assessment is effective.

The Adaptive End of Year Test Model

Another through year assessment model purports to take advantage of the strengths of adaptive multi-stage testing. Essentially, the through year information collected in each assessment event provides an estimate of student ability at the time of that assessment; the most recent estimate of student ability is used to place the student within an adaptive assessment (See Fig. 2). The adaptive end of year test model does not change the traditional end of year summative claim so much as it purports to collect the information to support that claim more efficiently.

Figure 2: Example of through year “multi-stage adaptive” approach.

In this model, the only evidence to inform end of year summative claims is provided by the end of year assessment. The other through year assessments are providing what may be thought of as a “suggestion to improve efficiency” of the end of year assessment. On the surface, it would appear that this model should provide summative information comparable to a traditional end of year summative assessment, and perhaps do so more efficiently.

However, I think the adaptive through year assessment is providing information largely superfluous to the end of year summative determination. Here are the assumptions or conditions that must be in place for the end of year assessment to provide the summative information needed:

• The end of year assessment should be able to make an accurate determination of student ability, regardless of whether a student completed all, some, or none of the through year assessments. In other words, the through year information is not essential.
• The end of year test should be able to make an accurate determination of student ability, regardless of the starting estimate from the through year assessments. In other words, the through year information does not determine anything in the end of year results; it may make it more efficient. An adaptive test that is starting with an estimate of student ability—especially if that estimate is coming from outside the current test—should have enough adaptive range to “recover” if the starting estimate does not agree with the performance on the adaptive test itself.
• The end of year assessment must assess the grade-level content thoroughly to get a fair and accurate estimate of student ability at the end of the year, no matter the results from the through year assessment. This condition is true anytime, but most obviously when through year results may or may not be based on on-grade level content.

The increase in efficiency afforded by use of the through year assessment results likely is modest. When all of the conditions for content coverage are met, the increase in efficiency might be about 10 minutes in a typical end of year CAT test with good items. The placement prior to starting the end of year assessment into levels such as “high,” “medium,” and “low”—typical for stage adaptive tests—might be equally well done by a teacher rating.

Given the conditions above and the fact that in the adaptive model nothing from the through year assessments is incorporated into the end of year assessment results, one must ask whether the adaptive end of year test model, as described, is in fact a through year design at all.

In summary, the end-of-year assessment is best thought of as a stand-alone assessment, with little to no contribution to summative information from the preceding through year assessments.

This view frees the through year assessments to focus on providing primarily instructional information. How they can do so will have to be the subject of another post. But this adaptive model is a good design of a balanced assessment system: two different types of assessments—end of year and through year—providing two different types of assessment information serving two different purposes: summative and instructional.

Would it Work to Use a Through Year Assessment for Both Instructional and Summative Purposes?

In my previous post, I conclude that providing both instructional and end of year summative information—specifically, summative information about what a student knows and can do at the end of the year— is not possible. In this post, I show that through year assessments can be designed to provide useful information, including information that can be used for summative purposes, if adjustments are made to the summative claim. I believe, however, that through year assessments may most productively be thought of within the context of a balanced assessment system, where different types of assessments serve different assessment purposes, including informing summative and instructional interpretations and actions.