On the Content Validity of Performance Assessments: Centrality of Domain Specification

Abstract

Performance assessments--such as writing prompts in language arts, hands-on experiments in science, or portfolios in mathematics--are being heralded as “authentic” assessments (e.g., Wiggins, 1989). They play a central role in the rhetoric, if not the reality, of proposed state and national testing programs (e.g., Bush, 1991; see Shavelson, Baxter & Pine, 1992). Unfortunately, in the headlong pursuit of testing reform, technical considerations may be pushed aside (e.g., Linn, Baker, & Dunbar, 1991; Shavelson & Baxter, 1992). Technical considerations include reliability and validity of performance assessments, and the interchangeability of alternative methods of measuring performance (e.g., hands-on investigations, computer simulations, short-answer questions). Empirical research has, to date, focused primarily on interrater and intertask reliability (e.g., Dunbar, Koretz, & Hoover, 1991; Shavelson, Baxter, Pine, & Yuré, 1991; Shavelson, Baxter, Pine, Yuré, Goldman, & Smith, 1991). The findings are consistent. Interrater reliability is not a problem. Raters can be trained to score performance reliably in real time or from surrogates such as notebooks (e.g., Baxter, Shavelson, Goldman, & Pine, 1992; Shavelson, et al., in press). Task-sampling variability, however, is large, substantially reducing the reliability of the measurements (e.g., Dunbar et al., 1991; Linn et al., 1991; Shavelson, Baxter, & Gao, in press). If performance assessments are to prove useful in practice, task-sampling variability must be addressed.