Perspectives on Methodological Issues

Abstract

In this chapter the authors have surveyed the methodological perspectives seen as important for assessing twenty-first century skills. Some of those issues are specific to twenty-first century skills, but the majority would apply more generally to the assessment of other psychological and educational variables. The narrative of the paper initially follows the logic of assessment development, commencing by defining constructs to be assessed, designing tasks that can be used to generate informative student responses, coding/valuing of those responses, delivering the tasks and gathering the responses, and modeling the responses in accordance with the constructs. The paper continues with a survey of the strands of validity evidence that need to be established, and a discussion of specific issues that are prominent in this context, such as the need to resolve issues of generality versus contextual specificity; the relationships of classroom to large-scale assessments; and the possible roles for technological advances in assessing these skills. There is also a brief segment discussing some issues that arise with respect to specific types of variables involved in the assessment of twenty-first century skills. The chapter concludes with a listing of particular challenges that are regarded as being prominent at the time of writing. There is an annexure that describes specific approaches to assessment design that are useful in the development of new assessments.

Annex: Assessment Design Approaches

Evidence-Centered Design

Design, in general, is a prospective activity; it is an evolving plan for creating an object with desired functionality or esthetic value. It is prospective because it takes place prior to the creation of the object. That is, a design and the resulting object are two different things (Mitchell 1990, pp. 37–38):

when we describe the forms of buildings we refer to extant constructions of physical materials in physical space, but when we describe designs we make claims about something else—constructions of imagination. More precisely, we refer to some sort of model—a drawing, physical scale model, structure of information in computer memory, or even a mental model—rather than to a real building.

The idea of design is, of course, equally applicable to assessments, and Mitchell’s distinction just noted is equally applicable. The design of an assessment and the resulting assessment-as-implemented are different entities. Under the best of circumstance, the design is sound and the resulting assessment satisfies the design, as evidenced empirically through the administration of the assessment. Under less ideal circumstances, the design may not be sound—in which case only by a miracle will the resulting assessment be sound or useful—or the implementation of the assessment is less than ideal. In short, merely using a design process in no way guarantees that the resulting assessment will be satisfactory, but it would be foolish to implement an assessment without a thorough design effort as a preamble.

An approach to assessment design that is gaining momentum is ECD, evidence-centered design (Mislevy et al. 2003b). The approach is based on the idea that the design of an assessment can be facilitated or optimized by taking into consideration the argument we wish to make in support of the proposed score interpretation or inference from the assessment. In its barest form a proposed score interpretation takes the following form: Given that the students has obtained score X, it follows that the student knows and can do Y.

There is no reason for anyone to accept such an assertion at face value. It would be sensible to expect an elaboration of the reasons, an argument, before we accept the conclusion or, if necessary, challenge it. A Toulmian argument, whereby the reasons for the above interpretation are explicated and potential counterarguments are addressed, is at the heart of ECD. ECD focuses on that argument primarily, up to the test score level, by explicating what the intended conclusions or inferences based on scores will be, and, given those inferences as the goal of the assessment, determine the observation of student performance that would leads us to those conclusions. Such an approach is in line with current thinking about validation, where a distinction is made between (1) a validity argument, the supporting reasoning for a particular score interpretation, and (2) the appraisal of that argument. ECD turns the validity argument on its head to find out what needs to be the case, what must be true of the assessment—what should the design of the assessment be—so that the score interpretations that we would like to reach in the end will have a better chance of being supported.

For example, suppose we are developing an assessment to characterize student’s mastery of information technology. If we wish to reach conclusions about this, we need to carefully define what we mean by “command of information technology,” including what behavior on the part of students would convince us that they have acquired mastery. With that definition in hand, we can then proceed to devise a series of tasks that will elicit student behavior or performance indicative of different levels of command of information technology, as we have defined it. Then, as the assessment is implemented, trials need to be conducted to verify that, indeed, the items produced according to the design elicit the evidence that will be needed to support that interpretation.

Approaching assessment development this way means that we have well-defined expectations of what the data from the assessment will look like. For example, what the difficulty of the items will be, how strongly they will intercorrelate, and how the scores will relate to other test scores and background variables. Those expectations are informed by the knowledge about student learning and developmental considerations that were the basis of the design of the assessment; if they are not met, there will be work to be done to find out where the design is lacking or whether the theoretical information used in the design was inadequate.

The process of reconciling design expectations with empirical reality parallels the scientific method’s emphasis on hypothesis testing aided by suitable experimental designs. It should be pointed out, however, that an argument based solely on positive confirmatory evidence is not sufficiently compelling. Ruling out alternative interpretations of positive confirmatory evidence would add considerable weight to an argument, as would a failed attempt to challenge the argument. Such challenges can take a variety of forms in an assessment context. For example, Loevinger (1957) argued that items that explicitly aim to measure a different construct should be included, at least experimentally, to ensure that performance in those items is not explained equally well by the postulated construct.

ECD is highly prospective about the process for implementing the assessment so that the desired score interpretations can be supported in the end. Essentially, ECD prescribes an order of design events. First, the purpose of the assessment needs to be explicated to make it clear what sort of inferences need to be drawn from performance on the test. Once those target inferences are enumerated, the second step is to identify the types of evidence needed to support them. Finally, the third step is to conceive of means of eliciting the evidence needed to support the target inferences. These three steps are associated with corresponding models: a student model, an evidence model, and a series of task models. Note that, according to ECD, task models, from which items would be produced, are the last to be formulated. This is an important design principle, especially since when undertaking the development of an assessment, there is a strong temptation to “start writing items” before we have a good grasp of what the goals of the assessment are. Writing items without first having identified the target inferences, and the evidence required to support them, risks producing many items that are not optimal or even failing to produce the items that are needed to support score interpretation (see e.g., Pellegrino et al. (1999), Chap. 5). For example, producing overly hard or easy items may be suboptimal if decisions or inferences are desired for students having a broad range of proficiency. Under the best of circumstances, starting to write items before we have a firm conception of the goals of the assessment leads to many wasted items that, in the end, do not fit well into the assessment. Under the worst of circumstances, producing items in this manner can permanently hobble the effectiveness of an assessment because we have to make do with the items that are available.

The importance of a design perspective has grown as a result of the shift to so-called standards-based reporting. Standards-based reporting evolved from earlier efforts at criterion-referenced testing (Glaser 1963) intended to attach a specific interpretations to test scores, especially scores that would define different levels of achievement. Since the early 1990s, the National Assessment of Educational Progress (NAEP) in the USA has relied on achievement levels (Bourque 2009). In the USA, tests oriented to inform accountability decisions have followed in NAEP’s footsteps in reporting scores in terms of achievement or performance levels. This, however, does not imply that achievement levels are defined equivalently (Braun and Qian 2007) in different jurisdictions. While it is true that the definition of achievement levels need not, for legitimate policy reasons, be equivalent across jurisdictions in practice in the USA, there has not been a good accounting of the variability across states. A likely reason is that the achievement levels are defined by cutscores that are typically arrived at by an expert panel after the assessment has been implemented (Bejar et al. 2007). However, unless the achievement levels have been defined as part of the design effort, rather than leaving them to be based on the assessment as implemented, there is a good chance that there will be a lack of alignment between the intended achievement levels and the levels that emerge from the cutscore setting process. The cutscore setting panel has the duty to produce the most sensible cutscores it can. However, if the assessment was developed without these cutscores in mind, the panel will still need to produce a set of cutscores to fit the assessment as it exists. The fact that the panel is comprised of subject matter experts cannot possibly compensate for an assessment that was not designed to specifically support the desired inferences.

Whether the assessment outcomes are achievement level or scores, an important further consideration is the temporal span assumed by the assessment. In a K-12 context, the assessment is focused on a single grade, and, typically, the assessment is administered toward the end of the year. A drawback of a single end-of-year assessment is that there is not an opportunity to utilize the assessment information to improve student achievement, at least not directly (Stiggins 2002). An alternative is to distribute assessments during the year (Bennett and Gitomer 2009); a major advantage of this is the opportunity it gives to act upon the assessment results that occur earlier in the year. Some subjects, notably mathematics and the language arts, can extend over several years, and the yearly end-of-year assessments could be viewed interim assessments. Consider first the simpler case where instruction is completed within a year and there is an end-of-year assessment. In this case, achievement levels can be unambiguously defined as the levels of knowledge expected after 1 year of instruction. For subjects that require a multiyear sequence or for subjects that distribute the assessment across several measurement occasions within a year, at least two approaches are available. One of these defines the achievement levels in a bottom-up fashion. The achievement levels for the first measurement occasion are defined first, followed by the definitions for subsequent measurement occasions. So long as the process is carried out in a coordinated fashion, the resulting sets of achievement levels should exhibit what has been called coherence (Wilson 2004). The alternative approach is top-down; in this case, the achievement levels at the terminal point of instruction are defined first. For example, in the USA, it is common to define so-called “exit criteria” for mathematics and language art subjects that, in principle, define what students should have learned by, say, Grade 10. With those exit definitions at hand, it is possible to work backwards and define achievement levels for earlier measurement occasions in a coherent manner.

Operationalization Issues

The foregoing considerations are some of the critical information in determining achievement levels, which, according to Fig. 3.18, are the foundation on which the assessment rests, along with background knowledge about student learning and developmental considerations. For clarity, Fig. 3.18 outlines the “work flow” for assessment at one point in time, but in reality, at least for some subject matters, the design of “an” assessment really entails the simultaneous design of several. That complexity is captured in Fig. 3.18 under Developmental considerations; as the figure shows, achievement levels are set by those developmental considerations and a competency model, which summarizes what we know about how students learn in the domain to be assessed (NRC 2001).

Fig. 3.18

The ECD framework

The achievement levels are fairly abstract characterizations of what students are expected to achieve. Those expectations need to be recast to make them more concrete, by means of evidence models and task models. Evidence models spell out the student behavior that would be evidence of having acquired the skills and knowledge called for by the achievement levels. Evidence models are, in turn, specifications for the tasks or items that will actually elicit the evidence called for. Once the achievement levels, task models, and evidence models are established, the design proceeds by defining task specifications and performance level descriptors (PLDs), which contain all the preceding information in a form that lends itself to formulating the test specifications. These three components should be seen as components of an iterative process. As the name implies, task specifications are very specific descriptions of the tasks that will potentially comprise the assessment. It would be prudent to produce specifications for more tasks than can possibly be used in the assessment to allow for the possibility that some of them will not work out well. PLDs are (tentative) narratives of what students at each achievement levels can be said to know and are able to do.

A change to any of these components requires revisiting the other two; in practice, test specifications cannot be finalized without information about pragmatic constraints, such as budgets, testing time available, and so on. A requirement to shorten testing time would trigger changes to the test specifications, which in turn could trigger changes to the task specifications. Utmost care is needed in this process. Test specifications determine test-level attributes like reliability and decision consistency and need to be carefully thought through. An assessment that does classify students into achievement levels with sufficient consistency is a failure, no matter how soundly and carefully the achievement levels have been defined, since the uncertainty that will necessarily be attached to student-level and policy-level decisions based on such assessment will diminish its value.

This is an iterative process that aims at an optimal design, subject to relevant pragmatic and psychometric constraints. Note that among the psychometric constraints is incorporated the goal of achieving maximal discrimination in the region of the scale where the eventual cutscores are likely to be located. This is also an iterative process, ideally supplemented by field trials. Once the array of available tasks or task models is known and the constraints are agreed upon, a test blueprint can be formulated, which should be sufficiently detailed so that preliminary cutscores corresponding to the performance standards can be formulated. After the assessment is administered, it will be possible to evaluate whether the preliminary cutscores are well supported or need adjustment in light of the data that are available. At that juncture, the role of the standard setting panel is to accept the preliminary cutscores or to adjust them in the light of new information.

The BEAR Assessment System

As mentioned before, the assessment structure plays a key role in the in the study and the educational implementation of learning progressions. Although there are several alternative approaches that could be used to model, this section focuses in the BEAR Assessment System (BAS; Wilson 2005; Wilson and Sloane 2000), a measurement approach that will allow us to represent one of the various forms in which LPs could be conceived or measured.

The BEAR Assessment System is based on the idea that good assessment addresses the need for sound measurement by way of four principles: (1) a developmental perspective; (2) the match between instruction and assessment; (3) management by instructors to allow appropriate feedback, feedforward, and follow-up; and (4) generation of quality evidence. These four principles, with the four building blocks that embody them, are shown in Fig. 3.19. They serve as the basis of a model that is rooted in our knowledge of cognition and learning in each domain and that supports the alignment of instruction, curriculum, and assessment—all aspects recommended by the NRC (2001) as important components of educational assessment.

Fig. 3.19

The principles and building blocks of the BEAR Assessment System

Principle 1: A Developmental Perspective

A “developmental perspective” on student learning highlights two crucial ideas: (a) the need to characterize the evolution of learners over time and (b) the need for assessments that are “tailored” to the characteristics of different learning theories and learning domains.

The first element, portraying the evolution of learners over time, emphasizes the definition of relevant constructs based on the development of student mastery of particular concepts and skills over time, as opposed to making a single measurement at some final or supposedly significant point of time. Additionally, it promotes assessments based on “psychologically plausible” pathways of increasing proficiency, as opposed to attempt to assess contents based on logical approaches to the structure of disciplinary knowledge.

Much of the strength of the BEAR Assessment System is related to the second element, the emphasis on providing tools to model many different kinds of learning theories and learning domains, which avoids the “one-size-fits-all” development assessment approach that has rarely satisfied educational needs. What is to be measured and how it is to be valued in each BEAR assessment application is drawn from the expertise and learning theories of the teachers, curriculum developers, and assessment developers involved in the process of creating the assessments.

The developmental perspective assumes that student performance on a given learning progression can be traced over the course of instruction, facilitating a more developmental perspective on student learning. Assessing the growth of students’ understanding of particular concepts and skills requires a model of how student learning develops over a certain period of (instructional) time; this growth perspective helps one to move away from “one shot” testing situations and cross-sectional approaches to defining student performance, toward an approach that focuses on the process of learning and on an individual’s progress through that process. Clear definitions of what students are expected to learn and a theoretical framework of how that learning is expected to unfold as the student progresses through the instructional material (i.e., in terms of learning performances) are necessary to establish the construct validity of an assessment system.

Building Block 1: Construct Maps

Construct maps (Wilson 2005) embody this first of the four principles: a developmental perspective on assessing student achievement and growth. A construct map is a well-thought-out and researched ordering of qualitatively different levels of performance focusing on one characteristic that organizes clear definitions of the expected student progress. Thus, a construct map defines what is to be measured or assessed in terms general enough to be interpretable within a curriculum, and potentially across curricula, but specific enough to guide the development of the other components. When instructional practices are linked to the construct map, then the construct map also indicates the aims of the teaching.

Construct maps are derived in part from research into the underlying cognitive structure of the domain and in part from professional judgments about what constitutes higher and lower levels of performance or competence, but are also informed by empirical research into how students respond to instruction or perform in practice (NRC 2001).

Construct maps are one model of how assessments can be integrated with instruction and accountability. They provide a way for large-scale assessments to be linked in a principled way to what students are learning in classrooms, while having the potential at least to remain independent of the content of a specific curriculum.

The idea of using construct maps as the basis for assessments offers the possibility of gaining significant efficiency in assessment: Although each new curriculum prides itself on bringing something new to the subject matter, the truth is that most curricula are composed of a common stock of content. And, as the influence of national and state standards increases, this will become truer and make them easier to codify. Thus, we might expect innovative curricula to have one, or perhaps even two, variables that do not overlap with typical curricula, but the rest will form a fairly stable set of variables that will be common across many curricula.

Principle 2: Match Between Instruction and Assessment

The main motivation for the progress variables so far developed is that they serve as a framework for the assessments and a method for making measurement possible. However, this second principle makes clear that the framework for the assessments and the framework for the curriculum and instruction must be one and the same. This emphasis is consistent with research in the design of learning environments, which suggests that instructional settings should coordinate their focus on the learner (incorporated in Principle 1) with both knowledge-centered and assessment-centered environments (NRC 2000).

Building Block 2: The Items Design

The items design process governs the coordination between classroom instruction and assessment. The critical element to ensure this in the BEAR Assessment System is that each assessment task and typical student response is matched to particular levels of proficiency within at least one construct map.

When using this assessment system within a curriculum, a particularly effective mode of assessment is what is called embedded assessment. This means that opportunities to assess student progress and performance are integrated into the instructional materials and are (from the student’s point of view) virtually indistinguishable from the day-to-day classroom activities.

It is useful to think of the metaphor of a stream of instructional activity and student learning, with the teacher dipping into the stream of learning from time to time to evaluate student progress and performance. In this model or metaphor, assessment then becomes part of the teaching and learning process, and we can think of it as being assessment for learning (AfL; Black et al. 2003).

If assessment is also a learning event, then it does not take time away from instruction unnecessarily, and the number of assessment tasks can be more readily increased so as to improve the reliability of the results (Linn and Baker 1996). Nevertheless, for assessment to become fully and meaningfully embedded in the teaching and learning process, the assessment must be linked to the curriculum and not be seen as curriculum-independent as is the rhetoric for traditional norm-referenced tests (Wolf and Reardon 1996).

Principle 3: Management by Teachers

For information from the assessment tasks and the BEAR analysis to be useful to instructors and students, it must be couched in terms that are directly related to the instructional goals behind the progress variables. Open-ended tasks, if used, must all be scorable—quickly, readily, and reliably.

Building Block 3: The Outcome Space

The outcome space is the set of categorical outcomes into which student performances are categorized, for all the items associated with a particular progress variable. In practice, these are presented as scoring guides for student responses to assessment tasks, which are meant to help make the performance criteria for the assessments clear and explicit (or “transparent and open” to use Glaser’s (1963) terms)—not only to the teachers but also to the students and parents, administrators, or other “consumers” of assessment results. In fact, we strongly recommend to teachers that they share the scoring guides with administrators, parents, and students, as a way of helping them understand what types of cognitive performance are expected and to model the desired processes.

Scoring guides are the primary means by which the essential element of teacher professional judgment is implemented in the BEAR Assessment System. These are supplemented by “exemplars” of student work at every scoring level for each task and variable combination, and “blueprints,” which provide the teachers with a layout indicating opportune times in the curriculum to assess the students on the different variables.

Principle 4: Evidence of High Quality Assessment

Technical issues of reliability and validity, fairness, consistency, and bias can quickly sink any attempt to measure along a progress variable, as described above, or even to develop a reasonable framework that can be supported by evidence. To ensure comparability of results across time and context, procedures are needed to (a) examine the coherence of information gathered using different formats, (b) map student performances onto the progress variables, (c) describe the structural elements of the accountability system—tasks and raters—in terms of the achievement variables, and (d) establish uniform levels of system functioning, in terms of quality control indices such as reliability.

Building Block 4: Wright Maps

Wright maps represent this principle of evidence of high quality. Wright maps are graphical and empirical representations of a construct map, showing how it unfolds or evolves in terms of increasingly sophisticated student performances.

They are derived from empirical analyses of student data on sets of assessment tasks. They show on an ordering of these assessment tasks from relatively easy tasks to more difficult ones. A key feature of these maps is that both students and tasks can be located on the same scale, giving student proficiency the possibility of substantive interpretation, in terms of what the student knows and can do and where the student is having difficulty. The maps can be used to interpret the progress of one particular student or the pattern of achievement of groups of students ranging from classes to nations.

Wright maps can be very useful in large-scale assessments, providing information that is not readily available through numerical score averages and other traditional summary information—they are used extensively, for example, in reporting on the PISA assessments (OECD 2005). Moreover, Wright maps can be seamlessly interpreted as representations of learning progressions, quickly mapping the statistical results back to the initial construct, providing the necessary evidence to explore questions about the structure of the learning progression, serving as the basis for improved versions of the original constructs.