Skip to main content
SpringerLink
Log in

Measuring What Matters in a Digital Age: Technology and the Design of Assessments for Multisource Comprehension

  • Chapter
  • First Online:
Book cover Ubiquitous and Mobile Learning in the Digital Age

Abstract

This chapter provides a discussion of issues associated with research and development to effect a more productive connection between technology and the design and deployment of assessments that can measure what matters and support learning in a digital world. Assessment is first discussed as a process of reasoning from evidence, emphasizing its necessary connections to theory and research on cognition and learning. An evidence-centered design process, which allows one to go from theory and research on cognition to actual assessment development, is described and then subsequently illustrated. Consideration is also given to the affordances of technology for expanding the scope of what we assess and how, and ways in which the information derived from a formative assessment process can then be used to support the processes of teaching and learning. To illustrate these ideas, this chapter then focuses on explicating a cognitive model of multisource comprehension and then using that model to design and deploy technology-based assessments of components of multisource comprehension. The process of applying evidence-centered design is discussed for components such as sourcing and analysis and synthesis in the content of multiple digital text sources. Illustrations are provided of technology-based tasks for assessing aspects of sourcing. This chapter concludes with a discussion of the opportunities that currently exist in a digital world to make assessment an integral part of learning environments and some of technology’s affordances to make such environments more productive and effective for all learners.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For a more complete description of the process of developing multiple source comprehension assessments with illustrations of the technology-based tasks and results, the reader should consult Goldman et al. (2012).

  2. 2.

    Our use of the term interpretive model most closely matches the evaluation component of the evidence model described by Mislevy and Haertel (2006) and is focused on the evidence rules for determining the salient features of student work to be derived from the tasks and that form the basis for claims about student competence. For our present purposes, we did not attempt to develop a formal measurement model that is considered by Mislevy and Haertel (2006) as the second major component of the evidence model.

  3. 3.

    This scenario is adapted from one originally developed in Pellegrino et al. (2001).

References

  • Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia. Journal of Educational Psychology, 96, 523–535.

    Article  Google Scholar 

  • Bennett, R. E. (2008). Technology for large-scale assessment (ETS Report No. RM-08-10). Princeton, NJ: Educational Testing Service.

    Google Scholar 

  • Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2004). Assessment for learning – Putting it into practice. Buckingham: Open University Press.

    Google Scholar 

  • Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. London: King’s College.

    Google Scholar 

  • Braasch, J. L. G., Lawless, K. A., Goldman, S. R., Manning, F., Gomez, K. W., & MacLeod, S. (2009). Evaluating search results: An empirical analysis of middle school students’ use of source attributes to select useful sources. Journal of Educational Computing Research, 41(1), 63–82.

    Article  Google Scholar 

  • Bransford, J. D., Brown, A. L., Cocking, R. R., Donovan, M. S., & Pellegrino, J. W. (Eds.). (2000). How people learn: Brain, mind, experience, and school (Expandedth ed.). Washington, DC: National Academy Press.

    Google Scholar 

  • Bråten, I., & Strømsø, H. (2006). Effects of personal epistemology on the understanding of multiple texts. Reading Psychology, 27, 457–484.

    Article  Google Scholar 

  • Bråten, I., & Strømsø, H. (2010). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes: A Multidisciplinary Journal, 47(1), 1–31.

    Article  Google Scholar 

  • Brem, S. K., Russell, J., & Weems, L. (2001). Science on the Web: Student evaluations of scientific arguments. Discourse Processes, 32, 191–213.

    Google Scholar 

  • Britt, M. A., & Aglinskas, C. (2002). Improving student’s ability to use source information. Cognition & Instruction, 20(40), 485–522.

    Article  Google Scholar 

  • Brown, J., Hinze, S., & Pellegrino, J. W. (2008). Technology and formative assessment. In T. Good (Ed.), 21st Century education. Vol 2. Technology (pp. 245–255). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Coiro, J. (2009). Rethinking online reading assessment. Educational Leadership, 66(6), 59–63.

    Google Scholar 

  • Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42(2), 214–257.

    Article  Google Scholar 

  • Council of Chief State School Officers (CCSSO) and the National Governors Association (NGA). (2010). The Common Core State Standards for English Language Arts & Literacy in history/social studies, science, and technical subjects. Retrieved July 16, 2010, from http://www.corestandards.org/the-standards/english-language-arts-standards.

  • Foltz, P. W., Britt, M. A., & Perfetti, C. A. (1996). Reasoning from multiple texts: An automatic analysis of readers’ situation models. In G. W. Cottrell (Ed.), Proceedings of the 18th annual Cognitive Science Society (pp. 110–115). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Gil, L., Braten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31(1), 30–68.

    Article  Google Scholar 

  • Goldman, S. R. (2004). Cognitive aspects of constructing meaning through and across multiple texts. In N. Shuart-Ferris & D. M. Bloome (Eds.), Uses of intertextuality in classroom and educational research (pp. 313–347). Greenwich, CT: Information Age.

    Google Scholar 

  • Goldman, S. R., Lawless, K. A., Gomez, K. W., Braasch, J. L. G., MacLeod, S., & Manning, F. (2010). Literacy in the digital world: Comprehending and learning from multiple sources. In M. G. McKeown & L. Kucan (Eds.), Bringing reading researchers to life (pp. 257–284). New York: Guilford.

    Google Scholar 

  • Goldman, S. R., Lawless, K. A., & Manning, F. (in press). Research and development of multiple source comprehension assessment. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts. New York: Routledge.

    Google Scholar 

  • Goldman, S. R., Lawless, K. A., Pellegrino, J. W., Braasch, J. L. G., Manning, F. H., & Gomez, K. (2012). A technology for assessing multiple source comprehension: An essential skill of the 21st century. In M. Mayrath, J. Clarke-Midura, & D. H. Robinson (Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 171–207). Charlotte, NC: Information Age.

    Google Scholar 

  • Henry, L. A. (2006). SEARCHing for an answer: The critical role of new literacies while reading on the Internet. The Reading Teacher, 59, 614–627.

    Article  Google Scholar 

  • Ito, M., Horst, H., Bittanti, M., Boyd, D., Stephenson, B., Lange, P., et al. (2008). Living and learning with new media: Summary of findings from the digital youth project. The John D. and Catherine T. MacArthur Foundation Reports on Digital Media and Learning. Cambridge: MIT Press.

    Google Scholar 

  • Kim, H. J. J., & Millis, K. (2006). The influence of sourcing and relatedness on event integration. Discourse Processes, 41(1), 51–65.

    Article  Google Scholar 

  • Lawless, K. A., Goldman, S., R., Gomez, K., Manning, F., & Braasch, J. (still in press). Assessing multiple source comprehension through Evidence Centered Design. In J. P. Sabatini & E. R. Albro (Eds.), Assessing reading in the 21st century: Aligning and applying advances in the reading and measurement sciences. Lanham, MD: Rowman & Littlefield.

    Google Scholar 

  • Lawless, K. A., Schrader, P. G., & Mayall, H. J. (2007). Acquisition of information online: Knowledge, navigation and learning outcomes. Journal of Literacy Research, 39 (3), 289–306.

    Google Scholar 

  • Lawless, K. A., Schrader, P. G., & Mayall, H. J. (2007). Acquisition of information online: Knowledge, navigation and learning outcomes. Journal of Literacy Research, 39(3), 289–306.

    Google Scholar 

  • Lee, C. D., & Spratley, A. (2010). Reading in the disciplines: The challenges of adolescent literacy. New York, NY: Carnegie Corporation of New York.

    Google Scholar 

  • Lenhart, A., Rainie, L., & Lewis, O. (2001). Teenage life online: The rise of the Instant-Message generation and the Internet’s impact on friendships and family relationships. Pew Internet & American Life Project. Retrieved July 10, 2010, from http://www.pewinternet.org/Reports/2001/Teenage-Life-Online.aspx.

  • Mateos, M., & Solé, I. (2009). Synthesising information from various texts: A study of procedures and products at different educational levels. European Journal of Psychology of Education – EJPE (Instituto Superior De Psicologia Aplicada), 24(4), 435–451.

    Google Scholar 

  • Mislevy, R. J., Steinberg, L., & Almond, R (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspective, 1, 3–67.

    Google Scholar 

  • Mislevy, R. J., & Haertel, G. (2006). Implications of evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25(4), 6–20.

    Article  Google Scholar 

  • Mislevy, R. J., & Haertel, G. (2006). Implications of evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25 (4), 6–20.

    Article  Google Scholar 

  • Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design: Layers, concepts, and terminology. In S. Downing & T. Haladyna (Eds.), Handbook of test development (pp. 61–90). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Moje, E. B., & O’Brien, D. G. (Eds.). (2001). Constructions of literacy: Studies of teaching and learning in and out of secondary classrooms. Mahwah, NJ: Erlbaum.

    Google Scholar 

  • National Center on Education and the Economy (NCEE). (2007). Tough choices or tough times. The report of the New Commission on the Skills of the American Workforce. Washington DC: National center on Education and the Economy.

    Google Scholar 

  • National Research Council (NRC). (2007). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Committee on Prospering in the Global Economy of the 21st Century: An agenda for American science and technology; Committee on Science, Engineering, and Public Policy. Washington, DC: National Academies Press.

    Google Scholar 

  • National Research Council (NRC). (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a Conceptual Framework for New K-12 Science Education Standards, Board on Science Education. Washington, DC: National Academies Press.

    Google Scholar 

  • New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66, 60–92.

    Google Scholar 

  • Organization for Economic Co-operation and Development (OECD). (2002). Reading for change: Performance and engagement across countries. Paris: OECD.

    Google Scholar 

  • Organization for Economic Cooperation and Development (OECD). (2004). Learning for tomorrow’s world: First results from PISA 2003. Paris: Author.

    Google Scholar 

  • Organization for Economic Co-operation and Development (OECD). (2006). Assessing scientific, reading and mathematical literacy: A framework for PISA 2006. Research. Paris: Author.

    Google Scholar 

  • Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.

    Google Scholar 

  • Pellegrino, J. W., & Quellmalz, E. S. (2010). Perspectives on the integration of technology and assessment. Journal of Research on Technology and Education, 43(2), 119–134.

    Google Scholar 

  • Perez, S. (2009). “Who’s online and what are they doing”. Pew Internet and American Life Project. Retrieved July 10, 2010, from http://pewinternet.org/Media-Mentions/2009/Whos-Online-and-What-Are-They-Doing-There.aspx.

  • Foltz, P. W., Britt, M. A., & Perfetti, C. A. (1996). Reasoning from multiple texts: An automatic analysis of readers’ situation models. In G. W. Cottrell (Ed.), Proceedings of the 18th annual Cognitive Science Society (pp. 110–115). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Phillips, L. M., & Norris, S. P. (1999). Interpreting popular reports of science: What happens when the reader’s world meets the world on paper? International Journal of Science Education, 21, 317–327.

    Article  Google Scholar 

  • Quellmalz, E., & Pellegrino, J. W. (2009). Technology and testing. Science, 323, 75–79.

    Article  Google Scholar 

  • Recker, M., Walker, A., & Lawless, K. (2003). What do you recommend? Implementation and analyses of collaborative information filtering of web resources for education. Instructional Science, 31(4–5), 299–316.

    Article  Google Scholar 

  • Rouet, J. F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493.

    Article  Google Scholar 

  • Salmerón, L., Baccino, T., Cañas, J. J., Madrid, R. I., & Fajardo, I. (2009). Do graphical overviews facilitate or hinder comprehension in hypertext? Computers & Education, 53, 1308–1319.

    Article  Google Scholar 

  • Salmerón, L., Gil, L., Bråten, I., & Strømsø, H. (2010). Comprehension effects of signalling relationships between documents in search engines. Computers in Human Behavior, 26(3), 419–426.

    Article  Google Scholar 

  • Sawyer, R. K. (Ed.). (2006). Cambridge handbook of the learning sciences. New York: Cambridge University Press.

    Google Scholar 

  • Schum, D. (1987). Evidence and inference for the intelligence analyst. Lantham, MD: University of America Press.

    Google Scholar 

  • Shanahan, T., & Shanahan, C. (2008). Teaching disciplinary literacy to adolescents: Rethinking content-area literacy. Harvard Educational Review, 78(1), 40–59.

    Google Scholar 

  • Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of documents models. Computer-Supported Collaborative Learning, 2, 191–210.

    Article  Google Scholar 

  • Stiggins, R. (2005). From formative assessment to assessment for learning: A path to success in standards-based schools. Phi Delta Kappan, 87(3), 324–328.

    Google Scholar 

  • Strømsø, H. I., Bråten, I., & Britt, M. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20(3), 192–204.

    Article  Google Scholar 

  • U.S. Department of Education. (2010). Transforming American education: Learning powered by technology. Washington, DC: U.S. Department of Education. http://www.ed.gov/technology/netp-2010.

  • VanSledright, B. A. (2002). Confronting history’s interpretive paradox while teaching fifth graders to investigate the past. American Educational Research Journal, 39, 1089–1115.

    Article  Google Scholar 

  • Vendlinski, T., & Stevens, R. (2002). Assessing student problem-solving skills with complex computer-based tasks. Journal of Technology, Learning, and Assessment, 1(3). http://www.jtla.org.

  • Wade, S. E., & Moje, E. B. (2000). The role of text in classroom learning. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. III, pp. 609–627). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Wallace, R. M., Kupperman, J., & Krajcik, J. (2000). Science on the Web: Students online in a sixth-grade classroom. Journal of the Learning Sciences, 9, 75–105.

    Article  Google Scholar 

  • Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106.

    Article  Google Scholar 

  • Wiliam, D. (2007). Keeping learning on track. In F. K. Lester Jr. (Ed.), Second handbook of mathematics teaching and learning (pp. 1051–1098). Greenwich, CT: Information Age.

    Google Scholar 

  • Williams, K., & Gomez, L. (2002). Presumptive literacies in technology-integrated science curriculum. In G. Stahl (Ed.), Computer support for collaborative learning: Foundations for a CSCL community (pp. 599–600). Mahwah, NJ: Erlbaum.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Illinois, Chicago, IL, USA

    James W. Pellegrino

Authors
  1. James W. Pellegrino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James W. Pellegrino .

Editor information

Editors and Affiliations

  1. Dept. Technology in Education &, Digital Systems, University of Piraeus, Androutsou St. 150, Piraeus, 185 32, Greece

    Demetrios G. Sampson

  2. Universidade Aberta, Rua de Junqueria 86, Lisboa, 1300-344, Portugal

    Pedro Isaias

  3. , Department of Educational Science, University of Freiburg, Remaprtstr. 11, Freiburg, 79085, Baden-Württemberg, Germany

    Dirk Ifenthaler

  4. , Department of Learning Technologies, C, University of North Texas, North Elm 3940, Denton, 76207-7102, Texas, USA

    J. Michael Spector

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Pellegrino, J.W. (2013). Measuring What Matters in a Digital Age: Technology and the Design of Assessments for Multisource Comprehension. In: Sampson, D., Isaias, P., Ifenthaler, D., Spector, J. (eds) Ubiquitous and Mobile Learning in the Digital Age. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3329-3_17

Download citation

Publish with us

Policies and ethics

Search

Navigation