Skip to main content
SpringerLink
Log in

Challenges of Assessing Collaborative Problem Solving

  • Chapter
  • First Online:
Assessment and Teaching of 21st Century Skills

Part of the book series: Educational Assessment in an Information Age ((EAIA))

Abstract

An assessment of Collaborative Problem Solving (CPS) proficiency was developed by an expert group for the PISA 2015 international evaluation of student skills and knowledge. The assessment framework defined CPS skills by crossing three major CPS competencies with four problem solving processes that were adopted from PISA 2012 Complex Problem Solving to form a matrix of 12 specific skills. The three CPS competencies are (1) establishing and maintaining shared understanding, (2) taking appropriate action, and (3) establishing and maintaining team organization. For the assessment, computer-based agents provide the means to assess students by varying group composition and discourse across multiple collaborative situations within a short period of time. Student proficiency is then measured by the extent to which students respond to requests and initiate actions or communications to advance the group goals. This chapter identifies considerations and challenges in the design of a collaborative problem solving assessment for large-scale testing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Aronson, E., & Patnoe, S. (1997). The jigsaw classroom: Building cooperation in the classroom. New York: Addison Wesley Longman.

    Google Scholar 

  • Asterhan, C. S., & Schwarz, B. B. (2009). Argumentation and explanation in conceptual change: Indications from protocol analyses of peer-to-peer dialog. Cognitive Science, 33, 374–400.

    Article  Google Scholar 

  • Austin, J. R. (2003). Transactive memory in organizational groups: The effects of content, consensus, specialization, and accuracy on group performance. Journal of Applied Psychology, 88, 866–878.

    Article  Google Scholar 

  • Barth, C. M., & Funke, J. (2010). Negative affective environments improve complex solving performance. Cognition and Emotion, 24, 1259–1268.

    Article  Google Scholar 

  • Biswas, G., Jeong, H., Kinnebrew, J., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning, 5, 123–152.

    Article  Google Scholar 

  • Brannick, M. T., & Prince, C. (1997). An overview of team performance measurement. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement: Theory methods and applications (pp. 3–16). Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Cannon-Bowers, J. A., & Salas, E. (2001). Reflections on shared cognition. Journal of Organizational Behavior, 22, 195–202.

    Article  Google Scholar 

  • Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–149). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22, 221–239.

    Article  Google Scholar 

  • Dillenbourg, P. (1999). Collaborative learning: Cognitive and computational approaches. Advances in learning and instruction series. New York: Elsevier Science.

    Google Scholar 

  • Dillenbourg, P., & Traum, D. (2006). Sharing solutions: Persistence and grounding in multi-modal collaborative problem solving. The Journal of the Learning Sciences, 15, 121–151.

    Article  Google Scholar 

  • Fan, X., McNeese, M., & Yen, J. (2010). NDM-based cognitive agents for supporting decision-making teams. Human Computer Interaction, 25, 195–234.

    Article  Google Scholar 

  • Fiore, S., & Schooler, J. W. (2004). Process mapping and shared cognition: Teamwork and the development of shared problem models. In E. Salas & S. M. Fiore (Eds.), Team cognition: Understanding the factors that drive process and performance (pp. 133–152). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Fiore, S. M., Rosen, M., Salas, E., Burke, S., & Jentsch, F. (2008). Processes in complex team problem solving: Parsing and defining the theoretical problem space. In M. Letsky, N. Warner, S. M. Fiore, & C. Smith (Eds.), Macrocognition in teams: Theories and methodologies (pp. 143–163). London: Ashgate Publishers.

    Google Scholar 

  • Fiore, S. M., Rosen, M. A., Smith-Jentsch, K. A., Salas, E., Letsky, M., & Warner, N. (2010). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52, 203–224.

    Article  Google Scholar 

  • Fiore, S. M., Wiltshire, T. J., Oglesby, J. M., O’Keefe, W. S., & Salas, E. (2014). Complex collaborative problem solving in mission control. Aviation, Space, and Environmental Medicine, 85, 456–461.

    Article  Google Scholar 

  • Foltz, P. W., & Martin, M. J. (2008). Automated communication analysis of teams. In E. Salas, G. F. Goodwin, & S. Burke (Eds.), Team effectiveness in complex organisations and systems: Cross-disciplinary perspectives and approaches (pp. 411–431). New York: Routledge.

    Google Scholar 

  • Franklin, S., & Graesser, A. (1997). Is it an agent, or just a program? A taxonomy for autonomous agents. InProceedings of the agent theories, architectures, and languages workshop (pp. 21–35). Berlin: Springer-Verlag.

    Chapter  Google Scholar 

  • Funke, J. (2010). Complex problem solving: A case for complex cognition? Cognitive Processing, 11, 133–142.

    Article  Google Scholar 

  • Graesser, A. C., Gernsbacher, M. A., & Goldman, S. (Eds.). (2003). Handbook of discourse processes. Mahwah: Erlbaum.

    Google Scholar 

  • Graesser, A. C., McNamara, D. S., & Rus, V. (2011). Computational modeling of discourse and conversation. In M. Spivey, M. Joanisse, & K. McRae (Eds.), The Cambridge handbook of psycholinguistics (pp. 558–572). Cambridge: Cambridge University Press.

    Google Scholar 

  • Graesser, A. C., Li, H., & Forsyth, C. (2014). Learning by communicating in natural language with conversational agents. Current Directions in Psychological Science, 23, 374–380.

    Article  Google Scholar 

  • Greiff, S., Würstenburg, S., Csapó, B., Demetriou, A., Hautamäki, J., Graesser, A. C., & Martin, R. (2014). Domain-general problem solving skills and education in the 21st century. Educational Research Review, 13, 74–83.

    Google Scholar 

  • Griffin, P., Care, E., & McGaw, B. (2012). The changing role of education and schools. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching 21st century skills (pp. 1–15). Heidelberg: Springer.

    Chapter  Google Scholar 

  • Halpern, D. F., Millis, K., Graesser, A. C., Butler, H., Forsyth, C., & Cai, Z. (2012). Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity, 7, 93–100.

    Article  Google Scholar 

  • Hsu, J. L., & Chou, H. W. (2009). The effects of communicative genres on intra-group conflict in virtual student teams. International Journal of Distance Education Technologies, 7, 1–22.

    Article  Google Scholar 

  • Jackson, G. T., & McNamara, D. S. (2013). Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105, 1036–1049.

    Article  Google Scholar 

  • Johnson, L. W., & Valente, A. (2008). Tactical language and culture training systems: Using artificial intelligence to teach foreign languages and cultures. In M. Goker & K. Haigh (Eds.), Proceedings of the twentieth conference on innovative applications of artificial intelligence (pp. 1632–1639). Menlo Park: AAAI Press.

    Google Scholar 

  • Klein, G. (2008). Naturalistic decision making. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50, 456–460.

    Article  Google Scholar 

  • Klein, C., DeRouin, R. E., & Salas, E. (2006). Uncovering workplace interpersonal skills: A review, framework, and research agenda. In G. P. Hodgkinson & J. K. Ford (Eds.), International review of industrial and organisational psychology (pp. 80–126). New York: Wiley.

    Google Scholar 

  • Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19, 335–353.

    Article  Google Scholar 

  • Lehman, B., D’Mello, S. K., Strain, A., Mills, C., Gross, M., Dobbins, A., Wallace, P., Millis, K., & Graesser, A. C. (2013). Inducing and tracking confusion with contradictions during complex learning. International Journal of Artificial Intelligence in Education, 22, 85–105.

    Google Scholar 

  • Lewis, K., Lange, D., & Gillis, L. (2005). Transactive memory systems, learning, and learning transfer. Organization Science, 16, 581–598.

    Article  Google Scholar 

  • Lipshitz, R., Klein, G., Orasanu, J., & Salas, E. (2001). Taking stock of naturalistic decision making. Journal of Behavioral Decision Making, 14, 331–352.

    Article  Google Scholar 

  • Littlepage, G. E., Hollingshead, A. B., Drake, L. R., & Littlepage, A. M. (2008). Transactive memory and performance in work groups: Specificity, communication, ability differences, and work allocation. Group Dynamics: Theory, Research, and Practice, 12, 223–241.

    Article  Google Scholar 

  • Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.

    Book  Google Scholar 

  • Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., & Halpern, D. (2011). Operation ARIES!: A serious game for teaching scientific inquiry. In M. Ma, A. Oikonomou, & J. Lakhmi (Eds.), Serious games and edutainment applications (pp. 169–195). London: Springer.

    Chapter  Google Scholar 

  • Mullins, D., Rummel, N., & Spada, H. (2011). Are two heads always better than one? Differential effects of collaboration on students’ computer-supported learning in mathematics. International Journal of Computer-Supported Collaborative Learning, 6, 421–443.

    Article  Google Scholar 

  • Nash, J. M., Collins, B. N., Loughlin, S. E., Solbrig, M., Harvey, R., Krishnan-Sarin, S., Unger, J., Miner, C., Rukstalis, M., Shenassa, E., Dube, C., & Spirito, A. (2003). Training the transdisciplinary scientist: A general framework applied to tobacco use behavior. Nicotine & Tobacco Research, 5, 41–53.

    Google Scholar 

  • National Assessment of Educational Progress. (2014). Abridged technology and engineering literacy framework for the national assessment of educational progress. Washington, DC: National Academies Press.

    Google Scholar 

  • National Research Council. (2011). Assessing 21st century skills. Washington, DC: National Academies Press.

    Google Scholar 

  • O’Neil, H. F., Chuang, S. H., & Baker, E. L. (2010). Computer-based feedback for computer-based collaborative problem-solving. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 261–279). New York: Springer-Verlag.

    Chapter  Google Scholar 

  • OECD. (2009). Problem solving in technology-rich environments: A conceptual framework. OECD Education Working Papers, 36, OECD Publishing. Retrieved from http://search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage=en&cote=edu/wkp(2009)15

  • OECD. (2010). PISA 2012 field trial problem solving framework. Paris: OECD. Retrieved from http://www.oecd.org/dataoecd/8/42/46962005.pdf.

    Google Scholar 

  • OECD (2013). PISA 2015 collaborative problem solving framework. Paris: OECD. Retrieved from http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20 Collaborative%20Problem%20Solving%20Framework%20.pdf.

  • Rice, R. E. (2008). Unusual routines: Organizational (non)sensemaking. Journal of Communication, 58, 1–19.

    Article  Google Scholar 

  • Rosé, C., Wang, Y. C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning, 3, 237–271.

    Article  Google Scholar 

  • Rosen, Y. (2014). Comparability of conflict opportunities in human-to-human and human-to-agent online collaborative problem solving. Technology, Knowledge and Learning, 19, 147–174.

    Article  Google Scholar 

  • Rosen, Y., & Foltz, P. (2014). Assessing collaborative problem solving through automated technologies. Research and Practice in Technology Enhanced Learning, 9, 389–410.

    Google Scholar 

  • Rosen, Y., & Rimor, R. (2009). Using a collaborative database to enhance students’ knowledge construction. Interdisciplinary Journal of E-Learning and Learning Objects, 5, 187–195.

    Google Scholar 

  • Rouet, J.-F. (2006). The skills of document use. Mahwah: Erlbaum.

    Google Scholar 

  • Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50, 696–735.

    Article  Google Scholar 

  • Salas, E., Cooke, N. J., & Rosen, M. A. (2008). On teams, teamwork, and team performance: Discoveries and developments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50, 540–547.

    Article  Google Scholar 

  • Schwartz, D. L. (1995). The emergence of abstract representations in dyad problem solving. The Journal of the Learning Sciences, 4, 321–354.

    Article  Google Scholar 

  • Shaffer, D. W. (2012). Models of situated action: Computer games and the problem of transfer. In C. Steinkuehler, K. Squire, & S. Barab (Eds.), Games, learning, and society: Learning and meaning in the digital age (pp. 403–433). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Shaffer, D. W., & Gee, J. P. (2012). The right kind of GATE: Computer games and the future of assessment. In M. Mayrath, D. Robinson, & J. Clarke-Midura (Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research. Charlotte: Information Age Publishing.

    Google Scholar 

  • Sonnentag, S., & Lange, I. (2002). The relationship between high performance and knowledge about how to master cooperation situations. Applied Cognitive Psychology, 16, 491–508.

    Article  Google Scholar 

  • Stasser, G., & Titus, W. (2003). Hidden profiles: A brief history. Psychological Inquiry, 14, 304–313.

    Article  Google Scholar 

  • Stewart, C. O., Setlock, L. D., & Fussell, S. R. (2007). Conversational argumentation in decision making: Chinese and US participants in face-to-face and instant-messaging interactions. Discourse Processes, 44, 113–139.

    Article  Google Scholar 

  • Theiner, G., & O’Connor, T. (2010). The emergence of group cognition. In A. Corradini & T. O’Connor (Eds.), Emergence in science and philosophy (pp. 79–117). New York: Routledge.

    Google Scholar 

  • Van der Sluis, I., & Krahmer, E. (2007). Generating multimodal references. Discourse Processes, 44, 145–174.

    Article  Google Scholar 

  • Wildman, J. L., Shuffler, M. L., Lazzara, E. H., Fiore, S. M., Burke, C. S., Salas, E., & Garven, S. (2012). Trust development in swift starting action teams: A multilevel framework. Group & Organization Management, 37, 137–170.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Memphis, Memphis, TN, USA

    Arthur C. Graesser, Carol Forsyth & Mae-Lynn Germany

  2. University of Colorado, Boulder, and Pearson, Boulder, CO, USA

    Peter W. Foltz

  3. Harvard University, Cambridge, MA, USA

    Yigal Rosen

  4. University of Wisconsin, Madison, WI, USA

    David Williamson Shaffer

Authors
  1. Arthur C. Graesser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter W. Foltz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yigal Rosen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Williamson Shaffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carol Forsyth
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mae-Lynn Germany
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arthur C. Graesser .

Editor information

Editors and Affiliations

  1. Brookings Institution, Washington DC, Washington, USA

    Esther Care

  2. Melbourne Graduate School of Education, University of Melbourne, Parkville, Victoria, Australia

    Patrick Griffin

  3. University of California, Berkeley, California, USA

    Mark Wilson

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

C. Graesser, A., Foltz, P.W., Rosen, Y., Shaffer, D.W., Forsyth, C., Germany, ML. (2018). Challenges of Assessing Collaborative Problem Solving. In: Care, E., Griffin, P., Wilson, M. (eds) Assessment and Teaching of 21st Century Skills. Educational Assessment in an Information Age. Springer, Cham. https://doi.org/10.1007/978-3-319-65368-6_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-65368-6_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-65366-2

  • Online ISBN: 978-3-319-65368-6

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation