Skip to main content
SpringerLink
Log in

Development of Scalable Assessment for Collaborative Problem-Solving

  • Chapter
  • First Online:
International Handbook of Computer-Supported Collaborative Learning

Part of the book series: Computer-Supported Collaborative Learning Series ((CULS,volume 19))

  • 2775 Accesses

Abstract

While the field of computer-supported collaborative learning (CSCL) is focused primarily on the development of computational artifacts and social interaction, the key research advances in collaborative problem-solving (CPS) domain are associated with competency model development and assessment at scale. Numerous research reports indicate that CPS competency is increasingly important in today’s complex interconnected world, therefore, of increasing interest in teaching and assessing with students. However, learning and assessment design, data analytics, and reporting on CPS competency, specifically in CSCL setting encapsulates multiple opportunities and challenges. This chapter introduces a spectrum of approaches for CPS competency development and scalable assessment to advance the theory and practice of measuring CPS with the focus on recent work at the nonprofits ACT and Educational Testing Service.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Adejumo, G., Duimering, P. R., & Zhong, Z. (2008). A balance theory approach to group problem solving. Social Networks, 30(1), 83–99.

    Article  Google Scholar 

  • Andrews, J. J., Kerr, D., Mislevy, R. J., von Davier, A. A., Hao, J., & Liu, L. (2017). Modeling collaborative interaction patterns in a simulation-based task. Journal of Educational Measurement, 54(1), 54–69.

    Article  Google Scholar 

  • Andrews-Todd, J., Jackson, G. T., & Kurzum, C. (2019). Collaborative problem solving assessment in an online mathematics task (research report no. RR-19-24). Educational Testing Service.

    Google Scholar 

  • Andrews-Todd, J., & Kerr, D. (2019). Application of ontologies for assessing collaborative problem solving skills. International Journal of Testing, 19, 172–187. https://doi.org/10.1080/15305058.2019.1573823.

    Article  Google Scholar 

  • Asterhan, C. S. C., Schwartz, B. B., & Cohen-Eliyahu, N. (2014). Outcome feedback during collaborative learning: Contingencies between feedback and dyad composition. Learning and Instruction, 34, 1–10. https://doi.org/10.1016/j.learninstruc.2014.07.003.

    Article  Google Scholar 

  • Avouris, N., Dimitracopoulou, A., & Komis, V. (2003). On analysis of collaborative problem solving: An object-oriented approach. Computers in Human Behavior, 19(2), 147–167.

    Article  Google Scholar 

  • Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 17–66). New York, NY: Springer.

    Google Scholar 

  • Camara, W., O’Connor, R., Mattern, K., & Hanson, M. A. (2015). Beyond academics: A holistic framework for enhancing education and workplace success. ACT Research Report Series, 2015(4). ACT.

    Google Scholar 

  • Care, E., & Kim, H. (2018). Assessment of twenty-first century skills: The issue of authenticity. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and teaching of 21st century skills: Research and applications (pp. 21–39). New York, NY: Springer.

    Google Scholar 

  • Cooke, N. J., Kiekel, P. A., Salas, E., Stout, R., Bowers, C., & Cannon-Bowers, J. (2003). Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics: Theory, Research and Practice, 7(3), 179–219.

    Article  Google Scholar 

  • Flor, M., Yoon, S.-Y., Hao, J., Liu, L., & von Davier, A. A. (2016). Automated classification of collaborative problem-solving interactions in simulated science tasks. In Proceedings of the 11th workshop on innovative use of NLP for building educational applications (pp. 31–41). Association for Computational Linguistics.

    Google Scholar 

  • Foltz, P. W., & Martin, M. J. (2008). Automated communication analysis of teams. In E. Salas, G. F. Goodwin, & S. Burke (Eds.), Team effectiveness in complex organizations and systems: Cross-disciplinary perspectives and approaches (pp. 411–431). New York, NY: Routledge.

    Google Scholar 

  • Graesser, A. C., Fiore, S. M., Greiff, S., Andrews-Todd, J., Foltz, P. W., & Hesse, F. W. (2018a). Advancing the science of collaborative problem solving. Psychological Science in the Public Interest, 19, 59–92.

    Article  Google Scholar 

  • Graesser, A. C., Foltz, P. W., Rosen, Y., Shaffer, D. W., Forsyth, C., & Germany, M. (2018b). Challenges of assessing collaborative problem solving. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and teaching of 21st century skills (pp. 75–91). Springer. https://doi.org/10.1007/978-3-319-65368-6_5.

  • Graesser, A. C., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45(4), 298–322.

    Article  Google Scholar 

  • Griffin, P., Care, E., & McGaw, B. (2012a). The changing role of education and schools. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching 21st century skills (pp. 1–15). New York, NY: Springer.

    Google Scholar 

  • Griffin, P., McGaw, B., & Care, E. (Eds.). (2012b). Assessment and teaching of 21st century skills. New York, NY: Springer.

    Google Scholar 

  • Halpin, P. F., von Davier, A. A., Hao, J., & Liu, L. (2017). Measuring student engagement during collaboration. Journal of Educational Measurement, 54(1), 70–84.

    Article  Google Scholar 

  • Hao, J., Chen, L., Flor, M., Liu, L., & von Davier, A. A., (2017a), CPS-rater: automated sequential annotation for conversations in collaborative problem-solving activities (ETS research report no. RR-17-58). Educational Testing Service.

    Google Scholar 

  • Hao, J., Liu, L., Kyllonen, P., Flor, M., & von Davier, A. A. (2019). Psychometric considerations and a general scoring strategy for assessments of collaborative problem solving (research report no. RR-19-41). Educational Testing Service.

    Google Scholar 

  • Hao, J., Liu, L., von Davier, A. A., & Kyllonen, P. C. (2017b). Initial steps towards a standardized assessment for CPS: Practical challenges and strategies. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative assessment of collaboration. New York, NY: Springer.

    Google Scholar 

  • Hao, J., Liu, L., von Davier, A. A., Lederer, N., Zapata-Rivera, D., Jkal, P., & Bakkenson, M. (2017c). EPCAL: ETS platform for collaborative assessment and learning (ETS research report no. RR-17-49). Educational Testing Service.

    Google Scholar 

  • Hao, J., & Mislevy, R. J. (2019). Characterizing interactive communications in computer-supported collaborative problem-solving tasks: A conditional transition profile approach. Frontiers in Psychology, 10, 1011. https://doi.org/10.3389/fpsyg.2019.01011.

    Article  Google Scholar 

  • Hao, J., Smith, L., Mislevy, R., von Davier, A. A., & Bauer, M. (2016). Taming log files from game/simulation-based assessments: Data models and data analysis tools. ETS Research Report Series, 2016(1), 1–17. https://doi.org/10.1002/ets2.12096.

    Article  Google Scholar 

  • Kim, Y., Almond, R., & Shute, V. (2016). Applying evidence-centered design for the development of game-based assessments in physics playground. International Journal of Testing, 16(2), 142–163.

    Article  Google Scholar 

  • Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19(3), 335–353.

    Article  Google Scholar 

  • Liu, L., Hao, J., von Davier, A. A., Kyllonen, P., & Zapata-Rivera, D. (2016). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on computational tools for real-world skill development. IGI-Global.

    Google Scholar 

  • Looi, C.-K., & Dillenbourg, P. (2013). How will collaborative problem solving be assessed at international scale? (invited panel). In N. Rummel, M. Kapur, M. Nathan, & S. Puntambekar (Eds.), 10th International conference on computer-supported collaborative learning (CSCL) 2013, conference proceedings: Vol. 2. Short papers, panels, posters, Demos & Community Events. ISLS.

    Google Scholar 

  • Ludvigsen, S., Lund, K., & Oshima, J. (this volume). A conceptual stance on CSCL history. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning. Cham: Springer.

    Google Scholar 

  • Martin-Raugh, M. P., Kyllonen, P. C., Hao, J., Bacall, A., Becker, D., Kurzum, C., Yang, Z., Yan, F., & Barnwell, P. (2020). Negotiation as an interpersonal skill: Generalizability of negotiation outcomes and tactics across contexts at the individual and collective levels. Computers in Human Behavior, 104, 105966.

    Article  Google Scholar 

  • McCaslin, M., & Burross, H. L. (2011). Research on individual differences within a sociocultural perspective: Co-regulation and adaptive learning. Teachers College Record, 113(2), 325–349.

    Article  Google Scholar 

  • Medina, R., & Stahl, G. (this volume). Analysis of group practices. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning. Cham: Springer.

    Google Scholar 

  • Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.

    Article  Google Scholar 

  • Mestre, J. (2002). Transfer of learning: Issues and research agenda (National Science Foundation Report No. NSF03-212).

    Google Scholar 

  • Mislevy, R., Steinberg, L., & Almond, R. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspective, 1(1), 3–62.

    Google Scholar 

  • O’Neil, H. F., Chung, G., & Brown, R. (1997). Use of networked simulations as a context to measure team competencies. In H. F. O’Neil Jr. (Ed.), Workforce readiness: Competencies and assessment (pp. 411–452). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • OECD. (2017a). PISA 2015 collaborative problem-solving framework. Paris, France: OECD Publishing.

    Google Scholar 

  • OECD. (2017b). PISA 2015 results: Collaborative problem solving. Paris, France: OECD Publishing.

    Google Scholar 

  • Pai, H., Sears, D. A., & Maeda, Y. (2015). Effects of small-group learning on transfer: A meta-analysis. Educational Psychology Review, 27(1), 79–102.

    Article  Google Scholar 

  • Rosen, Y. (2017). Assessing students in human-to-agent settings to inform collaborative problem solving learning. Journal of Educational Measurement, 54(1), 36–53.

    Article  Google Scholar 

  • Rosen, Y., & Foltz, P. W. (2014). Assessing collaborative problem solving through automated technologies. Research & Practice in Technology Enhanced Learning, 9(3), 389–410.

    Google Scholar 

  • Rosen, Y., Wolf, I., & Stoeffler, K. (2020). Fostering collaborative problem solving skills in science: The Animalia project. Computers in Human Behavior, 104, 105922. https://doi.org/10.1016/j.chb.2019.02.018.

    Article  Google Scholar 

  • Salomon, G. (Ed.). (1997). Distributed cognitions: Psychological and educational considerations. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Scoular, C., Care, E., & Awwal, N. (2017). An approach to scoring collaboration in online game environments. Electronic Journal of e-Learning, 15(4), 335–342.

    Google Scholar 

  • Sottilare, R. A., Burke, C. S., Salas, E., Sinatra, A. M., Johnston, J. H., & Gilbert, S. B. (2018). Designing adaptive instruction for teams: A meta-analysis. International Journal of Artificial Intelligence in Education, 28, 225–264.

    Article  Google Scholar 

  • Stahl, G. (2013). Workshop presentation: Assessing the PISA assessment of CSCL. Presented at the international conference of computer-supported collaborative learning (CSCL), Madison, WI.

    Google Scholar 

  • Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning. In R. K. Sawyer (EB22d.), Cambridge handbook of the learning sciences. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Stahl, G., Ludvigsen, S., Law, N., & Cress, U. (2014). CSCL artifacts. International Journal of Computer-Supported Collaborative Learning, 9(3), 237–245.

    Article  Google Scholar 

  • Stoeffler, K., Rosen, Y., Bolsinova, M., & von Davier, A. A. (2020). Gamified performance assessment of collaborative problem solving skills. Computers in Human Behavior, 104, 106036. https://doi.org/10.1016/j.chb.2019.05.033.

    Article  Google Scholar 

  • Stoeffler, K., Rosen, Y., & von Davier, A. A. (2017). Exploring the measurement of collaborative problem solving using a human-agent educational game. In LAK’17: Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 570–571). Association for Computing Machinery. doi: https://doi.org/10.1145/3027385.3029464.

  • von Davier, A. A., & Halpin, P. F. (2013). Collaborative problem solving and the assessment of cognitive skills: Psychometric considerations (research report no. RR-13-41). Educational Testing Service.

    Google Scholar 

  • von Davier, A. A., Hao, J., Liu, L., & Kyllonen, P. (2017). Interdisciplinary research agenda in support of assessment of collaborative problem solving: Lessons learned from developing a collaborative science assessment prototype. Computers in Human Behavior, 76, 631–640.

    Article  Google Scholar 

  • World Economic Forum. (2015). New vision for education: Unlocking the potential of technology. Vancouver, BC: British Columbia Teachers’ Federation.

    Google Scholar 

  • Zhu, M., & Andrews-Todd, J. (2019). Understanding the connections of collaborative problem solving skills in a simulation-based task through network analysis [paper presentation]. International Conference on Computer Supported Collaborative Learning, Lyon, France.

    Google Scholar 

  • Zhu, M., & Zhang, M. (2017). Network analysis of conversation data for engineering professional skills assessment (research report no. RR-17-59). Educational Testing Service.

    Google Scholar 

Further Readings

  • Care, E., & Griffin, P. (2014). Assessment of collaborative problem solving. Research and Practice in Technology Enhanced Learning, 9, 367–388. This paper outlines the process, within the ATC21S project, of moving from the definition of collaborative problem-solving to its assessment. It describes the development of learning progressions, assessment tools, and scoring methods, tested with 4056 adolescents from six countries: Australia, Costa Rica, USA, Finland, the Netherlands, and Singapore.

    Google Scholar 

  • Care, E., Griffin, P., & Wilson, M. (Eds.). (2017). Assessment and teaching of 21st century skills: Research and application. Springer. Following Griffin and Care (2015), this edited volume includes 15 chapters that showcase research and application related to two 21st-century skills, collaborative problem-solving and learning in digital networks. The research reviewed includes data generated from the ACTC21S school trials as well as a case study of preservice teacher education in Finland.

    Google Scholar 

  • Dingler, C., von Davier, A. A., & Hao, J., (2017). Methodological challenges in measuring collaborative problem-solving skills over time. In E. Salas, W. B. Vessey, L. B. Landon (Eds.), Team dynamics over time (Vol. 18, pp. 51–70). Emerald Publishing. This chapter provides an interdisciplinary survey of recent developments in measurement of teamwork and collaboration in educational contexts. It discusses a range of methods for collecting and analyzing teamwork data and compares five frameworks for measuring collaborative problem-solving over time (from PISA, ATC21S, ETS, ACT, and von Davier & Halpin, 2013). Preliminary results of the assessments developed from these frameworks show innovative task designs, data-mining techniques, and novel applications of stochastic models.

    Google Scholar 

  • Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills. Springer. This edited volume includes 15 chapters on the ATC21S project, providing an overview of the method and framework for assessment, along with considerations for how collaborative tasks are delivered, scored, calibrated, and interpreted. Fieldwork chapters review the project implementation in the participating countries (Australia, Costa Rica, USA, Finland, the Netherlands, and Singapore). The concluding chapters address teaching and policy considerations related to collaborative learning.

    Google Scholar 

  • Song, Y. (2018). Improving primary students’ collaborative problem solving competency in project-based science learning with productive failure instructional design in a seamless learning environment. Education Technology Research and Development, 66, 979–1008. https://doi.org/10.1007/s11423-018-9600-3. This article reports an empirical study using mixed research methods to investigate primary students’ collaborative problem-solving competency in project-based learning with versus without productive failure instructional design. Data collection included student products from their coursework, as well as reflections, focus group interviews, and domain pre- and posttests. Results indicated that students in the class with productive failure instructional design showed multiple benefits, suggesting this as a fruitful avenue for further investigation.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. BrainPOP, New York, NY, USA

    Yigal Rosen

  2. ACT, Inc., Iowa City, IA, USA

    Kristin Stoeffler & Vanessa Simmering

  3. Educational Testing Service, Princeton, NJ, USA

    Jiangang Hao

  4. Duolingo, Pittsburgh, PA, USA

    Alina von Davier

  5. EdAstra Tech LLC, Boston, MA, USA

    Alina von Davier

Authors
  1. Yigal Rosen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kristin Stoeffler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vanessa Simmering
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiangang Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alina von Davier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yigal Rosen .

Editor information

Editors and Affiliations

  1. Leibniz-Institut für Wissensmedien, Tübingen, Germany

    Ulrike Cress

  2. Language Technologies Institute and Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA

    Carolyn Rosé

  3. Department of Administration Leadership, and Technology, New York University, New York, NY, USA

    Alyssa Friend Wise

  4. Faculty of Informatics, Shizuoka University, Hamamatsu-shi, Shizuoka, Japan

    Jun Oshima

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Rosen, Y., Stoeffler, K., Simmering, V., Hao, J., von Davier, A. (2021). Development of Scalable Assessment for Collaborative Problem-Solving. In: Cress, U., Rosé, C., Wise, A.F., Oshima, J. (eds) International Handbook of Computer-Supported Collaborative Learning. Computer-Supported Collaborative Learning Series, vol 19. Springer, Cham. https://doi.org/10.1007/978-3-030-65291-3_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-65291-3_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-65290-6

  • Online ISBN: 978-3-030-65291-3

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation