Skip to main content
SpringerLink
Log in

Initial Steps Towards a Standardized Assessment for Collaborative Problem Solving (CPS): Practical Challenges and Strategies

  • Chapter
  • First Online:
Innovative Assessment of Collaboration

Part of the book series: Methodology of Educational Measurement and Assessment ((MEMA))

Abstract

Collaborative problem-solving (CPS) skill is an important 21st century skill (Griffin, McGaw, and Care, 2012). However, assessing CPS, particularly in a standardized way, is challenging. The type of collaboration, size of teams, and assessment domain all need to be properly considered when developing a CPS assessment. In this chapter, we outline some practical challenges for developing a large-scale, standardized assessment for CPS and present some strategies to address those challenges. We illustrate these strategies with the Collaborative Science Assessment Prototype (CSAP) developed at Educational Testing Service.

This work was conducted while Alina A. von Davier was employed with Educational Testing Service.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We introduced all the components in this study in this chapter, but the findings from several of them (e.g., personality, general science-knowledge test) won't be reported here.

References

  • Adams, R., Vista, A., Scoular, C., Awwal, N., Griffin, P., & Care, E. (2015). Automatic coding procedures. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 115–132). New York, NY, USA: Springer.

    Google Scholar 

  • Andrews, J., Kerr, D., Mislevy, R., Von Davier, A. A., Hao, J., & Liu, L. (in press). Using a simulation-based task to explore gender and cultural differences in collaboration. Journal of Educational Measurement.

    Google Scholar 

  • Barron, B. (2003). When smart groups fail. The Journal of the Learning Sciences, 12(3), 307–359.

    Article  Google Scholar 

  • Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.

    Article  Google Scholar 

  • Care, E., & Griffin, P. (2014). An approach to assessment of collaborative problem solving. Research & Practice in Technology Enhanced Learning, 9(3), 367–388.

    Google Scholar 

  • Cohen, E. G., Lotan, R. A., Scarloss, B. A., & Arellano, A. R. (1999). Complex instruction: Equity in cooperative learning classrooms. Theory Into Practice, 38(2), 80–86.

    Article  Google Scholar 

  • DeChurch, L. A., & Mesmer-Magnus, J. R. (2010). The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology, 95(1), 32–53.

    Article  Google Scholar 

  • Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder, & S. Barnes (Eds.), Technology-enhanced learning (pp. 3–19). New York, NY, USA: Springer.

    Google Scholar 

  • Dillenbourg, P., & Traum, D. (2006). Sharing solutions: Persistence and grounding in multimodal collaborative problem solving. Journal of the Learning Sciences, 15(1), 121–151.

    Article  Google Scholar 

  • Feng, S., Stewart, J., Clewley, D., & Graesser, A. C. (2015). Emotional, epistemic, and neutral feedback in autotutor trialogues to improve reading comprehension. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th international conference on artificial intelligence in education (pp. 570–573). New York, NY, USA: Springer.

    Google Scholar 

  • Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3), 613–619.

    Article  Google Scholar 

  • Flor, M., Yoon, S.-Y., Hao, J., Liu, L., & von Davier, A. (2016). Automated classification of collaborative problem solving interactions in simulated science tasks. In Proceedings of 11th workshop on innovative use of NLP for building educational applications (pp. 31–41). Stroudsburg, PA, USA: Association for Computational Linguistics.

    Google Scholar 

  • Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37(6), 504–528.

    Article  Google Scholar 

  • Griffin, P., McGaw, B., & Care, E. (Eds.). (2012). Assessment and teaching of 21st century skills: Methods and approach. New York, NY, USA: Springer.

    Google Scholar 

  • Halpin, P. F., von Davier, A. A., Hao, J., & Liu, L. (in press). Measuring student engagement during collaboration. Journal of Educational Measurement.

    Google Scholar 

  • Hao, J., Liu, L., von Davier, A., & Kyllonen, P. (2015). Assessing collaborative problem solving with simulation based tasks. In O. Lindwall, P. Hakkinen, T. Koschmann, P. Tchounkikine, & S. Ludvigsen (Eds.), Exploring the material conditions of learning: The computer supported collaborative learning (CSCL) conference 2015 (Vol. 1, pp. 544–547).

    Google Scholar 

  • Hao, J., Smith, L., Mislevy, R., von Davier, A., & Bauer, M. (2016a). Taming log files from game and simulation-based assessment: Data model and data analysis tool (Research Report No. RR-16-10). Princeton, NJ, USA: Educational Testing Service.

    Google Scholar 

  • Hao, J., Liu, L., von Davier, A., Kyllonen, P., & Kitchen, C., (2016b). Collaborative problem-solving skills versus collaboration outcomes: findings from statistical analysis and data mining. Proceedings of the 9th International Conference on Educational Data Mining.

    Google Scholar 

  • Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 37–56). New York, NY, USA: Springer.

    Google Scholar 

  • Ho, T. K. (1995). Random decision forests. (In Proceedings of the third international conference on document analysis and recognition (Vol. 1, pp. 278–282). Los Alamitos, CA: IEEE Computer Society Press.

    Google Scholar 

  • Järvelä, S., Volet, S., & Järvenoja, H. (2010). Research on motivation in collaborative learning: Moving beyond the cognitive-situative divide and combining individual and social processes. Educational Psychologist, 45(1), 15–27.

    Article  Google Scholar 

  • Karau, S. J., & Williams, K. D. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65(4), 681–706.

    Article  Google Scholar 

  • Kerr, D., Andrews, J., & Mislevy, R. (in press). The in-task assessment framework: Extracting evidence of proficiency from in-task behavior. In A. A. Rupp & J. Leighton (Eds.), Handbook of cognition and assessment: Frameworks, methods, and applications. New York, NY, USA: Wiley.

    Google Scholar 

  • Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. In CHI ‘08: Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453–456). New York, NY, USA: ACM.

    Google Scholar 

  • Koschmann, T. D. (1996). CSCL: Theory and practice of an emerging paradigm. New York, NY, USA: Routledge.

    Google Scholar 

  • Kreijns, K., Kirschner, P. A., & Jochems, W. (2002). The sociability of computer-supported collaborative learning environments. Educational Technology & Society, 5(1), 8–22.

    Google Scholar 

  • Liu, L., Hao, J., von Davier, A. A., Kyllonen, P., & Zapata-Rivera, D. (2015). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 344–359). Hershey, PA, USA: IGI Global.

    Google Scholar 

  • Lotan, R. A. (2003). Group-worthy tasks. Educational Leadership, 60(6), 72–75.

    Google Scholar 

  • Mislevy, R. J., & Riconscente, M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 61–90). New York, NY, USA: Routledge.

    Google Scholar 

  • National Assessment of Educational Progress. (2013). Questionnaires for students, teachers, and schools. Retrieved July 20, 2016, from https://nces.ed.gov/nationsreportcard/bgquest.aspx

  • O’Neil, H. F., Jr. (Ed.). (2014). Workforce readiness: Competencies and assessment. New York, NY, USA: Psychology Press.

    Google Scholar 

  • Organization for Economic Co-operation and Development. (2013). PISA 2015 draft collaborative problem solving assessment framework. Paris, France: Author.

    Google Scholar 

  • Roschelle, J. (1992). Learning by collaborating: Convergent conceptual change. Journal of the Learning Sciences, 2(3), 235–276.

    Article  Google Scholar 

  • Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O’Malley (Ed.), Computer supported collaborative learning (pp. 69–97). New York, NY, USA: Springer.

    Chapter  Google Scholar 

  • Rundgren, C.-J., Rundgren, S.-N. C., Tseng, Y.-H., Lin, P.-L., & Chang, C.-Y. (2012). Are you slim? Developing an instrument for civic scientific literacy measurement (SLiM) based on media coverage. Public Understanding of Science, 21(6), 759–773.

    Article  Google Scholar 

  • Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265–283.

    Article  Google Scholar 

  • Sottilare, R. A., Brawner, K. W., Goldberg, B. S., & Holden, H. K. (2012). The generalized intelligent framework for tutoring (gift). Orlando, FL: US Army Research Laboratory-Human Research & Engineering Directorate (ARL-HRED).

    Google Scholar 

  • Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge (acting with technology). Cambridge, MA, USA: MIT Press.

    Google Scholar 

  • Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 409–426). Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Sycara, K., Gelfand, M., & Abbe, A. (2013). Models for intercultural collaboration and negotiation. New York, NY, USA: Springer.

    Book  Google Scholar 

  • Van den Bossche, P., Gijselaers, W. H., Segers, M., & Kirschner, P. A. (2006). Social and cognitive factors driving teamwork in collaborative learning environments team learning beliefs and behaviors. Small Group Research, 37(5), 490–521.

    Article  Google Scholar 

  • von Davier, A. A., & Halpin, P. F. (2013). Collaborative problem solving and the assessment of cognitive skills: Psychometric considerations (Research Report No. RR-13-41). Princeton, NJ, USA: Educational Testing Service.

    Google Scholar 

  • Webb, N. M., Nemer, K. M., Chizhik, A. W., & Sugrue, B. (1998). Equity issues in collaborative group assessment: Group composition and performance. American Educational Research Journal, 35(4), 607–651.

    Article  Google Scholar 

  • Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686–688.

    Article  Google Scholar 

  • Zapata-Rivera, D., Jackson, T., Liu, L., Bertling, M., Vezzu, M., & Katz, I. R. (2014). Assessing science inquiry skills using trialogues. In S. Trausan-Matu, K. E Boyer, M. Crosby, & K. Panourgia (Eds.), Intelligent tutoring systems: 12th International Conference, ITS 2014, Honolulu, HI, USA, June 5–9, 2014 (pp. 625–626). New York, NY, USA: Springer.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Educational Testing Service, Princeton, NJ, 08541, USA

    Jiangang Hao, Lei Liu & Patrick C. Kyllonen

  2. ACT, Iowa City, IA, 52243, USA

    Alina A. von Davier

Authors
  1. Jiangang Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alina A. von Davier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patrick C. Kyllonen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiangang Hao .

Editor information

Editors and Affiliations

  1. ACT , Iowa City, Iowa, USA

    Alina A. von Davier

  2. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Mengxiao Zhu

  3. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Patrick C. Kyllonen

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Hao, J., Liu, L., von Davier, A.A., Kyllonen, P.C. (2017). Initial Steps Towards a Standardized Assessment for Collaborative Problem Solving (CPS): Practical Challenges and Strategies. In: von Davier, A., Zhu, M., Kyllonen, P. (eds) Innovative Assessment of Collaboration. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-33261-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-33261-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-33259-8

  • Online ISBN: 978-3-319-33261-1

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation