Abstract
Collaborative problem solving (CPS) is a critical and necessary skill in educational settings and the workforce. The assessment of CPS in the Programme for International Student Assessment (PISA) 2015 focuses on the cognitive and social skills related to problem solving in collaborative scenarios: establishing and maintaining shared understanding, taking appropriate actions to solve problems, and establishing and maintaining group organization. This chapter draws on measures of the CPS domain in PISA 2015 to address the development and implications of CPS items, challenges, and solutions related to item design, as well as computational models for CPS data analysis in large-scale assessments. Measuring CPS skills is not only a challenge compared to measuring individual skills but also an opportunity to make the cognitive processes in teamwork observable. An example of a released CPS unit in PISA 2015 will be used for the purpose of illustration. This study also discusses future perspectives in CPS analysis using multidimensional scaling, in combination with process data from log files, to track the process of students’ learning and collaborative activities.
This work was conducted while Matthias von Davier was employed with Educational Testing Service.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For instance, “planning and executing” is usually referred to as knowledge application in complex problem-solving research; see Wüstenberg, Greiff, and Funke (2012).
- 2.
In the PISA context, a task that, in turn, might be composed of several items is considered as one unit.
- 3.
The sample size for each country/language group was required to be 1950 students.
References
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.
Autor, D. H., Levy, F., & Murnane, R. J. (2003). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics, 118, 1279–1333.
Cannon-Bowers, J. A., & Salas, E. (2001). Reflections on shared cognition. Journal of Organizational Behavior, 22, 195–202.
Cascio, W. F. (1995). Whither industrial and organizational psychology in a changing world of work? American Psychologist, 50, 928–939.
Dillenbourg, P. (Ed.) (1999). Collaborative learning: Cognitive and computational approaches (Advances in Learning and Instruction Series) New York, NY: Elsevier Science.
Dillenbourg, P., & Traum, D. (2006). Sharing solutions: Persistence and grounding in multi-modal collaborative problem solving. Journal of the Learning Sciences, 15, 121–151.
Fiore, S. M., & Schooler, J. W. (2004). Process mapping and shared cognition: Teamwork and the development of shared problem models. In E. Salas & S. M. Fiore (Eds.), Team cognition: Understanding the factors that drive process and performance (pp. 133–152). Washington, DC: American Psychological Association.
Formann, A. K. (1985). Constrained latent class models: Theory and applications. British Journal of Mathematical and Statistical Psychology, 38, 87–111.
Formann, A. K. (1989). Constrained latent class models: Some further applications. British Journal of Mathematical and Statistical Psychology, 42, 37–54.
Formann, A. K. (1992). Linear logistic latent class analysis for polytomous data. Journal of the American Statistical Association, 87, 476–486.
Goldhammer, F., Naumann, J., Selter, A., Toth, K., Rolke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608–626.
Goos, M., Manning, A., & Salomons, A. (2009). Job polarization in Europe. American Economic Review, 99, 58–63.
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H., Ventura, M., Olney, A., et al. (2004). AutoTutor: A tutor with dialogue in natural language. Behavioral Research Methods, Instruments, and Computers, 36, 180–192.
Greiff, S., Wüstenberg, S., Csapo, B., Demetriou, A., Hautamäki, J., Graesser, A. C., et al. (2014). Domain-general problem solving skills and education in the 21st century. Educational Research Review, 13, 74–83.
Griffin, P., McGaw, B., & Care, E. (Eds.). (2012). Assessment and teaching of 21st century skills. Dordrecht, Netherlands: Springer.
He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 749–776). Hershey, PA: Information Science Reference.
Hirschberg, D. S. (1975). A linear space algorithm for computing maximal common subsequences. Communications of the ACM, 18, 341–343.
Hirschberg, D. S. (1977). Algorithms for the longest common subsequence problem. Journal of the ACM, 24(4), 664–675.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, CA: Erlbaum.
Organisation for Economic Co-operation and Development. (2009). PIAAC problem solving in technology-rich environments: A conceptual framework. OECD Education Working Paper No. 36. Paris, France: Author.
Organisation for Economic Co-operation and Development (2010). PISA 2012 Field Trial Problem Solving Framework. Paris, France: Author. Accessed July 6, 2016 http://www.oecd.org/dataoecd/8/42/46962005.pdf
Organisation for Economic Co-operation and Development. (2013). PISA 2015: Draft collaborative problem solving framework. Paris, France: Author.
Organisation for Economic Co-operation and Development. (2015a). PISA 2015 released field trial cognitive items. Paris, France: Author.
Organisation for Economic Co-operation and Development. (2015b). PISA 2015 field trial analysis report: Outcomes of the cognitive assessment (JT03371930). Paris, France: Author.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237–247.
Sonamthiang, S., Cercone, N., & Naruedomkul, K. (2007). Discovering hierarchical patterns of students’ learning behavior in intelligent tutoring systems. In Institute of Electrical and Electronics Engineers (Ed.), Proceedings of the 2007 IEEE International Conference on Granular Computing (pp. 485–489). Los Alamitos, CA: IEEE Computer Society Press.
Sukkarieh, J. Z., von Davier, M., & Yamamoto, K. (2012). From biology to education: Scoring and clustering multilingual text sequences and other sequential tasks (Research Report No. RR-12-25). Princeton, NJ: Educational Testing Service.
Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. New York, NY: Cambridge University Press.
Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving. More than reasoning? Intelligence, 40, 1–14.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this chapter
Cite this chapter
He, Q., von Davier, M., Greiff, S., Steinhauer, E.W., Borysewicz, P.B. (2017). Collaborative Problem Solving Measures in the Programme for International Student Assessment (PISA). In: von Davier, A., Zhu, M., Kyllonen, P. (eds) Innovative Assessment of Collaboration. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-33261-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-33261-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-33259-8
Online ISBN: 978-3-319-33261-1
eBook Packages: EducationEducation (R0)