Skip to main content
SpringerLink
Log in

Development of the social metacognition inventory for online collaborative argumentation: construct validity and reliability

  • Research Article
  • Published:
Educational technology research and development Aims and scope Submit manuscript

Abstract

At present, with the rapid development of the internet and the gradual promotion of online collaborative learning, the social regulation of learning is receiving increasing attention, which involves socially shared metacognition, one facet of social metacognition. To date, social regulation of learning or socially shared metacognition have been widely studied using qualitative approaches. Although a variety of scales have been developed to measure metacognition in traditional individual learning, little work has been done to develop a scale to measure social metacognition in collaborative learning contexts. This study originally developed a social metacognition inventory consisting of 24 indicators by referring to the literature for assessing beliefs of other persons (BOP), awareness of other persons’ thinking (AOPT), judgment of other persons’ emotions (JOPE), co-regulation of each other’s thinking (CREOT), and evaluation of other persons’ thinking (EOPT). After EFA using 218 undergraduates’ questionnaires of social metacognition in collaborative argumentation on a social psychological issue from a Sino-Foreign Cooperative Educational Institution, 17 indicators showed good factorability and reliability. After CFA using another 300 questionnaires on social metacognition in collaborative argumentation about the pandemic received from undergraduates who come from 52 countries in the International College of Education, among the 17 indicators derived from the first sample’s EFA, three indicators had high correlation with others. Finally, based on the reviews of three experts, these three indicators were deleted. The remaining 14 indicators formed good construct validity with acceptable convergent and discriminant validities. In addition, the multi-group invariance test demonstrated that the structural model of the Social Metacognition Inventory has better configural invariance, which indicates that it can be generalized to other online collaborative argumentation contexts. The Social Metacognition Inventory can be used to quantify social metacognition in online collaborative argumentation when administering a large-scale experiment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The data of the present study are available upon request by sending e-mails to the corresponding author.

References

  • Arbuckle, J. L. (2009). IBM SPSS Amos 19 user’s guide. Version 19. SmallWaters Corporation.

    Google Scholar 

  • Akyol, Z., Garrison, D. R., & Vaughan, N. (2012). The development of a metacognition questionnaire for online and blended communities of inquiry. Paper presented at the Annual Meeting of the American Educational Research Association. Vancouver

  • Biasutti, M., & Frate, S. (2018). Group metacognition in online collaborative learning: Validity and reliability of the group metacognition scale(GMS). Educational Technology Research and Development, 66(6), 1321–1338.

    Article  Google Scholar 

  • Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology (pp. 77–165). Erlbaum Associates.

    Google Scholar 

  • Brown, J. D. (2009). Choosing the right number of components or factors in PCA and EFA. JALT Testing & Evaluation SIG Newsletter, 13(2), 19–23.

    Google Scholar 

  • Brown, T. (2006). Confirmatory factor analysis for applied research. Guildford Press.

    Google Scholar 

  • Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming (2nd ed.). Taylor and Francis.

    Google Scholar 

  • Chen, F. F. (2007). Sensitivity of goodness of fifit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504.

    Article  Google Scholar 

  • Chen, S., & McDunn, B. A. (2022). Metacognition: History, measurements, and the role in early childhood development and education. Learning and Motivation, 78, 101786.

    Article  Google Scholar 

  • Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255.

    Article  Google Scholar 

  • Dindar, M., Järvelä, S., & Järvenoja, H. (2020). Interplay of metacognitive experiences and performance in collaborative problem solving. Computers & Education, 154, 103922.

    Article  Google Scholar 

  • Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychological Review, 20, 391–409.

    Article  Google Scholar 

  • Efklides, A. (2006). Metacognition and affect: What can metacognitive experiences tell us about the learning process? Educational Research Review, 1(1), 3–14.

    Article  Google Scholar 

  • Efklides, A. (2008). Metacognition defining its facets and levels of functioning in relation to self-regulation and co-regulation. European Psychologist, 13(4), 277–287.

    Article  Google Scholar 

  • Efklides, A., Papadaki, M., Papantoniou, G., & Kiosseoglou, G. (1998). Individual differences in feelings of diffificulty: The case of school mathematics. European Journal of Psychology of Education, 8, 207–226.

    Article  Google Scholar 

  • Engeström, Y., & Sannino, A. (2010). Studies of expansive learning: Foundations, findings and future challenges. Educational Research Review, 5, 1–24.

    Article  Google Scholar 

  • Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Orienta-Konsultit.

    Google Scholar 

  • Erneling, C. E. (2010). Towards discursive education: Philosophy, technology, and modern education.

  • Fabricius, W. V., & Schwanenflugel, P. J. (1994). The older child’s theory of mind. In A. Demetriou & A. Efklides (Eds.), Intelligence, mind, and reasoning: Structure and development (pp. 111–132). Elsevier.

    Chapter  Google Scholar 

  • Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. Resnick (Ed.), The nature of intelligence (pp. 231–236). Lawrence Erlbaum Associates.

    Google Scholar 

  • Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of psychological inquiry. American Psychologist, 34, 906–911.

    Article  Google Scholar 

  • Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. E. Weinert & R. H. Kluwe (Eds.), Metacognition, motivation and understanding (pp. 21–30). Lawrence Erlbaum Associates.

    Google Scholar 

  • Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.

    Article  Google Scholar 

  • Garrison, D. R., & Akyol, Z. (2013). Toward the development of a metacognition construct for communities of inquiry. The Internet and Higher Education, 17, 84–89.

    Article  Google Scholar 

  • Garrison, D. R., & Akyol, Z. (2015). Toward the development of a metacognition construct for communities of inquiry. The Internet and Higher Education, 24, 66–71.

    Article  Google Scholar 

  • Gascoine, L., Higgins, S., & Wall, K. (2017). The assessment of metacognition in children aged 4–16 years: A systematic review. Review of Education, 5(1), 3–57.

    Article  Google Scholar 

  • Goldberg, T., & Schwarz, B. B. (2016). Harnessing emotions to deliberative argumentation in classroom discussions on historical issues in multi-cultural contexts. Frontline Learning Research, 4(4), 7–19.

    Article  Google Scholar 

  • Goos, M., Galbraith, P., & Renshaw, P. (2002). Socially mediated metacognition: Creating collaborative zones of proximal development in small group problem solving. Educational Studies in Mathematics, 49(2), 193–223.

    Article  Google Scholar 

  • Grau, V., & Whitebread, D. (2012). Self and social regulation of learning during collaborative activities in the classroom: The interplay of individual and group cognition. Learning and Instruction, 22(6), 401–412.

    Article  Google Scholar 

  • Greene, J. A. (2018). Self-regulation in education. Routledge.

    Google Scholar 

  • Hadwin, A., & Oshige, M. (2011). Self-regulation, co-regulaiton, and socially shared regulation: Exploring perspectives of social in self-regulated learning theory. Teachers College Record, 113(2), 240–264.

    Article  Google Scholar 

  • Hadwin, A., Järvelä, S., & Miller, M. (2018). Self-regulation, co-regulation, and shared regulation in collaborative learning environments. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (pp. 83–106). Routledge/Taylor & Francis Group.

    Google Scholar 

  • Hernández-Sellés, N., Pablo-César, M.-C., & González-Sanmamed, M. (2019). Computer-supported collaborative learning: An analysis of the relationship between interaction, emotional support and online collaborative tools. Computers & Education, 138, 1–12.

    Article  Google Scholar 

  • Huang, C.-C. (2009). Knowledge sharing and group cohesiveness on performance: An empirical study of technology R&D teams in Taiwan. Technovation, 29(11), 786–797.

    Article  Google Scholar 

  • Hurme, T.-R., Merenluoto, K., & Järvelä, S. (2009). Socially shared metacognition of pre-service primary teachers in a computer-supported mathematics course and their feelings of task difficulty: A case study. Educational Research and Evaluation, 15(3), 503–524.

    Article  Google Scholar 

  • Iiskala, T., Vauras, M., & Lehtinen, E. (2004). Socially-shared metacognition in peer learning? Hellenic Journal of Psychology, 1(2), 147–178.

    Google Scholar 

  • Iiskala, T., Vauras, M., Lehtinen, E., & Salonen, P. (2011). Socially shared metacognition of dyads of pupils in collaborativemathematical problem-solving processes. Learning and Instruction, 21, 379–393.

    Article  Google Scholar 

  • Isohätälä, J., Järvenoja, H., & Järvelä, S. (2017). Socially shared regulation of learning and participation in social interaction in collaborative learning. International Journal of Educational Research, 81, 11–24.

    Article  Google Scholar 

  • Isohätälä, J., Näykki, P., Järvelä, S., & Baker, M. J. (2018). Striking a balance: Socio-emotional processes during argumentation in collaborative learning interaction. Learning, Culture and Social Interaction, 16, 1–19.

    Article  Google Scholar 

  • Janssen, J., Erkens, G., & Kanselaar, G. (2007). Visualization of agreement and discussion processes during computer-supported collaborative learning. Computers in Human Behavior, 23, 1105–1125.

    Article  Google Scholar 

  • Järvelä, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. Educational Psychologist, 48(1), 25–39.

    Article  Google Scholar 

  • Järvelä, S., Kirschner, P. A., Panadero, E., Malmberg, J., Phielix, C., Jaspers, J., & Järvenoja, H. (2014). Enhancing socially shared regulation in collaborative learning groups: designing for CSCL regulation tools. Educational Technology Research and Development, 63(1), 125–142.

    Article  Google Scholar 

  • Jonassen, D. H., & Kim, B. (2009). Arguing to learn and learning to argue: Design justifications and guidelines. Educational Technology Research and Development, 58(4), 439–457.

    Article  Google Scholar 

  • Jost, J. T., Kruglanski, A. W., & Nelson, T. O. (1998). Social metacognition: An expansionist review. Personality & Social Psychology Review, 2(2), 137–154.

    Article  Google Scholar 

  • Lahdenperä, J., Rämö, J., & Postareff, L. (2022). Student-centred learning environments supporting undergraduate mathematics students to apply regulated learning: A mixed-methods approach. The Journal of Mathematical Behavior, 66, 100949.

    Article  Google Scholar 

  • Lakin, S. (2009). Socially mediated metacognition and learning to write. Thinking Skills and Creativity, 4(3), 149–159.

    Article  Google Scholar 

  • Lim, J. Y., & Lim, K. Y. (2020). Co-regulation in collaborative learning: Grounded in achievement goal theory. International Journal of Educational Research, 103, 101621.

    Article  Google Scholar 

  • Lobczowski, N. G., Allen, E. M., Firetto, C. M., Greene, J. A., & Murphy, P. K. (2020). An exploration of social regulation of learning during scientific argumentation discourse. Contemporary Educational Psychology, 63, 101925.

    Article  Google Scholar 

  • Lobczowski, N. G., Lyons, K., Greene, J. A., & McLaughlin, J. E. (2021). Socially shared metacognition in a project-based learning environment: A comparative case study. Learning, Culture and Social Interaction, 30, 100543.

    Article  Google Scholar 

  • Lories, G., Dardenne, B., & Yzerbyt, V. Y. (1998). From social cognition to metacognition. In V. Y. Yzerbyt, G. Lories, & B. Dardenne (Eds.), Metacognition: Cognitive and social dimensions (pp. 1–15). Sage.

    Google Scholar 

  • Lyons, K. M., Lobczowski, N. G., Greene, J. A., Whitley, J., & McLaughlin, J. E. (2021). Using a design-based research approach to develop and study a web-based tool to support collaborative learning. Computers & Education, 161, 104064.

    Article  Google Scholar 

  • Malmberg, J., Järvelä, S., Järvenoja, H., & Panadero, E. (2015). Promoting socially shared regulation of learning in CSCL: Progress of socially shared regulation among high- and low-performing groups. Computers in Human Behavior, 52, 562–572.

    Article  Google Scholar 

  • Mänty, K., Järvenoja, H., & Törmänen, T. (2020). Socio-emotional interaction in collaborative learning: Combining individual emotional experiences and group-level emotion regulation. International Journal of Educational Research, 102, 101589.

    Article  Google Scholar 

  • Metcalfe, J., & Shimamura, A. P. (1994). Metacognition. MIT Press.

    Book  Google Scholar 

  • Nelson, T. O. (Ed.). (1992). Metacognition: Core readings. Allyn & Bacon.

    Google Scholar 

  • Nelson, T. O. (1996). Consciousness and metacognition. American Psychologist, 51, 102–116.

    Article  Google Scholar 

  • Nelson, T. O., Kruglanski, A. W., & Jost, T. (1998). Knowing the self and others: Progress in metacognitive social psychology. In V. Yzerbyt, G. Lories, & B. Dardenne (Eds.), Metacognition: Cognitive and social dimensions (pp. 69–89). Sage.

    Chapter  Google Scholar 

  • Noroozi, O., Weinberger, A., Biemans, H. J. A., Mulder, M., & Chizari, M. (2012). Argumentation-based computer supported collaborative learning (ABCSCL): A synthesis of 15 years of research. Educational Research Review, 7(2), 79–106.

    Article  Google Scholar 

  • O’Neil, H. F., & Abedi, J. (1996). Reliability and validity of a state metacognitive inventory: Potential for alternative assessment. The Journal of Educational Research, 89(4), 234–245.

    Article  Google Scholar 

  • Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 452–502). Elsevier Academic Press.

    Google Scholar 

  • Pintrich, P. R., Wolters, C. A., & Baxter, G. P. (2000). Assessing metacognition and self-regulated learning. In J. C. Impara, G. Schraw, & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 43–97). University of Nebraska-Lincoln.

    Google Scholar 

  • Polo, C., Lund, K., Plantin, C., & Niccolai, G. P. (2016). Group emotions: The social and cognitive functions of emotions in argumentation. International Journal of Computer-Supported Collaborative Learning, 11, 123–156.

    Article  Google Scholar 

  • Raubenheimer, J. (2004). An item selection procedure to maximize scale reliability and validity. SA Journal of Industrial Psychology, 30(4), 59–64.

    Article  Google Scholar 

  • Robinson, K. (2013). The interrelationship of emotion and cognition when students undertake collaborative group work online: An interdisciplinary approach. Computers & Education, 62, 298–307.

    Article  Google Scholar 

  • Salonen, P., Vauras, M., & Efklides, A. (2005). Social interaction-what can it tell us about metacognition and coregulation in learning? European Psychologist, 10(3), 199–208.

    Article  Google Scholar 

  • Schmitt, N., & Kuljanin, G. (2008). Measurement invariance: Review of practice and implications. Human Resource Management Review, 18, 210–222.

    Article  Google Scholar 

  • Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460–475.

    Article  Google Scholar 

  • Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26(1), 113–125.

    Article  Google Scholar 

  • Schraw, G., & Moshman, D. (1995). Metacognitive theories. Educational Psychology Review, 7(4), 351–371.

    Article  Google Scholar 

  • Snedecor, G. W., & Cochran, W. G. (1989). Statistical methods (8th ed.). Iowa State University Press.

    Google Scholar 

  • Ursavaş, Ö. F., Yalçın, Y., & Bakır, E. (2019). The effect of subjective norms on preservice and in-service teachers’ behavioural intentions to use technology: A multigroup multimodel study. British Journal of Educational Technology, 50(5), 2501–2519.

    Article  Google Scholar 

  • Vandergriff, I. (2013). Emotive communication online: A contextual analysis of computer-mediated communication (CMC) cues. Journal of Pragmatics, 51, 1–12.

    Article  Google Scholar 

  • Veenman, M. V. (2011). Alternative assessment of strategy use with self-report instruments: A discussion. Metacognition and Learning, 6(2), 205–211.

    Article  Google Scholar 

  • Veenman, M. V. J., & Elshout, J. J. (1999). Changes in the relation between cognitive and metacognitive skills during the acquisition of expertise. European Journal of Psychology of Education, 14(4), 509–523.

    Article  Google Scholar 

  • Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838.

    Article  Google Scholar 

  • Yong, A. G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79–94.

    Article  Google Scholar 

  • Zhang, Z., Liu, T., & Lee, C. B. (2021). Language learners’ enjoyment and emotion regulation in online collaborative learning. System, 98, 102478.

    Article  Google Scholar 

  • Zimmerman, B. J. (2013). From cognitive modeling to self-regulation: A social cognitive career path. Educational Psychologist, 48(3), 135–147.

    Article  Google Scholar 

Download references

Acknowledgements

This study was granted by Project No. Y201839174 from Zhejiang Provincial Education Department and Project No. 18jg20 from Wenzhou University, People’s Republic of China.

Author information

Authors and Affiliations

  1. Department of Educational Technology, Wenzhou University, South Campus, Higher Education Region, Wenzhou, 325035, China

    Xiao-Li Zheng, Xin-Yan Gu, Wen-Hua Lai & Feng Wang

  2. Department of Library and Information Science, Research and Development Center for Physical Education, Health, and Information Technology, Fu Jen Catholic University, Taipei, Taiwan

    Yun-Fang Tu

  3. Graduate Institute of Digital Learning and Education, National Taiwan University of Science and Technology, Taipei, 106, Taiwan

    Gwo-Jen Hwang

  4. Graduate Institute of Educational Information and Measurement, National Taichung University of Education, Taichung, 403514, Taiwan

    Gwo-Jen Hwang

  5. College of Electrical and Electronic Engineering, Wenzhou University, Wenzhou, China

    Feng Wang

Authors
  1. Xiao-Li Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin-Yan Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen-Hua Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yun-Fang Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gwo-Jen Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Feng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gwo-Jen Hwang.

Ethics declarations

Conflict of interest

The authors would like to declare that there is no conflict of interest in this study.

Consent for participant

The participants were protected by hiding their personal information during the research process. They knew that the participation was voluntary and they could withdraw from the study at any time.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: The finalized 14-item social metacognition inventory (SMI)

Appendix: The finalized 14-item social metacognition inventory (SMI)

Considering what generally happened in your mind and behavior during online collaborative argumentation, please indicate the extent of your agreement/disagreement with the statements by using the following inventory:

Item

Strongly disagree

Disagree

Neutral

Agree

Strongly agree

I believe that team members can take their own responsibilities for the group work

     

I believe that team members have the ability to work together in a group

     

I can follow other members’ responses

     

I can make a response to other members’ responses

     

I can focus attention on other members’ ideas, understanding, or comments

     

I can judge other persons’ emotions by reading the emoticons on QQ

     

I can judge other persons’ emotions by reading the text on QQ

     

I set goals to achieve a high level of collaborative argumentation for the group work

     

I ask questions or request extra information to deepen my thinking

     

I challenge myself or team members for better solutions

     

I give advice to other members to help our collaborative argumentation

     

I can judge whether or not the claims proposed by others are correct

     

I can judge whether or not the justifications put forward by others are clear and persuasive

     

I can judge whether or not the counterarguments proposed by others are correct

     

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, XL., Gu, XY., Lai, WH. et al. Development of the social metacognition inventory for online collaborative argumentation: construct validity and reliability. Education Tech Research Dev 71, 949–971 (2023). https://doi.org/10.1007/s11423-023-10220-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-023-10220-5

Keywords

Search

Navigation