Skip to main content
SpringerLink
Log in

Investigating the mechanisms of analytics-supported reflective assessment for fostering collective knowledge

  • Published:
Journal of Computing in Higher Education Aims and scope Submit manuscript

Abstract

Helping students gradually develop collective knowledge is critical but generally faces great challenges. Employing a quasi-experimental design, this study investigated the impacts and mechanisms of analytics-supported reflective assessment on the collective knowledge advancement of undergraduates. The experimental group (n = 55) engaged in Knowledge Building inquiries with facilitation through analytics-supported reflective assessment, while the comparison class (n = 38) pursued Knowledge Building inquiries facilitated by portfolio-supported reflective assessment. This study found that analytics-supported reflective assessment positively and significantly influenced undergraduates’ collective knowledge advancement. Path analysis revealed the mechanisms of analytics-supported reflective assessment for supporting undergraduates’ collective knowledge advancement—the undergraduates’ metacognitive engagement and cognitive engagement influenced each other, further influencing their contribution to collective knowledge advancement and domain understanding. This study holds significant practical implications for fostering students’ knowledge building, inquiry, and metacognition by designing technology-enhanced learning environments as collaborative and metacognitive tools. Additionally, the study offers insights into the processes and mechanisms of reflective assessment, contributing to an understanding of how it enhances students’ development of higher-order skills.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Arbuckle, J. L. (2010). IBM SPSS Amos 19 user’s guide. Amos Development Corporation.

  • Bereiter, C., Chan, C. K. K., Hong, H.-Y., Lee, J., Khanlari, A., Lin, P. Y., Chai, C. S., Tsai, C. C., Scardamalia, M., Tan, S. C., Tong, Y., van Aalst, J., Zhang, J., & Zhang, Y. (2019). The roles of knowledge in knowledge creation. In K. Lund, G. P. Niccolai, E. Lavoué, C. Hmelo-Silver, G. Gweon, & M. Baker (Eds.), A wide lens: Combining embodied, enactive, extended, and embedded learning in collaborative settings, 13th International Conference on Computer Supported Collaborative Learning (CSCL) 2019 (Vol. 2, pp. 767–774). International Society of the Learning Sciences.

    Google Scholar 

  • Borge, M., Ong, Y. S., & Rosé, C. P. (2018). Learning to monitor and regulate collective thinking processes. International Journal of Computer-Supported Collaborative Learning, 13(1), 61–92.

    Article  Google Scholar 

  • Byrne, B. M. (2001). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Erlbaum.

    Google Scholar 

  • Cacciamani, S., Perrucci, V., & Fujita, N. (2021). Promoting students’ collective cognitive responsibility through concurrent, embedded and transformative assessment in blended higher education courses. Technology, Knowledge and Learning, 26(4), 1169–1194.

    Article  Google Scholar 

  • Chan, C. K. K., & Chan, Y.-Y. (2011). Students’views of collaboration and online participation in knowledge forum. Computers & Education, 57(1), 1445–1457.

    Article  Google Scholar 

  • Chen, B., & Hong, H.-Y. (2016). Schools as knowledge-building organizations: Thirty years of design research. Educational Psychologist, 51(2), 266–288.

    Article  Google Scholar 

  • D’Mello, S., Dieterle, E., & Duckworth, A. (2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123.

    Article  Google Scholar 

  • Damşa, C. I. (2014). The multi-layered nature of small-group learning: Productive interactions in object-oriented collaboration. International Journal of Computer-Supported Collaborative Learning, 9, 247–281.

    Article  Google Scholar 

  • Hong, H. Y., Chen, B., & Chai, C. S. (2016). Exploring the development of college students’ epistemic views during their knowledge building activities. Computers & Education, 98, 1–13.

    Article  Google Scholar 

  • Hung, S. T. A. (2019). Creating digital stories: EFL learners’ engagement, cognitive and metacognitive skills. Educational Technology & Society, 22(2), 26–37.

    Google Scholar 

  • Järvelä, S., Gašević, D., Seppänen, T., Pechenizkiy, M., & Kirschner, P. A. (2020). Bridging learning sciences, machine learning and affective computing for understanding cognition and affect in collaborative learning. British Journal of Educational Technology, 51(6), 2391–2406.

    Article  Google Scholar 

  • Jiang, J. P., Hu, J. Y., Zhang, Y. B., & Yin, X. C. (2022). Fostering college students’ critical thinking skills through peer assessment in the knowledge building community. Interactive Learning Environments, 1–17.

  • Keith, T. Z. (2014). Multiple regression and beyond: An introduction to multiple regression and structural equation modeling (2nd ed.). Routledge.

  • Lee, E. Y., Chan, C. K. K., & van Aalst, J. (2006). Students assessing their own collaborative knowledge building. International Journal of Computer-Supported Collaborative Learning, 1, 57–87.

    Article  Google Scholar 

  • Lei, C., & Chan, C. K. K. (2018). Developing metadiscourse through reflective assessment in knowledge building environments. Computers & Education, 126, 153–169.

    Article  Google Scholar 

  • Lin, F., & Chan, C. K. K. (2018). Promoting elementary students’ epistemology of science through computer-supported knowledge-building discourse and epistemic reflection. International Journal of Science Education, 40(6), 668–687.

    Article  Google Scholar 

  • Oshima, J., & Hoppe, H. U. (2021). Finding meaning in log-file data. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning. Computer-supported collaborative learning series. (Vol. 19). Cham: Springer.

    Google Scholar 

  • Oshima, J., Ohsaki, A., Yamada, Y., & Oshima, R. (2017). Collective knowledge advancement and conceptual understanding of complex scientific concepts in the jigsaw instruction. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), Making a difference: Prioritizing equity and access in CSCL, 12 th International conference on computer supported collaborative learning (CSCL) 2017 (Vol. 1, pp. 57–64). International Society of the Learning Sciences.

    Google Scholar 

  • Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge building discourse explorer: A social network analysis application for knowledge building discourse. Educational Technology Research and Development, 60, 903–921.

    Article  Google Scholar 

  • Oshima, J., Oshima, R., & Saruwatari, S. (2020). Analysis of students’ ideas and conceptual artifacts in knowledge-building discourse. British Journal of Educational Technology, 51(4), 1308–1321.

    Article  Google Scholar 

  • Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. Handbook of research on student engagement (pp. 259–282). Springer.

    Chapter  Google Scholar 

  • Radović, S., Firssova, O., Hummel, H. G., & Vermeulen, M. (2023). The case of socially constructed knowledge through online collaborative reflection. Studies in Continuing Education, 45(2), 168–187.

    Article  Google Scholar 

  • Raes, A., Schellens, T., & De Wever, B. (2014). Web-based collaborative inquiry to bridge gaps in secondary science education. Journal of the Learning Sciences, 23(3), 316–347.

    Article  Google Scholar 

  • Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 97–116). Cambridge University Press.

    Google Scholar 

  • Scardamalia, M., & Bereiter, C. (2014). Knowledge building and knowledge creation: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 397–417). Cambridge University Press.

    Chapter  Google Scholar 

  • Scardamalia, M., & Bereiter, C. (2021). Knowledge building: Advancing the state of community knowledge. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning. Computer-supported collaborative learning series. (Vol. 19). Cham: Springer.

    Google Scholar 

  • Scardmalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 67–98). Open Court.

    Google Scholar 

  • Shaffer, D. W. (2018). Epistemic network analysis: Understanding learning by using big data for thick description. In International handbook of the learning sciences (pp. 520–531). Routledge.

  • Siemens, G., & Gasevic, D. (2012). Guest editorial-learning and knowledge analytics. Journal of Educational Technology & Society, 15(3), 1–2.

    Google Scholar 

  • Tan, S. C., Chan, C., & Bielaczyc, K. (2021). Knowledge building: Aligning education with needs for knowledge creation in the digital age. Educational Technology Research and Development, 69, 2243–2266.

    Article  Google Scholar 

  • Tao, D., & Zhang, J. (2021). Agency to transform: How did a grade 5 community co-configure dynamic knowledge building practices in a yearlong science inquiry? International Journal of Computer-Supported Collaborative Learning, 16, 403–434.

    Article  Google Scholar 

  • Tong, Y., & Chan, C. K. K. (2019). Developing productive discourse through collective inquiry of knowledge-building principles. In K. Lund, G. Niccolai, E. Lavoué, C. Hmelo-Silver, G. Gweon, & M. Baker (Eds.), A wide lens: Combining embodied, enactive, extended, and embedded learning in collaborative settings, 13th international conference on computer supported collaborative learning CSCL 2019 (Vol. 1, pp. 534–541). International Society of the Learning Sciences.

    Google Scholar 

  • Toth, E. E., Suthers, D. D., & Lesgold, A. M. (2002). “Mapping to know”: The effects of representational guidance and reflective assessment on scientific inquiry. Science Education, 86(2), 264–286.

    Article  Google Scholar 

  • Van Aalst, J., & Chan, C. K. K. (2007). Student-directed assessment of knowledge building using electronic portfolios. The Journal of the Learning Sciences, 16, 175–220.

    Article  Google Scholar 

  • White, B., & Frederiksen, J. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16, 3–118.

    Article  Google Scholar 

  • White, B., & Frederiksen, J. (2005). A theoretical framework and approach for fostering metacognitive development. Educational Psychologist, 40(4), 211–223.

    Article  Google Scholar 

  • Wise, A. F., & Schwartz, B. (2017). Visions of CSCL: Eight provocations for the future of the field. International Journal of Computer-Supported Collaborative Learning, 12, 423–467.

    Article  Google Scholar 

  • Yang, Y., Feng, X., Zhu, G., & Sun, D. (2023). Exploring the mechanisms of data-supported reflective assessment for pre-service teachers’ knowledge building. Interactive Learning Environments, 1–18.

  • Yang, Y., Feng, X., Zhu, G., & Xie, K. (2024). Effects and mechanisms of analytics-assisted reflective assessment in fostering undergraduates' collective epistemic agency in computer-supported collaborative inquiry. Journal of Computer Assisted Learning, 1–25.

  • Yang, Y. (2019). Reflective assessment for epistemic agency of academically low-achieving students. Journal of Computer Assisted Learning, 35(4), 459–475.

    Article  Google Scholar 

  • Yang, Y., Chen, Q., Yu, Y., Feng, X., & van Aalst, J. (2020a). Collective reflective assessment for shared epistemic agency by undergraduates in knowledge building. British Journal of Educational Technology, 51(4), 1136–1154.

    Article  Google Scholar 

  • Yang, Y., Du, Y., van Aalst, J., Sun, D., & Ouyang, F. (2020b). Self-directed reflective assessment for collective empowerment among pre-service teachers. British Journal of Educational Technology, 51(6), 1961–1981.

    Article  Google Scholar 

  • Yang, Y., van Aalst, J., & Chan, C. K. (2020c). Dynamics of reflective assessment and knowledge building for academically low-achieving students. American Educational Research Journal, 57(3), 1241–1289.

    Article  Google Scholar 

  • Yang, Y., van Aalst, J., Chan, C. K., & Tian, W. (2016). Reflective assessment in knowledge building by students with low academic achievement. International Journal of Computer-Supported Collaborative Learning, 11, 281–311.

    Article  Google Scholar 

  • Yang, Y., Yuan, K., Feng, X., Li, X., & van Aalst, J. (2022a). Fostering low-achieving students’ productive disciplinary engagement through knowledge-building inquiry and reflective assessment. British Journal of Educational Technology, 53(6), 1511–1529.

    Article  Google Scholar 

  • Yang, Y., Zhu, G., Sun, D., & Chan, C. K. (2022b). Collaborative analytics-supported reflective assessment for scaffolding pre-service teachers’ collaborative inquiry and knowledge building. International Journal of Computer-Supported Collaborative Learning, 17(2), 249–292.

    Article  Google Scholar 

  • Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of 9-and 10-year-olds. Educational Technology Research and Development, 55, 117–145.

    Article  Google Scholar 

  • Zhang, J., Tao, D., Chen, M. H., Sun, Y., Judson, D., & Naqvi, S. (2018). Co-organizing the collective journey of inquiry with idea thread mapper. Journal of the Learning Sciences, 27, 390–430.

    Article  Google Scholar 

  • Zhang, J., Tian, Y., Yuan, G., & Tao, D. (2022). Epistemic agency for costructuring expansive knowledge-building practices. Science Education, 106(4), 890–923.

    Article  Google Scholar 

  • Zhao, K., & Chan, C. K. K. (2014). Fostering collective and individual learning through knowledge building. International Journal of Computer-Supported Collaborative Learning, 9(1), 63–95.

    Article  Google Scholar 

Download references

Acknowledgements

We thank the instructor and students for their participation. This research is supported by grants from University-Level Educational Reformation Research Projects (CCNUTEIII 2021-11, CCNUAI&FE2022-03-15) of Central China Normal University, the National Natural Science Foundation of the People’s Republic of China (Grant No. 62107020), Ministry of Education of the People’s Republic of China (Grant Nos. 21YJA880078, and 22YJC880058), and Collaborative Innovation Center for Informatization and Balanced Development of K-12 Education by MOE (Ministry of Education of the People’s Republic of China) and Hubei Province (xtzd2022-002).

Author information

Authors and Affiliations

  1. Faculty of Artificial Intelligence in Education, Central China Normal University, Hubei, China

    Yuqin Yang, Yewen Chen & Shiyan Pang

  2. Center for Future Education Research, Southern University of Science and Technology, Shenzhen, China

    Xueqi Feng

  3. Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong, China

    Daner Sun

Authors
  1. Yuqin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yewen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xueqi Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daner Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shiyan Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xueqi Feng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

See Table 

Table 3 Demographic details of the participating undergraduates
Full size table

3.

Appendix 2

See Table 

Table 4 Sample prompt sheet for using KBDeX
Full size table

4.

Appendix 3

See Table 

Table 5 Coding scale for rating students’ responses in pre-post tests on understanding of collaborative inquiry
Full size table

5.

Appendix 4

See Table 

Table 6 Coding Framework for the Analysis and Characterization of Knowledge Building Actions
Full size table

6.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Chen, Y., Feng, X. et al. Investigating the mechanisms of analytics-supported reflective assessment for fostering collective knowledge. J Comput High Educ 36, 242–273 (2024). https://doi.org/10.1007/s12528-024-09398-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12528-024-09398-1

Keywords

Search

Navigation