Multimodal Learning Analytics in a Laboratory Classroom

  • Man Ching Esther ChanEmail author
  • Xavier Ochoa
  • David Clarke
Part of the Intelligent Systems Reference Library book series (ISRL, volume 158)


Sophisticated research approaches and tools can help researchers to investigate the complex processes involved in learning in various settings. The use of video technology to record classroom practices, in particular, can be a powerful way for capturing and studying learning and related phenomena within a social setting such as the classroom. This chapter outlines several multimodal techniques to analyze the learning activities in a laboratory classroom. The video and audio recordings were processed automatically to obtain information rather than requiring manual coding. Moreover, these automated techniques are able to extract information with an efficiency that is beyond the capabilities of human-coders, providing the means to deal analytically with the multiple modalities that characterize the classroom. Once generated, the information provided by the different modalities is used to explain and predict high-level constructs such as students’ attention and engagement. This chapter not only presents the results of the analysis, but also describes the setting, hardware and software needed to replicate this analytical approach.



This research was conducted with Science of Learning Research Centre funding provided by the Australian Research Council Special Initiatives Grant (SR120300015) and the Discovery Projects funding scheme (DP170102541). We would like to thank the students, parents, teachers, and school staff for their invaluable support of this project. We are also very grateful to our technical team, Cameron Mitchell and Peter (Reggie) Bowman, for their expertise in operating the Science of Learning Research Classroom facility for the project.


  1. 1.
    Amidon, E., Flanders, N.A.: The effects of direct and indirect teacher influence on dependent-prone students learning geometry. J. Educ. Psychol. 52(6), 286–291 (1961). Scholar
  2. 2.
    Antonakis, J., Bendahan, S., Jacquart, P., Lalive, R.: On making causal claims: a review and recommendations. Leadersh. Q. 21(6), 1086–1120 (2010). Scholar
  3. 3.
    Beeby, T., Burkhardt, H., Fraser, R.: Systematic Classroom Analysis Notation. Shell Centre for Mathematics Education, Nottingham, England (1979)Google Scholar
  4. 4.
    Bezdek, J.C., Ehrlich, R., Full, W.: FCM: the fuzzy c-means clustering algorithm. Comput. Geosci. 10(2), 191–203 (1984). Scholar
  5. 5.
    Blikstein, P.: Multimodal learning analytics. In: Suthers, D., Verbert, K., Duval, E., Ochoa, X. (eds.) Proceedings of the Third International Conference on Learning Analytics and Knowledge, Leuven, Belgium, pp. 102–106 (2013)Google Scholar
  6. 6.
    Bourke, S.F.: The teaching and learning of mathematics: national report of the second phase of the IEA Classroom Environment Study (ACER research monograph No. 25). Australian Council for Educational Research, Hawthorn, Victoria (1984)Google Scholar
  7. 7.
    Campbell, D.T.: Assessing the impact of planned social change (Occasional Paper Series No. 8). Western Michigan University, College of Education, Evaluation Center, Kalamazoo, MI (1976).
  8. 8.
    Chan, M.C.E., Baker, A., Slee, R., Williamson, J.: Educational engagement through the middle years of schooling: Report for the In2COmmuinity Applied Learning Project. Victoria Institute, Melbourne, Australia (2015).
  9. 9.
    Chan, M.C.E., Clarke, D.J.: Video-based research in a laboratory classroom. In: Xu, L., Aranda, G., Clarke, D. (eds.) Video-Based Research in Education: Cross-Disciplinary Perspectives. Routledge, New York (in press)Google Scholar
  10. 10.
    Chan, M.C.E., Clarke, D.J.: Learning research in a laboratory classroom: Complementarity and commensurability in juxtaposing multiple interpretive accounts. In: Dooley, T., Gueudet, G. (eds.) Proceedings of the Congress of European Research in Mathematics Education, Dublin, Ireland, pp. 2713–2720 (2017)Google Scholar
  11. 11.
    Chan, M.C.E., Clarke, D.J.: Structured affordances in the use of open-ended tasks to facilitate collaborative problem solving. ZDM Int. J. Math. Educ. 49, 951–963 (2017). Scholar
  12. 12.
    Chan, M.C.E., Clarke, D.J., Cao, Y.: The social essentials of learning: an experimental investigation of collaborative problem solving and knowledge construction in mathematics classrooms in Australia and China. Math. Educ. Res. J. 30(1), 39–50 (2017). Scholar
  13. 13.
    Clarke,D.J.: International comparative studies in mathematics education. Chapter 5 In: Bishop, A.J.,Clements, M.A.,Keitel, C.,Kilpatrick, J., Leung, F.K.S. (eds.) Second InternationalHandbook of Mathematics Education, pp. 145–186. Dordrecht, The Netherlands: Kluwer Academic Publishers (2003)Google Scholar
  14. 14.
    Clarke, D.J.: Studying the classroom negotiation of meaning: Complementary accounts methodology, Chapter 7. In: Teppo, A. (ed.) Qualitative Research Methods in Mathematics Education. Monograph Number 9 of the Journal for Research in Mathematics Education, pp. 98–111. NCTM, Reston, VA (1998)Google Scholar
  15. 15.
    Clarke, D.J. (ed.): Perspectives on Practice and Meaning in Mathematics and Science Classrooms. Kluwer Academic Publishers, Dordrecht, The Netherlands (2001)Google Scholar
  16. 16.
    Clarke, D.J.: Using cross-cultural comparison to interrogate the logic of classroom research in mathematics education. In: Kaur, B., Ho, W.K., Toh, T.L., Choy, B.H. (eds.) Proceedings of the 41st Conference of the International Group for the Psychology of Mathematics Education, vol. 1, pp. 13–28. PME, Singapore (2017)Google Scholar
  17. 17.
    Clarke, D.J., Keitel, C., Shimizu, Y. (eds.): Mathematics Classrooms in Twelve Countries: The Insider’s Perspective. Sense Publishers, Rotterdam, The Netherlands (2006)Google Scholar
  18. 18.
    Clarke, D.J., Mitchell, C., Bowman, P.: Optimising the use of available technology to support international collaborative research in mathematics classrooms. In: Janik, T., Seidel, T. (eds.) The Power of Video Studies in Investigating Teaching and Learning in the Classroom, pp. 39–60. Waxmann, New York (2009)Google Scholar
  19. 19.
    Clarke, D.J., Xu, L.H., Wan, M.E.V.: Spoken mathematics as an instructional strategy: The public discourse of mathematics classrooms in different countries. In: Kaur, B., Anthony, G., Ohtani, M., Clarke, D. (eds.) Student Voice in Mathematics Classrooms Around the World, pp. 13–31. Sense Publishers, Rotterdam, The Netherlands (2013)CrossRefGoogle Scholar
  20. 20.
    Cobb, P., Bauersfeld, H.: The Emergence of Mathematical Meaning: Interaction in Classroom Cultures. L. Erlbaum Associates, Hillsdale, NJ (1995)Google Scholar
  21. 21.
    Dringus, L.P.: Learning analytics considered harmful. J. Asynchronous Learn. Netw. 16(3), 87–100 (2012)Google Scholar
  22. 22.
    Erlwanger, S.H.: Case studies of children’s conceptions of mathematics. J. Child. Math. Behav. 1(3), 157–283 (1975)Google Scholar
  23. 23.
    Ferguson, R.: Learning analytics: drivers, developments and challenges. Int. J. Technol. Enhanc. Learn. 4(5–6), 304–317 (2012)CrossRefGoogle Scholar
  24. 24.
    Ferreiro, E., Teberosky, A.: Literacy before schooling (K. Goodman Castro Trans.) [Los sistemas de escritura en el desarrollo del niño]. Heinemann Educational Books, Exeter, NH (Original work published 1979) (1982)Google Scholar
  25. 25.
    Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: potential of the concept, state of the evidence. Rev. Educ. Res. 74(1), 59–109 (2004)CrossRefGoogle Scholar
  26. 26.
    Frydenberg, E., Ainley, M., Russell, V.J.: Student motivation and engagement. Sch. Issues Dig. 2 (2005)Google Scholar
  27. 27.
    Gibbs, R., Poskitt, J.: Student Engagement in the Middle Years of Schooling (Years 7–10): A Literature Review. New Zealand Ministry of Education, Wellington (2010).
  28. 28.
    Good, T.L., Grouws, D.A.: Teaching effects: a process-product study in fourth-grade mathematics classrooms. J. Teach. Educ. 28(3), 49–54 (1977). Scholar
  29. 29.
    Hart, L.E.: Classroom processes, sex of student, and confidence in learning mathematics. J. Res. Math. Educ. 20(3), 242–260 (1989). Scholar
  30. 30.
    Helme, S., Clarke, D.: Cognitive engagement in the mathematics classroom. In: Clarke, D. (ed.) Perspectives on Practice and Meaning in Mathematics and Science Classrooms, pp. 131–153. Kluwer Academic Publishers, Dordrecht, The Netherlands (2001)Google Scholar
  31. 31.
    Hiebert, J., Gallimore, R., Garnier, H., Givvin, K.B., Hollingsworth, H., Jacobs, J., et al.: Teaching Mathematics in Seven Countries: Results from the TIMSS 1999 Video Study (NCES 2003–013 Revised). U.S. Department of Education, National Center for Education Statistics, Washington, DC (2003).
  32. 32.
    Janík, T., Seidel, T. (eds.): The Power of Video Studies in Investigating Teaching and Learning in the Classroom. Waxmann, Münster, Germany (2009)zbMATHGoogle Scholar
  33. 33.
    Kleinsmith, A., Bianchi-Berthouze, N.: Recognizing affective dimensions from body posture. In: Paiva, A., Prada, R., Picard, R.W. (eds.) Proceedings of the Affective Computing and Intelligent Interaction Conference, Lisbon, Portugal, pp. 48–58 (2007)Google Scholar
  34. 34.
    Lim, F.V., O’Halloran, K.L., Podlasov, A.: Spatial pedagogy: mapping meanings in the use of classroom space. Camb. J. Educ. 42(2), 235–251 (2012)CrossRefGoogle Scholar
  35. 35.
    Marshall, S.: Exploring the ethical implications of MOOCs. Distance Educ. 35(2), 250–262 (2014). Scholar
  36. 36.
    Mota, S., Picard, R.W.: Automated posture analysis for detecting learner’s interest level. In: Martinez, A.M., Tan, H.Z. (eds.) Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop, Madison, WI, vol. 5, pp. 49–54, June 2003Google Scholar
  37. 37.
    Ochoa, X., Chiluiza, K., Méndez, G., Luzardo, G., Guamán, B., Castells, J.: Expertise estimation based on simple multimodal features. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 583–590. ACM, Dec 2013Google Scholar
  38. 38.
    Ochoa, X.: Multimodal learning analytics. In: Lang, C., Siemens, G., Wise, A.F., Gaševic, D. (eds.) The Handbook of Learning Analytics, pp. 129–141. Society for Learning Analytics Research (SoLAR), Alberta, Canada (2017)CrossRefGoogle Scholar
  39. 39.
    Ochoa, X., Worsley, M.: Augmenting learning analytics with multimodal sensory data. J. Learn. Anal. 3(2), 213–219 (2016)CrossRefGoogle Scholar
  40. 40.
    Overholt, G.: Ethnography and education: limitations and sources of error. J. Thought 15(3), 11–20 (1980)Google Scholar
  41. 41.
    Paiva, A., Prada, R., Picard, R.W. (eds.): Conference Proceedings of the Second Affective Computing and Intelligent Interaction Conference. Springer, Lisbon, Portugal (2007)Google Scholar
  42. 42.
    Piaget, J.: The Language and Thought of the Child. Routledge & Kegan Paul, London (1926)Google Scholar
  43. 43.
    Peterson, P.L., Fennema, E.: Effective teaching, student engagement in classroom activities, and sex-related differences in learning mathematics. Am. Educ. Res. J. 22(3), 309–335 (1985). Scholar
  44. 44.
    Raca, M., Dillenbourg, P.: System for assessing classroom attention. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge, pp. 265–269. ACM, Apr 2013Google Scholar
  45. 45.
    Raca, M., Dillenbourg, P.: System for assessing classroom attention. In: Suthers, D., Verbert, K., Duval, E., Ochoa, X. (eds.) Proceedings of the Third International Conference on Learning Analytics and Knowledge, Leuven, Belgium, pp. 265–269, Apr 2013Google Scholar
  46. 46.
    Raca, M., Tormey, R., Dillenbourg, P.: Sleepers’ lag-study on motion and attention. In: Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, Indianapolis, IN, pp. 36–43, Mar 2014Google Scholar
  47. 47.
    Sfard, A., Kieran, C.: Cognition as communication: Rethinking learning-by-talking through multi-faceted analysis of students’ mathematical interactions. Mind Cult. Act. 8(1), 42–76 (2001). Scholar
  48. 48.
    Shernoff, D.J.: Optimal Learning Environments to Promote Student Engagement. Springer, New York (2013)CrossRefGoogle Scholar
  49. 49.
    Skinner, B.F.: The science of learning and the art of teaching. Harv. Educ. Rev. 24, 86–97 (1954)Google Scholar
  50. 50.
    Skinner, E.A., Belmont, M.J.: Motivation in the classroom: reciprocal effects of teacher behavior and student engagement across the school year. J. Educ. Psychol. 85(4), 571–581 (1993)CrossRefGoogle Scholar
  51. 51.
    Stigler, J.W., Hiebert, J.: The Teaching Gap: Best Ideas from the World’s Teachers for Improving Education in the Classroom. Free Press, New York (2009)Google Scholar
  52. 52.
    Taylor, L., Parsons, J.: Improving student engagement. Curr. Issues Educ. 14(1) (2011)Google Scholar
  53. 53.
    Ulewicz, M., Beatty, A.: The Power of Video Technology in International Comparative Research in Education. National Academy Press, Washington, DC (2001)Google Scholar
  54. 54.
    Vygotsky, L.S.: Mind in Society: The Development of Higher Psychological Processes. Harvard University Press, Cambridge, MA (1978)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Man Ching Esther Chan
    • 1
    Email author
  • Xavier Ochoa
    • 2
  • David Clarke
    • 1
  1. 1.Melbourne Graduate School of EducationThe University of MelbourneVictoriaAustralia
  2. 2.New York UniversityNew YorkUSA

Personalised recommendations