The Multimodal Matrix as a Quantitative Ethnography Methodology

  • Simon Buckingham ShumEmail author
  • Vanessa Echeverria
  • Roberto Martinez-Maldonado
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1112)


This paper seeks to contribute to the emerging field of Quantitative Ethnography (QE) by demonstrating its utility to solve a complex challenge in Learning Analytics: the provision of timely feedback to collocated teams and their coaches. We define two requirements that extend the QE concept in order to operationalise it such a design process, namely, the use of co-design methodologies, and the availability of automated analytics workflow to close the feedback loop. We introduce the Multimodal Matrix as a data modelling approach that can integrate theoretical concepts about teamwork with contextual insights about specific work practices, enabling the analyst to map between higher order codes and low-level sensor data, with the option add the results of manually performed analyses. This is implemented in software as a workflow for rapid data modelling, analysis and interactive visualisation, demonstrated in the context of nursing teamwork simulations. We propose that this exemplifies how a QE methodology can underpin collocated activity analytics, at scale, with in-principle applications to embodied, collocated activities beyond our case study.


Multimodal Learning analytics Teamwork CSCL Sense making 


  1. 1.
    Alhadad, S.S.J.: Visualizing data to support judgement, inference, and decision making in learning analytics: insights from cognitive psychology and visualization science. J. Learn. Anal. 5(2), 60–85 (2018)CrossRefGoogle Scholar
  2. 2.
    Bakhshinategh, B., Zaiane, O.R., Elatia, S., Ipperciel, D.: Educational data mining applications and tasks: a survey of the last 10 years. Educ. Inf. Technol. 23(1), 537–553 (2018)CrossRefGoogle Scholar
  3. 3.
    Behrens, J.T., Dicerbo, K.E., Foltz, P.W.: Assessment of complex performances in digital environments. Ann. Am. Acad. Polit. Soc. Sci. 683(1), 217–232 (2019)CrossRefGoogle Scholar
  4. 4.
    Bodily, R., Verbert, K.: Trends and issues in student-facing learning analytics reporting systems research. In: Proceedings of the International Learning Analytics and Knowledge Conference, LAK 2017, pp. 309–318 (2017)Google Scholar
  5. 5.
    Braithwaite, J.J., Watson, D.G., Jones, R., Rowe, M.: A guide for analysing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments. Psychophysiology 49(1), 1017–1034 (2013)Google Scholar
  6. 6.
    Buckingham Shum, S., Crick, R.D.: Learning analytics for 21st century competencies. J. Learn. Anal. 3(2), 6–21 (2016)CrossRefGoogle Scholar
  7. 7.
    Chandrasegaran, S., Bryan, C., Shidara, H., Chuang, T.-Y., Ma, K.-L.: TalkTraces: real-time capture and visualization of verbal content in meetings. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2019, pp. 577:571–577:514 (2019)Google Scholar
  8. 8.
    Chappel, S.E., Verswijveren, S.J.J.M., Aisbett, B., Considine, J., Ridgers, N.D.: Nurses’ occupational physical activity levels: a systematic review. Int. J. Nurs. Stud. 73(August), 52–62 (2017)CrossRefGoogle Scholar
  9. 9.
    Csanadi, A., Eagan, B., Kollar, I., Shaffer, D.W., Fischer, F.: When coding-and-counting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research. Int. J. Comput.-Support. Collab. Learn. 13(4), 419–438 (2018)CrossRefGoogle Scholar
  10. 10.
    Di Mitri, D., Schneider, J., Klemke, R., Specht, M., Drachsler, H.: Read between the lines: an annotation tool for multimodal data for learning. In: Proceedings of the International Conference on Learning Analytics and Knowledge, LAK 2019, pp. 51–60 (2019)Google Scholar
  11. 11.
    Echeverria, V.: Designing and validating automated feed-back for collocated teams using multimodal learning analytics. Ph.D. in Learning Analytics, University of Technology Sydney (UTS), Sydney, Australia (in preparation)Google Scholar
  12. 12.
    Echeverria, V., Martinez-Maldonado, R., Buckingham Shum, S.: Towards collaboration translucence: giving meaning to multimodal group data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2019, p. 39 (2019)Google Scholar
  13. 13.
    Echeverria, V., Martinez-Maldonado, R., Chiluiza, K., Buckingham Shum, S.: DBCollab: automated feedback for face-to-face group database design. In: Proceedings of the International Conference on Computers in Education, ICCE 2017, pp. 156–165 (2017)Google Scholar
  14. 14.
    Echeverria, V., Martinez-Maldonado, R., Power, T., Hayes, C., Buckingham Shum, S.: Where is the nurse? Towards automatically visualising meaningful team movement in healthcare education. In: Penstein Rosé, C., et al. (eds.) AIED 2018. LNCS (LNAI), vol. 10948, pp. 74–78. Springer, Cham (2018). Scholar
  15. 15.
    Gašević, D., Dawson, S., Siemens, G.: Let’s not forget: learning analytics are about learning. TechTrends 59(1), 64–71 (2015)CrossRefGoogle Scholar
  16. 16.
    Gašević, D., Joksimović, S., Eagan, B.R., Shaffer, D.W.: SENS: network analytics to combine social and cognitive perspectives of collaborative learning. Comput. Hum. Behav. 92, 562–577 (2018)CrossRefGoogle Scholar
  17. 17.
    Gibson, A., Martinez-Maldonado, R.: That dashboard looks nice, but what does it mean?: towards making meaning explicit in learning analytics design. In: Proceedings of the Australian Conference on Computer-Human Interaction, OzCHI 2017, pp. 528–532 (2017)Google Scholar
  18. 18.
    Green, A., Stawicki, S.P., Firstenberg, M.S.: Medical error and associated harm-the the critical role of team communication and coordination. In: Vignettes in Patient Safety, pp. 1–13. IntechOpen, London (2018)Google Scholar
  19. 19.
    Herder, T., et al.: Supporting teachers’ intervention in students’ virtual collaboration using a network based model. In: Proceedings of the International Conference on Learning Analytics and Knowledge, LAK 2018, pp. 21–25 (2018)Google Scholar
  20. 20.
    Jivet, I., Scheffel, M., Specht, M., Drachsler, H.: License to evaluate: preparing learning analytics dashboards for educational practice. In: Proceedings of the International Learning Analytics and Knowledge Conference, LAK 2018, pp. 31–40 (2018)Google Scholar
  21. 21.
    Martinez-Maldonado, R.: I spent more time with that team: making spatial pedagogy visible using positioning sensors. In: Proceedings of the International Conference on Learning Analytics & Knowledge, LAK 2019, pp. 21–25 (2019)Google Scholar
  22. 22.
    Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., Yacef, K.: Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop. Int. J. Comput.-Support. Collab. Learn. 8(4), 455–485 (2013)CrossRefGoogle Scholar
  23. 23.
    Martinez-Maldonado, R., Echeverria, V., Santos, O.C., Dos Santos, A.D.P., Yacef, K.: Physical learning analytics: a multimodal perspective. In: Proceedings of the International Conference on Learning Analytics and Knowledge, LAK 2018, pp. 375–379 (2018)Google Scholar
  24. 24.
    Martinez-Maldonado, R., Goodyear, P., Kay, J., Thompson, K., Carvalho, L.: An actionable approach to understand group experience in complex, multi-surface spaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 2062–2074 (2016)Google Scholar
  25. 25.
    Matcha, W., Gasevic, D., Pardo, A.: A systematic review of empirical studies on learning analytics dashboards: a self-regulated learning perspective. IEEE Trans. Learn. Technol. (2019, in press)Google Scholar
  26. 26.
    Mislevy, R.J., Behrens, J.T., Dicerbo, K.E., Levy, R.: Design and discovery in educational assessment: evidence-centered design, psychometrics, and educational data mining. J. Educ. Data Mining 4(1), 11–48 (2012)Google Scholar
  27. 27.
    Müller, L., Rivera-Pelayo, V., Kunzmann, C., Schmidt, A.: From stress awareness to coping strategies of medical staff: supporting reflection on physiological data. In: Salah, A.A., Lepri, B. (eds.) HBU 2011. LNCS, vol. 7065, pp. 93–103. Springer, Heidelberg (2011). Scholar
  28. 28.
    Ochoa, X.: Multimodal learning analytics. In: The Handbook of Learning Analytics, pp. 129–141. SOLAR, Alberta (2017)CrossRefGoogle Scholar
  29. 29.
    Ochoa, X., Chiluiza, K., Granda, R., Falcones, G., Castells, J., Guamán, B.: Multimodal transcript of face-to-face group-work activity around interactive tabletops. In: Proceedings of the CROSS-MMLA Workshop on Multimodal Learning Analytics Across Spaces, pp. 1–6 (2018)Google Scholar
  30. 30.
    Perera, D., Kay, J., Koprinska, I., Yacef, K., Zaïane, O.R.: Clustering and sequential pattern mining of online collaborative learning data. IEEE Trans. Knowl. Data Eng. 21(6), 759–772 (2008)CrossRefGoogle Scholar
  31. 31.
    Prieto-Alvarez, C., Martinez-Maldonado, R., Shum, S.B.: Mapping learner-data journeys: evolution of a visual co-design tool. In: Proceedings of the ACM Australian Computer-Human Interaction Conference, OzCHI 2018, pp. 205–214 (2018)Google Scholar
  32. 32.
    Prieto-Alvarez, C.G., Martinez-Maldonado, R., Anderson, T.D.: Co-designing learning analytics tools with learners. In: Learning Analytics in the Classroom: Translating Learning Analytics for Teachers, pp. 93–110. Routledge, London (2018)Google Scholar
  33. 33.
    Sarter, N.B.: Multimodal information presentation: design guidance and research challenges. Int. J. Ind. Ergon. 36(5), 439–445 (2006)CrossRefGoogle Scholar
  34. 34.
    Scherr, R.E., Hammer, D.: Student behavior and epistemological framing: examples from collaborative active-learning activities in physics. Cogn. Instr. 27(2), 147–174 (2009)CrossRefGoogle Scholar
  35. 35.
    Shaffer, D.W.: Epistemic frames for epistemic games. Comput. Educ. 46(3), 223–234 (2006)CrossRefGoogle Scholar
  36. 36.
    Shaffer, D.W.: Quantitative Ethnography. Cathcart Press, Madison (2017)Google Scholar
  37. 37.
    Shute, V.J., Ventura, M.: Stealth Assessment: Measuring and Supporting Learning in Video Games. MIT Press, Cambridge (2013)CrossRefGoogle Scholar
  38. 38.
    Taylor, S., Jaques, N., Chen, W., Fedor, S., Sano, A., Picard, R.: Automatic identification of artifacts in electrodermal activity data. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC 2015, pp. 1934–1937 (2015)Google Scholar
  39. 39.
    Wise, A., Knight, S., Buckingham Shum, S.: Collaborative learning analytics. In: Cress, U., Rosé, C., Wise, A., Oshima, J. (eds.) International Handbook of Computer-Supported Collaborative Learning. Springer, Cham (in press)Google Scholar
  40. 40.
    Worsley, M., Blikstein, P.: A multimodal analysis of making. Int. J. Artif. Intell. Educ. 28(3), 385–419 (2018)CrossRefGoogle Scholar
  41. 41.
    Zhang, Z., Sarcevic, A.: Constructing awareness through speech, gesture, gaze and movement during a time-critical medical task. In: Boulus-Rødje, N., Ellingsen, G., Bratteteig, T., Aanestad, M., Bjørn, P. (eds.) ECSCW 2015: Proceedings of the 14th European Conference on Computer Supported Cooperative Work, 19-23 September 2015, Oslo, Norway, pp. 163–182. Springer, Cham (2015). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Simon Buckingham Shum
    • 1
    Email author
  • Vanessa Echeverria
    • 1
    • 2
  • Roberto Martinez-Maldonado
    • 1
  1. 1.Connected Intelligence CentreUniversity of Technology SydneySydneyAustralia
  2. 2.Escuela Superior Politécnica del Litoral, ESPOLGuayaquilEcuador

Personalised recommendations