An Architecture and Data Model to Process Multimodal Evidence of Learning

  • Shashi Kant ShankarEmail author
  • Adolfo Ruiz-Calleja
  • Luis P. Prieto
  • María Jesús Rodríguez-Triana
  • Pankaj Chejara
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11841)


In learning situations that do not occur exclusively online, the analysis of multimodal evidence can help multiple stakeholders to better understand the learning process and the environment where it occurs. However, Multimodal Learning Analytics (MMLA) solutions are often not directly applicable outside the specific data gathering setup and conditions they were developed for. This paper focuses specifically on authentic situations where MMLA solutions are used by multiple stakeholders (e.g., teachers and researchers). In this paper, we propose an architecture to process multimodal evidence of learning taking into account the situation’s contextual information. Our adapter-based architecture supports the preparation, organisation, and fusion of multimodal evidence, and is designed to be reusable in different learning situations. Moreover, to structure and organise such contextual information, a data model is proposed. Finally, to evaluate the architecture and the data model, we apply them to four authentic learning situations where multimodal learning data was collected collaboratively by teachers and researchers.


MMLA Architecture Data model Multimodal Learning Analytics 



This research has been partially funded by the European Union via the European Regional Development Fund and in the context of CEITER and Next-Lab (Horizon 2020 Research and Innovation Programme, grant agreements no. 669074 and 731685).


  1. 1.
    Bano, M., Zowghi, D.: User involvement in software development and system success: a systematic literature review. In: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, pp. 125–130. ACM (2013)Google Scholar
  2. 2.
    Buschmann, F., Henney, K., Schmidt, D.C.: Pattern-Oriented Software Architecture, on Patterns and Pattern Languages, vol. 5. Wiley, Hoboken (2007)Google Scholar
  3. 3.
    Chatti, M.A., et al.: Learning analytics: challenges and future research directions. eleed 10(1) (2014)Google Scholar
  4. 4.
    Chen, L., Feng, G., Leong, C.W., Joe, J., Kitchen, C., Lee, C.M.: Designing an automated assessment of public speaking skills using multimodal cues. J. Learn. Anal. 3(2), 261–281 (2016)CrossRefGoogle Scholar
  5. 5.
    Di Mitri, D., Schneider, J., Klemke, R., Specht, M., Drachsler, H.: Read between the lines: an annotation tool for multimodal data for learning. In: Proceedings of the 9th International Conference on Learning Analytics & Knowledge, pp. 51–60. ACM (2019)Google Scholar
  6. 6.
    Farrell, T., Mikroyannidis, A., Alani, H.: “We’re seeking relevance”: qualitative perspectives on the impact of learning analytics on teaching and learning. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., Pérez-Sanagustín, M. (eds.) EC-TEL 2017. LNCS, vol. 10474, pp. 397–402. Springer, Cham (2017). Scholar
  7. 7.
    Grafsgaard, J., Wiggins, J., Boyer, K.E., Wiebe, E., Lester, J.: Predicting learning and affect from multimodal data streams in task-oriented tutorial dialogue. In: Proceedings of the 7th International Conference on Educational Data Mining. Citeseer (2014)Google Scholar
  8. 8.
    Hernández-García, Á., Conde, M.Á.: Dealing with complexity: educational data and tools for learning analytics. In: Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality, pp. 263–268. ACM (2014)Google Scholar
  9. 9.
    Krafzig, D., Banke, K., Slama, D.: Enterprise SOA: Service-Oriented Architecture Best Practices. Prentice Hall Professional, Upper Saddle River (2005)Google Scholar
  10. 10.
    Miller, H.G., Mork, P.: From data to decisions: a value chain for big data. IT Prof. 15(1), 57–59 (2013). Scholar
  11. 11.
    Ochoa, X.: Multimodal learning analytics. In: Handbook of Learning Analytics, pp. 129–141 (2017)CrossRefGoogle Scholar
  12. 12.
    Ochoa, X., Worsley, M.: Augmenting learning analytics with multimodal sensory data. J. Learn. Anal. 3(2), 213–219 (2016)CrossRefGoogle Scholar
  13. 13.
    Prieto, L.P., Rodríguez-Triana, M.J., Martínez-Maldonado, R., Dimitriadis, Y., Gašević, D.: Orchestrating learning analytics (OrLA): supporting inter-stakeholder communication about adoption of learning analytics at the classroom level. Australas. J. Educ. Technol. 4(35) (2019).
  14. 14.
    Rodríguez-Triana, M.J., Prieto, L.P., Martínez-Monés, A., Asensio-Pérez, J.I., Dimitriadis, Y.: The teacher in the loop: customizing multimodal learning analytics for blended learning. In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge, pp. 417–426. ACM (2018)Google Scholar
  15. 15.
    Rodriguez Triana, M.J., et al.: Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review. Int. J. Technol. Enhanced Learn. 9(2–3), 126–150 (2017)CrossRefGoogle Scholar
  16. 16.
    Schneider, J., Börner, D., Van Rosmalen, P., Specht, M.: Augmenting the senses: a review on sensor-based learning support. Sensors 15(2), 4097–4133 (2015)CrossRefGoogle Scholar
  17. 17.
    Schneider, J., Di Mitri, D., Limbu, B., Drachsler, H.: Multimodal learning hub: a tool for capturing customizable multimodal learning experiences. In: Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H., Elferink, R., Scheffel, M. (eds.) EC-TEL 2018. LNCS, vol. 11082, pp. 45–58. Springer, Cham (2018). Scholar
  18. 18.
    Shankar, S.K., Prieto, L.P., Rodríguez-Triana, M.J., Ruiz-Calleja, A.: A review of multimodal learning analytics architectures. In: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 212–214. IEEE (2018)Google Scholar
  19. 19.
    Tarr, P., Ossher, H., Harrison, W., Sutton, S.M.: N degrees of separation: multi-dimensional separation of concerns. In: Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No. 99CB37002), pp. 107–119. IEEE (1999)Google Scholar
  20. 20.
    Worsley, M.: Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. In: Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge, pp. 1–4. ACM (2014)Google Scholar
  21. 21.
    Worsley, M., Blikstein, P.: Leveraging multimodal learning analytics to differentiate student learning strategies. In: Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, pp. 360–367. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Shashi Kant Shankar
    • 1
    Email author
  • Adolfo Ruiz-Calleja
    • 2
  • Luis P. Prieto
    • 1
  • María Jesús Rodríguez-Triana
    • 1
  • Pankaj Chejara
    • 1
  1. 1.Tallinn UniversityTallinnEstonia
  2. 2.GSIC-EMIC GroupUniversity of ValladolidValladolidSpain

Personalised recommendations