Multimedia Tools and Applications

, Volume 77, Issue 13, pp 16903–16926 | Cite as

Video-based learners’ observed attention estimates for lecture learning gain evaluation

  • Urban BurnikEmail author
  • Janez Zaletelj
  • Andrej Košir


A significant problem in the field of higher education is maintaining learners’ attention during lectures, which is known to significantly affect their learning outcomes. Attention management is commonly associated with the individual ability of a lecturer to track and respond to the common behaviour of an auditorium; it lacks a detailed estimation of the intra-variability of individual learners’ attention during the course of the lecture. This paper suggests an objective and non-intrusive evaluation of learners’ attention against learning outcomes by introducing an observed attention estimate (OAE). The procedure uses human annotations based on visual cues with a supporting video recording/playback system and a web-based annotation system. This proposed procedure enables us to estimate the attention level of individual learners as observed by human annotators for given time intervals associated with specific concepts covered by the lecture. As part of the procedure, we use an inventory-based lecture gain evaluation and representation based on a novel learning gain matrix. This procedure allows for a detailed analysis of the lecture time flow with regard to learners’ attention. We have verified the applicability of the procedure on a small-scale case study.


Attention metering Observed attention estimate Multimedia annotation system Lecture efficiency Learning management Learning outcomes Learning gain Teaching advisory systems 



The authors would like to thank the students in the digital signal processing course at the University of Ljubljana for taking part in the study. Special thanks goes to John R. Buck and Kathleen E. Wage for giving us access to the SSCI used in this study.


This study was partially funded by Slovenian Research Agency (P2–0246 B).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. ERIC. doi:
  2. 2.
    Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose---application in an e-learning environment. Multimed Tools Appl 41:469–493. CrossRefGoogle Scholar
  3. 3.
    Bao L (2006) Theoretical comparisons of average normalized gain calculations. Am J Phys 74:917–922. CrossRefGoogle Scholar
  4. 4.
    Bligh, D. A. (1998). What's the Use of Lectures? Intellect (UK) doi:
  5. 5.
    Cain J, Black EP, Rohr J (2009) An Audience Response System Strategy to Improve Student Motivation, Attention, and Feedback. Am J Pharm Educ 73:21. CrossRefGoogle Scholar
  6. 6.
    Chen C-M, Wang J-Y, Yu C-M (2015) Assessing the Attention Levels of Students by Using a Novel Attention Aware System based on Brainwave Signals. (T. Matsuo, K. Hashimoto, T. Mine, & S. Hirokawa, Eds.) Br J Educ Technol:379–384.
  7. 7.
    D'Mello SK (2016) On the influence of an iterative affect annotation approach on interobserver and self-observer reliability. IEEE Trans Affect Comput 7(2):136–149CrossRefGoogle Scholar
  8. 8.
    von Eye A, Mun EY (2004) Analyzing rater agreement: Manifest variable methods. Lawrence Erlbaum Associates Publishers. doi:
  9. 9.
    Giannopoulos I, Schöning J, Krüger A, Raubal M (2016) Attention as an input modality for Post-WIMP interfaces using the viGaze eye tracking framework. Multimed Tools Appl 75:2913–2929. CrossRefGoogle Scholar
  10. 10.
    Gibbs G (2011) Student attention over an hour. Accessed 27 February 2017
  11. 11.
    Graesser A, McDaniel B, Chipman P, Witherspoon A, D'Mello SK, Gholson B (2006) Detection of emotions during learning with AutoTutor. In: Sun R, Miyake N (eds) Proceedings of the 28th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Austin, pp 285–290Google Scholar
  12. 12.
    Hake RR (1998) Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Am J Phys 66:64–74CrossRefGoogle Scholar
  13. 13.
    Hallgren KA (2012) Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol 8:23–34.  10.20982/tqmp.08.1.p023 CrossRefGoogle Scholar
  14. 14.
    Halloun IA, Hestenes D (1985) Common sense concepts about motion. Am J Phys 53:1056–1065. CrossRefGoogle Scholar
  15. 15.
    Hussey T, Smith P (2003) The Uses of Learning Outcomes. Teach High Educ 8:357–368. CrossRefGoogle Scholar
  16. 16.
    Johnstone AH, Percival F (1976) Attention Breaks in Lectures. Educ Chem 13:49–50Google Scholar
  17. 17.
    Kondermann, D. (2013). Ground Truth Design Principles: An Overview. Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications (pp. 5:1--5:4). New York: Association for Computing Machinery (ACM). doi:
  18. 18.
    Krippendorff K (2004) Content Analysis: An Introduction to Its Methodology. 2nd Ed. SAGE Publications, London pp 413Google Scholar
  19. 19.
    Malle BF, Pearce GE (2001) Attention to behavioral events during interaction: Two actor-observer gaps and three attempts to close them. J Pers Soc Psychol 81(2):278–294CrossRefGoogle Scholar
  20. 20.
    Martinez HP, Yannakakis GN, Hallam J (2014) Don't Classify Ratings of Affect; Rank Them. IEEE Trans Affect Comput 5:314–326. CrossRefGoogle Scholar
  21. 21.
    Marx JD, Cummings K (2007) Normalized change. Am J Phys 75:87–91. CrossRefGoogle Scholar
  22. 22.
    Matheson C (2008) The educational value and effectiveness of lectures. Clin Teach 5:218–221. CrossRefGoogle Scholar
  23. 23.
    Wilcox RR (2003) Applying Contemporary Statistical Techniques. Academic Press, London, pp 608Google Scholar
  24. 24.
    Perrenet JC, Bouhuijs PA, Smits JG (2000) The Suitability of Problem-based Learning for Engineering Education: Theory and practice. Teach High Educ 5:345–358. CrossRefGoogle Scholar
  25. 25.
    Porta, M., Ricotti, S., & Perez, C. J. (2012). Emotional e-learning through eye tracking. Proceedings of the 2012 I.E. Global Engineering Education Conference (EDUCON). Institute of Electrical and Electronics Engineers (IEEE). doi:
  26. 26.
    Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010) Learning From Crowds. J Mach Learn Res 11:1297–1322MathSciNetGoogle Scholar
  27. 27.
    Ried LD (2011) A Model for Curricular Quality Assessment and Improvement. Am J Pharm Educ 75:196. CrossRefGoogle Scholar
  28. 28.
    Risko EF, Anderson N, Sarwal A, Engelhardt M, Kingstone A (2012) Everyday Attention: Variation in Mind Wandering and Memory in a Lecture. Appl Cogn Psychol 26:234–242. CrossRefGoogle Scholar
  29. 29.
    Ruhl KL, Hughes CA, Schloss PJ (1987) Using the pause procedure to enhance lecture recall. TESE: J TED CEC 10:14–18. Google Scholar
  30. 30.
    Russell BC, Torralba A, Murphy KP, Freeman WT (2007) LabelMe: A Database and Web-Based Tool for Image Annotation. Int J Comput Vis 77:157–173. CrossRefGoogle Scholar
  31. 31.
    Tkalčič M, Odić A, Košir A (2013) The impact of weak ground truth and facial expressiveness on affect detection accuracy from time-continuous videos of facial expressions. Inf Sci 249:13–23. CrossRefGoogle Scholar
  32. 32.
    Wage KE, Buck JR, Wright CH, Welch TB (2005) The Signals and Systems Concept Inventory. IEEE Trans Educ 48:448–461. CrossRefGoogle Scholar
  33. 33.
    Wilson K, Korn JH (2007) Attention During Lectures: Beyond Ten Minutes. Teach Psychol 34:85–89. CrossRefGoogle Scholar
  34. 34.
    Yannakakis, G. N., & Martinez, H. P. (2015). Grounding truth via ordinal annotation. 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 574-580). Institute of Electrical and Electronics Engineers (IEEE). doi:
  35. 35.
    Young MS, Robinson S, Alberts P (2009) Students pay attention!: Combating the vigilance decrement to improve learning during lectures. Act Learn High Educ 10:41–55. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Faculty of Electrical EngineeringUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations