Video-based learners’ observed attention estimates for lecture learning gain evaluation


A significant problem in the field of higher education is maintaining learners’ attention during lectures, which is known to significantly affect their learning outcomes. Attention management is commonly associated with the individual ability of a lecturer to track and respond to the common behaviour of an auditorium; it lacks a detailed estimation of the intra-variability of individual learners’ attention during the course of the lecture. This paper suggests an objective and non-intrusive evaluation of learners’ attention against learning outcomes by introducing an observed attention estimate (OAE). The procedure uses human annotations based on visual cues with a supporting video recording/playback system and a web-based annotation system. This proposed procedure enables us to estimate the attention level of individual learners as observed by human annotators for given time intervals associated with specific concepts covered by the lecture. As part of the procedure, we use an inventory-based lecture gain evaluation and representation based on a novel learning gain matrix. This procedure allows for a detailed analysis of the lecture time flow with regard to learners’ attention. We have verified the applicability of the procedure on a small-scale case study.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  1. 1.

    Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. ERIC. doi:

  2. 2.

    Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose---application in an e-learning environment. Multimed Tools Appl 41:469–493.

    Article  Google Scholar 

  3. 3.

    Bao L (2006) Theoretical comparisons of average normalized gain calculations. Am J Phys 74:917–922.

    Article  Google Scholar 

  4. 4.

    Bligh, D. A. (1998). What's the Use of Lectures? Intellect (UK) doi:

  5. 5.

    Cain J, Black EP, Rohr J (2009) An Audience Response System Strategy to Improve Student Motivation, Attention, and Feedback. Am J Pharm Educ 73:21.

    Article  Google Scholar 

  6. 6.

    Chen C-M, Wang J-Y, Yu C-M (2015) Assessing the Attention Levels of Students by Using a Novel Attention Aware System based on Brainwave Signals. (T. Matsuo, K. Hashimoto, T. Mine, & S. Hirokawa, Eds.) Br J Educ Technol:379–384.

  7. 7.

    D'Mello SK (2016) On the influence of an iterative affect annotation approach on interobserver and self-observer reliability. IEEE Trans Affect Comput 7(2):136–149

    Article  Google Scholar 

  8. 8.

    von Eye A, Mun EY (2004) Analyzing rater agreement: Manifest variable methods. Lawrence Erlbaum Associates Publishers. doi:

  9. 9.

    Giannopoulos I, Schöning J, Krüger A, Raubal M (2016) Attention as an input modality for Post-WIMP interfaces using the viGaze eye tracking framework. Multimed Tools Appl 75:2913–2929.

    Article  Google Scholar 

  10. 10.

    Gibbs G (2011) Student attention over an hour. Accessed 27 February 2017

  11. 11.

    Graesser A, McDaniel B, Chipman P, Witherspoon A, D'Mello SK, Gholson B (2006) Detection of emotions during learning with AutoTutor. In: Sun R, Miyake N (eds) Proceedings of the 28th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Austin, pp 285–290

    Google Scholar 

  12. 12.

    Hake RR (1998) Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Am J Phys 66:64–74

    Article  Google Scholar 

  13. 13.

    Hallgren KA (2012) Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol 8:23–34. 10.20982/tqmp.08.1.p023

    Article  Google Scholar 

  14. 14.

    Halloun IA, Hestenes D (1985) Common sense concepts about motion. Am J Phys 53:1056–1065.

    Article  Google Scholar 

  15. 15.

    Hussey T, Smith P (2003) The Uses of Learning Outcomes. Teach High Educ 8:357–368.

    Article  Google Scholar 

  16. 16.

    Johnstone AH, Percival F (1976) Attention Breaks in Lectures. Educ Chem 13:49–50

    Google Scholar 

  17. 17.

    Kondermann, D. (2013). Ground Truth Design Principles: An Overview. Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications (pp. 5:1--5:4). New York: Association for Computing Machinery (ACM). doi:

  18. 18.

    Krippendorff K (2004) Content Analysis: An Introduction to Its Methodology. 2nd Ed. SAGE Publications, London pp 413

  19. 19.

    Malle BF, Pearce GE (2001) Attention to behavioral events during interaction: Two actor-observer gaps and three attempts to close them. J Pers Soc Psychol 81(2):278–294

    Article  Google Scholar 

  20. 20.

    Martinez HP, Yannakakis GN, Hallam J (2014) Don't Classify Ratings of Affect; Rank Them. IEEE Trans Affect Comput 5:314–326.

    Article  Google Scholar 

  21. 21.

    Marx JD, Cummings K (2007) Normalized change. Am J Phys 75:87–91.

    Article  Google Scholar 

  22. 22.

    Matheson C (2008) The educational value and effectiveness of lectures. Clin Teach 5:218–221.

    Article  Google Scholar 

  23. 23.

    Wilcox RR (2003) Applying Contemporary Statistical Techniques. Academic Press, London, pp 608

  24. 24.

    Perrenet JC, Bouhuijs PA, Smits JG (2000) The Suitability of Problem-based Learning for Engineering Education: Theory and practice. Teach High Educ 5:345–358.

    Article  Google Scholar 

  25. 25.

    Porta, M., Ricotti, S., & Perez, C. J. (2012). Emotional e-learning through eye tracking. Proceedings of the 2012 I.E. Global Engineering Education Conference (EDUCON). Institute of Electrical and Electronics Engineers (IEEE). doi:

  26. 26.

    Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010) Learning From Crowds. J Mach Learn Res 11:1297–1322

    MathSciNet  Google Scholar 

  27. 27.

    Ried LD (2011) A Model for Curricular Quality Assessment and Improvement. Am J Pharm Educ 75:196.

    Article  Google Scholar 

  28. 28.

    Risko EF, Anderson N, Sarwal A, Engelhardt M, Kingstone A (2012) Everyday Attention: Variation in Mind Wandering and Memory in a Lecture. Appl Cogn Psychol 26:234–242.

    Article  Google Scholar 

  29. 29.

    Ruhl KL, Hughes CA, Schloss PJ (1987) Using the pause procedure to enhance lecture recall. TESE: J TED CEC 10:14–18.

    Google Scholar 

  30. 30.

    Russell BC, Torralba A, Murphy KP, Freeman WT (2007) LabelMe: A Database and Web-Based Tool for Image Annotation. Int J Comput Vis 77:157–173.

    Article  Google Scholar 

  31. 31.

    Tkalčič M, Odić A, Košir A (2013) The impact of weak ground truth and facial expressiveness on affect detection accuracy from time-continuous videos of facial expressions. Inf Sci 249:13–23.

    Article  Google Scholar 

  32. 32.

    Wage KE, Buck JR, Wright CH, Welch TB (2005) The Signals and Systems Concept Inventory. IEEE Trans Educ 48:448–461.

    Article  Google Scholar 

  33. 33.

    Wilson K, Korn JH (2007) Attention During Lectures: Beyond Ten Minutes. Teach Psychol 34:85–89.

    Article  Google Scholar 

  34. 34.

    Yannakakis, G. N., & Martinez, H. P. (2015). Grounding truth via ordinal annotation. 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 574-580). Institute of Electrical and Electronics Engineers (IEEE). doi:

  35. 35.

    Young MS, Robinson S, Alberts P (2009) Students pay attention!: Combating the vigilance decrement to improve learning during lectures. Act Learn High Educ 10:41–55.

    Article  Google Scholar 

Download references


The authors would like to thank the students in the digital signal processing course at the University of Ljubljana for taking part in the study. Special thanks goes to John R. Buck and Kathleen E. Wage for giving us access to the SSCI used in this study.


This study was partially funded by Slovenian Research Agency (P2–0246 B).

Author information



Corresponding author

Correspondence to Urban Burnik.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Burnik, U., Zaletelj, J. & Košir, A. Video-based learners’ observed attention estimates for lecture learning gain evaluation. Multimed Tools Appl 77, 16903–16926 (2018).

Download citation


  • Attention metering
  • Observed attention estimate
  • Multimedia annotation system
  • Lecture efficiency
  • Learning management
  • Learning outcomes
  • Learning gain
  • Teaching advisory systems