A significant problem in the field of higher education is maintaining learners’ attention during lectures, which is known to significantly affect their learning outcomes. Attention management is commonly associated with the individual ability of a lecturer to track and respond to the common behaviour of an auditorium; it lacks a detailed estimation of the intra-variability of individual learners’ attention during the course of the lecture. This paper suggests an objective and non-intrusive evaluation of learners’ attention against learning outcomes by introducing an observed attention estimate (OAE). The procedure uses human annotations based on visual cues with a supporting video recording/playback system and a web-based annotation system. This proposed procedure enables us to estimate the attention level of individual learners as observed by human annotators for given time intervals associated with specific concepts covered by the lecture. As part of the procedure, we use an inventory-based lecture gain evaluation and representation based on a novel learning gain matrix. This procedure allows for a detailed analysis of the lecture time flow with regard to learners’ attention. We have verified the applicability of the procedure on a small-scale case study.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. ERIC. doi:https://doi.org/10.2307/2943957
Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose---application in an e-learning environment. Multimed Tools Appl 41:469–493. https://doi.org/10.1007/s11042-008-0240-1
Bao L (2006) Theoretical comparisons of average normalized gain calculations. Am J Phys 74:917–922. https://doi.org/10.1119/1.2213632
Bligh, D. A. (1998). What's the Use of Lectures? Intellect (UK) doi:https://doi.org/10.1080/03098268508708932
Cain J, Black EP, Rohr J (2009) An Audience Response System Strategy to Improve Student Motivation, Attention, and Feedback. Am J Pharm Educ 73:21. https://doi.org/10.5688/aj730221
Chen C-M, Wang J-Y, Yu C-M (2015) Assessing the Attention Levels of Students by Using a Novel Attention Aware System based on Brainwave Signals. (T. Matsuo, K. Hashimoto, T. Mine, & S. Hirokawa, Eds.) Br J Educ Technol:379–384. https://doi.org/10.1109/IIAI-AAI.2015.224
D'Mello SK (2016) On the influence of an iterative affect annotation approach on interobserver and self-observer reliability. IEEE Trans Affect Comput 7(2):136–149
von Eye A, Mun EY (2004) Analyzing rater agreement: Manifest variable methods. Lawrence Erlbaum Associates Publishers. doi:https://doi.org/10.4324/9781410611024
Giannopoulos I, Schöning J, Krüger A, Raubal M (2016) Attention as an input modality for Post-WIMP interfaces using the viGaze eye tracking framework. Multimed Tools Appl 75:2913–2929. https://doi.org/10.1007/s11042-014-2412-5
Gibbs G (2011) Student attention over an hour. https://epigeum.com/downloads/uct_accessible/uk/01_lecturing1/html/course_files/2_30.html. Accessed 27 February 2017
Graesser A, McDaniel B, Chipman P, Witherspoon A, D'Mello SK, Gholson B (2006) Detection of emotions during learning with AutoTutor. In: Sun R, Miyake N (eds) Proceedings of the 28th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Austin, pp 285–290
Hake RR (1998) Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Am J Phys 66:64–74
Hallgren KA (2012) Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol 8:23–34. 10.20982/tqmp.08.1.p023
Halloun IA, Hestenes D (1985) Common sense concepts about motion. Am J Phys 53:1056–1065. https://doi.org/10.1119/1.14031
Hussey T, Smith P (2003) The Uses of Learning Outcomes. Teach High Educ 8:357–368. https://doi.org/10.1080/13562510309399
Johnstone AH, Percival F (1976) Attention Breaks in Lectures. Educ Chem 13:49–50
Kondermann, D. (2013). Ground Truth Design Principles: An Overview. Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications (pp. 5:1--5:4). New York: Association for Computing Machinery (ACM). doi:https://doi.org/10.1145/2501105.2501114
Krippendorff K (2004) Content Analysis: An Introduction to Its Methodology. 2nd Ed. SAGE Publications, London pp 413
Malle BF, Pearce GE (2001) Attention to behavioral events during interaction: Two actor-observer gaps and three attempts to close them. J Pers Soc Psychol 81(2):278–294
Martinez HP, Yannakakis GN, Hallam J (2014) Don't Classify Ratings of Affect; Rank Them. IEEE Trans Affect Comput 5:314–326. https://doi.org/10.1109/taffc.2014.2352268
Marx JD, Cummings K (2007) Normalized change. Am J Phys 75:87–91. https://doi.org/10.1119/1.2372468
Matheson C (2008) The educational value and effectiveness of lectures. Clin Teach 5:218–221. https://doi.org/10.1111/j.1743-498X.2008.00238.x
Wilcox RR (2003) Applying Contemporary Statistical Techniques. Academic Press, London, pp 608
Perrenet JC, Bouhuijs PA, Smits JG (2000) The Suitability of Problem-based Learning for Engineering Education: Theory and practice. Teach High Educ 5:345–358. https://doi.org/10.1080/713699144
Porta, M., Ricotti, S., & Perez, C. J. (2012). Emotional e-learning through eye tracking. Proceedings of the 2012 I.E. Global Engineering Education Conference (EDUCON). Institute of Electrical and Electronics Engineers (IEEE). doi:https://doi.org/10.1109/educon.2012.6201145
Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010) Learning From Crowds. J Mach Learn Res 11:1297–1322
Ried LD (2011) A Model for Curricular Quality Assessment and Improvement. Am J Pharm Educ 75:196. https://doi.org/10.5688/ajpe7510196
Risko EF, Anderson N, Sarwal A, Engelhardt M, Kingstone A (2012) Everyday Attention: Variation in Mind Wandering and Memory in a Lecture. Appl Cogn Psychol 26:234–242. https://doi.org/10.1002/acp.1814
Ruhl KL, Hughes CA, Schloss PJ (1987) Using the pause procedure to enhance lecture recall. TESE: J TED CEC 10:14–18. https://doi.org/10.1177/088840648701000103
Russell BC, Torralba A, Murphy KP, Freeman WT (2007) LabelMe: A Database and Web-Based Tool for Image Annotation. Int J Comput Vis 77:157–173. https://doi.org/10.1007/s11263-007-0090-8
Tkalčič M, Odić A, Košir A (2013) The impact of weak ground truth and facial expressiveness on affect detection accuracy from time-continuous videos of facial expressions. Inf Sci 249:13–23. https://doi.org/10.1016/j.ins.2013.06.006
Wage KE, Buck JR, Wright CH, Welch TB (2005) The Signals and Systems Concept Inventory. IEEE Trans Educ 48:448–461. https://doi.org/10.1109/te.2005.849746
Wilson K, Korn JH (2007) Attention During Lectures: Beyond Ten Minutes. Teach Psychol 34:85–89. https://doi.org/10.1080/00986280701291291
Yannakakis, G. N., & Martinez, H. P. (2015). Grounding truth via ordinal annotation. 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 574-580). Institute of Electrical and Electronics Engineers (IEEE). doi:https://doi.org/10.1109/acii.2015.7344627
Young MS, Robinson S, Alberts P (2009) Students pay attention!: Combating the vigilance decrement to improve learning during lectures. Act Learn High Educ 10:41–55. https://doi.org/10.1177/1469787408100194
The authors would like to thank the students in the digital signal processing course at the University of Ljubljana for taking part in the study. Special thanks goes to John R. Buck and Kathleen E. Wage for giving us access to the SSCI used in this study.
This study was partially funded by Slovenian Research Agency (P2–0246 B).
Conflict of interest
The authors declare that they have no conflict of interest.
About this article
Cite this article
Burnik, U., Zaletelj, J. & Košir, A. Video-based learners’ observed attention estimates for lecture learning gain evaluation. Multimed Tools Appl 77, 16903–16926 (2018). https://doi.org/10.1007/s11042-017-5259-8
- Attention metering
- Observed attention estimate
- Multimedia annotation system
- Lecture efficiency
- Learning management
- Learning outcomes
- Learning gain
- Teaching advisory systems