Skip to main content
SpringerLink
Log in

Engagement Detection with Multi-Task Training in E-Learning Environments

  • Conference paper
  • First Online:
Image Analysis and Processing – ICIAP 2022 (ICIAP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13233))

Included in the following conference series:

Abstract

Recognition of user interaction, in particular engagement detection, became highly crucial for online working and learning environments, especially during the COVID-19 outbreak. Such recognition and detection systems significantly improve the user experience and efficiency by providing valuable feedback. In this paper, we propose a novel Engagement Detection with Multi-Task Training (ED-MTT) system which minimizes mean squared error and triplet loss together to determine the engagement level of students in an e-learning environment. The performance of this system is evaluated and compared against the state-of-the-art on a publicly available dataset as well as videos collected from real-life scenarios. The results show that ED-MTT achieves \(6\%\) lower MSE than the best state-of-the-art performance with highly acceptable training time and lightweight feature extraction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Code and pretrained model are available at https://github.com/CopurOnur/ED-MTT.

  2. 2.

    Note that the size of the hidden state is constant across all Bi-LSTM layers.

References

  1. Abedi, A., Khan, S.: Affect-driven engagement measurement from videos. arXiv preprint arXiv:2106.10882 (2021)

  2. Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.P.: OpenFace 2.0: facial behavior analysis toolkit. In: 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 59–66 (2018)

    Google Scholar 

  3. Booth, B.M., Ali, A.M., Narayanan, S.S., Bennett, I., Farag, A.A.: Toward active and unobtrusive engagement assessment of distance learners. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 470–476 (2017)

    Google Scholar 

  4. Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997)

    Article  MathSciNet  Google Scholar 

  5. Chang, C., Zhang, C., Chen, L., Liu, Y.: An ensemble model using face and body tracking for engagement detection. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 616–622 (2018)

    Google Scholar 

  6. Dewan, M.A.A., Lin, F., Wen, D., Murshed, M., Uddin, Z.: A deep learning approach to detecting engagement of online learners. In: 2018 IEEE SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI, pp. 1895–1902 (2018)

    Google Scholar 

  7. Dhall, A., Sharma, G., Goecke, R., Gedeon, T.: EmotiW 2020: driver gaze, group emotion, student engagement and physiological signal based challenges. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 784–789 (2020)

    Google Scholar 

  8. D’Mello, S.K., Craig, S.D., Graesser, A.C.: Multimethod assessment of affective experience and expression during deep learning. Int. J. Learn. Technol. 4(3–4), 165–187 (2009)

    Article  Google Scholar 

  9. Dong, X., Shen, J.: Triplet loss in siamese network for object tracking. In: Proceedings of the European Conference on Computer Vision (ECCV), September 2018

    Google Scholar 

  10. Grafsgaard, J., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.: Automatically recognizing facial expression: predicting engagement and frustration. In: Educational Data Mining 2013 (2013)

    Google Scholar 

  11. Gupta, A., D’Cunha, A., Awasthi, K., Balasubramanian, V.: DAiSEE: towards user engagement recognition in the wild. arXiv preprint arXiv:1609.01885 (2016)

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  13. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  14. Huang, T., Mei, Y., Zhang, H., Liu, S., Yang, H.: Fine-grained engagement recognition in online learning environment. In: 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), pp. 338–341 (2019)

    Google Scholar 

  15. Kamath, S., Singhal, P., Jeevan, G., Annappa, B.: Engagement analysis of students in online learning environments. In: Misra, R., Shyamasundar, R.K., Chaturvedi, A., Omer, R. (eds.) ICMLBDA 2021. LNNS, vol. 256, pp. 34–47. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-82469-3_4

    Chapter  Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  17. Lemaître, G., Nogueira, F., Aridas, C.K.: Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. J. Mach. Learn. Res. 18(17), 1–5 (2017)

    Google Scholar 

  18. Liao, J., Liang, Y., Pan, J.: Deep facial spatiotemporal network for engagement prediction in online learning. Appl. Intell. 51, 1–13 (2021)

    Article  Google Scholar 

  19. Murshed, M., Dewan, M.A.A., Lin, F., Wen, D.: Engagement detection in e-learning environments using convolutional neural networks. In: 2019 IEEE (DASC/PiCom/CBDCom/CyberSciTech), pp. 80–86. IEEE (2019)

    Google Scholar 

  20. Niu, X., et al.: Automatic engagement prediction with gap feature. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 599–603 (2018)

    Google Scholar 

  21. Rao, K.P., Rao, M.C.S.: Recognition of learners’ cognitive states using facial expressions in e-learning environments. J. Univ. Shanghai Sci. Technol. 93–103 (2020)

    Google Scholar 

  22. Roy, S., Etemad, A.: Self-supervised contrastive learning of multi-view facial expressions. In: Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 253–257 (2021)

    Google Scholar 

  23. Thiruthuvanathan, M., Krishnan, B., Rangaswamy, M.A.D.: Engagement detection through facial emotional recognition using a shallow residual convolutional neural networks. Int. J. Intell. Eng. Syst. 14, 236–247 (2021)

    Google Scholar 

  24. Thomas, C., Nair, N., Jayagopi, D.B.: Predicting engagement intensity in the wild using temporal convolutional network. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 604–610 (2018)

    Google Scholar 

  25. Thong Huynh, V., Kim, S.H., Lee, G.S., Yang, H.J.: Engagement intensity prediction withfacial behavior features. In: 2019 International Conference on Multimodal Interaction, pp. 567–571 (2019)

    Google Scholar 

  26. Balntas, V., Riba, E., Ponsa, D., Mikolajczyk, K.: Learning local feature descriptors with triplets and shallow convolutional neural networks. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 119.1–119.11. BMVA Press (2016)

    Google Scholar 

  27. Wang, K., Yang, J., Guo, D., Zhang, K., Peng, X., Qiao, Y.: Bootstrap model ensemble and rank loss for engagement intensity regression. In: 2019 International Conference on Multimodal Interaction, pp. 551–556 (2019)

    Google Scholar 

  28. Whitehill, J., Serpell, Z., Lin, Y., Foster, A., Movellan, J.R.: The faces of engagement: automatic recognition of student engagementfrom facial expressions. IEEE Trans. Affect. Comput. 5(1), 86–98 (2014)

    Article  Google Scholar 

  29. Wu, J., Yang, B., Wang, Y., Hattori, G.: Advanced multi-instance learning method with multi-features engineering and conservative optimization for engagement intensity prediction. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 777–783 (2020)

    Google Scholar 

  30. Wu, J., Zhou, Z., Wang, Y., Li, Y., Xu, X., Uchida, Y.: Multi-feature and multi-instance learning with anti-overfitting strategy for engagement intensity prediction. In: 2019 International Conference on Multimodal Interaction, pp. 582–588 (2019)

    Google Scholar 

  31. Yang, J., Wang, K., Peng, X., Qiao, Y.: Deep recurrent multi-instance learning with spatio-temporal features for engagement intensity prediction. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 594–598 (2018)

    Google Scholar 

  32. Zhu, B., Lan, X., Guo, X., Barner, K.E., Boncelet, C.: Multi-rate attention based GRU model for engagement prediction. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 841–848 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Sapienza University of Rome, 00185, Rome, Italy

    Onur Copur & Simone Scardapane

  2. Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, 44-100, Gliwice, Poland

    Mert Nakıp

  3. Barco NV, 8500, Kortrijk, Belgium

    Jürgen Slowack

Authors
  1. Onur Copur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mert Nakıp
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Simone Scardapane
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jürgen Slowack
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Onur Copur .

Editor information

Editors and Affiliations

  1. Boston University, Boston, MA, USA

    Stan Sclaroff

  2. National Research Council, Lecce, Italy

    Cosimo Distante

  3. National Research Council, Lecce, Italy

    Marco Leo

  4. University of Catania, Catania, Italy

    Giovanni M. Farinella

  5. Technische Universität München, Garching, Germany

    Federico Tombari

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Copur, O., Nakıp, M., Scardapane, S., Slowack, J. (2022). Engagement Detection with Multi-Task Training in E-Learning Environments. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13233. Springer, Cham. https://doi.org/10.1007/978-3-031-06433-3_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06433-3_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06432-6

  • Online ISBN: 978-3-031-06433-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation