Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information

Abstract

The boom of big data in education has provided an unrivalled opportunity for educators to evaluate the learners’ cognitive state. However, most existing cognitive state analysis methods focus on attention, ignoring the roles of emotion in human learning. Therefore, this study proposes an emotion-sensitive learning cognitive state analysis framework, which automatically estimates the learners’ attention based on head pose and emotion based on facial expression in a non-invasive way. The proposed framework includes two modules. In the first module, a multi-task learning implementation with a cascaded convolutional neural network (CNN) is presented for face detection, landmark location, and head pose estimation simultaneously. The located landmarks are used to align the faces as the preprocessing step for the facial expression analysis. The estimated head pose and landmarks are used to recognize the visual focus of attention of the learner. In the second module, an expression intensity ranking CNN is proposed to recognize the facial expression and evaluate its intensity using ordinal information of the sequences. Then, the learners’ emotions are estimated based on the facial expression. Experimental results show that this method can estimate a learner’s attention and emotion with correctness rates of 79.5% and 88.6%, respectively. The results obtained suggest that the method has strong potential as an alternative method for analyzing emotion-sensitive learning cognitive state.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. 1.

    Ashby FG, Isen AM, Turken AU (1999) A neuropsychological theory of positive affect and its influence on cognition. Psychol Rev 106(3):529–550

    Article  Google Scholar 

  2. 2.

    Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose-application in an e-learning environment. Multimed Tools Appl 41(3):469–493

    Article  Google Scholar 

  3. 3.

    Batista JC, Albiero V, Bellon ORP, Silva L (2017) Aumpnet: simultaneous action units detection and intensity estimation on multipose facial images using a single convolutional neural network. In: IEEE international conference on automatic face and gesture recognition, pp 866–871

  4. 4.

    Chen D, Hu Y, Wang L, Zomaya AY, Li X (2017) H-PARAFAC: Hierarchical parallel factor analysis of multidimensional big data. IEEE Trans Parallel Distrib Syst 28(4):1091–1104

    Article  Google Scholar 

  5. 5.

    Chen J, Luo N, Liu Y, Liu L, Zhang K, Kolodziej J (2016) A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing 98(1–2):215–233

    MathSciNet  Article  Google Scholar 

  6. 6.

    Chen S, Zhang C, Dong M, Le J, Rao M (2017) Using ranking-CNN for age estimation. In: IEEE conference on computer vision and pattern recognition, pp 742–751

  7. 7.

    Chu W, Ghahramani Z (2004) Gaussian processes for ordinal regression. J Mach Learn Res 6(3):1019–1041

    MathSciNet  MATH  Google Scholar 

  8. 8.

    Dong Y, Huang C, Liu W (2014) RankCNN: when learning to rank encounters the pseudo preference feedback. Comput Stand Interfaces 36(3):554–562

    Article  Google Scholar 

  9. 9.

    Ekman P (1978) Facial action coding system. A technique for the measurement of facial action. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  10. 10.

    Fan Y, Shen D, Davatzikos C (2006) Detecting cognitive states from FMRI images by machine learning and multivariate classification. In: Conference on computer vision and pattern recognition workshop, 2006. CVPRW ’06, p 89

  11. 11.

    Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions, vol 7432. Springer, Berlin

    Google Scholar 

  12. 12.

    Ke H, Chen D, Shah T, Liu X, Zhang X, Zhang L, Li X (2018) Cloud-aided online EEG classification system for brain healthcare: a case study of depression evaluation with a lightweight CNN. Software: Practice and Experience

  13. 13.

    Koelstra S, Pantic M (2008) Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics. IEEE Trans Med Imaging 18(8):712–721

    Google Scholar 

  14. 14.

    Köstinger M, Wohlhart P, Roth PM, Bischof H (2012) Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: IEEE international conference on computer vision workshops, pp 2144–2151

  15. 15.

    Li S, Deng W (2018) Deep facial expression recognition: A survey. arXiv:1804.08348

  16. 16.

    Liao CT, Chuang HJ, Lai SH (2012) Learning expression kernels for facial expression intensity estimation. In: IEEE international conference on acoustics, speech and signal processing, pp 2217–2220

  17. 17.

    Littlewort G, Bartlett MS, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression extracted automatically from video. Image Vis Comput 24(6):615–625

    Article  Google Scholar 

  18. 18.

    Liu M, Li S, Shan S, Wang R, Chen X (2014) Deeply learning deformable facial action parts model for dynamic expression analysis. Springer, Berlin

    Google Scholar 

  19. 19.

    Liu Y, Chen J, Zhang M, Rao C (2018) Student engagement study based on multi-cue detection and recognition in an intelligent learning environment. Multimed Tools Appl 77:28749–28775

    Article  Google Scholar 

  20. 20.

    Liu Y, Wang L, Li W (2017) Emotion analysis based on facial expression recognition in virtual learning environment. Int J Comput Commun Eng 6:49–56

    Article  Google Scholar 

  21. 21.

    Lucey P, Cohn JF, Prkachin KM, Solomon PE (2011) Painful data: the UNBC-McMaster shoulder pain expression archive database. In: IEEE international conference on automatic face and gesture recognition, pp 57–64

  22. 22.

    Mehrabian A, Wiener M (1967) Decoding of inconsistent communications. J Pers Soc Psychol 6(1):109–114

    Article  Google Scholar 

  23. 23.

    Odobez JM, Ba S (2007) A cognitive and unsupervised map adaptation approach to the recognition of the focus of attention from head pose. In: IEEE international conference on multimedia and expo, pp 1379–1382

  24. 24.

    Ranjan R, Patel VM, Chellappa R (2016) Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41:121–135

    Article  Google Scholar 

  25. 25.

    Shen L, Wang M, Shen R (2009) Affective e-learning: using “emotional” data to improve learning in pervasive learning environment. J Educ Technol Soc 12(2):176–189

    Google Scholar 

  26. 26.

    Shen WY, Lin HT (2017) Active sampling of pairs and points for large-scale linear bipartite ranking. J Mach Learn Res 29:388–403

    Google Scholar 

  27. 27.

    Siau K, Sheng H, Nah FH (2006) Use of a classroom response system to enhance classroom interactivity. IEEE Trans Educ 49(3):398–403

    Article  Google Scholar 

  28. 28.

    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations

  29. 29.

    Tang Y, Chen D, Wang L, Zomaya AY, Chen J, Liu H (2018) Bayesian tensor factorization for multi-way analysis of multi-dimensional EEG. Neurocomputing 318:162–174

    Article  Google Scholar 

  30. 30.

    Tian YL, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  31. 31.

    Wei C, Keerthi SS (2005) New approaches to support vector ordinal regression. In: International conference on machine learning, pp 145–152

  32. 32.

    Yang P, Liu Q, Metaxas DN (2010) Rankboost with l1 regularization for facial expression recognition and intensity estimation. In: IEEE international conference on computer vision, pp 1018–1025

  33. 33.

    Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: International conference on automatic face and gesture recognition, pp 211–216

  34. 34.

    Yun WH, Lee D, Park C, Kim J (2015) Automatic engagement level estimation of kids in a learning environment. Int J Mach Learn Comput 5(2):148–152

    Article  Google Scholar 

  35. 35.

    Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  36. 36.

    Zhao R, Gan Q, Wang S, Ji Q (2016) Facial expression intensity estimation using ordinal information. In: Computer vision and pattern recognition, pp 3466–3474

  37. 37.

    Zhao X, Liang X, Liu L, Li T, Han Y, Vasconcelos N, Yan S (2016) Peak-piloted deep network for facial expression recognition. In: European conference on computer vision, pp 425–442

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (Grant No. 2018YFB1004504), the National Natural Science Foundation of China (No. 61772380, No. 61273063), Foundation for Innovative Research Groups of Hubei Province (No. 2017CFA007), and Hubei Province Technological Innovation Major Project and Research Funds of CCNU from the Colleges’ Basic Research and Operation of MOE (Grant No. CCNU19Z02002).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jingying Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Xu, R., Chen, J., Han, J. et al. Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information. Computing 102, 765–780 (2020). https://doi.org/10.1007/s00607-019-00722-7

Download citation

Keywords

  • Learning cognitive state analysis
  • Multi-task learning
  • Head pose estimation
  • Facial expression recognition
  • Expression intensity evaluation

Mathematics Subject Classification

  • 65D19