Emotion Detection in E-learning Using Expectation-Maximization Deep Spatial-Temporal Inference Network

  • Jiangqin Xu
  • Zhongqiang Huang
  • Minghui Shi
  • Min JiangEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 650)


It is very useful for the E-learning systems to detect the students emotional state accurately, and this can remind the teacher in time to change the teaching rhythm or content to meet the student’s emotional changes for making the teaching effect optimization. In this paper, we propose an emotion detection method based on a deep learning approach, Expectation-maximization Deep Spatial-Temporal Inference Network (EM-DeSTIN). This method takes the student’s facial expression as input and combine with Support Vector Machine (SVM) to implement emotion classification and identification. Experimental results show that the proposed method improves the performance of detecting emotion in a noisy environment compared with other methods.


E-learning Emotion detection Deep learning 



This work was supported by the National Natural Science Foundation of China (No. 61003014 and No. 61673328), the National Social Science Foundation (15BYY082) and the Natural Science Foundation of Fujian Province of China (No. 2017J01128).


  1. 1.
    Daradoumis, T., Bassi, R., Xhafa, F., Caballé, S.: A review on massive e-learning (MOOC) design, delivery and assessment. In: 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC), pp. 208–213. IEEE (2013)Google Scholar
  2. 2.
    Zaíane, O.R.: Building a recommender agent for e-learning systems. In: Proceedings of the International Conference on Computers in Education, pp. 55–59. IEEE (2002)Google Scholar
  3. 3.
    Binali, H.H., Wu, C., Potdar, V.: A new significant area: emotion detection in e-learning using opinion mining techniques. In: 3rd IEEE International Conference on Digital Ecosystems and Technologies, DEST 2009, pp. 259–264. IEEE (2009)Google Scholar
  4. 4.
    Sylwester, R.: How emotions affect learning. Educ. Leadersh. 52(2), 60–65 (1994)Google Scholar
  5. 5.
    Oregan, K.: Emotion and e-learning. J. Asynchronous Learn. Netw. 7(3), 78–92 (2003)Google Scholar
  6. 6.
    Russell, J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110(1), 145 (2003)CrossRefGoogle Scholar
  7. 7.
    De Vicente, A., Pain, H.: Informing the detection of the students motivational state: an empirical study. In: International Conference on Intelligent Tutoring Systems, pp. 933–943. Springer (2002)Google Scholar
  8. 8.
    Torricelli, D., Goffredo, M., Conforto, S., Schmid, M.: An adaptive blink detector to initialize and update a view-basedremote eye gaze tracking system in a natural scenario. Pattern Recogn. Lett. 30(12), 1144–1150 (2009)CrossRefGoogle Scholar
  9. 9.
    Graesser, A.C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., Tutoring Research Group, et al.: Autotutor: a simulation of a human tutor. Cogn. Syst. Res. 1(1), 35–51 (1999)Google Scholar
  10. 10.
    Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G.: Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 18(1), 32–80 (2001)CrossRefGoogle Scholar
  11. 11.
    Matsugu, M., Mori, K., Mitari, Y., Kaneda, Y.: Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 16(5), 555–559 (2003)CrossRefGoogle Scholar
  12. 12.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  13. 13.
    Jiang, M., Ding, Y., Goertzel, B., Huang, Z., Zhou, C., Chao, F.: Improving machine vision via incorporating expectation-maximization into deep spatio-temporal learning. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 1804–1811. IEEE (2014)Google Scholar
  14. 14.
    Arel, I., Rose, D.C., Coop, R.: Destin: a scalable deep learning architecture with application to high-dimensional robust pattern recognition. In: AAAI Fall Symposium: Biologically Inspired Cognitive Architectures (2009)Google Scholar
  15. 15.
    Cappé, O., Moulines, E.: On-line expectation-maximization algorithm for latent data models. J. Roy. Stat. Soc.: Ser. B (Stat. Methodol.) 71(3), 593–613 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    LeCun, Y., et al.: Lenet-5, convolutional neural networks (2015).
  18. 18.
    Krizhevsky, A., Hinton, G.E.: Using very deep autoencoders for content-based image retrieval. In: ESANN (2011)Google Scholar
  19. 19.
    Jiang, M., Huang, W., Huang, Z., Yen, G.G.: Integration of global and local metrics for domain adaptation learning via dimensionality reduction. IEEE Trans. Cybern. 47(1), 38–51 (2017)CrossRefGoogle Scholar
  20. 20.
    Jiang, M., Zhou, C., Chen, S.: Embodied concept formation and reasoning via neural-symbolic integration. Neurocomputing 74(1), 113–120 (2010)CrossRefGoogle Scholar
  21. 21.
    Chao, F., Wang, Z., Shang, C., Meng, Q., Jiang, M., Zhou, C., Shen, Q.: A developmental approach to robotic pointing via human-robot interaction. Inf. Sci. 283, 288–303 (2014)CrossRefGoogle Scholar
  22. 22.
    Jiang, M., Yu, Y., Liu, X., Zhang, F., Hong, Q.: Fuzzy neural network based dynamic path planning. In: 2012 International Conference on Machine Learning and Cybernetics (ICMLC), vol. 1, pp. 326–330. IEEE (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Jiangqin Xu
    • 1
  • Zhongqiang Huang
    • 2
  • Minghui Shi
    • 2
  • Min Jiang
    • 2
    Email author
  1. 1.College of Foreign Languages and CulturesXiamen UniversityXiamenChina
  2. 2.Fujian Province Key Laboratory for Brain-inspired Computing Technique and Applications, School of Information and EngineeringXiamen UniversityXiamenChina

Personalised recommendations