Improving Automatic Affect Recognition on Low-Level Speech Features in Intelligent Tutoring Systems

  • Ruth JanningEmail author
  • Carlotta Schatten
  • Lars Schmidt-Thieme
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9307)


Currently, a lot of research in the field of intelligent tutoring systems is concerned with recognising student’s emotions and affects. The recognition is done by extracting features from information sources like speech, typing and mouse clicking behaviour or physiological sensors. According to the state-of-the-art support vector machines are the best performing classification models for those kinds of features. However, single classification models often do not deliver the best possible performance. Hence, we propose an approach for further improving the affect recognition performance, which is based on ideas from ensemble approaches and feature selection methods. The approach is proven by experiments on low-level speech features extracted from data which was collected in a study with German students solving mathematical tasks. In these experiments the proposed approach reached on average an affect recognition performance improvement of about 59 % in comparison to a single SVM.


Affect recognition Low-level speech features Intelligent tutoring systems Feature selection Ensemble Classification performance improvement 



The research leading to the results reported here has received funding from the European Union Seventh Framework Programme (FP7/2007 – 2013) under grant agreement no. 318051 – iTalk2Learn project (


  1. 1.
    Arroyo, I., Woolf, B.P., Burelson, W., Muldner, K., Rai, D., Tai, M.: A multimedia adaptive tutoring system for mathematics that addresses cognition, metacognition and affect. Int. J. Artif. Intell. Edu. 24, 387–426 (2014)CrossRefGoogle Scholar
  2. 2.
    Baker, R.S.J.D., Gowda, S., Wixon, M., Kalka, J., Wagner, A., Salvi, A., Aleven, V., Kusbit, G., Ocumpaugh, J. and Rossi, L.: Towards sensor-free affect detection in cognitive tutor algebra. In: Proceedings of the 5th International Conference on Educational Data Mining (EDM 2012), pp. 126–133 (2012)Google Scholar
  3. 3.
    Boser, B.E., Guyon, I., Vapnik, V.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 144–152. ACM Press (1992)Google Scholar
  4. 4.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)CrossRefGoogle Scholar
  5. 5.
    Cortes, C., Vapnik, V.: Support-vector network. Mach. Learn. 20, 273–297 (1995)zbMATHGoogle Scholar
  6. 6.
    Epp, C., Lippold, M., Mandryk, R.L.: Identifying emotional states using keystroke dynamics. In: Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems (CHI 2011), pp. 715–724 (2011)Google Scholar
  7. 7.
    Grawemeyer, B., Gutierrez-Santos, S., Holmes, W., Mavrikis, M., Rummel, N., Mazziotti, C. Janning, R.: Talk, tutor, explore, learn: intelligent tutoring and exploration for robust learning. In: Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED) (2015)Google Scholar
  8. 8.
    Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification. Technical report, Department of Computer Science, National Taiwan University (2011).
  9. 9.
    Hu, X., Tang, L., Tang, J., Liu, H.: Exploiting social relations for sentiment analysis in microblogging. In: Proceedings of the Sixth ACM WSDM Conference (WSDM 2013) (2013)Google Scholar
  10. 10.
    Janning, R., Schatten, C., Schmidt-Thieme, L.: Multimodal affect recognition for adaptive intelligent tutoring systems. In: Extended Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), pp. 171–178 (2014)Google Scholar
  11. 11.
    Janning, R., Schatten, C., Schmidt-Thieme, L.: Feature analysis for affect recognition supporting task sequencing in adaptive intelligent tutoring systems. In: Proceedings of the European Conference on Technology Enhanced Learning (EC-TEL 2014), pp. 179–192 (2014)Google Scholar
  12. 12.
    Janning, R., Schatten, C., Schmidt-Thieme, L., Backfried, G.: An SVM plait for improving affect recognition in intelligent tutoring systems. In: Proceedings of the IEEE International Conference on Tools with Artificial Intelligence (ICTAI) (2014)Google Scholar
  13. 13.
    Luz, S.: Automatic identification of experts and performance prediction in the multimodal math data corpus through analysis of speech interaction. In: Second International Workshop on Multimodal Learning Analytics, Sydney Australia (2013)Google Scholar
  14. 14.
    Mavrikis, M.: Data-driven modelling of students interactions in an ILE. In: Proceedings of the International Conference on Educational Data Mining (EDM 2008), pp. 87–96 (2008)Google Scholar
  15. 15.
    D’Mello, S.K., Craig, S.D., Witherspoon, A., McDaniel, B., Graesser, A.: Automatic detection of learners affect from conversational cues. User Model User-Adap Inter. 18, 45–80 (2008). doi: 10.1007/s11257-007-9037-6 CrossRefGoogle Scholar
  16. 16.
    D’Mello, S.K., Graesser, A.: Language and discourse are powerful signals of student emotions during tutoring. IEEE Trans. Learn. Technol. 5(4), 304–317 (2012)CrossRefGoogle Scholar
  17. 17.
    Moore, J.D., Tian, L., Lai, C.: Word-level emotion recognition using high-level features. In: Gelbukh, A. (ed.) CICLing 2014, Part II. LNCS, vol. 8404, pp. 17–31. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  18. 18.
    Morency, L.P., Oviatt, S., Scherer, S., Weibel, N., Worsley, M.: ICMI 2013 grand challenge workshop on multimodal learning analytics. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI 2013), pp. 373–378 (2013)Google Scholar
  19. 19.
    Pardos, Z.A., Baker, R.S.J.D., San Pedro, M., Gowda, S.M., Gowda, S.M.: Affective states and state tests: investigating how affect and engagement during the school year predict end-of-year learning outcomes. Inaugural issue J. Learn. Anal. 1(1), 107–128 (2014)CrossRefGoogle Scholar
  20. 20.
    Purandare, A., Litman, D.: Humor: prosody analysis and automatic recognition for F * R * I * E * N * D * S *. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pp. 208–215 (2006)Google Scholar
  21. 21.
    Qi, F., Bao, C., Liu, Y.: A novel two-step SVM classifier for voiced/unvoiced/silence classification of speech. In: International Symposium on Chinese Spoken Language Processing, pp. 77–80 (2004)Google Scholar
  22. 22.
    Sadegh, M., Ibrahim, R., Othman, Z.A.: Opinion mining and sentiment analysis: a survey. Int. J. Comput. Technol. 2(3), 171–178 (2012)Google Scholar
  23. 23.
    Saif, H., He, Y., Alani, H.: Semantic sentiment analysis of Twitter. In: Cudré-Mauroux, P., Heflin, J., Sirin, E., Tudorache, T., Euzenat, J., Hauswirth, M., Parreira, J.X., Hendler, J., Schreiber, G., Bernstein, A., Blomqvist, E. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 508–524. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  24. 24.
    San Pedro, M.O.C., Baker, R.S.J.D., Bowers, A., Heffernan, N.: Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In: Proceedings of the 6th International Conference on Educational Data Mining (EDM 2013), pp. 177–184 (2013)Google Scholar
  25. 25.
    Schuller, B., Batliner, A., Steidl, S., Seppi, D.: Recognising realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge. Speech Commun. 53, 1062–1087 (2011)CrossRefGoogle Scholar
  26. 26.
    Ting, K.M., Witten, I.H.: Issues in stacked generalization. J. Artif. Intel. Res. 10, 271–289 (1999)zbMATHGoogle Scholar
  27. 27.
    Worsley, M., Blikstein, P.: What’s an expert? using learning analytics to identify emergent markers of expertise through automated speech, sentiment and sketch analysis. In: Proceedings of the 4th International Conference on Educational Data Mining (EDM 2011), pp. 235–240 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Ruth Janning
    • 1
    Email author
  • Carlotta Schatten
    • 1
  • Lars Schmidt-Thieme
    • 1
  1. 1.Information Systems and Machine Learning Lab (ISMLL)University of HildesheimHildesheimGermany

Personalised recommendations