Affect Detection from Multichannel Physiology during Learning Sessions with AutoTutor

  • M. S. Hussain
  • Omar AlZoubi
  • Rafael A. Calvo
  • Sidney K. D’Mello
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6738)


It is widely acknowledged that learners experience a variety of emotions while interacting with Intelligent Tutoring Systems (ITS), hence, detecting and responding to emotions might improve learning outcomes. This study uses machine learning techniques to detect learners’ affective states from multichannel physiological signals (heart activity, respiration, facial muscle activity, and skin conductivity) during tutorial interactions with AutoTutor, an ITS with conversational dialogues. Learners were asked to self-report (both discrete emotions and degrees of valence/arousal) the affective states they experienced during their sessions with AutoTutor via a retrospective judgment protocol immediately after the tutorial sessions. In addition to mapping the discrete learning-centered emotions (e.g., confusion, frustration, etc) on a dimensional valence/arousal space, we developed and validated an automatic affect classifier using physiological signals. Results indicate that the classifier was moderately successful at detecting naturally occurring emotions during the AutoTutor sessions.


Affective computing emotion AutoTutor multichannel physiology learning interaction self reports 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Craig, S., Graesser, A., Sullins, J., Gholson, B.: Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Learning, Media and Technology 29, 241–250 (2004)Google Scholar
  2. 2.
    Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction 19, 267–303 (2009)CrossRefGoogle Scholar
  3. 3.
    Calvo, R.A., D’Mello, S.: New perspectives on affect and learning technologies. Springer, New York (in preparation)Google Scholar
  4. 4.
    Kapoor, A., Picard, R.W.: Multimodal affect recognition in learning environments. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, Hilton, Singapore, pp. 677–682 (2005)Google Scholar
  5. 5.
    Calvo, R.A., D’Mello, S.: Affect Detection: An Interdisciplinary Review of Models, Methods, and their Applications. IEEE Transactions on Affective Computing 1, 18–37 (2010)CrossRefGoogle Scholar
  6. 6.
    D’Mello, S., Graesser, A.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction 20, 147–187 (2010)CrossRefGoogle Scholar
  7. 7.
    Arroyo, I., Cooper, D., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion Sensors Go To School. In: Proceeding of the 2009 Conference on Artificial Intelligence in Education, Amsterdam, vol. 200, pp. 17–24 (2009)Google Scholar
  8. 8.
    Lehman, B., Matthews, M., D’Mello, S.K., Person, N.: What are you feeling? Investigating student affective states during expert human tutoring sessions. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 50–59. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Aghaei Pour, P., Hussain, M., AlZoubi, O., D’Mello, S., Calvo, R.: The Impact of System Feedback on Learners’ Affective and Physiological States. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 264–273. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Graesser, A.C., Chipman, P., Haynes, B.C., Olney, A.: AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education 48, 612–618 (2005)CrossRefGoogle Scholar
  11. 11.
    Russell, J.A., Barrett, L.F.: Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology 76, 805–819 (1999)CrossRefGoogle Scholar
  12. 12.
    Lichtenstein, A., Oehme, A., Kupschick, S., Jürgensohn, T.: Comparing Two Emotion Models for Deriving Affective States from Physiological Data. In: Peter, C., Beale, R. (eds.) Affect and Emotion in Human-Computer Interaction. LNCS, vol. 4868, pp. 35–50. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  13. 13.
    Kort, B., Reilly, R., Picard, R.W.: An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion. In: IEEE International Conference on Advanced Learning Technologies, Madison, Wisconsin, pp. 43–46 (2001)Google Scholar
  14. 14.
    Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 1175–1191 (2001)CrossRefGoogle Scholar
  15. 15.
    Wagner, J., Kim, J., Andre, E.: From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: IEEE International Conference on Multimedia and Expo., ICME 2005, Amsterdam, The Netherlands, pp. 940–943 (2005)Google Scholar
  16. 16.
    Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): Technical manual and affective ratings. The Center for Research in Psychophysiology, University of Florida, Gainesville, FL (1995)Google Scholar
  17. 17.
    Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39, 1161–1178 (1980)CrossRefGoogle Scholar
  18. 18.
    Wagner, J., Kim, J., Andre, E.: From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: IEEE International Conference on Multimedia and Expo. 2005, Amsterdam, The Netherlands, pp. 940–943 (2005)Google Scholar
  19. 19.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann Series in Data Management Systems. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  20. 20.
    Kuncheva, L.I.: Combining pattern classifiers: methods and algorithms. Wiley-Interscience, Hoboken (2004)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • M. S. Hussain
    • 1
    • 2
  • Omar AlZoubi
    • 2
  • Rafael A. Calvo
    • 2
  • Sidney K. D’Mello
    • 3
  1. 1.National ICT Australia (NICTA)EveleighAustralia
  2. 2.School of Electrical and Information EngineeringUniversity of SydneyAustralia
  3. 3.Institute for Intelligent SystemsUniversity of MemphisMemphisUSA

Personalised recommendations