Modeling Confusion: Facial Expression, Task, and Discourse in Task-Oriented Tutorial Dialogue

  • Joseph F. Grafsgaard
  • Kristy Elizabeth Boyer
  • Robert Phillips
  • James C. Lester
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6738)


Recent years have seen a growing recognition of the importance of affect in learning. Efforts are being undertaken to enable intelligent tutoring systems to recognize and respond to learner emotion, but the field has not yet seen the emergence of a fully contextualized model of learner affect. This paper reports on a study of learner affect through an analysis of facial expression in human task-oriented tutorial dialogue. It extends prior work through in-depth analyses of a highly informative facial action unit and its interdependencies with dialogue utterances and task structure. The results demonstrate some ways in which learner facial expressions are dependent on both dialogue and task context. The findings also hold design implications for affect recognition and tutorial strategy selection within tutorial dialogue systems.


Affect tutorial dialogue tutorial strategies 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Woolf, B.P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D.G., Picard, R.W.: Affect-Aware Tutors: Recognizing and Responding to Student Affect. International Journal of Learning Technology 4, 129–164 (2009)CrossRefGoogle Scholar
  2. 2.
    D’Mello, S.K., Lehman, B., Person, N.: Monitoring Affect States During Effortful Problem Solving Activities. Int. J. Artif. Intell. Educ. 20 (2010)Google Scholar
  3. 3.
    Baker, R.S.J.d., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments. International Journal of Human-Computer Studies 68, 223–241 (2010)CrossRefGoogle Scholar
  4. 4.
    Lehman, B., D’Mello, S., Person, N.: The intricate dance between cognition and emotion during expert tutoring. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6095, pp. 433–442. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Afzal, S., Robinson, P.: Natural Affect Data - Collection and Annotation in a Learning Context. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 1–7 (2009)Google Scholar
  6. 6.
    Robison, J.L., McQuiggan, S.W., Lester, J.C.: Evaluating the Consequences of Affective Feedback in Intelligent Tutoring Systems. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 37–42 (2009)Google Scholar
  7. 7.
    Graesser, A.C., Olde, B.A.: How Does One Know Whether a Person Understands a Device? The Quality of the Questions the Person Asks When the Device Breaks Down. Journal of Educational Psychology 95, 524–536 (2003)CrossRefGoogle Scholar
  8. 8.
    D’Mello, S.K., Picard, R.W., Graesser, A.C.: Toward an Affect-Sensitive AutoTutor. IEEE Intelligent Systems 22, 53–61 (2007)CrossRefGoogle Scholar
  9. 9.
    Cooper, D.G., Muldner, K., Arroyo, I., Woolf, B.P., Burleson, W.: Ranking Feature Sets for Emotion Models used in Classroom Based Intelligent Tutoring Systems. User Modeling, Adaptation, and Personalization, 135–146 (2010)Google Scholar
  10. 10.
    Kapoor, A., Burleson, W., Picard, R.W.: Automatic Prediction of Frustration. International Journal of Human-Computer Studies 65, 724–736 (2007)CrossRefGoogle Scholar
  11. 11.
    Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City, USA (2002)Google Scholar
  12. 12.
    Craig, S.D., D’Mello, S.K., Witherspoon, A., Graesser, A.: Emote Aloud During Learning with AutoTutor: Applying the Facial Action Coding System to Cognitive-Affective States During Learning. Cognition & Emotion 22, 777–788 (2008)CrossRefGoogle Scholar
  13. 13.
    McDaniel, B.T., D’Mello, S.K., King, B.G., Chipman, P., Tapp, K., Graesser, A.C.: Facial Features for Affective State Detection in Learning Environments. In: Proceedings of the 29th Annual Meeting of the Cognitive Science Society, pp. 467–472 (2007)Google Scholar
  14. 14.
    Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System: Investigator’s Guide. A Human Face, Salt Lake City, USA (2002)Google Scholar
  15. 15.
    Cohn, J.F., Zlochower, A.J., Lien, J., Kanade, T.: Automated Face Analysis by Feature Point Tracking Has High Concurrent Validity with Manual FACS Coding. Psychophysiology 36, 35–43 (1999)CrossRefGoogle Scholar
  16. 16.
    Conati, C., Maclaren, H.: Empirically Building and Evaluating a Probabilistic Model of User Affect. User Modeling and User-Adapted Interaction 19, 267–303 (2009)CrossRefGoogle Scholar
  17. 17.
    McQuiggan, S.W., Lee, S., Lester, J.C.: Early Prediction of Student Frustration. In: Proceedings of the Second International Conference on Affective Computing and Intelligent Interactions, pp. 698–709 (2007)Google Scholar
  18. 18.
    Burleson, W.: Affective Learning Companions: Strategies for Empathetic Agents with Real-Time Multimodal Affective Sensing to Foster Meta-Cognitive and Meta-Affective Approaches to Learning, Motivation, and Perseverance. MIT Ph.D. thesis (2006)Google Scholar
  19. 19.
    Kaliouby, R., Robinson, P.: The Emotional Hearing Aid: An Assistive Tool for Children with Asperger Syndrome. Universal Access in the Information Society 4, 121–134 (2005)CrossRefGoogle Scholar
  20. 20.
    Afzal, S., Robinson, P.: Modelling Affect in Learning Environments - Motivation and Methods. In: Proceedings of the International Conference on Advanced Learning Technologies (2010)Google Scholar
  21. 21.
    D’Mello, S.K., Graesser, A.C.: Multimodal Semi-Automated Affect Detection from Conversational Cues, Gross Body Language, and Facial Features. User Modeling and User-Adapted Interaction 20, 147–187 (2010)CrossRefGoogle Scholar
  22. 22.
    Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M. D., Vouk, M. A., Lester, J. C.: Characterizing the effectiveness of tutorial dialogue with hidden markov models. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 55–64. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  23. 23.
    Boyer, K.E., Lahti, W.J., Phillips, R., Wallis, M.D., Vouk, M.A., Lester, J.C.: An Empirically-Derived Question Taxonomy for Task-Oriented Tutorial Dialogue. In: Proceedings of the Second Workshop on Question Generation, pp. 9–16 (2009)Google Scholar
  24. 24.
    Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper-Row, NY (1990)Google Scholar
  25. 25.
    Calvo, R.A., D’Mello, S.K.: Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing 1, 18–37 (2010)CrossRefGoogle Scholar
  26. 26.
    Russell, J.A.: Core Affect and the Psychological Construction of Emotion. Psychological Review 110, 145–172 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Joseph F. Grafsgaard
    • 1
  • Kristy Elizabeth Boyer
    • 1
  • Robert Phillips
    • 1
    • 2
  • James C. Lester
    • 1
  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA
  2. 2.Applied Research Associates, Inc.RaleighUSA

Personalised recommendations