Abstract
Recent years have seen a growing recognition of the importance of affect in learning. Efforts are being undertaken to enable intelligent tutoring systems to recognize and respond to learner emotion, but the field has not yet seen the emergence of a fully contextualized model of learner affect. This paper reports on a study of learner affect through an analysis of facial expression in human task-oriented tutorial dialogue. It extends prior work through in-depth analyses of a highly informative facial action unit and its interdependencies with dialogue utterances and task structure. The results demonstrate some ways in which learner facial expressions are dependent on both dialogue and task context. The findings also hold design implications for affect recognition and tutorial strategy selection within tutorial dialogue systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Woolf, B.P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D.G., Picard, R.W.: Affect-Aware Tutors: Recognizing and Responding to Student Affect. International Journal of Learning Technology 4, 129–164 (2009)
D’Mello, S.K., Lehman, B., Person, N.: Monitoring Affect States During Effortful Problem Solving Activities. Int. J. Artif. Intell. Educ. 20 (2010)
Baker, R.S.J.d., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments. International Journal of Human-Computer Studies 68, 223–241 (2010)
Lehman, B., D’Mello, S., Person, N.: The intricate dance between cognition and emotion during expert tutoring. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6095, pp. 433–442. Springer, Heidelberg (2010)
Afzal, S., Robinson, P.: Natural Affect Data - Collection and Annotation in a Learning Context. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 1–7 (2009)
Robison, J.L., McQuiggan, S.W., Lester, J.C.: Evaluating the Consequences of Affective Feedback in Intelligent Tutoring Systems. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 37–42 (2009)
Graesser, A.C., Olde, B.A.: How Does One Know Whether a Person Understands a Device? The Quality of the Questions the Person Asks When the Device Breaks Down. Journal of Educational Psychology 95, 524–536 (2003)
D’Mello, S.K., Picard, R.W., Graesser, A.C.: Toward an Affect-Sensitive AutoTutor. IEEE Intelligent Systems 22, 53–61 (2007)
Cooper, D.G., Muldner, K., Arroyo, I., Woolf, B.P., Burleson, W.: Ranking Feature Sets for Emotion Models used in Classroom Based Intelligent Tutoring Systems. User Modeling, Adaptation, and Personalization, 135–146 (2010)
Kapoor, A., Burleson, W., Picard, R.W.: Automatic Prediction of Frustration. International Journal of Human-Computer Studies 65, 724–736 (2007)
Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City, USA (2002)
Craig, S.D., D’Mello, S.K., Witherspoon, A., Graesser, A.: Emote Aloud During Learning with AutoTutor: Applying the Facial Action Coding System to Cognitive-Affective States During Learning. Cognition & Emotion 22, 777–788 (2008)
McDaniel, B.T., D’Mello, S.K., King, B.G., Chipman, P., Tapp, K., Graesser, A.C.: Facial Features for Affective State Detection in Learning Environments. In: Proceedings of the 29th Annual Meeting of the Cognitive Science Society, pp. 467–472 (2007)
Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System: Investigator’s Guide. A Human Face, Salt Lake City, USA (2002)
Cohn, J.F., Zlochower, A.J., Lien, J., Kanade, T.: Automated Face Analysis by Feature Point Tracking Has High Concurrent Validity with Manual FACS Coding. Psychophysiology 36, 35–43 (1999)
Conati, C., Maclaren, H.: Empirically Building and Evaluating a Probabilistic Model of User Affect. User Modeling and User-Adapted Interaction 19, 267–303 (2009)
McQuiggan, S.W., Lee, S., Lester, J.C.: Early Prediction of Student Frustration. In: Proceedings of the Second International Conference on Affective Computing and Intelligent Interactions, pp. 698–709 (2007)
Burleson, W.: Affective Learning Companions: Strategies for Empathetic Agents with Real-Time Multimodal Affective Sensing to Foster Meta-Cognitive and Meta-Affective Approaches to Learning, Motivation, and Perseverance. MIT Ph.D. thesis (2006)
Kaliouby, R., Robinson, P.: The Emotional Hearing Aid: An Assistive Tool for Children with Asperger Syndrome. Universal Access in the Information Society 4, 121–134 (2005)
Afzal, S., Robinson, P.: Modelling Affect in Learning Environments - Motivation and Methods. In: Proceedings of the International Conference on Advanced Learning Technologies (2010)
D’Mello, S.K., Graesser, A.C.: Multimodal Semi-Automated Affect Detection from Conversational Cues, Gross Body Language, and Facial Features. User Modeling and User-Adapted Interaction 20, 147–187 (2010)
Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M. D., Vouk, M. A., Lester, J. C.: Characterizing the effectiveness of tutorial dialogue with hidden markov models. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 55–64. Springer, Heidelberg (2010)
Boyer, K.E., Lahti, W.J., Phillips, R., Wallis, M.D., Vouk, M.A., Lester, J.C.: An Empirically-Derived Question Taxonomy for Task-Oriented Tutorial Dialogue. In: Proceedings of the Second Workshop on Question Generation, pp. 9–16 (2009)
Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper-Row, NY (1990)
Calvo, R.A., D’Mello, S.K.: Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing 1, 18–37 (2010)
Russell, J.A.: Core Affect and the Psychological Construction of Emotion. Psychological Review 110, 145–172 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Grafsgaard, J.F., Boyer, K.E., Phillips, R., Lester, J.C. (2011). Modeling Confusion: Facial Expression, Task, and Discourse in Task-Oriented Tutorial Dialogue. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds) Artificial Intelligence in Education. AIED 2011. Lecture Notes in Computer Science(), vol 6738. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21869-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-642-21869-9_15
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-21868-2
Online ISBN: 978-3-642-21869-9
eBook Packages: Computer ScienceComputer Science (R0)