Abstract
Affect plays a vital role in learning. During tutoring, particular affective states may benefit or detract from student learning. A key cognitive-affective state is confusion, which has been positively associated with effective learning. Although identifying episodes of confusion presents significant challenges, recent investigations have identified correlations between confusion and specific facial movements. This paper builds on those findings to create a predictive model of learner confusion during task-oriented human-human tutorial dialogue. The model leverages textual dialogue, task, and facial expression history to predict upcoming confusion within a hidden Markov modeling framework. Analysis of the model structure also reveals meaningful modes of interaction within the tutoring sessions. The results demonstrate that because of its predictive power and rich qualitative representation, the model holds promise for informing the design of affective-sensitive tutoring systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bloom, B.S.: The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher 13, 4–16 (1984)
D’Mello, S.K., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., Graesser, A.C.: A Time For Emoting: When Affect-Sensitivity Is and Isn’t Effective at Promoting Deep Learning. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010, Part I. LNCS, vol. 6094, pp. 245–254. Springer, Heidelberg (2010)
Koedinger, K.R., Anderson, J.R., Hadley, W.H., Mark, M.A.: Intelligent Tutoring Goes To School in the Big City. Intl. Jl. of Artificial Intelligence in Education 8, 30–43 (1997)
D’Mello, S., Olney, A., Person, N.: Mining Collaborative Patterns in Tutorial Dialogues. Jl. of Educational Data Mining 2, 1–37 (2010)
D’Mello, S.K., Lehman, B., Person, N.: Monitoring Affect States During Effortful Problem Solving Activities. International Jl. of Artificial Intelligence in Education 20 (2010)
Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.M.: Emotion Sensors Go To School. In: 14th International Conference on Artificial Intelligence in Education (2009)
Burleson, W.: Affective Learning Companions: Strategies for Empathetic Agents with Real-Time Multimodal Affective Sensing to Foster Meta-Cognitive and Meta-Affective Approaches to Learning, Motivation, and Perseverance. MIT Ph.D. thesis (2006)
McQuiggan, S.W., Robison, J.L., Lester, J.C.: Affective Transitions in Narrative-Centered Learning Environments. Educational Technology & Society 13, 40–53 (2010)
Craig, S.D., Graesser, A.C., Sullins, J., Gholson, B.: Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Jl. of Educational Media 29, 241–250 (2004)
McDaniel, B.T., D’Mello, S.K., King, B.G., Chipman, P., Tapp, K., Graesser, A.C.: Facial Features for Affective State Detection in Learning Environments. In: Proceedings of the 29th Annual Meeting of the Cognitive Science Society, pp. 467–472 (2007)
Russell, J.A., Bachorowski, J., Fernandez-Dols, J.: Facial and vocal expressions of emotion. Annual Review of Psychology 54, 329–349 (2003)
Afzal, S., Robinson, P.: Natural Affect Data - Collection & Annotation in a Learning Context. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 1–7 (2009)
Graesser, A.C., Olde, B.A.: How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Jl. of Educational Psychology 95, 524–536 (2003)
Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City, USA (2002)
Boyer, K.E., Ha, E.Y., Wallis, M., Phillips, R., Vouk, M., Lester, J.: Discovering Tutorial Dialogue Strategies with Hidden Markov Models. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education, pp. 141–148 (2009)
Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M.D., Vouk, M.A., Lester, J.C.: Characterizing the Effectiveness of Tutorial Dialogue with Hidden Markov Models. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 55–64. Springer, Heidelberg (2010)
D’Mello, S.K., Craig, S.D., Graesser, A.C.: Multi-Method Assessment of Affective Experience and Expression during Deep Learning. International Jl. of Learning Technology 4, 165–187 (2009)
Baker, R.S.J.d., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments. International Jl. of Human-Computer Studies 68, 223–241 (2010)
Picard, R.W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., Machover, T., Resnick, M., Roy, D., Strohecker, C.: Affective Learning — A Manifesto. BT Technology Jl. 22, 253–269 (2004)
Woolf, B.P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D.G., Picard, R.W.: Affect-aware tutors: recognising and responding to student affect. International Jl. of Learning Technology 4, 129–164 (2009)
Calvo, R.A., D’Mello, S.K.: Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing 1, 18–37 (2010)
D’Mello, S.K., Graesser, A.: Multimodal Semi-Automated Affect Detection from Conversational Cues, Gross Body Language, and Facial Features. User Modeling and User-Adapted Interaction 20, 147–187 (2010)
Grafsgaard, J.F., Boyer, K.E., Phillips, R., Lester, J.C.: Modeling Confusion: Facial Expression, Task, and Discourse in Task-Oriented Tutorial Dialogue. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 98–105. Springer, Heidelberg (2011)
Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77, 257–286 (1989)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A Survey Of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 39–58 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Grafsgaard, J.F., Boyer, K.E., Lester, J.C. (2011). Predicting Facial Indicators of Confusion with Hidden Markov Models. In: D’Mello, S., Graesser, A., Schuller, B., Martin, JC. (eds) Affective Computing and Intelligent Interaction. ACII 2011. Lecture Notes in Computer Science, vol 6974. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24600-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-24600-5_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-24599-2
Online ISBN: 978-3-642-24600-5
eBook Packages: Computer ScienceComputer Science (R0)