Toward a Machine Learning Framework for Understanding Affective Tutorial Interaction

  • Joseph F. Grafsgaard
  • Kristy Elizabeth Boyer
  • James C. Lester
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7315)


Affect and cognition intertwine throughout human experience. Research into this interplay during learning has identified relevant cognitive-affective states, but recognizing them poses significant challenges. Among multiple promising approaches for affect recognition, analyzing facial expression may be particularly informative. Descriptive computational models of facial expression and affect, such as those enabled by machine learning, aid our understanding of tutorial interactions. Hidden Markov modeling, in particular, is useful for encoding patterns in sequential data. This paper presents a descriptive hidden Markov model built upon facial expression data and tutorial dialogue within a task-oriented human-human tutoring corpus. The model reveals five frequently occurring patterns of affective tutorial interaction across text-based tutorial dialogue sessions. The results show that hidden Markov modeling holds potential for the semi-automated understanding of affective interaction, which may contribute to the development of affect-informed intelligent tutoring systems.


Affect hidden Markov models tutorial dialogue 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    D’Mello, S.K., Calvo, R.A.: Significant Accomplishments, New Challenges, and New Perspectives. In: Calvo, R.A., D’Mello, S.K. (eds.) New Perspectives on Affect and Learning Technologies, pp. 255–271. Springer, New York (2011)CrossRefGoogle Scholar
  2. 2.
    D’Mello, S.K., Lehman, B., Person, N.: Monitoring Affect States During Effortful Problem Solving Activities. International Journal of Artificial Intelligence in Education 20 (2010)Google Scholar
  3. 3.
    D’Mello, S.K., Graesser, A.: Modeling Cognitive-Affective Dynamics with Hidden Markov Models. In: Proceedings of the 32nd Annual Cognitive Science Society, pp. 2721–2726 (2010)Google Scholar
  4. 4.
    Johns, J., Woolf, B.: A Dynamic Mixture Model to Detect Student Motivation and Proficiency. In: Proceedings of the 21st National Conference on Artificial Intelligence, pp. 163–168 (2006)Google Scholar
  5. 5.
    Grafsgaard, J.F., Boyer, K.E., Lester, J.C.: Predicting Facial Indicators of Confusion with Hidden Markov Models. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part I. LNCS, vol. 6974, pp. 97–106. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M., Vouk, M., Lester, J.: Characterizing the Effectiveness of Tutorial Dialogue with Hidden Markov Models. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 55–64. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City, USA (2002)Google Scholar
  8. 8.
    Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77, 257–286 (1989)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Joseph F. Grafsgaard
    • 1
  • Kristy Elizabeth Boyer
    • 1
  • James C. Lester
    • 1
  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA

Personalised recommendations