Recognizing and Responding to Student Affect

  • Beverly Woolf
  • Toby Dragon
  • Ivon Arroyo
  • David Cooper
  • Winslow Burleson
  • Kasia Muldner
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5612)


This paper describes the use of wireless sensors to recognize student emotion and the use of pedagogical agents to respond to students with these emotions. Minimally invasive sensor technology has reached such a maturity level that students engaged in classroom work can us sensors while using a computer-based tutor. The sensors, located on each of 25 student’s chair, mouse, monitor, and wrist, provide data about posture, movement, grip tension, facially expressed mental states and arousal. This data has demonstrated that intelligent tutoring systems can provide adaptive feedback based on an individual student’s affective state. We also describe the evaluation of emotional embodied animated pedagogical agents and their impact on student motivation and achievement. Empirical studies show that students using the agents increased their math value, self-concept and mastery orientation.


intelligent tutoring systems wireless sensors student emotion pedagogical agents 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wentzel, K., Asher, S.R.: Academic lives of neglected, rejected, popular, and controversial children. Child Development 66, 754–763 (1995)CrossRefGoogle Scholar
  2. 2.
    Royer, J.M., Walles, R.: Influences of gender, motivation and socioeconomic status on mathematics performance. In: Berch, D.B., Mazzocco, M.M.M. (eds.) Why is math so hard for some children, pp. 349–368. Paul. H. Brookes Publishing Co., Baltimore (2007)Google Scholar
  3. 3.
    Conati, C., Maclare, H.: Evaluating a Probabilistic Model of Student Affect. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 55–66. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  4. 4.
    D’Mello, S., Graesser, A.: Mind and Body: Dialogue and Posture for Affect Detection in Learning Environments. Paper presented at the Frontiers in Artificial Intelligence and Applications (2007)Google Scholar
  5. 5.
    McQuiggan, S., Lester, J.: Diagnosing Self-Efficacy in Intelligent Tutoring Systems: An Empirical Study. In: Ikeda, M., Ashley, K., Chan, T.W. (eds.) Eighth International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan (2006)Google Scholar
  6. 6.
    Graesser, A.C., Chipman, P., King, B., McDaniel, B., D’Mello, S.: Emotions and Learning with AutoTutor. In: Luckin, R., Koedinger, K., Greer, J. (eds.) 13th International Conference on Artificial Intelligence in Education (AIED 2007), pp. 569–571. IOS Press, Amsterdam (2007)Google Scholar
  7. 7.
    D’Mello, S.K., Picard, R.W., Graesser, A.C.: Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems. IEEE Intelligent Systems 22(4), 53–61 (2007)CrossRefGoogle Scholar
  8. 8.
    Conati, C., Mclaren, H.: Evaluating A Probabilistic Model of Student Affect. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 55–66. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Burleson, W.: Affective Learning Companions: Strategies for Empathetic Agents with Real-Time Multimodal Affective Sensing to Foster Meta-Cognitive Approaches to Learning, Motivation, and Perseverance. MIT PhD thesis (2006),
  10. 10.
    Picard, R.W., Scheirer, J.: The galvactivator: A glove that senses and communicates skin conductivity. In: 9th International Conference on Human-Computer Interaction, New Orleans, August 2001, pp. 1538–1542 (2001)Google Scholar
  11. 11.
    Strauss, M., Reynolds, C., Hughes, S., Park, K., McDarby, G., Picard, R.: The handwave bluetooth skin conductance sensor. In: Tao, J., Tan, T., Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 699–706. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    Qi, Y., Picard, R.: Context-sensitive bayesian classifiers and application to mouse pressure pattern classification. In: 16th International Conference on Pattern Recognition, Proceedings, vol. 3, pp. 448–451 (2002)Google Scholar
  13. 13.
    Dennerlein, J., Becker, T., Johnson, P., Reynolds, C., Picard, R.W.: Frustrating computer users increases exposure to physical factors. In: Proceedings of International Ergonomics Association, Seoul, Korea, pp. 24–29 (2003)Google Scholar
  14. 14.
    Mota, S., Picard, R.W.: Automated posture analysis for detecting learner’s interest level. In: Computer Vision and Pattern Recognition Workshop, vol. 5, p. 49 (2003)Google Scholar
  15. 15.
    Kapoor, A., Burleson, W., Picard, R.W.: Automatic prediction of frustration. International Journal of Human-Computer Studies 65(8), 724–736 (2007)CrossRefGoogle Scholar
  16. 16.
    Burleson, W., Picard, R.W.: Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intelligent Systems 22(4), 62–69 (2007)CrossRefGoogle Scholar
  17. 17.
    el Kaliouby, R.: Mind-reading Machines: the automated inference of complex mental states from video. PhD thesis, University of Cambridge (2005)Google Scholar
  18. 18.
    Arroyo, I., Cooper, D., Burleson, W., Woolf, B.P., Muldner, K.: Empathetic Pedagogical Agents. Submitted to AIED (2009)Google Scholar
  19. 19.
    Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Mehranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing Disengagement With Non Invasive Interventions. In: International Conference on Artificial Intelligence in Education, Marina del Rey, CA (2007)Google Scholar
  20. 20.
    Dragon, T., Arroyo, I., Woolf, B.P., Burleson, W., El Kaliouby, R., Eydgahi, H.: Viewing Student Affect and Learning through Classroom Observation and Physical Sensors. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 29–39. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  21. 21.
    Graham, S., Weiner, B.: Theories and principles of motivation. In: Berliner, D., Calfee, R. (eds.) Handbook of Educational Psychology, pp. 63–84. Macmillan, New York (1996)Google Scholar
  22. 22.
    Zimmerman, B.J.: Self-Efficacy: An Essential Motive to Learn. Contemporary Educational Psychology 25, 82–91 (2000)CrossRefGoogle Scholar
  23. 23.
    Picard, R.W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., Machover, T., Resnick, M., Roy, D., Strohecker, C.: Affective Learning–A Manifesto. BT Journal 2(4), 253–269 (2004)Google Scholar
  24. 24.
    Bickmore, T., Picard, R.W.: Establishing and Maintaining Long-Term Human-Computer Relationships. Transactions on Computer-Human Interaction 12(2), 293–327 (2004)CrossRefGoogle Scholar
  25. 25.
    Reeves, B., Nass, C.: The media equation: How people treat computers, television and new media like real people and places. CSLI, New York (1998)Google Scholar
  26. 26.
    Klein, J., Moon, Y., Picard, R.W.: This Computer Responds to User Frustration: Theory, Design, Results, and Implications. Interacting with Computers 14(2), 119–140 (2002)CrossRefGoogle Scholar
  27. 27.
    Prendinger, H., Ishizuka, M.: The Empathic Companion: A Character-Based Interface that Addresses Users’ Affective States. Applied Artificial Intelligence 19(3-4), 267–285 (2005)CrossRefGoogle Scholar
  28. 28.
    Bickmore, T., Picard, R.W.: Establishing and Maintaining Long-Term Human-Computer Relationships. Transactions on Computer-Human Interaction 12(2), 293–327 (2004)CrossRefGoogle Scholar
  29. 29.
    Baylor, A.: The Impact of Pedagogical Agent Image on Affective Outcomes. In: Proceedings of Workshop on Affective Interactions: Computers in the Affective Loop, International Conference on Intelligent User Interfaces, San Diego, CA (2005)Google Scholar
  30. 30.
    Dweck, C.: Messages that motivate: How praise molds students’ beliefs, motivation, and performance (In Surprising Ways). In: Aronson, J. (ed.) Improving academic achievement. Academic Press, New York (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Beverly Woolf
    • 1
  • Toby Dragon
    • 1
  • Ivon Arroyo
    • 1
  • David Cooper
    • 1
  • Winslow Burleson
    • 2
  • Kasia Muldner
    • 2
  1. 1.Department of Computer ScienceUniversity of Massachusetts AmherstAmherstUSA
  2. 2.School of Computer Science and Informatics/Arts, Media and EngineeringArizona State UniversityTempeUSA

Personalised recommendations