User Modeling and User-Adapted Interaction

, Volume 27, Issue 1, pp 119–158 | Cite as

Affective learning: improving engagement and enhancing learning with affect-aware feedback

  • Beate GrawemeyerEmail author
  • Manolis MavrikisEmail author
  • Wayne Holmes
  • Sergio Gutiérrez-Santos
  • Michael Wiedmann
  • Nikol Rummel


This paper describes the design and ecologically valid evaluation of a learner model that lies at the heart of an intelligent learning environment called iTalk2Learn. A core objective of the learner model is to adapt formative feedback based on students’ affective states. Types of adaptation include what type of formative feedback should be provided and how it should be presented. Two Bayesian networks trained with data gathered in a series of Wizard-of-Oz studies are used for the adaptation process. This paper reports results from a quasi-experimental evaluation, in authentic classroom settings, which compared a version of iTalk2Learn that adapted feedback based on students’ affective states as they were talking aloud with the system (the affect condition) with one that provided feedback based only on the students’ performance (the non-affect condition). Our results suggest that affect-aware support contributes to reducing boredom and off-task behavior, and may have an effect on learning. We discuss the internal and ecological validity of the study, in light of pedagogical considerations that informed the design of the two conditions. Overall, the results of the study have implications both for the design of educational technology and for classroom approaches to teaching, because they highlight the important role that affect-aware modelling plays in the adaptive delivery of formative feedback to support learning.


Affective learning Bayesian networks Formative feedback Learner modelling 



This research received funding from the European Union Seventh Framework Programme (FP7/2007–2013) under Grant Agreement No. 318051—iTalk2Learn project. Thanks to all our iTalk2Learn colleagues for their support and ideas.


  1. Acee, T.W., Kim, H., Kim, H.J., Kim, J.I., Chu, H.N.R., Kim, M., Cho, Y.J., Wicker, F.W.: Academic boredom in under- and over-challenging situations. Contemp. Educ. Psychol. 35(1), 17–27 (2010)CrossRefGoogle Scholar
  2. Askeland, M.: Sound-based strategy training in multiplication. Eur. J. Spec. Needs Educ. 27(2), 201–217 (2012)CrossRefGoogle Scholar
  3. Azevedo, R., Witherspoon, A., Chauncey, A., Burkett, C., Fike, A: MetaTutor: a metacognitive tool for enhancing self-regulated learning. In: Proceedings of the AAAI Fall Symposium on Cognitive and Metacognitive Educational Systems, Association for the Advancement of Artificial Intelligence (AAAI) Press., Menlo Park, CA, USA, pp 14–19 (2009)Google Scholar
  4. Baker, R.S.J.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), pp. 1059–1068 (2007)Google Scholar
  5. Baker, R.S.J., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. Int. J. Hum. Comput. Stud. 68(4), 223–241 (2010)CrossRefGoogle Scholar
  6. Basu, S., Biswas, G., Kinnebrew, J.: Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model User-Adap. Inter. (2017). doi: 10.1007/s11257-017-9187-0
  7. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)CrossRefGoogle Scholar
  8. Burleson, W., Picard, R.: Evidence for gender specific approaches to the development of emotionally intelligent learning companions. IEEE Intell. Syst. 22(4), 62–69 (2007)CrossRefGoogle Scholar
  9. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.: Highlighting interventions and user differences: Informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), pp. 1835–1844 (2014)Google Scholar
  10. Chi, M.T.H.: Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In: Glaser, R. (ed.) Advances in Instructional Psychology, pp. 161–238. Lawrence Erbaum Associates, Mahwah (2000)Google Scholar
  11. Conati, C., MacLaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User Adapt. Interact. 19, 267–303 (2009)CrossRefGoogle Scholar
  12. Cowie, R., Douglas-Cowie, E., Apolloni, B., Romano, A., Fellenz, W.: What a neural net needs to know about emotion words. In: Mastorakis, N. (ed.) Computational Intelligence and Applications, pp. 109–114. World Scientific Engineering Society (1999)Google Scholar
  13. Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper and Row, New York (1990)Google Scholar
  14. del Soldato, T., du Boulay, B.: Implementation of motivational tactics in tutoring systems. J. Artif. Intell. Educ. 6(4), 337–378 (1995)Google Scholar
  15. D’Mello, S., Graesser, A.: AutoTutor and affective AutoTutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans. Interact. Intell. Syst. 2(4), 1–38 (2013)CrossRefGoogle Scholar
  16. D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., Graesser, A.: A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In: 10th International Conference on Intelligent Tutoring Systems (ITS 2010) (2010)Google Scholar
  17. D’Mello, S.K., Graesser, A.C.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User Adapt. Interact. 20(2), 147–187 (2010)CrossRefGoogle Scholar
  18. D’Mello, S.K., Kory, J.: A review and meta-analysis of multimodal affect detection systems. ACM Comput. Surv. 47(3), 43:1–43:36 (2015)Google Scholar
  19. D’Mello, S.K., Craig, S.D., Gholson, B., Franklin, S., Picard, R.W., Graesser, A.C.: Integrating affect sensors in an intelligent tutoring system. In: Affective Interactions: The Computer in the Affective Loop Workshop at the International Conference on Intelligent User Interfaces, pp. 7–13 (2005)Google Scholar
  20. D’Mello, S.K., Lehman, B., Pekrun, R., Graesser, A.C.: Confusion can be beneficial for learning. Learn. Instr. 29(1), 153–170 (2014)CrossRefGoogle Scholar
  21. Epp, C., Lippold, M., Mandryk, R.L.: Identifying emotional states using keystroke dynamics. In: Annual Conference on Human Factors in Computing Systems, pp. 715–724 (2011)Google Scholar
  22. Forbes-Riley, K., Litman, D.: Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Commun. 53(9–10), 1115–1136 (2011a)CrossRefGoogle Scholar
  23. Forbes-Riley, K., Litman, D.: Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Comput. Speech Lang. 25(1), 105–126 (2011b)CrossRefGoogle Scholar
  24. Grawemeyer, B., Mavrikis, M., Hansen, A., Mazziotti, C., Gutiérrez-Santos, S.: Employing speech to contribute to modelling and adapting to students’ affective states. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 568–569 (2014)Google Scholar
  25. Grawemeyer, B., Holmes, W., Gutiérrez-Santos, S., Hansen, A., Loibl, K., Mavrikis, M.: Light-bulb moment? towards adaptive presentation of feedback based on students’ affective state. In: Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ‘15). ACM, New York, NY, USA, pp. 400–404 (2015a)Google Scholar
  26. Grawemeyer, B., Mavrikis, M., Holmes, W., Hansen, A., Loibl, K., Gutiérrez-Santos, S.: Affect matters: Exploring the impact of feedback during mathematical tasks in an exploratory environment. In: Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED 2015). Lecture Notes in Computer Science. Springer, Berlin, pp. 595–599 (2015b)Google Scholar
  27. Gutiérrez-Santos, S., Mavrikis, M., Magoulas, G.: A separation of concerns for engineering intelligent support for exploratory learning environments. J. Res. Pract. Inf. Technol. 44(3), 347–360 (2012)Google Scholar
  28. Hattie, J., Timperley, H.: The power of feedback. Revi. Educ. Res. 77(1), 81–112 (2007)CrossRefGoogle Scholar
  29. Hayes, A.F.: Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach. Guilford Press, New York (2013)Google Scholar
  30. Holmes, W., Mavrikis, M., Hansen, A., Grawemeyer, B.: Purpose and Level of Feedback in an Exploratory Learning Environment for Fractions. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) Artificial Intelligence in Education. Lecture Notes in Computer Science. Springer, Berlin, pp. 620–623. (2015). doi: 10.1007/978-3-319-19773-9_76
  31. Janning, R., Schatten, C., Schmidt-Thieme, L.: Feature analysis for affect recognition supporting task sequencing in adaptive intelligent tutoring systems. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 179–192 (2014)Google Scholar
  32. Janning, R., Schatten, C., Schmidt-Thieme, L.: Perceived task-difficulty recognition from log-file information for the use in adaptive intelligent tutoring systems. Int. J. Artif. Intell. Educ. 26(3), 855–876 (2016)CrossRefGoogle Scholar
  33. Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of the 12th International Conference of Intelligent Tutoring Systems (ITS 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 29–38 (2014)Google Scholar
  34. Jiang, D., Cui, Y., Zhang, F.P.X., Ganzalez, I., Sahli, H.: Audio visual emotion recognition based on triple-stream dynamic Bayesian network models. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Lecture Notes in Computer Science. Springer, Berlin, pp. 609–618 (2011)Google Scholar
  35. Kaliouby, R.E., Robinson, P.: Real-time inference of complex mental states from facial expressions and head gestures. In: Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops 2004) (2004)Google Scholar
  36. Kirschner, P., Sweller, J., Clark, R.E.: Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educ Psychol. 41(2), 75–86 (2006)CrossRefGoogle Scholar
  37. Kort, B., Reilly, R., Picard, R.W.: An affective model of the interplay between emotions and learning. In: Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT ‘01). IEEE Computer Society, pp. 43–46 (2001)Google Scholar
  38. Lang, P.J., Greenwald, M.K., Bradley, M.M., Hamm, A.O.: Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3), 261–273 (1993)CrossRefGoogle Scholar
  39. Long, Y., Aleven, V.: Enhancing learning outcomes through self-regulated learning support with an open learner model. User Model User-Adap. Inter. (2017). doi: 10.1007/s11257-016-9186-6
  40. Mavrikis, M., Maciocia, A., Lee, J.: Towards predictive modelling of student affect from web-based interactions. In: Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work, pp. 169–176. IOS Press, Amsterdam (2007).
  41. Mavrikis, M., Geraniou, E., Noss, R., Hoyles, C.: Revisiting pedagogic strategies for supporting students’ learning in mathematical microworlds. In: Proceedings of the International Workshop on Intelligent Support for Exploratory Environments at EC-TEL, vol. 8, pp. 41–50 (2008)Google Scholar
  42. Mavrikis, M., Gutiérrez-Santos, S., Geraniou, E., Noss, R.: Design requirements, student perception indicators and validation metrics for intelligent exploratory learning environments. Pers. Ubiquitous Comput. 17(8), 1605–1620 (2013)CrossRefGoogle Scholar
  43. Mavrikis, M., Grawemeyer, B., Hansen, A., Gutiérrez-Santos, S.: Exploring the potential of speech recognition to support problem solving and reflection—wizards go to school in the elementary maths classroom. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 263–276 (2014)Google Scholar
  44. Mazziotti, C., Holmes, W., Wiedmann, M., Loibl, K., Rummel, N., Mavrikis, M., Hansen, A., Grawemeyer, B.: Robust student knowledge: adapting to individual student needs as they explore the concepts and practice the procedures of fractions. In: Workshop on Intelligent Support in Exploratory and Open-Ended Learning Environments Learning Analytics for Project Based and Experiential Learning Scenarios at the 17th International Conference on Artificial Intelligence in Education (AIED 2015), pp. 32–40 (2015)Google Scholar
  45. Nasoz, F., Alvarez, K., Lisetti, C.L., Finkelstein, N.: Emotion recognition from physiological signals for presence technologies. Int. J. Cogn. Technol. Work 6(1), 4–14 (2003)CrossRefGoogle Scholar
  46. Ocumpaugh, J., Baker, R.S.J., Rodrigo, M.M.T.: Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual Version 1.0. Technical Report, EdLab., New York, Ateneo Laboratory for the Learning Sciences, Manila (2012)Google Scholar
  47. Paleari, M., Benmokhtar, R., Huet, B.: Evidence theory-based multimodal emotion recognition. In: Proceedings of the 15th International Multimedia Modeling Conference (MMM 2009). Lecture Notes in Computer Science, Springer, Berlin, pp. 435–446 (2009)Google Scholar
  48. Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20, 383–453 (2010)CrossRefGoogle Scholar
  49. Pekrun, R.: The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ. Psychol. Rev. 18(4), 315–341 (2006)CrossRefGoogle Scholar
  50. Piaget, J.: Organization and pathology of thought: Selected sources. Principal Factors Determining Intellectual Evolution from Childhood to Adult Life, pp. 154–175. Columbia University Press, New York (1951)Google Scholar
  51. Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User Adapt. Interact. 18(1), 125–173 (2008)CrossRefGoogle Scholar
  52. Rowe, J.P., Mott, B.W., McQuiggan, S.W., Robison, J.L., Lee, S., Lester, J.C.: Crystal island: a narrative-centered learning environment for eighth grade microbiology. In: Workshop on Intelligent Educational Games at the 14th International Conference on Artificial Intelligence in Education (AIED 2009), pp. 11–19 (2009)Google Scholar
  53. Rummel, N., Mavrikis, M., Wiedmann, M., Loibl, K., Mazziotti, C., Holmes, W., Hansen, A.: Combining exploratory learning with structured practice to foster conceptual and procedural fractions knowledge. In: Proceedings of the 12th International Conference of the Learning Sciences (ICLS 2016), pp. 58–65 (2016)Google Scholar
  54. Sail-Labs: SAIL LABS Technology GmbH (2016).
  55. Santos, O.C., Saneiro, M., Salmeron-Majadas, S., Boticario, J.G.: A methodological approach to elicit affective educational recommendations. In: Proceedings of the 14th International Conference on Advanced Learning Technologies (ICALT 2014), pp. 529–533 (2014)Google Scholar
  56. Schuller, B., Müller, R., Lang, M., Rigoll, G.: Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensemble. In: Proceedings of the 9th European Conference on Speech Communication and Technology (Interspeech 2005), pp. 805–808 (2005)Google Scholar
  57. Shen, L., Wang, M., Shen, R.: Affective e-learning: using emotional data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12(2), 176–189 (2009)Google Scholar
  58. Shute, V.J.: Focus on formative feedback. Rev. Educ. Res. 78(1), 153–189 (2008)CrossRefGoogle Scholar
  59. Sweller, J., van Merrienboer, J.G., Paas, G.W.: Cognitive architecture and instructional design. Educ. Psychol. Rev. 10, 251–296 (1998)CrossRefGoogle Scholar
  60. Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The mars and venus effect: the influence of user gender on the effectiveness of adaptive task support. In: Proceedings of the 23rd International Conference on User Modeling, Adaptation and Personalization (UMAP 2015), pp. 265–276 (2015)Google Scholar
  61. Vanlehn, K.: The behavior of tutoring systems. Int. J. Artif. Intell. Educ. 16(3):227–265. (2006)
  62. VanLehn, K., Burleson, W., Girard, S., Chavez-Echeagaray, E., Gonzalez-Sanchez, J., Hidalgo-Pontet, Y., Zhang, L.: The affective meta-tutoring project: lessons learned. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), pp. 84–93 (2014)Google Scholar
  63. Vogt, T., André, E.: Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proceedings of the 2005 IEEE International Conference on Multimedia and Expo (ICME 2005), pp. 474–477 (2005)Google Scholar
  64. Vyzas, E., Picard, R.W.: Affective pattern classification. In: Proceedings of the AAAI Fall Symposium Series: Emotional and Intelligent: The Tangled Knot of Cognition, pp. 23–25 (1998)Google Scholar
  65. Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., Narayanan, S.S.: Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. In: Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH ‘10), pp. 2362–2365 (2010)Google Scholar
  66. Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. Int. J. Learn. Technol. 4(3–4), 129–164 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2017

Authors and Affiliations

  • Beate Grawemeyer
    • 1
    Email author
  • Manolis Mavrikis
    • 2
    Email author
  • Wayne Holmes
    • 3
  • Sergio Gutiérrez-Santos
    • 1
  • Michael Wiedmann
    • 4
  • Nikol Rummel
    • 4
  1. 1.BBK Knowledge Lab, Department of Computer Science and Information SystemsBirkbeck, University of LondonLondonUK
  2. 2.UCL Knowledge Lab, UCL Institute of EducationUniversity College LondonLondonUK
  3. 3.Institute of Educational TechnologyThe Open UniversityMilton KeynesUK
  4. 4.Institute of Educational ResearchRuhr-Universität BochumBochumGermany

Personalised recommendations