Advertisement

Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Affective learning: improving engagement and enhancing learning with affect-aware feedback

Abstract

This paper describes the design and ecologically valid evaluation of a learner model that lies at the heart of an intelligent learning environment called iTalk2Learn. A core objective of the learner model is to adapt formative feedback based on students’ affective states. Types of adaptation include what type of formative feedback should be provided and how it should be presented. Two Bayesian networks trained with data gathered in a series of Wizard-of-Oz studies are used for the adaptation process. This paper reports results from a quasi-experimental evaluation, in authentic classroom settings, which compared a version of iTalk2Learn that adapted feedback based on students’ affective states as they were talking aloud with the system (the affect condition) with one that provided feedback based only on the students’ performance (the non-affect condition). Our results suggest that affect-aware support contributes to reducing boredom and off-task behavior, and may have an effect on learning. We discuss the internal and ecological validity of the study, in light of pedagogical considerations that informed the design of the two conditions. Overall, the results of the study have implications both for the design of educational technology and for classroom approaches to teaching, because they highlight the important role that affect-aware modelling plays in the adaptive delivery of formative feedback to support learning.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

References

  1. Acee, T.W., Kim, H., Kim, H.J., Kim, J.I., Chu, H.N.R., Kim, M., Cho, Y.J., Wicker, F.W.: Academic boredom in under- and over-challenging situations. Contemp. Educ. Psychol. 35(1), 17–27 (2010)

  2. Askeland, M.: Sound-based strategy training in multiplication. Eur. J. Spec. Needs Educ. 27(2), 201–217 (2012)

  3. Azevedo, R., Witherspoon, A., Chauncey, A., Burkett, C., Fike, A: MetaTutor: a metacognitive tool for enhancing self-regulated learning. In: Proceedings of the AAAI Fall Symposium on Cognitive and Metacognitive Educational Systems, Association for the Advancement of Artificial Intelligence (AAAI) Press., Menlo Park, CA, USA, pp 14–19 (2009)

  4. Baker, R.S.J.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), pp. 1059–1068 (2007)

  5. Baker, R.S.J., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. Int. J. Hum. Comput. Stud. 68(4), 223–241 (2010)

  6. Basu, S., Biswas, G., Kinnebrew, J.: Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model User-Adap. Inter. (2017). doi:10.1007/s11257-017-9187-0

  7. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)

  8. Burleson, W., Picard, R.: Evidence for gender specific approaches to the development of emotionally intelligent learning companions. IEEE Intell. Syst. 22(4), 62–69 (2007)

  9. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.: Highlighting interventions and user differences: Informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), pp. 1835–1844 (2014)

  10. Chi, M.T.H.: Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In: Glaser, R. (ed.) Advances in Instructional Psychology, pp. 161–238. Lawrence Erbaum Associates, Mahwah (2000)

  11. Conati, C., MacLaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User Adapt. Interact. 19, 267–303 (2009)

  12. Cowie, R., Douglas-Cowie, E., Apolloni, B., Romano, A., Fellenz, W.: What a neural net needs to know about emotion words. In: Mastorakis, N. (ed.) Computational Intelligence and Applications, pp. 109–114. World Scientific Engineering Society (1999)

  13. Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper and Row, New York (1990)

  14. del Soldato, T., du Boulay, B.: Implementation of motivational tactics in tutoring systems. J. Artif. Intell. Educ. 6(4), 337–378 (1995)

  15. D’Mello, S., Graesser, A.: AutoTutor and affective AutoTutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans. Interact. Intell. Syst. 2(4), 1–38 (2013)

  16. D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., Graesser, A.: A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In: 10th International Conference on Intelligent Tutoring Systems (ITS 2010) (2010)

  17. D’Mello, S.K., Graesser, A.C.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User Adapt. Interact. 20(2), 147–187 (2010)

  18. D’Mello, S.K., Kory, J.: A review and meta-analysis of multimodal affect detection systems. ACM Comput. Surv. 47(3), 43:1–43:36 (2015)

  19. D’Mello, S.K., Craig, S.D., Gholson, B., Franklin, S., Picard, R.W., Graesser, A.C.: Integrating affect sensors in an intelligent tutoring system. In: Affective Interactions: The Computer in the Affective Loop Workshop at the International Conference on Intelligent User Interfaces, pp. 7–13 (2005)

  20. D’Mello, S.K., Lehman, B., Pekrun, R., Graesser, A.C.: Confusion can be beneficial for learning. Learn. Instr. 29(1), 153–170 (2014)

  21. Epp, C., Lippold, M., Mandryk, R.L.: Identifying emotional states using keystroke dynamics. In: Annual Conference on Human Factors in Computing Systems, pp. 715–724 (2011)

  22. Forbes-Riley, K., Litman, D.: Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Commun. 53(9–10), 1115–1136 (2011a)

  23. Forbes-Riley, K., Litman, D.: Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Comput. Speech Lang. 25(1), 105–126 (2011b)

  24. Grawemeyer, B., Mavrikis, M., Hansen, A., Mazziotti, C., Gutiérrez-Santos, S.: Employing speech to contribute to modelling and adapting to students’ affective states. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 568–569 (2014)

  25. Grawemeyer, B., Holmes, W., Gutiérrez-Santos, S., Hansen, A., Loibl, K., Mavrikis, M.: Light-bulb moment? towards adaptive presentation of feedback based on students’ affective state. In: Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ‘15). ACM, New York, NY, USA, pp. 400–404 (2015a)

  26. Grawemeyer, B., Mavrikis, M., Holmes, W., Hansen, A., Loibl, K., Gutiérrez-Santos, S.: Affect matters: Exploring the impact of feedback during mathematical tasks in an exploratory environment. In: Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED 2015). Lecture Notes in Computer Science. Springer, Berlin, pp. 595–599 (2015b)

  27. Gutiérrez-Santos, S., Mavrikis, M., Magoulas, G.: A separation of concerns for engineering intelligent support for exploratory learning environments. J. Res. Pract. Inf. Technol. 44(3), 347–360 (2012)

  28. Hattie, J., Timperley, H.: The power of feedback. Revi. Educ. Res. 77(1), 81–112 (2007)

  29. Hayes, A.F.: Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach. Guilford Press, New York (2013)

  30. Holmes, W., Mavrikis, M., Hansen, A., Grawemeyer, B.: Purpose and Level of Feedback in an Exploratory Learning Environment for Fractions. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) Artificial Intelligence in Education. Lecture Notes in Computer Science. Springer, Berlin, pp. 620–623. (2015). doi:10.1007/978-3-319-19773-9_76

  31. Janning, R., Schatten, C., Schmidt-Thieme, L.: Feature analysis for affect recognition supporting task sequencing in adaptive intelligent tutoring systems. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 179–192 (2014)

  32. Janning, R., Schatten, C., Schmidt-Thieme, L.: Perceived task-difficulty recognition from log-file information for the use in adaptive intelligent tutoring systems. Int. J. Artif. Intell. Educ. 26(3), 855–876 (2016)

  33. Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of the 12th International Conference of Intelligent Tutoring Systems (ITS 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 29–38 (2014)

  34. Jiang, D., Cui, Y., Zhang, F.P.X., Ganzalez, I., Sahli, H.: Audio visual emotion recognition based on triple-stream dynamic Bayesian network models. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Lecture Notes in Computer Science. Springer, Berlin, pp. 609–618 (2011)

  35. Kaliouby, R.E., Robinson, P.: Real-time inference of complex mental states from facial expressions and head gestures. In: Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops 2004) (2004)

  36. Kirschner, P., Sweller, J., Clark, R.E.: Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educ Psychol. 41(2), 75–86 (2006)

  37. Kort, B., Reilly, R., Picard, R.W.: An affective model of the interplay between emotions and learning. In: Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT ‘01). IEEE Computer Society, pp. 43–46 (2001)

  38. Lang, P.J., Greenwald, M.K., Bradley, M.M., Hamm, A.O.: Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3), 261–273 (1993)

  39. Long, Y., Aleven, V.: Enhancing learning outcomes through self-regulated learning support with an open learner model. User Model User-Adap. Inter. (2017). doi:10.1007/s11257-016-9186-6

  40. Mavrikis, M., Maciocia, A., Lee, J.: Towards predictive modelling of student affect from web-based interactions. In: Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work, pp. 169–176. IOS Press, Amsterdam (2007). http://dl.acm.org/citation.cfm?id=1563601.1563632

  41. Mavrikis, M., Geraniou, E., Noss, R., Hoyles, C.: Revisiting pedagogic strategies for supporting students’ learning in mathematical microworlds. In: Proceedings of the International Workshop on Intelligent Support for Exploratory Environments at EC-TEL, vol. 8, pp. 41–50 (2008)

  42. Mavrikis, M., Gutiérrez-Santos, S., Geraniou, E., Noss, R.: Design requirements, student perception indicators and validation metrics for intelligent exploratory learning environments. Pers. Ubiquitous Comput. 17(8), 1605–1620 (2013)

  43. Mavrikis, M., Grawemeyer, B., Hansen, A., Gutiérrez-Santos, S.: Exploring the potential of speech recognition to support problem solving and reflection—wizards go to school in the elementary maths classroom. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 263–276 (2014)

  44. Mazziotti, C., Holmes, W., Wiedmann, M., Loibl, K., Rummel, N., Mavrikis, M., Hansen, A., Grawemeyer, B.: Robust student knowledge: adapting to individual student needs as they explore the concepts and practice the procedures of fractions. In: Workshop on Intelligent Support in Exploratory and Open-Ended Learning Environments Learning Analytics for Project Based and Experiential Learning Scenarios at the 17th International Conference on Artificial Intelligence in Education (AIED 2015), pp. 32–40 (2015)

  45. Nasoz, F., Alvarez, K., Lisetti, C.L., Finkelstein, N.: Emotion recognition from physiological signals for presence technologies. Int. J. Cogn. Technol. Work 6(1), 4–14 (2003)

  46. Ocumpaugh, J., Baker, R.S.J., Rodrigo, M.M.T.: Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual Version 1.0. Technical Report, EdLab., New York, Ateneo Laboratory for the Learning Sciences, Manila (2012)

  47. Paleari, M., Benmokhtar, R., Huet, B.: Evidence theory-based multimodal emotion recognition. In: Proceedings of the 15th International Multimedia Modeling Conference (MMM 2009). Lecture Notes in Computer Science, Springer, Berlin, pp. 435–446 (2009)

  48. Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20, 383–453 (2010)

  49. Pekrun, R.: The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ. Psychol. Rev. 18(4), 315–341 (2006)

  50. Piaget, J.: Organization and pathology of thought: Selected sources. Principal Factors Determining Intellectual Evolution from Childhood to Adult Life, pp. 154–175. Columbia University Press, New York (1951)

  51. Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User Adapt. Interact. 18(1), 125–173 (2008)

  52. Rowe, J.P., Mott, B.W., McQuiggan, S.W., Robison, J.L., Lee, S., Lester, J.C.: Crystal island: a narrative-centered learning environment for eighth grade microbiology. In: Workshop on Intelligent Educational Games at the 14th International Conference on Artificial Intelligence in Education (AIED 2009), pp. 11–19 (2009)

  53. Rummel, N., Mavrikis, M., Wiedmann, M., Loibl, K., Mazziotti, C., Holmes, W., Hansen, A.: Combining exploratory learning with structured practice to foster conceptual and procedural fractions knowledge. In: Proceedings of the 12th International Conference of the Learning Sciences (ICLS 2016), pp. 58–65 (2016)

  54. Sail-Labs: SAIL LABS Technology GmbH (2016). http://www.sail-labs.com

  55. Santos, O.C., Saneiro, M., Salmeron-Majadas, S., Boticario, J.G.: A methodological approach to elicit affective educational recommendations. In: Proceedings of the 14th International Conference on Advanced Learning Technologies (ICALT 2014), pp. 529–533 (2014)

  56. Schuller, B., Müller, R., Lang, M., Rigoll, G.: Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensemble. In: Proceedings of the 9th European Conference on Speech Communication and Technology (Interspeech 2005), pp. 805–808 (2005)

  57. Shen, L., Wang, M., Shen, R.: Affective e-learning: using emotional data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12(2), 176–189 (2009)

  58. Shute, V.J.: Focus on formative feedback. Rev. Educ. Res. 78(1), 153–189 (2008)

  59. Sweller, J., van Merrienboer, J.G., Paas, G.W.: Cognitive architecture and instructional design. Educ. Psychol. Rev. 10, 251–296 (1998)

  60. Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The mars and venus effect: the influence of user gender on the effectiveness of adaptive task support. In: Proceedings of the 23rd International Conference on User Modeling, Adaptation and Personalization (UMAP 2015), pp. 265–276 (2015)

  61. Vanlehn, K.: The behavior of tutoring systems. Int. J. Artif. Intell. Educ. 16(3):227–265. http://dl.acm.org/citation.cfm?id=1435351.1435353 (2006)

  62. VanLehn, K., Burleson, W., Girard, S., Chavez-Echeagaray, E., Gonzalez-Sanchez, J., Hidalgo-Pontet, Y., Zhang, L.: The affective meta-tutoring project: lessons learned. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), pp. 84–93 (2014)

  63. Vogt, T., André, E.: Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proceedings of the 2005 IEEE International Conference on Multimedia and Expo (ICME 2005), pp. 474–477 (2005)

  64. Vyzas, E., Picard, R.W.: Affective pattern classification. In: Proceedings of the AAAI Fall Symposium Series: Emotional and Intelligent: The Tangled Knot of Cognition, pp. 23–25 (1998)

  65. Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., Narayanan, S.S.: Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. In: Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH ‘10), pp. 2362–2365 (2010)

  66. Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. Int. J. Learn. Technol. 4(3–4), 129–164 (2009)

Download references

Acknowledgements

This research received funding from the European Union Seventh Framework Programme (FP7/2007–2013) under Grant Agreement No. 318051—iTalk2Learn project. Thanks to all our iTalk2Learn colleagues for their support and ideas.

Author information

Correspondence to Beate Grawemeyer or Manolis Mavrikis.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Grawemeyer, B., Mavrikis, M., Holmes, W. et al. Affective learning: improving engagement and enhancing learning with affect-aware feedback. User Model User-Adap Inter 27, 119–158 (2017). https://doi.org/10.1007/s11257-017-9188-z

Download citation

Keywords

  • Affective learning
  • Bayesian networks
  • Formative feedback
  • Learner modelling