Skip to main content

Advertisement

SpringerLink
Log in

Affective learning: improving engagement and enhancing learning with affect-aware feedback

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

This paper describes the design and ecologically valid evaluation of a learner model that lies at the heart of an intelligent learning environment called iTalk2Learn. A core objective of the learner model is to adapt formative feedback based on students’ affective states. Types of adaptation include what type of formative feedback should be provided and how it should be presented. Two Bayesian networks trained with data gathered in a series of Wizard-of-Oz studies are used for the adaptation process. This paper reports results from a quasi-experimental evaluation, in authentic classroom settings, which compared a version of iTalk2Learn that adapted feedback based on students’ affective states as they were talking aloud with the system (the affect condition) with one that provided feedback based only on the students’ performance (the non-affect condition). Our results suggest that affect-aware support contributes to reducing boredom and off-task behavior, and may have an effect on learning. We discuss the internal and ecological validity of the study, in light of pedagogical considerations that informed the design of the two conditions. Overall, the results of the study have implications both for the design of educational technology and for classroom approaches to teaching, because they highlight the important role that affect-aware modelling plays in the adaptive delivery of formative feedback to support learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Acee, T.W., Kim, H., Kim, H.J., Kim, J.I., Chu, H.N.R., Kim, M., Cho, Y.J., Wicker, F.W.: Academic boredom in under- and over-challenging situations. Contemp. Educ. Psychol. 35(1), 17–27 (2010)

    Article  Google Scholar 

  • Askeland, M.: Sound-based strategy training in multiplication. Eur. J. Spec. Needs Educ. 27(2), 201–217 (2012)

    Article  Google Scholar 

  • Azevedo, R., Witherspoon, A., Chauncey, A., Burkett, C., Fike, A: MetaTutor: a metacognitive tool for enhancing self-regulated learning. In: Proceedings of the AAAI Fall Symposium on Cognitive and Metacognitive Educational Systems, Association for the Advancement of Artificial Intelligence (AAAI) Press., Menlo Park, CA, USA, pp 14–19 (2009)

  • Baker, R.S.J.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), pp. 1059–1068 (2007)

  • Baker, R.S.J., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. Int. J. Hum. Comput. Stud. 68(4), 223–241 (2010)

    Article  Google Scholar 

  • Basu, S., Biswas, G., Kinnebrew, J.: Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model User-Adap. Inter. (2017). doi:10.1007/s11257-017-9187-0

  • Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)

    Article  Google Scholar 

  • Burleson, W., Picard, R.: Evidence for gender specific approaches to the development of emotionally intelligent learning companions. IEEE Intell. Syst. 22(4), 62–69 (2007)

    Article  Google Scholar 

  • Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.: Highlighting interventions and user differences: Informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), pp. 1835–1844 (2014)

  • Chi, M.T.H.: Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In: Glaser, R. (ed.) Advances in Instructional Psychology, pp. 161–238. Lawrence Erbaum Associates, Mahwah (2000)

    Google Scholar 

  • Conati, C., MacLaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User Adapt. Interact. 19, 267–303 (2009)

    Article  Google Scholar 

  • Cowie, R., Douglas-Cowie, E., Apolloni, B., Romano, A., Fellenz, W.: What a neural net needs to know about emotion words. In: Mastorakis, N. (ed.) Computational Intelligence and Applications, pp. 109–114. World Scientific Engineering Society (1999)

  • Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper and Row, New York (1990)

    Google Scholar 

  • del Soldato, T., du Boulay, B.: Implementation of motivational tactics in tutoring systems. J. Artif. Intell. Educ. 6(4), 337–378 (1995)

    Google Scholar 

  • D’Mello, S., Graesser, A.: AutoTutor and affective AutoTutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans. Interact. Intell. Syst. 2(4), 1–38 (2013)

    Article  Google Scholar 

  • D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., Graesser, A.: A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In: 10th International Conference on Intelligent Tutoring Systems (ITS 2010) (2010)

  • D’Mello, S.K., Graesser, A.C.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User Adapt. Interact. 20(2), 147–187 (2010)

    Article  Google Scholar 

  • D’Mello, S.K., Kory, J.: A review and meta-analysis of multimodal affect detection systems. ACM Comput. Surv. 47(3), 43:1–43:36 (2015)

    Google Scholar 

  • D’Mello, S.K., Craig, S.D., Gholson, B., Franklin, S., Picard, R.W., Graesser, A.C.: Integrating affect sensors in an intelligent tutoring system. In: Affective Interactions: The Computer in the Affective Loop Workshop at the International Conference on Intelligent User Interfaces, pp. 7–13 (2005)

  • D’Mello, S.K., Lehman, B., Pekrun, R., Graesser, A.C.: Confusion can be beneficial for learning. Learn. Instr. 29(1), 153–170 (2014)

    Article  Google Scholar 

  • Epp, C., Lippold, M., Mandryk, R.L.: Identifying emotional states using keystroke dynamics. In: Annual Conference on Human Factors in Computing Systems, pp. 715–724 (2011)

  • Forbes-Riley, K., Litman, D.: Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Commun. 53(9–10), 1115–1136 (2011a)

    Article  Google Scholar 

  • Forbes-Riley, K., Litman, D.: Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Comput. Speech Lang. 25(1), 105–126 (2011b)

    Article  Google Scholar 

  • Grawemeyer, B., Mavrikis, M., Hansen, A., Mazziotti, C., Gutiérrez-Santos, S.: Employing speech to contribute to modelling and adapting to students’ affective states. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 568–569 (2014)

  • Grawemeyer, B., Holmes, W., Gutiérrez-Santos, S., Hansen, A., Loibl, K., Mavrikis, M.: Light-bulb moment? towards adaptive presentation of feedback based on students’ affective state. In: Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ‘15). ACM, New York, NY, USA, pp. 400–404 (2015a)

  • Grawemeyer, B., Mavrikis, M., Holmes, W., Hansen, A., Loibl, K., Gutiérrez-Santos, S.: Affect matters: Exploring the impact of feedback during mathematical tasks in an exploratory environment. In: Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED 2015). Lecture Notes in Computer Science. Springer, Berlin, pp. 595–599 (2015b)

  • Gutiérrez-Santos, S., Mavrikis, M., Magoulas, G.: A separation of concerns for engineering intelligent support for exploratory learning environments. J. Res. Pract. Inf. Technol. 44(3), 347–360 (2012)

    Google Scholar 

  • Hattie, J., Timperley, H.: The power of feedback. Revi. Educ. Res. 77(1), 81–112 (2007)

    Article  Google Scholar 

  • Hayes, A.F.: Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach. Guilford Press, New York (2013)

    Google Scholar 

  • Holmes, W., Mavrikis, M., Hansen, A., Grawemeyer, B.: Purpose and Level of Feedback in an Exploratory Learning Environment for Fractions. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) Artificial Intelligence in Education. Lecture Notes in Computer Science. Springer, Berlin, pp. 620–623. (2015). doi:10.1007/978-3-319-19773-9_76

  • Janning, R., Schatten, C., Schmidt-Thieme, L.: Feature analysis for affect recognition supporting task sequencing in adaptive intelligent tutoring systems. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 179–192 (2014)

  • Janning, R., Schatten, C., Schmidt-Thieme, L.: Perceived task-difficulty recognition from log-file information for the use in adaptive intelligent tutoring systems. Int. J. Artif. Intell. Educ. 26(3), 855–876 (2016)

    Article  Google Scholar 

  • Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of the 12th International Conference of Intelligent Tutoring Systems (ITS 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 29–38 (2014)

  • Jiang, D., Cui, Y., Zhang, F.P.X., Ganzalez, I., Sahli, H.: Audio visual emotion recognition based on triple-stream dynamic Bayesian network models. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Lecture Notes in Computer Science. Springer, Berlin, pp. 609–618 (2011)

  • Kaliouby, R.E., Robinson, P.: Real-time inference of complex mental states from facial expressions and head gestures. In: Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops 2004) (2004)

  • Kirschner, P., Sweller, J., Clark, R.E.: Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educ Psychol. 41(2), 75–86 (2006)

    Article  Google Scholar 

  • Kort, B., Reilly, R., Picard, R.W.: An affective model of the interplay between emotions and learning. In: Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT ‘01). IEEE Computer Society, pp. 43–46 (2001)

  • Lang, P.J., Greenwald, M.K., Bradley, M.M., Hamm, A.O.: Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3), 261–273 (1993)

    Article  Google Scholar 

  • Long, Y., Aleven, V.: Enhancing learning outcomes through self-regulated learning support with an open learner model. User Model User-Adap. Inter. (2017). doi:10.1007/s11257-016-9186-6

  • Mavrikis, M., Maciocia, A., Lee, J.: Towards predictive modelling of student affect from web-based interactions. In: Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work, pp. 169–176. IOS Press, Amsterdam (2007). http://dl.acm.org/citation.cfm?id=1563601.1563632

  • Mavrikis, M., Geraniou, E., Noss, R., Hoyles, C.: Revisiting pedagogic strategies for supporting students’ learning in mathematical microworlds. In: Proceedings of the International Workshop on Intelligent Support for Exploratory Environments at EC-TEL, vol. 8, pp. 41–50 (2008)

  • Mavrikis, M., Gutiérrez-Santos, S., Geraniou, E., Noss, R.: Design requirements, student perception indicators and validation metrics for intelligent exploratory learning environments. Pers. Ubiquitous Comput. 17(8), 1605–1620 (2013)

    Article  Google Scholar 

  • Mavrikis, M., Grawemeyer, B., Hansen, A., Gutiérrez-Santos, S.: Exploring the potential of speech recognition to support problem solving and reflection—wizards go to school in the elementary maths classroom. In: Proceedings of the 9th European Conference on Technology Enhanced Learning (EC-TEL 2014). Lecture Notes in Computer Science. Springer, Berlin, pp. 263–276 (2014)

  • Mazziotti, C., Holmes, W., Wiedmann, M., Loibl, K., Rummel, N., Mavrikis, M., Hansen, A., Grawemeyer, B.: Robust student knowledge: adapting to individual student needs as they explore the concepts and practice the procedures of fractions. In: Workshop on Intelligent Support in Exploratory and Open-Ended Learning Environments Learning Analytics for Project Based and Experiential Learning Scenarios at the 17th International Conference on Artificial Intelligence in Education (AIED 2015), pp. 32–40 (2015)

  • Nasoz, F., Alvarez, K., Lisetti, C.L., Finkelstein, N.: Emotion recognition from physiological signals for presence technologies. Int. J. Cogn. Technol. Work 6(1), 4–14 (2003)

    Article  Google Scholar 

  • Ocumpaugh, J., Baker, R.S.J., Rodrigo, M.M.T.: Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual Version 1.0. Technical Report, EdLab., New York, Ateneo Laboratory for the Learning Sciences, Manila (2012)

  • Paleari, M., Benmokhtar, R., Huet, B.: Evidence theory-based multimodal emotion recognition. In: Proceedings of the 15th International Multimedia Modeling Conference (MMM 2009). Lecture Notes in Computer Science, Springer, Berlin, pp. 435–446 (2009)

  • Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20, 383–453 (2010)

    Article  Google Scholar 

  • Pekrun, R.: The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ. Psychol. Rev. 18(4), 315–341 (2006)

    Article  Google Scholar 

  • Piaget, J.: Organization and pathology of thought: Selected sources. Principal Factors Determining Intellectual Evolution from Childhood to Adult Life, pp. 154–175. Columbia University Press, New York (1951)

    Google Scholar 

  • Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User Adapt. Interact. 18(1), 125–173 (2008)

    Article  Google Scholar 

  • Rowe, J.P., Mott, B.W., McQuiggan, S.W., Robison, J.L., Lee, S., Lester, J.C.: Crystal island: a narrative-centered learning environment for eighth grade microbiology. In: Workshop on Intelligent Educational Games at the 14th International Conference on Artificial Intelligence in Education (AIED 2009), pp. 11–19 (2009)

  • Rummel, N., Mavrikis, M., Wiedmann, M., Loibl, K., Mazziotti, C., Holmes, W., Hansen, A.: Combining exploratory learning with structured practice to foster conceptual and procedural fractions knowledge. In: Proceedings of the 12th International Conference of the Learning Sciences (ICLS 2016), pp. 58–65 (2016)

  • Sail-Labs: SAIL LABS Technology GmbH (2016). http://www.sail-labs.com

  • Santos, O.C., Saneiro, M., Salmeron-Majadas, S., Boticario, J.G.: A methodological approach to elicit affective educational recommendations. In: Proceedings of the 14th International Conference on Advanced Learning Technologies (ICALT 2014), pp. 529–533 (2014)

  • Schuller, B., Müller, R., Lang, M., Rigoll, G.: Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensemble. In: Proceedings of the 9th European Conference on Speech Communication and Technology (Interspeech 2005), pp. 805–808 (2005)

  • Shen, L., Wang, M., Shen, R.: Affective e-learning: using emotional data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12(2), 176–189 (2009)

    Google Scholar 

  • Shute, V.J.: Focus on formative feedback. Rev. Educ. Res. 78(1), 153–189 (2008)

    Article  Google Scholar 

  • Sweller, J., van Merrienboer, J.G., Paas, G.W.: Cognitive architecture and instructional design. Educ. Psychol. Rev. 10, 251–296 (1998)

    Article  Google Scholar 

  • Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The mars and venus effect: the influence of user gender on the effectiveness of adaptive task support. In: Proceedings of the 23rd International Conference on User Modeling, Adaptation and Personalization (UMAP 2015), pp. 265–276 (2015)

  • Vanlehn, K.: The behavior of tutoring systems. Int. J. Artif. Intell. Educ. 16(3):227–265. http://dl.acm.org/citation.cfm?id=1435351.1435353 (2006)

  • VanLehn, K., Burleson, W., Girard, S., Chavez-Echeagaray, E., Gonzalez-Sanchez, J., Hidalgo-Pontet, Y., Zhang, L.: The affective meta-tutoring project: lessons learned. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), pp. 84–93 (2014)

  • Vogt, T., André, E.: Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proceedings of the 2005 IEEE International Conference on Multimedia and Expo (ICME 2005), pp. 474–477 (2005)

  • Vyzas, E., Picard, R.W.: Affective pattern classification. In: Proceedings of the AAAI Fall Symposium Series: Emotional and Intelligent: The Tangled Knot of Cognition, pp. 23–25 (1998)

  • Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., Narayanan, S.S.: Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. In: Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH ‘10), pp. 2362–2365 (2010)

  • Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. Int. J. Learn. Technol. 4(3–4), 129–164 (2009)

    Article  Google Scholar 

Download references

Acknowledgements

This research received funding from the European Union Seventh Framework Programme (FP7/2007–2013) under Grant Agreement No. 318051—iTalk2Learn project. Thanks to all our iTalk2Learn colleagues for their support and ideas.

Author information

Authors and Affiliations

  1. BBK Knowledge Lab, Department of Computer Science and Information Systems, Birkbeck, University of London, Malet Street, London, WC1E 7HX, UK

    Beate Grawemeyer & Sergio Gutiérrez-Santos

  2. UCL Knowledge Lab, UCL Institute of Education, University College London, 23-29 Emerald Street, London, WC1N 3QS, UK

    Manolis Mavrikis

  3. Institute of Educational Technology, The Open University, Milton Keynes, MK7 6AA, UK

    Wayne Holmes

  4. Institute of Educational Research, Ruhr-Universität Bochum, Bochum, Germany

    Michael Wiedmann & Nikol Rummel

Authors
  1. Beate Grawemeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manolis Mavrikis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wayne Holmes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sergio Gutiérrez-Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael Wiedmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nikol Rummel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Beate Grawemeyer or Manolis Mavrikis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Grawemeyer, B., Mavrikis, M., Holmes, W. et al. Affective learning: improving engagement and enhancing learning with affect-aware feedback. User Model User-Adap Inter 27, 119–158 (2017). https://doi.org/10.1007/s11257-017-9188-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-017-9188-z

Keywords

Search

Navigation