Skip to main content
SpringerLink
Log in

Uses of Physiological Monitoring in Intelligent Learning Environments: A Review of Research, Evidence, and Technologies

  • Chapter
  • First Online:
Mind, Brain and Technology

Abstract

Two of the most important benefits of using computer-based learning environments are that (1) they allow for interactive learning experiences and (2) it becomes possible to assess learning at a fine level of granularity based on learner actions, choices, and performance. Using these assessments, modern systems can deliver tailored pedagogical interventions, such as through feedback or adjustment of problem difficulty, in order to enhance learning. To magnify their power to promote learning, researchers have also sought to capture additional information, such as physiological data, to make even finer-grained and nuanced pedagogical decisions. A wide range of technologies have been explored, such as depth-sensing cameras (e.g., Kinect), electrodermal activity (EDA) sensors, electroencephalography (EEG), posture/seat detectors, eye- and head-tracking cameras, and mouse-pressure sensors, among many others. This chapter provides an overview and examples of how these techniques have been used by educational technology researchers. We focus on how these additional inputs can improve technology-based pedagogical decision-making to support cognitive (i.e., knowledge and information processing), affective (i.e., emotions, including motivational factors), and metacognitive (i.e., attention and self-regulatory behaviors) aspects of learning. Though not a comprehensive review, we take a selective look at classic and recent developments in this area. The chapter concludes with a brief discussion of emerging topics and key open questions related to the use of physiological tracking in the context of education and learning with technology.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://nevermindgame.com/

References

  • Anderson, J. R., Betts, S., Ferris, J. L., & Fincham, J. M. (2010). Neural imaging to track mental states while using an intelligent tutoring system. Proceedings of the National Academy of Sciences, 107(15), 7018–7023. https://doi.org/10.1073/pnas.1000942107

    Article  Google Scholar 

  • Anderson, J. R., Betts, S., Ferris, J. L., & Fincham, J. M. (2012). Tracking children’s mental states while solving algebra equations. Human Brain Mapping, 33(11), 2650–2665.

    Article  Google Scholar 

  • Anderson, J. R., Corbett, A., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207.

    Article  Google Scholar 

  • Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of eductional outcomes. New York: Longman.

    Google Scholar 

  • Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. In Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 17–24).

    Google Scholar 

  • Azevedo, R., Johnson, A., Chauncey, A., & Burkett, C. (2010). Self-regulated learning with MetaTutor: Advancing the science of learning with metacognitive tools. In M. Khine & I. Saleh (Eds.), New science of learning: Computers, cognition, and colloboration in eduction (pp. 225–247). Amsterdam: Springer.

    Chapter  Google Scholar 

  • Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16.

    Article  Google Scholar 

  • Brawner, K. W., & Gonzalez, A. J. (2016). Modelling a learner’s affective state in real time to improve intelligent tutoring effectiveness. Theoretical Issues in Ergonomics Science, 17(2), 183–210.

    Article  Google Scholar 

  • Burleson, W., & Picard, R. W. (2007). Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intelligent Systems, 22(4), 62–69.

    Article  Google Scholar 

  • Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37.

    Article  Google Scholar 

  • Calvo, R. A., & D’Mello, S. K. (2011). New perspectives on affect and learning technologies. New York: Springer.

    Book  Google Scholar 

  • Castellano, G., Kessous, L., & Caridakis, G. (2008). Emotion recognition through multiple modalities: Face, body gesture, speech affect and emotion in human-computer interaction (pp. 92–103). Berlin: Springer.

    Google Scholar 

  • Chaffar, S., Derbali, L., & Frasson, C. (2009). Inducing positive emotional state in intelligent tutoring systems. Paper presented at the AIED.

    Google Scholar 

  • Chaouachi, M., Jraidi, I., & Frasson, C. (2015). Adapting to learners’ mental states using a physiological computing approach. In FLAIRS Conference (pp. 257–262).

    Google Scholar 

  • D’Mello, S. K. (2016). Giving eyesight to the blind: Towards attention-aware AIED. International Journal of Artificial Intelligence in Education, 26(2), 645–659.

    Article  Google Scholar 

  • D’Mello, S. K., Blanchard, N., Baker, R., Ocumpaugh, J., & Brawner, K. (2014). I feel your pain: A selective review of affect-sensitive instructional strategies. In R. A. Sottilare, A. C. Graesser, X. Hu, & B. Goldberg (Eds.), Design recommendations for intelligent tutoring systems: Adaptive instructional strategies (Vol. 2, pp. 35–48). Orlando, FL: US Army Research Laboratory.

    Google Scholar 

  • D’Mello, S. K., Dieterle, E., & Duckworth, A. (2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123.

    Article  Google Scholar 

  • D’Mello, S. K., & Graesser, A. C. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20, 147–187.

    Article  Google Scholar 

  • D’Mello, S. K., & Graesser, A. C. (2014). Feeling, thinking, and computing with affect-aware learning. In R. A. Calvo, S. K. D’Mello, J. Gratch, & A. Kappas (Eds.), Oxford library of psychology. The Oxford handbook of affective computing (pp. 419–434). New York: Oxford University Press.

    Google Scholar 

  • D’Mello, S. K., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., et al. (2010). A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (Vol. 6094, pp. 245–254). Heidelberg: Springer.

    Chapter  Google Scholar 

  • D’Mello, S. K., Mills, C., Bixler, R., & Bosch, N. (2017). Zone out no more: Mitigating mind wandering during computerized reading. In Proceedings of the 10th International Conference on Educational Data Mining (pp. 8–15).

    Google Scholar 

  • D’Mello, S. K., Olney, A., Williams, C., & Hays, P. (2012). Gaze tutor: A gaze-reactive intelligent tutoring system. International Journal of Human-Computer Studies, 70(5), 377–398.

    Article  Google Scholar 

  • Derbali, L., & Frasson, C. (2012). Assessment of learners’ motivation during interactions with serious games: A study of some motivational strategies in food-force. Advances in Human-Computer Interaction, 2012, 5.

    Article  Google Scholar 

  • Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3–4), 169–200.

    Article  Google Scholar 

  • Flavell, J. H. (1976). Metacognitive aspects of problem solving. In M. Resnick (Ed.), The nature of intelligence (pp. 231–236). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Graesser, A. C., Conley, M., & Olney, A. (2012). Intelligent tutoring systems. In K. R. Harris, S. Graham, & T. Urdan (Eds.), APA educational psychology handbook, Vol 3. Applications to learning and teaching (pp. 451–473). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Graesser, A. C., D’Mello, S. K., Hu, X., Cai, Z., Olney, A., & Morgan, B. (2012). AutoTutor. In P. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing: Indentification, investigation, and resolution (pp. 169–187). Hershey, PA: IGI Global.

    Chapter  Google Scholar 

  • Graesser, A. C., D’Mello, S. K., & Strain, A. C. (2014). Emotions in advanced learning technologies. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 473–493). New York, NY: Routledge.

    Google Scholar 

  • Gross, J. J., & Barrett, L. F. (2011). Emotion generation and emotion regulation: One or two depends on your point of view. Emotion Review, 3(1), 8–16.

    Article  Google Scholar 

  • Harley, J. M., Bouchet, F., Hussain, M. S., Azevedo, R., & Calvo, R. (2015). A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. Computers in Human Behavior, 48, 615–625. https://doi.org/10.1016/j.chb.2015.02.013

    Article  Google Scholar 

  • Harley, J. M., Lajoie, S. P., Frasson, C., & Hall, N. C. (2017). Developing emotion-aware, advanced learning technologies: A taxonomy of approaches and features. International Journal of Artificial Intelligence in Education, 27(2), 268–297. https://doi.org/10.1007/s40593-016-0126-8

    Article  Google Scholar 

  • Healey, J. (2015). Physiological sensing of emotion. In R. A. Calvo, S. K. D’Mello, J. Gratch, & A. Kappas (Eds.), Handbook of affective computing (pp. 204–216). New York: Oxford University Press.

    Google Scholar 

  • Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J. R., & D’Mello, S. K. (2017). Out of the Fr-Eye-ing pan: Towards gaze-based models of attention during learning with technology in the classroom. In M. Bielikova, E. Herder, F. Cena, & M. Desmarais (Eds.), Proceedings of the 2017 Conference on User Modeling, Adaptation, and Personalization (pp. 94–103). New York: ACM.

    Chapter  Google Scholar 

  • Kay, J. (2008). Lifelong learner modeling for lifelong personalized pervasive learning. IEEE Transactions on Learning Technologies, 1(4), 215–228.

    Article  Google Scholar 

  • Kort, B. (2009, May 10). Cognition, affect, and learning: The role of emotions in learning. Retrieved from http://knol.google.com/k/cognition-affect-and-learning.

  • Kulik, C.-L. C., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7(1–2), 75–94. https://doi.org/10.1016/0747-5632(91)90030-5

    Article  Google Scholar 

  • Kulik, J. A., & Fletcher, J. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78.

    Article  Google Scholar 

  • Lane, H. C. (2016). Pedagogical agents and affect: Molding positive learning interactions. In S. Y. Tettegah & M. Gartmeier (Eds.), Emotions, technology, design, & learning (pp. 47–61). London: Academic Press.

    Chapter  Google Scholar 

  • Lane, H. C., & Johnson, W. L. (2009). Intelligent tutoring and pedagogical experience manipulation in virtual learning environments. In D. Schmorrow, J. Cone, & D. Nicholson (Eds.), The handbook of virtual environments for training and education, Volume 2: VE components and training technologies (pp. 393–406). Westport, CT: Praeger Security International.

    Google Scholar 

  • Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901.

    Article  Google Scholar 

  • Mark, M. A., & Greer, J. E. (1995). The VCR tutor: Effective Instruction for device operation. The Journal of the Learning Sciences, 4(2), 209–246.

    Article  Google Scholar 

  • Mukhopadhyay, S. C. (2015). Wearable sensors for human activity monitoring: A review. IEEE Sensors Journal, 15(3), 1321–1330. https://doi.org/10.1109/JSEN.2014.2370945

    Article  Google Scholar 

  • Murphy, J. S., Carroll, M. B., Champney, R. K., & Padron, C. K. (2015). Investigating the role of physiological measurement in intelligent tutoring. Paper presented at the Generalized Intelligent Framework for Tutoring (GIFT) Users Symposium (GIFTSym2).

    Google Scholar 

  • Panksepp, J. (2004). Affective neuroscience: The foundations of human and animal emotions. New York: Oxford University Press..

    Google Scholar 

  • Partnership for 21st Century Skills (P21). 2012. Framework for 21st Century Learning. Retrieved from http://www.p21.org/overview/skills-framework.

  • Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259–282). New York: Springer.

    Chapter  Google Scholar 

  • Pham, P., & Wang, J. (2016). Adaptive review for mobile MOOC learning via implicit physiological signal sensing. Paper presented at the Proceedings of the 18th ACM International Conference on Multimodal Interaction.

    Google Scholar 

  • Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125. https://doi.org/10.1016/j.inffus.2017.02.003

    Article  Google Scholar 

  • Pour, P. A., Hussain, M. S., AlZoubi, O., D’Mello, S. K., & Calvo, R. A. (2010). The impact of system feedback on learners’ affective and physiological states. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (Vol. 6094, pp. 264–273). Berlin: Springer.

    Chapter  Google Scholar 

  • Pressey, S. L. (1926). A simple apparatus which gives tests and scores—And teaches. School and Society, 23(586), 373–376.

    Google Scholar 

  • Pressey, S. L. (1932). A third and fourth contribution toward the coming “industrial revolution” in education. School and Society, 36(934), 668–672.

    Google Scholar 

  • Rosenberg, E. L. (1998). Levels of analysis and the organization of affect. Review of General Psychology, 2(3), 247–270.

    Article  Google Scholar 

  • Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795

    Article  Google Scholar 

  • Simonov, P. V. (2013). The emotional brain: Physiology, neuroanatomy, psychology, and emotion. New York: Springer.

    Google Scholar 

  • Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285.

    Article  Google Scholar 

  • Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Psychology Review, 22(2), 123–138.

    Article  Google Scholar 

  • Taub, M., & Azevedo, R. (2018). How does prior knowledge influence eye fixations and sequences of cognitive and metacognitive SRL processes during learning with an intelligent tutoring system? International Journal of Artificial Intelligence in Education (pp 1–28). https://doi.org/10.1007/s40593-018-0165-4

  • VanLehn, K. (2006). The Behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.

    Google Scholar 

  • VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. https://doi.org/10.1080/00461520.2011.611369

    Article  Google Scholar 

  • VanLehn, K., Zhang, L., Burleson, W., Girard, S., & Hidago-Pontet, Y. (2017). Can a non-cognitive learning companion increase the effectiveness of a meta-cognitive learning strategy? IEEE Transactions on Learning Technologies, 10(3), 277–289.

    Article  Google Scholar 

  • Vygotsky, L. S. (1978). Zone of proximal development: A new approach. In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Minds in society: The development of higher psychological processes (pp. 84–91). Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Wenger, E. (1987). Artificial intelligence and tutoring systems. San Francisco, CA: Morgan Kaufmann.

    Google Scholar 

  • Woolf, B. P. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing E-learning. Amsterdam: Morgan Kaufmann.

    Google Scholar 

  • Woolf, B. P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D. G., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3–4), 129–164.

    Article  Google Scholar 

  • Yuksel, B. F., Oleson, K. B., Harrison, L., Peck, E. M., Afergan, D., Chang, R. et al. (2016). Learn piano with BACh: An adaptive learning interface that adjusts task difficulty based on brain state. Paper presented at the Proceedings of the 2016 Chi Conference on Human Factors in Computing Systems.

    Google Scholar 

  • Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the editors for their valuable feedback on this chapter, as well as the National Science Foundation, Institute of Education Sciences, and the US Department of Defense that have funded large portions of the work cited in this chapter.

Author information

Authors and Affiliations

  1. Department of Educational Psychology, College of Education, University of Illinois, Urbana-Champaign, Champaign, IL, USA

    H. Chad Lane

  2. Institute of Cognitive Science, University of Colorado, Boulder, Boulder, CO, USA

    Sidney K. D’Mello

Authors
  1. H. Chad Lane
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sidney K. D’Mello
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to H. Chad Lane .

Editor information

Editors and Affiliations

  1. College of Information, University of North Texas Computational Neuropsychology and Simulation, Denton, TX, USA

    Thomas D. Parsons

  2. Department of Learning Technologies, University of North Texas, Denton, TX, USA

    Lin Lin

  3. Department of Learning Technologies, Fort Worth Museum of Science and History, University of North Texas, Fort Worth, TX, USA

    Deborah Cockerham

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Association for Educational Communications and Technology

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Lane, H.C., D’Mello, S.K. (2019). Uses of Physiological Monitoring in Intelligent Learning Environments: A Review of Research, Evidence, and Technologies. In: Parsons, T.D., Lin, L., Cockerham, D. (eds) Mind, Brain and Technology. Educational Communications and Technology: Issues and Innovations. Springer, Cham. https://doi.org/10.1007/978-3-030-02631-8_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-02631-8_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-02630-1

  • Online ISBN: 978-3-030-02631-8

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation