Advertisement

Uses of Physiological Monitoring in Intelligent Learning Environments: A Review of Research, Evidence, and Technologies

  • H. Chad LaneEmail author
  • Sidney K. D’Mello
Chapter
Part of the Educational Communications and Technology: Issues and Innovations book series (ECTII)

Abstract

Two of the most important benefits of using computer-based learning environments are that (1) they allow for interactive learning experiences and (2) it becomes possible to assess learning at a fine level of granularity based on learner actions, choices, and performance. Using these assessments, modern systems can deliver tailored pedagogical interventions, such as through feedback or adjustment of problem difficulty, in order to enhance learning. To magnify their power to promote learning, researchers have also sought to capture additional information, such as physiological data, to make even finer-grained and nuanced pedagogical decisions. A wide range of technologies have been explored, such as depth-sensing cameras (e.g., Kinect), electrodermal activity (EDA) sensors, electroencephalography (EEG), posture/seat detectors, eye- and head-tracking cameras, and mouse-pressure sensors, among many others. This chapter provides an overview and examples of how these techniques have been used by educational technology researchers. We focus on how these additional inputs can improve technology-based pedagogical decision-making to support cognitive (i.e., knowledge and information processing), affective (i.e., emotions, including motivational factors), and metacognitive (i.e., attention and self-regulatory behaviors) aspects of learning. Though not a comprehensive review, we take a selective look at classic and recent developments in this area. The chapter concludes with a brief discussion of emerging topics and key open questions related to the use of physiological tracking in the context of education and learning with technology.

Keywords

Physiological sensing Affective computing Adaptive learning environments Learner modelling Intelligent tutoring systems 

Notes

Acknowledgements

We would like to thank the editors for their valuable feedback on this chapter, as well as the National Science Foundation, Institute of Education Sciences, and the US Department of Defense that have funded large portions of the work cited in this chapter.

References

  1. Anderson, J. R., Betts, S., Ferris, J. L., & Fincham, J. M. (2010). Neural imaging to track mental states while using an intelligent tutoring system. Proceedings of the National Academy of Sciences, 107(15), 7018–7023.  https://doi.org/10.1073/pnas.1000942107 CrossRefGoogle Scholar
  2. Anderson, J. R., Betts, S., Ferris, J. L., & Fincham, J. M. (2012). Tracking children’s mental states while solving algebra equations. Human Brain Mapping, 33(11), 2650–2665.CrossRefGoogle Scholar
  3. Anderson, J. R., Corbett, A., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207.CrossRefGoogle Scholar
  4. Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of eductional outcomes. New York: Longman.Google Scholar
  5. Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. In Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 17–24).Google Scholar
  6. Azevedo, R., Johnson, A., Chauncey, A., & Burkett, C. (2010). Self-regulated learning with MetaTutor: Advancing the science of learning with metacognitive tools. In M. Khine & I. Saleh (Eds.), New science of learning: Computers, cognition, and colloboration in eduction (pp. 225–247). Amsterdam: Springer.CrossRefGoogle Scholar
  7. Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16.CrossRefGoogle Scholar
  8. Brawner, K. W., & Gonzalez, A. J. (2016). Modelling a learner’s affective state in real time to improve intelligent tutoring effectiveness. Theoretical Issues in Ergonomics Science, 17(2), 183–210.CrossRefGoogle Scholar
  9. Burleson, W., & Picard, R. W. (2007). Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intelligent Systems, 22(4), 62–69.CrossRefGoogle Scholar
  10. Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37.CrossRefGoogle Scholar
  11. Calvo, R. A., & D’Mello, S. K. (2011). New perspectives on affect and learning technologies. New York: Springer.CrossRefGoogle Scholar
  12. Castellano, G., Kessous, L., & Caridakis, G. (2008). Emotion recognition through multiple modalities: Face, body gesture, speech affect and emotion in human-computer interaction (pp. 92–103). Berlin: Springer.Google Scholar
  13. Chaffar, S., Derbali, L., & Frasson, C. (2009). Inducing positive emotional state in intelligent tutoring systems. Paper presented at the AIED.Google Scholar
  14. Chaouachi, M., Jraidi, I., & Frasson, C. (2015). Adapting to learners’ mental states using a physiological computing approach. In FLAIRS Conference (pp. 257–262).Google Scholar
  15. D’Mello, S. K. (2016). Giving eyesight to the blind: Towards attention-aware AIED. International Journal of Artificial Intelligence in Education, 26(2), 645–659.CrossRefGoogle Scholar
  16. D’Mello, S. K., Blanchard, N., Baker, R., Ocumpaugh, J., & Brawner, K. (2014). I feel your pain: A selective review of affect-sensitive instructional strategies. In R. A. Sottilare, A. C. Graesser, X. Hu, & B. Goldberg (Eds.), Design recommendations for intelligent tutoring systems: Adaptive instructional strategies (Vol. 2, pp. 35–48). Orlando, FL: US Army Research Laboratory.Google Scholar
  17. D’Mello, S. K., Dieterle, E., & Duckworth, A. (2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123.CrossRefGoogle Scholar
  18. D’Mello, S. K., & Graesser, A. C. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20, 147–187.CrossRefGoogle Scholar
  19. D’Mello, S. K., & Graesser, A. C. (2014). Feeling, thinking, and computing with affect-aware learning. In R. A. Calvo, S. K. D’Mello, J. Gratch, & A. Kappas (Eds.), Oxford library of psychology. The Oxford handbook of affective computing (pp. 419–434). New York: Oxford University Press.Google Scholar
  20. D’Mello, S. K., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., et al. (2010). A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (Vol. 6094, pp. 245–254). Heidelberg: Springer.CrossRefGoogle Scholar
  21. D’Mello, S. K., Mills, C., Bixler, R., & Bosch, N. (2017). Zone out no more: Mitigating mind wandering during computerized reading. In Proceedings of the 10th International Conference on Educational Data Mining (pp. 8–15).Google Scholar
  22. D’Mello, S. K., Olney, A., Williams, C., & Hays, P. (2012). Gaze tutor: A gaze-reactive intelligent tutoring system. International Journal of Human-Computer Studies, 70(5), 377–398.CrossRefGoogle Scholar
  23. Derbali, L., & Frasson, C. (2012). Assessment of learners’ motivation during interactions with serious games: A study of some motivational strategies in food-force. Advances in Human-Computer Interaction, 2012, 5.CrossRefGoogle Scholar
  24. Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3–4), 169–200.CrossRefGoogle Scholar
  25. Flavell, J. H. (1976). Metacognitive aspects of problem solving. In M. Resnick (Ed.), The nature of intelligence (pp. 231–236). Hillsdale, NJ: Erlbaum.Google Scholar
  26. Graesser, A. C., Conley, M., & Olney, A. (2012). Intelligent tutoring systems. In K. R. Harris, S. Graham, & T. Urdan (Eds.), APA educational psychology handbook, Vol 3. Applications to learning and teaching (pp. 451–473). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  27. Graesser, A. C., D’Mello, S. K., Hu, X., Cai, Z., Olney, A., & Morgan, B. (2012). AutoTutor. In P. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing: Indentification, investigation, and resolution (pp. 169–187). Hershey, PA: IGI Global.CrossRefGoogle Scholar
  28. Graesser, A. C., D’Mello, S. K., & Strain, A. C. (2014). Emotions in advanced learning technologies. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 473–493). New York, NY: Routledge.Google Scholar
  29. Gross, J. J., & Barrett, L. F. (2011). Emotion generation and emotion regulation: One or two depends on your point of view. Emotion Review, 3(1), 8–16.CrossRefGoogle Scholar
  30. Harley, J. M., Bouchet, F., Hussain, M. S., Azevedo, R., & Calvo, R. (2015). A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. Computers in Human Behavior, 48, 615–625.  https://doi.org/10.1016/j.chb.2015.02.013 CrossRefGoogle Scholar
  31. Harley, J. M., Lajoie, S. P., Frasson, C., & Hall, N. C. (2017). Developing emotion-aware, advanced learning technologies: A taxonomy of approaches and features. International Journal of Artificial Intelligence in Education, 27(2), 268–297.  https://doi.org/10.1007/s40593-016-0126-8 CrossRefGoogle Scholar
  32. Healey, J. (2015). Physiological sensing of emotion. In R. A. Calvo, S. K. D’Mello, J. Gratch, & A. Kappas (Eds.), Handbook of affective computing (pp. 204–216). New York: Oxford University Press.Google Scholar
  33. Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J. R., & D’Mello, S. K. (2017). Out of the Fr-Eye-ing pan: Towards gaze-based models of attention during learning with technology in the classroom. In M. Bielikova, E. Herder, F. Cena, & M. Desmarais (Eds.), Proceedings of the 2017 Conference on User Modeling, Adaptation, and Personalization (pp. 94–103). New York: ACM.CrossRefGoogle Scholar
  34. Kay, J. (2008). Lifelong learner modeling for lifelong personalized pervasive learning. IEEE Transactions on Learning Technologies, 1(4), 215–228.CrossRefGoogle Scholar
  35. Kort, B. (2009, May 10). Cognition, affect, and learning: The role of emotions in learning. Retrieved from http://knol.google.com/k/cognition-affect-and-learning.
  36. Kulik, C.-L. C., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7(1–2), 75–94.  https://doi.org/10.1016/0747-5632(91)90030-5 CrossRefGoogle Scholar
  37. Kulik, J. A., & Fletcher, J. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78.CrossRefGoogle Scholar
  38. Lane, H. C. (2016). Pedagogical agents and affect: Molding positive learning interactions. In S. Y. Tettegah & M. Gartmeier (Eds.), Emotions, technology, design, & learning (pp. 47–61). London: Academic Press.CrossRefGoogle Scholar
  39. Lane, H. C., & Johnson, W. L. (2009). Intelligent tutoring and pedagogical experience manipulation in virtual learning environments. In D. Schmorrow, J. Cone, & D. Nicholson (Eds.), The handbook of virtual environments for training and education, Volume 2: VE components and training technologies (pp. 393–406). Westport, CT: Praeger Security International.Google Scholar
  40. Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901.CrossRefGoogle Scholar
  41. Mark, M. A., & Greer, J. E. (1995). The VCR tutor: Effective Instruction for device operation. The Journal of the Learning Sciences, 4(2), 209–246.CrossRefGoogle Scholar
  42. Mukhopadhyay, S. C. (2015). Wearable sensors for human activity monitoring: A review. IEEE Sensors Journal, 15(3), 1321–1330.  https://doi.org/10.1109/JSEN.2014.2370945 CrossRefGoogle Scholar
  43. Murphy, J. S., Carroll, M. B., Champney, R. K., & Padron, C. K. (2015). Investigating the role of physiological measurement in intelligent tutoring. Paper presented at the Generalized Intelligent Framework for Tutoring (GIFT) Users Symposium (GIFTSym2).Google Scholar
  44. Panksepp, J. (2004). Affective neuroscience: The foundations of human and animal emotions. New York: Oxford University Press..Google Scholar
  45. Partnership for 21st Century Skills (P21). 2012. Framework for 21st Century Learning. Retrieved from http://www.p21.org/overview/skills-framework.
  46. Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259–282). New York: Springer.CrossRefGoogle Scholar
  47. Pham, P., & Wang, J. (2016). Adaptive review for mobile MOOC learning via implicit physiological signal sensing. Paper presented at the Proceedings of the 18th ACM International Conference on Multimodal Interaction.Google Scholar
  48. Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125.  https://doi.org/10.1016/j.inffus.2017.02.003 CrossRefGoogle Scholar
  49. Pour, P. A., Hussain, M. S., AlZoubi, O., D’Mello, S. K., & Calvo, R. A. (2010). The impact of system feedback on learners’ affective and physiological states. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (Vol. 6094, pp. 264–273). Berlin: Springer.CrossRefGoogle Scholar
  50. Pressey, S. L. (1926). A simple apparatus which gives tests and scores—And teaches. School and Society, 23(586), 373–376.Google Scholar
  51. Pressey, S. L. (1932). A third and fourth contribution toward the coming “industrial revolution” in education. School and Society, 36(934), 668–672.Google Scholar
  52. Rosenberg, E. L. (1998). Levels of analysis and the organization of affect. Review of General Psychology, 2(3), 247–270.CrossRefGoogle Scholar
  53. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.  https://doi.org/10.3102/0034654307313795 CrossRefGoogle Scholar
  54. Simonov, P. V. (2013). The emotional brain: Physiology, neuroanatomy, psychology, and emotion. New York: Springer.Google Scholar
  55. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285.CrossRefGoogle Scholar
  56. Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Psychology Review, 22(2), 123–138.CrossRefGoogle Scholar
  57. Taub, M., & Azevedo, R. (2018). How does prior knowledge influence eye fixations and sequences of cognitive and metacognitive SRL processes during learning with an intelligent tutoring system? International Journal of Artificial Intelligence in Education (pp 1–28).  https://doi.org/10.1007/s40593-018-0165-4
  58. VanLehn, K. (2006). The Behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.Google Scholar
  59. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.  https://doi.org/10.1080/00461520.2011.611369 CrossRefGoogle Scholar
  60. VanLehn, K., Zhang, L., Burleson, W., Girard, S., & Hidago-Pontet, Y. (2017). Can a non-cognitive learning companion increase the effectiveness of a meta-cognitive learning strategy? IEEE Transactions on Learning Technologies, 10(3), 277–289.CrossRefGoogle Scholar
  61. Vygotsky, L. S. (1978). Zone of proximal development: A new approach. In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Minds in society: The development of higher psychological processes (pp. 84–91). Cambridge, MA: Harvard University Press.Google Scholar
  62. Wenger, E. (1987). Artificial intelligence and tutoring systems. San Francisco, CA: Morgan Kaufmann.Google Scholar
  63. Woolf, B. P. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing E-learning. Amsterdam: Morgan Kaufmann.Google Scholar
  64. Woolf, B. P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D. G., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3–4), 129–164.CrossRefGoogle Scholar
  65. Yuksel, B. F., Oleson, K. B., Harrison, L., Peck, E. M., Afergan, D., Chang, R. et al. (2016). Learn piano with BACh: An adaptive learning interface that adjusts task difficulty based on brain state. Paper presented at the Proceedings of the 2016 Chi Conference on Human Factors in Computing Systems.Google Scholar
  66. Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58.CrossRefGoogle Scholar

Copyright information

© Association for Educational Communications and Technology 2019

Authors and Affiliations

  1. 1.Department of Educational PsychologyCollege of Education, University of Illinois, Urbana-ChampaignChampaignUSA
  2. 2.Institute of Cognitive ScienceUniversity of Colorado, BoulderBoulderUSA

Personalised recommendations