Skip to main content
SpringerLink
Log in

Emotion Recognition from Physiological Sensor Data to Support Self-regulated Learning

  • Conference paper
  • First Online:
Computer Supported Education (CSEDU 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1220))

Included in the following conference series:

Abstract

In education, learners’ autonomy and agency have been emphasized across various domains. However, ability to self-regulate their learning by setting a goal, monitoring, regulating and evaluating their learning progress is not easy. With wearable sensor technology, various physiological and contextual data can be detected and collected. To provide learners with a context-aware personal learning support, we have researched physiological sensor data (EDA and ECG) by providing emotional stimulants to 70 students from two higher education institutes. We have analyzed our collected data using multiple methods (qualitative, quantitative, machine learning and fuzzy logic approaches) and found a relation between physiological sensor data and emotion that seems promising. Consecutively, we have investigated a learning support system for self-regulated learning and proposed three ideas with prototypes. Our future work will entail implementation of research findings to develop a learning companion system to support learners’ self-regulated learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    16SV7534K.

  2. 2.

    Electroencephalogram.

  3. 3.

    Electrocardiogram.

  4. 4.

    Photoplethysmography.

  5. 5.

    Electrodermal Activity.

  6. 6.

    Leibniz Institut für Wissensmedien, Tuebingen.

  7. 7.

    https://bitalino.com.

  8. 8.

    Electromyography.

References

  1. Zimmerman, B.J.: Becoming a self-regulated learner: an overview. Theory Pract. 41(2), 64–70 (2002)

    Article  Google Scholar 

  2. Azevedo, R., Taub, M., Mudrick, N.V., Millar, G.C., Bradbury, A.E., Price, M.J.: Using data visualizations to foster emotion regulation during self-regulated learning with advanced learning technologies. In: Buder, J., Hesse, F.W. (eds.) Informational Environments, pp. 225–247. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-64274-1_10

    Chapter  Google Scholar 

  3. Mohr, D.C., Zhang, M., Schueller, S.M.: Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annu. Rev. Clin. Psychol. 13, 23–47 (2017)

    Article  Google Scholar 

  4. Yun, H., Fortenbacher, A., Pinkwart, N.: Improving a mobile learning companion for self-regulated learning using sensors. In: Proceedings of the 9th International Conference on Computer Supported Education, CSEDU 2017, vol. 1 (2017)

    Google Scholar 

  5. Calvo, R.A., D’Mello, S., Gratch, J., Kappas, A. (eds.): The Oxford Handbook of Affective Computing. Oxford University Press, Oxford (2015)

    Google Scholar 

  6. Canzian, L., Musolesi, M.: Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1293–1304. ACM (2015)

    Google Scholar 

  7. Kreibig, S.D., Gendolla, G.H., Scherer, K.R.: Goal relevance and goal conduciveness appraisals lead to differential autonomic reactivity in emotional responding to performance feedback. Biol. Psychol. 91(3), 365–375 (2012)

    Article  Google Scholar 

  8. Pecchinenda, A.: The affective significance of skin conductance activity during a difficult problem-solving task. Cogn. Emot. 10(5), 481–504 (1996)

    Article  Google Scholar 

  9. Tomaka, J., Blascovich, J., Kelsey, R.M., Leitten, C.L.: Subjective, physiological, and behavioral effects of threat and challenge appraisal. J. Pers. Soc. Psychol. 65(2), 248 (1993)

    Article  Google Scholar 

  10. D’Mello, S.K.: A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. J. Educ. Psychol. 105, 1082–1099 (2013)

    Article  Google Scholar 

  11. Fairclough, S.H., Venables, L., Tattersall, A.: The influence of task demand and learning on the psychophysiological response. Int. J. Psychophysiol. 56(2), 171–184 (2005)

    Article  Google Scholar 

  12. Bradley, M.M., Lang, P.J.: Motivation and emotion. In: Cacioppo, J., Tssinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, Chap. 25, pp. 581–607. Oxford University Press, New York (2007)

    Chapter  Google Scholar 

  13. Levenson, R.W., Ekman, P., Friesen, W.V.: Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27(4), 363–384 (1990)

    Article  Google Scholar 

  14. Cacioppo, J.T., Berntson, G.G., Larsen, J.T., Poehlmann, K.M., Ito, T.A., et al.: The psychophysiology of emotion. In: Handbook of Emotions, vol, 2, pp. 173–191 (2000)

    Google Scholar 

  15. Mandryk, R.L., Atkins, M.S.: A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int. J. Hum Comput Stud. 65(4), 329–347 (2007)

    Article  Google Scholar 

  16. Vrana, S.R., Cuthbert, B.N., Lang, P.J.: Fear imagery and text processing. Psychophysiology 23(3), 247–253 (1986)

    Article  Google Scholar 

  17. Libby Jr., W.L., Lacey, B.C., Lacey, J.I.: Pupillary and cardiac activity during visual attention. Psychophysiology 10(3), 270–294 (1973)

    Article  Google Scholar 

  18. Ekman, P., Levenson, R.W., Friesen, W.V.: Autonomic nervous system activity distinguishes among emotions. Science 221(4616), 1208–1210 (1983)

    Article  Google Scholar 

  19. Chanel, G., Mühl, C.: Connecting brains and bodies: applying physiological computing to support social interaction. Interact. Comput. 27(5), 534–550 (2015)

    Article  Google Scholar 

  20. Lang, P.J.: The emotion probe: studies of motivation and attention. Am. Psychol. 50(5), 372 (1995)

    Article  Google Scholar 

  21. Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)

    Article  Google Scholar 

  22. Bradley, M.M., Lang, P.J.: The international affective picture system (IAPS) in the study of emotion and attention. In: Coan, J.A., Allen, J.J.B. (eds.) Handbook of Emotion Elicitation and Assessment, Chap. 29, pp. 29–46. Oxford University Press, New York (2007)

    Google Scholar 

  23. Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical report A-8. University of Florida, Gainesville, FL (2008)

    Google Scholar 

  24. Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161 (1980)

    Article  Google Scholar 

  25. Boucsein, W.: Electrodermal Activity. Springer, New York (2012). https://doi.org/10.1007/978-1-4614-1126-0

    Book  Google Scholar 

  26. Camm, A., et al.: Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task force of the European society of cardiology and the North American society of pacing and electrophysiology. Circulation 93(5), 1043–1065 (1996)

    Article  Google Scholar 

  27. Gruber, J., Mennin, D.S., Fields, A., Purcell, A., Murray, G.: Heart rate variability as a potential indicator of positive valence system disturbance: a proof of concept investigation. Int. J. Psychophysiol. 98(2), 240–248 (2015)

    Article  Google Scholar 

  28. Heathers, J., Goodwin, M.: Dead science in live psychology: a case study from heart rate variability (HRV) (2017)

    Google Scholar 

  29. Lanzetta, J.T., Cartwright-Smith, J., Eleck, R.E.: Effects of nonverbal dissimulation on emotional experience and autonomic arousal. J. Pers. Soc. Psychol. 33(3), 354 (1976)

    Article  Google Scholar 

  30. Winton, W.M., Putnam, L.E., Krauss, R.M.: Facial and autonomic manifestations of the dimensional structure of emotion. J. Exp. Soc. Psychol. 20(3), 195–216 (1984)

    Article  Google Scholar 

  31. Conati, C., Chabbal, R., Maclaren, H.: A study on using biometric sensors for monitoring user emotions in educational games. Technical report (2018)

    Google Scholar 

  32. Ferdinando, H., Seppänen, T., Alasaarela, E.: Emotion recognition using neighborhood components analysis and ECG/HRV-based features. In: De Marsico, M., di Baja, G.S., Fred, A. (eds.) ICPRAM 2017. LNCS, vol. 10857, pp. 99–113. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93647-5_6

    Chapter  Google Scholar 

  33. Ayata, D.D., Yaslan, Y., Kamaşak, M.: Emotion recognition via galvanic skin response: comparison of machine learning algorithms and feature extraction methods. Istanbul Univ.-J. Electr. Electr. Eng. 17(1), 3147–3156 (2017)

    Google Scholar 

  34. Minhad, K., Hamid Md Ali, S., Reaz, M.: A design framework for human emotion recognition using electrocardiogram and skin conductance response signals. J. Eng. Sci. Technol. 12(11), 3102–3119 (2017)

    Google Scholar 

  35. Schölkopf, B., Burges, C.J., Smola, A.J. (eds.): Advances in Kernel Methods: Support Vector Learning, pp. 327–352. MIT Press, Cambridge (1999)

    Google Scholar 

  36. Sammut, C., Webb, G.I.: Encyclopedia of Machine Learning and Data Mining, pp. 314–315. Springer, Heidelberg (2017)

    Book  Google Scholar 

  37. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Advances in Neural Information Processing Systems, pp. 2962–2970 (2015)

    Google Scholar 

  38. Kreibig, S.D.: Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84(3), 394–421 (2010)

    Article  Google Scholar 

  39. Scheibe, S., Fortenbacher, A.: Heart Rate Variability alsIndikatorfür den emotionalen Zustand eines Lernenden. In: Proceedings der Pre-Conference-Workshops der 17. E-Learning FachtagungInformatik co-located with 17th e-Learning Conference of the German Computer Society (DeLFI 2019) (2019)

    Google Scholar 

  40. Cleary, J.G., Trigg, L.E.: K*: an instance-based learner using an entropic distance measure. In: Machine Learning Proceedings 1995, pp. 108–114. Morgan Kaufmann (1995)

    Google Scholar 

  41. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: ICML, vol. 96, pp. 148–156 (1996)

    Google Scholar 

  42. Hyvärinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Netw. 13(4–5), 411–430 (2000)

    Article  Google Scholar 

  43. Stez, C., Anrich, B., Schumm, J., Marca, R., Troster, G., Elhlert, U.: Discriminating stress from cognitive load using a wearable EDA. IEEE Trans. Inf Technol. Biomed. 14(2), 410–417 (2010)

    Article  Google Scholar 

  44. Cox, E.: Fuzzy fundamentals. IEEE Spectr. 29(10), 58–61 (1992)

    Article  Google Scholar 

  45. Woolf, B.P.: Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing E-Learning. p. 225. Morgan Kaufmann (2010)

    Google Scholar 

  46. Gertner, A.S., VanLehn, K.: Andes: a coached problem solving environment for physics. In: Gauthier, G., Frasson, C., VanLehn, K. (eds.) ITS 2000. LNCS, vol. 1839, pp. 133–142. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45108-0_17

    Chapter  Google Scholar 

  47. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Burlington (2016)

    Google Scholar 

  48. Arroyo, I., Beck, J.E., Woolf, B.P., Beal, C.R., Schultz, K.: Macroadapting animalwatch to gender and cognitive differences with respect to hint interactivity and symbolism. In: Gauthier, G., Frasson, C., VanLehn, K. (eds.) ITS 2000. LNCS, vol. 1839, pp. 574–583. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45108-0_61

    Chapter  Google Scholar 

  49. Johns, J., Woolf, B.: A dynamic mixture model to detect student motivation and proficiency. In: Proceedings of the Twenty-First National Conference on Artificial Intelligence, pp. 2–8. AAAI Press, Boston (2006)

    Google Scholar 

  50. Azevedo, R., Witherspoon, A., Chauncey, A., Burkett, C., Fike, A.: MetaTutor: a Meta Cognitive tool for enhancing self-regulated learning. In: 2009 AAAI Fall Symposium Series (2009)

    Google Scholar 

  51. Koedinger, K.R., Aleven, V.A.W.M.M., Heffernan, N.: Toward a rapid development environment for cognitive tutors. In: Artificial Intelligence in Education: Shaping the Future of Learning through Intelligent Technologies, Proceedings of AI-ED, pp. 455–457 (2003)

    Google Scholar 

  52. Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 61–70. Springer, Heidelberg (2006). https://doi.org/10.1007/11774303_7

    Chapter  Google Scholar 

  53. Lallé, S., Conati, C., Azevedo, R.: Prediction of student achievement goals and emotion valence during interaction with pedagogical agents. In: Proceedings of the 17th International Conference on Autonomous Agents and Multi Agent Systems, pp. 1222–1231. International Foundation for Autonomous Agents and Multiagent Systems (2018)

    Google Scholar 

  54. McDuff, D., Karlson, A., Kapoor, A., Roseway, A., Czerwinski, M.: AffectAura: an intelligent system for emotional memory. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 849–858. ACM (2012)

    Google Scholar 

  55. Cernea, D., Weber, C., Ebert, A., Kerren, A.: Emotion-prints: interaction-driven emotion visualization on multi-touch interfaces. In: Visualization and Data Analysis 2015, vol. 9397, p. 93970A. International Society for Optics and Photonics (2015)

    Google Scholar 

  56. Silber-Varod, V., Eshet-Alkalai, Y., Geri, N.: Tracing research trends of 21st-century learning skills. Br. J. Educ. Technol. 50, 3099–3118 (2019)

    Article  Google Scholar 

  57. Yun, H., Fortenbacher, A., Helbig, R., Pinkwart, N.: In search of learning indicators: a study on sensor data and IAPS emotional pictures. In: Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019) (2019)

    Google Scholar 

  58. Schandry, R.: Heart beat perception and emotional experience. Psychophysiology 18(4), 483–488 (1981). https://doi.org/10.1111/j.1469-8986.1981.tb02486.x

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Applied Sciences Berlin, Wilhelminenhof Strasse 75A, 12459, Berlin, Germany

    Haeseon Yun, Albrecht Fortenbacher, René Helbig & Sven Geißler

  2. Humboldt University Berlin, Rudower Chaussee 25, 12489, Berlin, Germany

    Haeseon Yun & Niels Pinkwart

Authors
  1. Haeseon Yun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Albrecht Fortenbacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. René Helbig
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sven Geißler
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Niels Pinkwart
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haeseon Yun .

Editor information

Editors and Affiliations

  1. University of Illinois at Urbana-Champaign, Urbana, IL, USA

    H. Chad Lane

  2. Anderson Academic Commons, University of Denver, Denver, CO, USA

    Susan Zvacek

  3. School of Engineering, University of Ulster, Newtownabbey, UK

    James Uhomoibhi

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yun, H., Fortenbacher, A., Helbig, R., Geißler, S., Pinkwart, N. (2020). Emotion Recognition from Physiological Sensor Data to Support Self-regulated Learning. In: Lane, H.C., Zvacek, S., Uhomoibhi, J. (eds) Computer Supported Education. CSEDU 2019. Communications in Computer and Information Science, vol 1220. Springer, Cham. https://doi.org/10.1007/978-3-030-58459-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58459-7_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58458-0

  • Online ISBN: 978-3-030-58459-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation