Advertisement

“Hello Computer, How Am I Feeling?”, Case Studies of Neural Technology to Measure Emotions

  • Ian DalyEmail author
  • Duncan Williams
Chapter
Part of the Cognitive Science and Technology book series (CSAT)

Abstract

Emotion is a core part of the human experience. Many artistic and creative applications attempt to produce particular emotional experiences, for example, films, games, music, dance, and other visual arts. However, while emotional states are ubiquitous, they are also complex, proving difficult to describe to others by conventional psychometric means (e.g., traditional self-report mechanisms). Neural technology offers the potential to circumvent these difficulties by allowing the creation of a real-time, objective, metric of felt emotion to assist in emotionally driven experience design across a range of disciplines. This chapter discusses how neural technology based on the processing of the electroencephalogram may be used to measure human emotions in natural environments. We also present a set of case studies of applications that use neural technology to measure emotions. We are particularly interested in the use of neural technology to inform applications which can respond to the felt experience of the individual. We describe two case studies focused on driving scenarios and brain–computer music interfacing. The chapter concludes with a discussion of the challenges inherent in developing neural technology to measure emotion and a set of suggestions for future research directions in developing applications that use neural technology as an objective measure of emotion.

References

  1. Aftanas, L. I., Varlamov, A. A., Pavlov, S. V., Makhnev, V. P., & Reva, N. V. (2001). Affective picture processing: Event-related synchronization within individually defined human theta band is modulated by valence dimension. Neuroscience Letters, 303(2), 115–118.  https://doi.org/10.1016/S0304-3940(01)01703-7.CrossRefGoogle Scholar
  2. Barrett, L. F., Mesquita, B., Ochsner, K. N., & Gross, J. J. (2007). The experience of emotion. Annual Review of Psychology, 58, 373–403.  https://doi.org/10.1146/annurev.psych.58.110405.085709.CrossRefGoogle Scholar
  3. Benoit, A., Bonnaud, L., Caplier, A., Ngo, P., Lawson, L., Trevisan, D. G., … Chanel, G. (2009). Multimodal focus attention and stress detection and feedback in an augmented driver simulator. Personal and Ubiquitous Computing, 13(1), 33–41.  https://doi.org/10.1007/s00779-007-0173-0.CrossRefGoogle Scholar
  4. Blood, A. J., & Zatorre, R. J. (2001). Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proceedings of the National Academy of Sciences of the United States of America, 98(20), 11818–11823.  https://doi.org/10.1073/pnas.191355898.CrossRefGoogle Scholar
  5. Bradt, J., Magee, W. L., Dileo, C., Wheeler, B. L., & McGilloway, E. (2010). Music therapy for acquired brain injury. The Cochrane Database of Systematic Reviews, (7), CD006787.  https://doi.org/10.1002/14651858.CD006787.pub2.
  6. Canli, T., Desmond, J. E., Zhao, Z., Glover, G., & Gabrieli, J. D. (1998). Hemispheric asymmetry for emotional stimuli detected with fMRI. Neuroreport, 9(14), 3233–3239. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/9831457.
  7. Chu, C.-C., Tranel, D., & Damasio, H. (1994). How reliable are occipital asymmetry measurements? Neuropsychologia, 32(12), 1503–1513.  https://doi.org/10.1016/0028-3932(94)90122-8.CrossRefGoogle Scholar
  8. Coan, J. A., & Allen, J. J. (2004). Frontal EEG asymmetry as a moderator and mediator of emotion. Biological Psychology, 67(1–2), 7–50.  https://doi.org/10.1016/j.biopsycho.2004.03.002.CrossRefGoogle Scholar
  9. Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., & Schröder, M. (2000). “FEELTRACE”: An instrument for recording perceived emotion in real time. In Proceedings of the ISCA Workshop on Speech and Emotion (pp. 19–24). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.7528.
  10. Daly, I., Billinger, M., Scherer, R., & Mueller-Putz, G. (2013). On the automated removal of artifacts related to head movement from the EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 21(3), 427–434.CrossRefGoogle Scholar
  11. Daly, I., Ho, A., Marcon, J., Hwang, F., Williams, D., Kirke, A., … Nasuto, S. (2017). Affective brain computer music interfacing: A case study of use by an individual with Huntington’s disease. In Proceedings of the Graz Brain-Computer Interfacing Conference 2017.Google Scholar
  12. Daly, I., Malik, A., Hwang, F., Roesch, E., Weaver, J., Kirke, A., … Nasuto, S. J. (2014). Neural correlates of emotional responses to music: An EEG study. Neuroscience Letters, 573, 52–57.  https://doi.org/10.1016/j.neulet.2014.05.003.CrossRefGoogle Scholar
  13. Daly, I., Malik, A., Weaver, J., Hwang, F., Nasuto, S. J., Williams, D., … Miranda, E. (2015). Identifying music-induced emotions from EEG for use in brain-computer music interfacing. In 6th Affective Computing and Intelligent Interaction.Google Scholar
  14. Daly, I., Williams, D., Hallowell, J., Hwang, F., Kirke, A., Malik, A., … Nasuto, S. J. (2015). Music-induced emotions can be predicted from a combination of brain activity and acoustic features. Brain and Cognition, 101, 1–11.  https://doi.org/10.1016/j.bandc.2015.08.003.CrossRefGoogle Scholar
  15. Daly, I., Williams, D., Kirke, A., Weaver, J., Malik, A., Hwang, F., … Nasuto, S. J. (2016a). Affective brain–computer music interfacing. Journal of Neural Engineering, 13(4), 046022.  https://doi.org/10.1088/1741-2560/13/4/046022.CrossRefGoogle Scholar
  16. Daly, I., Williams, D., Kirke, A., Weaver, J., Malik, A., Hwang, F., … Nasuto, S. J. (2016b). An affective brain-computer music interface. In Proceedings of the 6th International Brain-Computer Interface Meeting, organized by the BCI Society.  https://doi.org/10.3217/978-3-85125-467-9-227.
  17. Darwin, C. (1998). The expression of the emotions in man and animals. Oxford University Press.Google Scholar
  18. Davidson, R. J., Abercrombie, H., Nitschke, J. B., & Putnam, K. (1999). Regional brain function, emotion and disorders of emotion. Current Opinion in Neurobiology, 9(2), 228–234.  https://doi.org/10.1016/S0959-4388(99)80032-4.CrossRefGoogle Scholar
  19. Eerola, T. (2012). Modeling listeners’ emotional response to music. Topics in Cognitive Science, 4(4), 607–624.  https://doi.org/10.1111/j.1756-8765.2012.01188.x.CrossRefGoogle Scholar
  20. Effron, D. A., Niedenthal, P. M., Gil, S., & Droit-Volet, S. (2006). Embodied temporal perception of emotion. Emotion, 6(1), 1–9.  https://doi.org/10.1037/1528-3542.6.1.1.CrossRefGoogle Scholar
  21. Erkkilä, J., Punkanen, M., Fachner, J., Ala-Ruona, E., Pöntiö, I., Tervaniemi, M., … Gold, C. (2011). Individual music therapy for depression: Randomised controlled trial. The British Journal of Psychiatry : The Journal of Mental Science, 199(2), 132–139.  https://doi.org/10.1192/bjp.bp.110.085431.CrossRefGoogle Scholar
  22. Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K.-R., et al. (2012). Enhanced performance by a hybrid NIRS-EEG brain computer interface. NeuroImage, 59(1), 519–529.  https://doi.org/10.1016/j.neuroimage.2011.07.084.CrossRefGoogle Scholar
  23. Fingelkurts, A. A., Fingelkurts, A. A., Rytsälä, H., Suominen, K., Isometsä, E., & Kähkönen, S. (2007). Impaired functional connectivity at EEG alpha and theta frequency bands in major depression. Human Brain Mapping, 28(3), 247–261.  https://doi.org/10.1002/hbm.20275.CrossRefGoogle Scholar
  24. Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007). The world of emotions is not two-dimensional. Psychological Science, 18(12), 1050–1057.  https://doi.org/10.1111/j.1467-9280.2007.02024.x.CrossRefGoogle Scholar
  25. Healey, J. A., & Picard, R. W. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156–166.  https://doi.org/10.1109/TITS.2005.848368.CrossRefGoogle Scholar
  26. Heller, W. (1993). Neuropsychological mechanisms of individual differences in emotion, personality, and arousal. Neuropsychology, 7(4), 476–489.  https://doi.org/10.1037/0894-4105.7.4.476.CrossRefGoogle Scholar
  27. Hou, J., Song, B., Chen, A. C. N., Sun, C., Zhou, J., Zhu, H., et al. (2017). Review on Neural correlates of emotion regulation and music: Implications for emotion dysregulation. Frontiers in Psychology, 8, 501.  https://doi.org/10.3389/fpsyg.2017.00501.CrossRefGoogle Scholar
  28. Hunter, P. G., & Schellenberg, E. G. (2010). Music and emotion. In Music perception, springer handbook of auditory research (pp. 129–164).Google Scholar
  29. Ishii, R., Canuet, L., Ishihara, T., Aoki, Y., Ikeda, S., Hata, M., … Takeda, M. (2014). Frontal midline theta rhythm and gamma power changes during focused attention on mental calculation: An MEG beamformer analysis. Frontiers in Human Neuroscience, 8, 406.  https://doi.org/10.3389/fnhum.2014.00406.
  30. Jones, N. A., & Fox, N. A. (1992). Electroencephalogram asymmetry during emotionally evocative films and its relation to positive and negative affectivity. Brain and Cognition, 20(2), 280–299. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1449758.
  31. Kassam, K. S., Markey, A. R., Cherkassky, V. L., Loewenstein, G., & Just, M. A. (2013). Identifying emotions on the basis of neural activation. PLoS ONE, 8(6), e66032.  https://doi.org/10.1371/journal.pone.0066032.CrossRefGoogle Scholar
  32. Kim, H.-G., Cheon, E.-J., Bai, D.-S., Lee, Y. H., & Koo, B.-H. (2018). Stress and heart rate variability: A meta-analysis and review of the literature. Psychiatry Investigation, 15(3), 235–245.  https://doi.org/10.30773/pi.2017.08.17.CrossRefGoogle Scholar
  33. Koelsch, S. (2010). Towards a neural basis of music-evoked emotions. Trends in Cognitive Sciences, 14(3), 131–137.  https://doi.org/10.1016/j.tics.2010.01.002.CrossRefGoogle Scholar
  34. Koelsch, S. (2014). Brain correlates of music-evoked emotions. Nature Reviews Neuroscience, 15(3), 170–180.  https://doi.org/10.1038/nrn3666.CrossRefGoogle Scholar
  35. Kreibig, S. D. (2010). Autonomic nervous system activity in emotion: A review. Biological Psychology, 84(3), 394–421.  https://doi.org/10.1016/j.biopsycho.2010.03.010.CrossRefGoogle Scholar
  36. Lee, Y.-Y., & Hsieh, S. (2014). Classifying different emotional states by means of EEG-based functional connectivity patterns. PLoS ONE, 9(4), e95415.  https://doi.org/10.1371/journal.pone.0095415.CrossRefGoogle Scholar
  37. Livingstone, R. S., & Thompson, W. F. (2009). The emergence of music from the theory of mind. Musicae Scientiae, 13(2 Suppl), 83–115.  https://doi.org/10.1177/1029864909013002061.CrossRefGoogle Scholar
  38. Luu, L., & Dinh, A. (2018). Artifact noise removal techniques on seismocardiogram using two tri-axial accelerometers. Sensors, 18(4), 1067.  https://doi.org/10.3390/s18041067.CrossRefGoogle Scholar
  39. Mehrabian, A. (1996). Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament. Current Psychology, 14(4), 261–292.  https://doi.org/10.1007/BF02686918.MathSciNetCrossRefGoogle Scholar
  40. Mier, D., Lis, S., Neuthe, K., Sauer, C., Esslinger, C., Gallhofer, B., & Kirsch, P. (2010). The involvement of emotion recognition in affective theory of mind. Psychophysiology, 47(6), no-no.  https://doi.org/10.1111/j.1469-8986.2010.01031.x.
  41. Müller-Putz, G. R., Breitwieser, C., Cincotti, F., Leeb, R., Schreuder, M., Leotta, F., et al. (2011). Tools for brain-computer interaction: A general concept for a hybrid BCI. Front Neuroinform, 5, 30.  https://doi.org/10.3389/fninf.2011.00030.
  42. Nicolaou, N., Malik, A., Daly, I., Weaver, J., Hwang, F., Kirke, A., … Nasuto, S. J. (2017). Directed motor-auditory EEG connectivity is modulated by music tempo. Frontiers in Human Neuroscience, 11, 502.  https://doi.org/10.3389/fnhum.2017.00502.
  43. Nicolas-Alonso, L. F., & Gomez-Gil, J. (2012). Brain computer interfaces, a review. Sensors (Basel, Switzerland), 12(2), 1211–1279.  https://doi.org/10.3390/s120201211.CrossRefGoogle Scholar
  44. Niedermeyer, E., & Da Silva, F. H. L. (2005). Electroencephalography: Basic principles, clinical applications, and related fields. Lippincott Williams & Wilkins.Google Scholar
  45. Ollander, S., Godin, C., Charbonnier, S., & Campagne, A. (2016). Selection of the most relevant physiological features for classifying emotion. In 3rd International Conference on Physiological Computing Systems (PhyCS 2016). Retrieved from https://hal.archives-ouvertes.fr/hal-01378328.
  46. Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. Cambridge: Cambridge University Press.  https://doi.org/10.1017/CBO9780511571299.CrossRefGoogle Scholar
  47. Palmiero, M., & Piccardi, L. (2017). Frontal EEG asymmetry of mood: A mini-review. Frontiers in Behavioral Neuroscience, 11, 224.  https://doi.org/10.3389/fnbeh.2017.00224.CrossRefGoogle Scholar
  48. Pfurtscheller, G., Allison, B. Z., Brunner, C., Bauernfeind, G., Solis-Escalante, T., Scherer, R., … Birbaumer, N. (2010). The hybrid BCI. Frontiers in Neuroprosthetics, 4(30).Google Scholar
  49. Pridmore, S. A. (1990). The prevalence of Huntington’s disease in Tasmania. The Medical Journal of Australia, 153(3), 133–134. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2142982.CrossRefGoogle Scholar
  50. Puglisi-Allegra, S., & Oliverio, A. (1990). Psychobiology of stress. Netherlands: Springer.CrossRefGoogle Scholar
  51. Raichle, M. E. (2011). The restless brain. Brain Connectivity, 1(1), 3–12.  https://doi.org/10.1089/brain.2011.0019.CrossRefGoogle Scholar
  52. Reeve, J. (2014). Understanding motivation and emotion (Vol. 3). Wiley. Retrieved from https://books.google.com/books?hl=en&lr=&id=zfOyBwAAQBAJ&pgis=1.
  53. Rigas, G., Goletsis, Y., & Fotiadis, D. I. (2012). Real-time driver’s stress event detection. IEEE Transactions on Intelligent Transportation Systems, 13(1), 221–234.  https://doi.org/10.1109/TITS.2011.2168215.CrossRefGoogle Scholar
  54. Rogenmoser, L., Zollinger, N., Elmer, S., & Jäncke, L. (2016). Independent component processes underlying emotions during natural music listening. Social Cognitive and Affective Neuroscience, 11(9), 1428–1439.  https://doi.org/10.1093/scan/nsw048.CrossRefGoogle Scholar
  55. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178.CrossRefGoogle Scholar
  56. Sacks, O. (2006). The power of music. Brain: A Journal of Neurology, 129(Pt 10), 2528–2532.  https://doi.org/10.1093/brain/awl234.CrossRefGoogle Scholar
  57. Sammler, D., Grigutsch, M., Fritz, T., & Koelsch, S. (2007). Music and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology, 44(2), 293–304.  https://doi.org/10.1111/j.1469-8986.2007.00497.x.CrossRefGoogle Scholar
  58. Schaffer, C. E., Davidson, R. J., & Saron, C. (1983). Frontal and parietal electroencephalogram asymmetry in depressed and nondepressed subjects. Biological Psychiatry, 18(7), 753–762. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/6615936.
  59. Schier, M. A. (2000). Changes in EEG alpha power during simulated driving: A demonstration. International Journal of Psychophysiology : Official Journal of the International Organization of Psychophysiology, 37(2), 155–162. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10832002.CrossRefGoogle Scholar
  60. Schimmack, U., & Grob, A. (2000). Dimensional models of core affect: A quantitative comparison by means of structural equation modeling. European Journal of Personality, 14(4), 21.CrossRefGoogle Scholar
  61. Schmidt, L. A., & Trainor, L. J. (2001). Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition and Emotion, 15(4), 487–500.  https://doi.org/10.1080/02699930126048.CrossRefGoogle Scholar
  62. Schomer, L., & Lopes de Silva, F. (Eds.). (2011). Niedermeyer’s electroencephalography: Basic principles, clinical applications, and related fields (6th ed.). Lippincott Williams & Wilkins.Google Scholar
  63. Silberman, E. K., & Weingartner, H. (1986). Hemispheric lateralization of functions related to emotion. Brain and Cognition, 5(3), 322–353.  https://doi.org/10.1016/0278-2626(86)90035-7.CrossRefGoogle Scholar
  64. Skouras, S., Gray, M., Critchley, H., & Koelsch, S. (2013). FMRI scanner noise interaction with affective neural processes. PLoS ONE, 8(11), e80564.  https://doi.org/10.1371/journal.pone.0080564.CrossRefGoogle Scholar
  65. Sporns, O. (2007). Brain connectivity. Scholarpedia, 2(10), 4695. Retrieved from http://www.scholarpedia.org/article/Brain_connectivity.CrossRefGoogle Scholar
  66. Stikic, M., Johnson, R. R., Tan, V., & Berka, C. (2014). EEG-based classification of positive and negative affective states. Brain-Computer Interfaces, 1(2), 99–112.  https://doi.org/10.1080/2326263X.2014.912883.CrossRefGoogle Scholar
  67. Vuoskoski, J. K., & Eerola, T. (2011). Measuring music-induced emotion: A comparison of emotion models, personality biases, and intensity of experiences. Musicae Scientiae, 15(2), 159–173.  https://doi.org/10.1177/1029864911403367.CrossRefGoogle Scholar
  68. Webber, R., Aha, D. W., Muñoz-Ávila, H., & Breslow, L. A. (2000). Advances in case-based reasoning. Advances in Case-Based Reasoning, 1898, 322–334.  https://doi.org/10.1007/3-540-44527-7.CrossRefGoogle Scholar
  69. Williams, D., Kirke, A., Miranda, E., Daly, I., Hwang, F., Weaver, J., et al. (2017). Affective calibration of musical feature sets in an emotionally intelligent music composition system. ACM Transactions on Applied Perception, 14(3), 1–13.  https://doi.org/10.1145/3059005.CrossRefGoogle Scholar
  70. Williams, D., Nasuto, S., Kirke, A., Miranda, E., Daly, I., Hallowell, J., … Hwang, F. (2015). Investigating perceived emotional correlates of rhythmic density in algorithmic music composition. ACM Transactions on Applied Perception, 12(3), 1–21.  https://doi.org/10.1145/2749466.CrossRefGoogle Scholar
  71. Winkler, I., Jäger, M., & Vojkan Mihajlović, T. T. (2010). Frontal EEG asymmetry based classification of emotional valence using common spatial patterns. World Academy of Science, Engineering and Technology, 69. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.190.6511.
  72. Zentner, M., Grandjean, D., & Scherer, K. R. (2008). Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion, 8(4), 494–521.  https://doi.org/10.1037/1528-3542.8.4.494.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Brain-Computer Interfacing and Neural Engineering Lab, Department of Computer Science and Electronic EngineeringUniversity of EssexColchesterUK
  2. 2.Digital Creativity Labs (Computer Science Department)University of YorkYorkUK

Personalised recommendations