Multi-modal Affect Induction for Affective Brain-Computer Interfaces

  • Christian Mühl
  • Egon L. van den Broek
  • Anne-Marie Brouwer
  • Femke Nijboer
  • Nelleke van Wouwe
  • Dirk Heylen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6974)


Reliable applications of affective brain-computer interfaces (aBCI) in realistic, multi-modal environments require a detailed understanding of the processes involved in emotions. To explore the modality-specific nature of affective responses, we studied neurophysiological responses (i.e., EEG) of 24 participants during visual, auditory, and audiovisual affect stimulation. The affect induction protocols were validated by participants’ subjective ratings and physiological responses (i.e., ECG). Coherent with literature, we found modality-specific responses in the EEG: posterior alpha power decreases during visual stimulation and increases during auditory stimulation, anterior alpha power tends to decrease during auditory stimulation and to increase during visual stimulation. We discuss the implications of these results for multi-modal aBCI.


affective brain-computer interfaces emotion ECG EEG visual auditory multi-modal 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aftanas, L.I., Reva, N.V., Varlamov, A.A., Pavlov, S.V., Makhnev, V.P.: Analysis of evoked eeg synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics. Neuroscience and Behavioral Physiology 34(8), 859–867 (2004)CrossRefGoogle Scholar
  2. 2.
    Baumgartner, T., Esslen, M., Jancke, L.: From emotion perception to emotion experience: Emotions evoked by pictures and classical music. International Journal of Psychophysiology 60(1), 34–43 (2006)CrossRefGoogle Scholar
  3. 3.
    Bradley, M.M., Lang, P.J.: Affective reactions to acoustic stimuli. Psychophysiology 37(2), 204–215 (2000)CrossRefGoogle Scholar
  4. 4.
    Bradley, M.M., Lang, P.J.: The international affective digitized sounds (IADS-2): Affective ratings of sounds and instruction manual. Technical report, University of Florida, Center for Research in Psychophysiology, Gainesville, Fl, USA (2007)Google Scholar
  5. 5.
    Brefczynski-Lewis, J., Lowitszch, S., Parsons, M., Lemieux, S., Puce, A.: Audiovisual non-verbal dynamic faces elicit converging fMRI and ERP responses. Brain Topography 21(3-4), 193–206 (2009)CrossRefGoogle Scholar
  6. 6.
    Codispoti, M., Ferrari, V., Bradley, M.M.: Repetitive picture processing: autonomic and cortical correlates. Brain Research 1068(1), 213–220 (2006)CrossRefGoogle Scholar
  7. 7.
    Ethofer, T., Van De Ville, D., Scherer, K., Vuilleumier, P.: Decoding of emotional information in voice-sensitive cortices. Current Biology 19(12), 1028–1033 (2009)CrossRefGoogle Scholar
  8. 8.
    Grandjean, D., Sander, D., Pourtois, G., Schwartz, S., Seghier, M.L., Scherer, K., Vuilleumier, P.: The voices of wrath: brain responses to angry prosody in meaningless speech. Nature Neuroscience 8(2), 145–146 (2005)CrossRefGoogle Scholar
  9. 9.
    Guntekin, B., Basar, E.: Emotional face expressions are differentiated with brain oscillations. International Journal of Psychophysiology 64(1), 91–100 (2007)CrossRefGoogle Scholar
  10. 10.
    Hadjikhani, N., de Gelder, B.: Seeing fearful body expressions activates the fusiform cortex and amygdala. Current Biology 13(24), 2201–2205 (2003)CrossRefGoogle Scholar
  11. 11.
    Hajcak, G., MacNamara, A., Olvet, D.M.: Event-related potentials, emotion, and emotion regulation: An integrative review. Developmental Neuropsychology 35(2), 129–155 (2010)CrossRefGoogle Scholar
  12. 12.
    Hari, R., Salmelin, R., Makela, J.P., Salenius, S., Helle, M.: MEG cortical rhythms. International Journal of Psychophysiology 26, 51–62 (1997)CrossRefGoogle Scholar
  13. 13.
    Jensen, O., Mazaheri, A.: Shaping functional architecture by oscillatory alpha activity: gating by inhibition. Frontiers in Human Neuroscience 4 (2010)Google Scholar
  14. 14.
    Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)CrossRefGoogle Scholar
  15. 15.
    Kober, H., Feldman Barrett, L., Joseph, J., Bliss-Moreau, E., Lindquist, K., Wager, T.D.: Functional grouping and cortical-subcortical interactions in emotion: a meta-analysis of neuroimaging studies. NeuroImage 42(2), 998–1031 (2008)CrossRefGoogle Scholar
  16. 16.
    Krause, C.M.: Cognition- and memory-related erd/ers responses in the auditory stimulus modality. Progress in brain research 159, 197–207 (2006)CrossRefGoogle Scholar
  17. 17.
    Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): Technical manual and affective ratings. Technical report, University of Florida, Center for Research in Psychophysiology, Gainesville, Fl, USA (1999)Google Scholar
  18. 18.
    Näätänena, R., Paavilainen, P., Rinne, T., Alho, K.: The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clinical Neurophysiology 118(12), 2544–2590 (2007)CrossRefGoogle Scholar
  19. 19.
    Panksepp, J., Bernatzky, G.: Emotional sounds and the brain: the neuro-affective foundations of musical appreciation. Behavioral Processes 60, 133–155 (2002)CrossRefGoogle Scholar
  20. 20.
    Pfurtscheller, G., Lopes da Silva, F.H.: Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology 110(11), 1842–1857 (1999)CrossRefGoogle Scholar
  21. 21.
    Picard, R.W.: Affective Computing. The MIT Press, Cambridge (1997)CrossRefGoogle Scholar
  22. 22.
    Pourtois, G., Vuilleumier, P.: Dynamics of emotional effects on spatial attention in the human visual cortex. Progress in Brain Research, vol. 156, pp. 67–91. Elsevier, Amsterdam (2006)Google Scholar
  23. 23.
    Sabatinelli, D., Lang, P.J., Keil, A., Bradley, M.M.: Emotional perception: Correlation of functional MRI and event-related potentials. Cerebral Cortex 17(5), 1085–1091 (2007)CrossRefGoogle Scholar
  24. 24.
    Sander, D., Grandjean, D., Scherer, K.R.: A systems approach to appraisal mechanisms in emotion. Neural Networks 18(4), 317–352 (2005)CrossRefGoogle Scholar
  25. 25.
    Vuilleumier, P.: How brains beware: neural mechanisms of emotional attention. Trends in Cognitive Sciences 9(12), 585–594 (2005)CrossRefGoogle Scholar
  26. 26.
    Wang, J., Nicol, T., Skoe, E., Sams, M., Kraus, N.: Emotion modulates early auditory response to speech. Journal of Cognitive Neuroscience 21(11), 2121–2128 (2009)CrossRefGoogle Scholar
  27. 27.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 39–58 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Christian Mühl
    • 1
  • Egon L. van den Broek
    • 1
    • 3
    • 4
  • Anne-Marie Brouwer
    • 2
  • Femke Nijboer
    • 1
  • Nelleke van Wouwe
    • 2
  • Dirk Heylen
    • 1
  1. 1.Human Media InteractionUniversity of TwenteEnschedeThe Netherlands
  2. 2.TNO Behavioural and Societal SciencesSoesterbergThe Netherlands
  3. 3.Human-Centered Computing ConsultancyViennaAustria
  4. 4.Karakter University CenterRadboud University Medical CenterNijmegenThe Netherlands

Personalised recommendations