Abstract
To obtain further evidence to support spontaneous attention to vocal tones in Japanese, in this study, we developed a new set of stimuli, including positive and negative emotions, and investigated responses to facial and vocal displays of emotion by using behavioral and event-related potential (ERP) measurements. Japanese participants either judged face as pleasant or unpleasant while ignoring the attendant voice or judged vocal tone as pleasant or unpleasant as while ignoring the attendant face. As predicted, an interference effect by to-be-ignored information, which was computed by subtracting accuracy for incongruous face-voice stimuli from accuracy for congruous ones, was greater in face judgments than in voice judgments, suggesting an attentional bias for vocal tone. Moreover, corresponding to this pattern in accuracy, a positive ERP occurring at approximately 200 ms after the onset of the stimulus (called P2) was more likely to be elicited by incongruous stimuli than by congruous ones, and the difference in P2 amplitude was greater in face judgments than in voice judgments. A negative ERP occurring relatively late in the processing (called N400) was also more likely to be elicited by incongruous stimuli than by congruous stimuli; however, the size was not significantly different between these judgments. Our findings are the first to provide evidence that Japanese pay spontaneous attention to emotional vocal tones over emotional faces at behavioral and neural levels.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The data of this study are available on request to the corresponding author.
Notes
Although Tanaka et al. (2010) did not report the results of response time, Liu et al. (2015) analyzed them and found that the interference effect by to-be-ignored channels (i.e., response time was significantly longer for incongruous stimuli than congruous ones) was influenced by neither judgment nor culture. For exploratory purposes, we performed a 2 (judgment condition) × 2 (face) × 2 (voice) ANOVA on mean response times and found a significant interaction between face and voice, F(1, 46) = 69.46, p < .001, η2p = .60. Response time was slower for incongruous stimuli than for congruous stimuli when vocal tone was positive (Ms = 1,137 ms vs. 1,083 ms, F(1, 46) = 18.02, p < .001, η2p = .28) and negative (Ms = 1099 ms vs. 1006 ms, F(1, 46) = 75.28, p < .001, η2p = .62). The difference between incongruous and congruous stimuli was also significant when faces were negative (F(1, 46) = 53.92, p < .001, η2p = .54), whereas it was not significant when faces were positive (F(1, 46) = 1.02, p = .32, η2p = .02). However, the interaction was not qualified with the judgment condition, F(1, 46) = 0.12, p = .73, η2p = .003, which was consistent with Liu et al. (2015).
Preliminary analyses including gender on accuracy and response time showed no significant effect of gender (ps > .09). Thus, we did not include gender in the analysis of accuracy reported in the main text.
To see a potential effect of gender, we performed a 2 (gender) × 2 (judgment) × 2 (congruity) ANOVA on P2 and N400 amplitudes. There was no significant effect of gender in the analysis of P2 (ps > .27). In contrast, only gender x judgment x congruity interaction was significant in the analysis of N400, F(1, 44) = 4.56, p = .038, η2p = .09. To see the detail of the interaction, we performed further a 2 × 2 ANOVA on N400 for each judgmental condition. In the facial judgment condition, there was no significant effect of gender (ps > .51). In contrast, in the vocal judgment condition, the gender x congruity interaction was almost significant, F(1, 44) = 4.03, p = .051, η2p = .08. The simple main effect analysis showed only the significant main effect of congruency in female participants, F(1, 22) = 14.86, p < .001, η2p = .40. Thus, incongruity-based N400 tended to be rather evident in the vocal judgment in female participants. On the other hand, we did not find an expected gender difference, such as larger N400 in female than in male, in the facial judgment.
References
Bjornsdottir, R. T., Tskhay, K. O., Ishii, K., & Rule, N. O. (2017). Cultural differences in perceiving and processing emotions: a holistic approach to person perception. Culture and Brain, 5, 105–124.
de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14, 289–311.
Gray, H. M., Ishii, K., & Ambady, N. (2011). Misery loves company: When sadness increases the desire for social connectedness. Personality and Social Psychology Bulletin, 37, 1438–1448.
Hall, E. T. (1976). Beyond culture. Doubleday.
Ishii, K. (2013). Culture and the mode of thought: A review. Asian Journal of Social Psychology, 16, 123–132.
Ishii, K., Kobayashi, Y., & Kitayama, S. (2010). Interdependence modulates the brain response to word-voice incongruity. Social Cognitive and Affective Neuroscience, 5, 307–317.
Ishii, K., Reyes, J. A., & Kitayama, S. (2003). Spontaneous attention to word content versus emotional tone: differences among three cultures. Psychological Science, 14, 39–46.
Kitayama, S., & Ishii, K. (2002). Word and Voice: Spontaneous attention to emotional speech in two cultures. Cognition and Emotion, 16, 29–59.
Kotz, S. A., & Paulmann, S. (2011). Emotion, language, and the brain. Language and Linguistics Compass, 5, 108–125.
Liu, P., Rigoulot, S., & Pell, M. D. (2015). Culture modulates the brain response to human expressions of emotion: Electrophysiological evidence. Neuropsychologia, 67, 1–13.
Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG-and MEG-data. Journal of Neuroscience Methods, 164, 177–190.
Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224–253.
Masuda, T., Gonzalez, R., Kwan, L., & Nisbett, R. E. (2008). Culture and aesthetic preference: Comparing the attention to context of East Asians and Americans. Personality and Social Psychology Bulletin, 34, 1260–1275.
Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: Holistic vs. analytic cognition. Psychological Review, 108, 291–310.
Paulmann, S., Bleichner, M., & Kotz, S. A. (2013). Valence, arousal, and task effects in emotional prosody processing. Frontiers in Psychology, 4, 345.
Pickett, C. L., Gardner, W. L., & Knowles, M. (2004). Getting a cue: The need to belong and enhanced sensitivity to social cues. Personality and Social Psychology Bulletin, 30, 1095–1107.
Schirmer, A., & Kotz, S. A. (2003). ERP evidence for a gender-specific Stroop effect in emotional speech. Journal of Cognitive Neuroscience, 15, 1135–1148.
Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends in Cognitive Sciences, 10, 24–30.
Stroop, J. R. (1935). Studies of interference in serial verbal reaction. Journal of Experimental Psychology, 18, 643–662.
Tadel, F., Baillet, S., Mosher, J. C., Pantazis, D., & Leahy, R. M. (2011). Brainstorm: A user-friendly application for MEG/EEG analysis. Computational Intelligence and Neuroscience, 2011, 879716.
Takagi, S., Hiramatsu, S., & Tanaka, A. (2014). Emotion perception from audiovisual expression by faces and voices. Cognitive Studies, 21, 344–362. (In Japanese with English abstract).
Tanaka, A., Koizumi, A., Imai, H., Hiramatsu, S., Hiramoto, E., & de Gelder, B. (2010). I feel your voice: Cultural differences in the multisensory perception of emotion. Psychological Science, 21, 1259–1262.
Whorf, B. L. (1956). Language, thought, and reality: Selected writings of Benjamin Lee Whorf. Wiley.
Zhou, Y., & Han, S. (2021). Neural dynamics of pain expression processing: Alpha-band synchronization to same-race pain but desynchronization to other-race pain. NeuroImage, 224, 117400.
Acknowledgements
The research was supported by Topic-Setting Program to Advance Cutting-Edge Humanities and Social Sciences Research Area Cultivation (#D-4), the Japan Society for the Promotion Science.
Funding
The research was supported by Topic-Setting Program to Advance Cutting-Edge Humanities and Social Sciences Research Area Cultivation (#D-4), the Japan Society for the Promotion Science.
Author information
Authors and Affiliations
Contributions
YM and KI designed research. YM, AT, and KI prepared stimuli. YM performed research. YM, YN, and KI analyzed data and wrote the paper.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be constructed as a potential conflict of interest.
Ethical approval
This study was reviewed and approved by the ethics committees at Kobe University. The participants provided written informed consent at the beginning of the study. All responses were kept confidential.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mori, Y., Noguchi, Y., Tanaka, A. et al. Neural responses to facial and vocal displays of emotion in Japanese people. Cult. Brain 10, 43–55 (2022). https://doi.org/10.1007/s40167-021-00101-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40167-021-00101-0