You Sound Like Your Counterpart: Interpersonal Speech Analysis

  • Jing HanEmail author
  • Maximilian Schmitt
  • Björn Schuller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11096)


In social interaction, people tend to mimic their conversational partners both when they agree and disagree. Research on this phenomenon is complex but not recent in theory, and related studies show that mimicry can enhance social relationships, increase affiliation and rapport. However, automatically recognising such a phenomenon is still in its early development. In this paper, we analyse mimicry in the speech domain and propose a novel method by using hand-crafted low-level acoustic descriptors and autoencoders (AEs). Specifically, for each conversation, two AEs are built, one for each speaker. After training, the acoustic features of one speaker are tested with the AE that is trained on the features of her counterpart. The proposed approach is evaluated on a database consisting of almost 400 subjects from 6 different cultures, recorded in-the-wild. By calculating the AE’s reconstruction errors of all speakers and analysing the errors at different times in their interactions, we show that, albeit to different degrees from culture to culture, mimicry arises in most interactions.


Affective computing Conversation analysis Computational paralinguistics 



The research leading to these results has received funding from the European Union’s Horizon 2020 Programme under GA No. 645094 (Innovation Action SEWA) and through the EFPIA Innovative Medicines Initiative under GA No. 115902 (RADAR-CNS).


  1. 1.
    Amiriparian, S., Freitag, M., Cummins, N., Schuller, B.: Sequence to sequence autoencoders for unsupervised representation learning from audio. In: Proceedings of the DCASE 2017 Workshop, Munich, Germany (2017)Google Scholar
  2. 2.
    Baltrušaitis, T., Robinson, P., Morency, L.P.: OpenFace: An open source facial behavior analysis toolkit. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, pp. 1–10 (2016)Google Scholar
  3. 3.
    Bilakhia, S., Petridis, S., Pantic, M.: Audiovisual detection of behavioural mimicry. In: Proceedings of Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, pp. 123–128 (2013)Google Scholar
  4. 4.
    Burgoon, J.K., Hubbard, A.E.: Cross-cultural and intercultural applications of expectancy violations theory and interaction adaptation theory. In: Gudykunst, W.B. (ed.) Theorizing about Intercultural Communication, pp. 149–171. Sage Publications, Beverly Hills (2005)Google Scholar
  5. 5.
    Chartrand, T.L., Bargh, J.A.: The chameleon effect: the perception-behavior link and social interaction. J. Pers. Soc. Psychol. 76(6), 893–910 (1999)CrossRefGoogle Scholar
  6. 6.
    Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., Cohen, D.: Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans. Affect. Comput. 3(3), 349–365 (2012)CrossRefGoogle Scholar
  7. 7.
    Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Nebraska Symposium on Motivation. University of Nebraska Press (1971)Google Scholar
  8. 8.
    Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Ishk, Ujjain (2003)Google Scholar
  9. 9.
    Eyben, F.: Real-Time Speech and Music Classification by Large Audio Feature Space Extraction. Springer, Switzerland (2016). Scholar
  10. 10.
    Eyben, F., Weninger, F., Groß, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia (MM), Barcelona, Spain, pp. 835–838 (2013)Google Scholar
  11. 11.
    Friesen, E., Ekman, P.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  12. 12.
    Gueguen, N., Jacob, C., Martin, A.: Mimicry in social interaction: its effect on human judgment and behavior. Eur. J. Soc. Sci. 8(2), 253–259 (2009)Google Scholar
  13. 13.
    Hess, U., Fischer, A.: Emotional mimicry as social regulation. Pers. Soc. Psychol. Rev. 17(2), 142–157 (2013)CrossRefGoogle Scholar
  14. 14.
    Hess, U., Fischer, A.: Emotional mimicry: why and when we mimic emotions. Soc. Pers. Psychol. Compass 8(2), 45–57 (2014)CrossRefGoogle Scholar
  15. 15.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, NV, pp. 3111–3119 (2013)Google Scholar
  16. 16.
    Parrill, F., Kimbara, I.: Seeing and hearing double: the influence of mimicry in speech and gesture on observers. J. Nonverbal Behav. 30(4), 157 (2006)CrossRefGoogle Scholar
  17. 17.
    Ringeval, F., et al.: AVEC 2018 workshop and challenge: Bipolar disorder and cross-cultural affect recognition. In: Proceedings of the 8th Annual Workshop on Audio/Visual Emotion Challenge, Seoul, Korea (2018, to appear)Google Scholar
  18. 18.
    Ringeval, F., et al.: AVEC 2017: Real-life depression, and affect recognition workshop and challenge. In: Proc. of the 7th Annual Workshop on Audio/Visual Emotion Challenge, Mountain View, CA, pp. 3–9 (2017)Google Scholar
  19. 19.
    Schmitt, M., Ringeval, F., Schuller, B.: At the border of acoustics and linguistics: bag-of-audio-words for the recognition of emotions in speech. In: Proceedings of INTERSPEECH, San Francisco, CA, pp. 495–499 (2016)Google Scholar
  20. 20.
    Schuller, B., et al.: The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. In: Proceedings of INTERSPEECH, Lyon, France, pp. 148–152 (2013)Google Scholar
  21. 21.
    Scissors, L.E., Gill, A.J., Gergle, D.: Linguistic mimicry and trust in text-based CMC. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, San Diego, CA, pp. 277–280 (2008)Google Scholar
  22. 22.
    Stivers, T., et al.: Universals and cultural variation in turn-taking in conversation. Proc. Nat. Acad. Sci. U.S.A 106(26), 10587–10592 (2009)CrossRefGoogle Scholar
  23. 23.
    Sun, X., Nijholt, A., Truong, K.P., Pantic, M.: Automatic visual mimicry expression analysis in interpersonal interaction. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Colorado Springs, CO, pp. 40–46 (2011)Google Scholar
  24. 24.
    Swaab, R.I., Maddux, W.W., Sinaceur, M.: Early words that work: When and how virtual linguistic mimicry facilitates negotiation outcomes. J. Exper. Soc. Psychol. 47(3), 616–621 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jing Han
    • 1
    Email author
  • Maximilian Schmitt
    • 1
  • Björn Schuller
    • 1
    • 2
  1. 1.ZD.B Chair of Embedded Intelligence for Health Care and WellbeingUniversity of AugsburgAugsburgGermany
  2. 2.GLAM – Group on Language, Audio & MusicImperial College LondonLondonUK

Personalised recommendations