Advertisement

RAMAS: Russian Multimodal Corpus of Dyadic Interaction for Affective Computing

  • Olga Perepelkina
  • Evdokia Kazimirova
  • Maria Konstantinova
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11096)

Abstract

Emotion expression encompasses various types of information, including face and eye movement, voice and body motion. Emotions collected from real conversations are difficult to classify using one channel. That is why multimodal techniques have recently become more popular in automatic emotion recognition. Multimodal databases that include audio, video, 3D motion capture and physiology data are quite rare. We collected The Russian Acted Multimodal Affective Set (RAMAS) − the first multimodal corpus in Russian language. Our database contains approximately 7 h of high-quality close-up video recordings of faces, speech, motion-capture data and such physiological signals as electro-dermal activity and photoplethysmogram. The subjects were 10 actors who played out interactive dyadic scenarios. Each scenario involved one of the basic emotions: Anger, Sadness, Disgust, Happiness, Fear or Surprise, and such characteristics of social interaction like Domination and Submission. In order to note emotions that subjects really felt during the process we asked them to fill in short questionnaires (self-reports) after each played scenario. The records were marked by 21 annotators (at least five annotators marked each scenario). We present our multimodal data collection, annotation process, inter-rater agreement analysis and the comparison between self-reports and received annotations. RAMAS is an open database that provides research community with multimodal data of faces, speech, gestures and physiology interrelation. Such material is useful for various investigations and automatic affective systems development.

Keywords

Affective computing Multimodal affect recognition Multimodal database Russian emotion database 

Notes

Acknowledgements

Supported by Neurodata Lab LLC. The authors would like to thank Elena Arkova for finding the actors and helping with the scenarios and experimental procedure, and Irina Vetrova for evaluating the emotional intelligence of the annotators with MSCEIT v 2.0 test.

References

  1. 1.
  2. 2.
  3. 3.
    Neurodata Lab LLC. http://www.neurodatalab.com
  4. 4.
  5. 5.
  6. 6.
    Anderson, A., Hsiao, T., Metsis, V.: Classification of emotional arousal during multimedia exposure. In: Proceedings of the 10th International Conference on Pervasive Technologies Related to Assistive Environments, pp. 181–184. ACM (2017)Google Scholar
  7. 7.
    Ayvaz, U., Gürüler, H., Devrim, M.O.: Use of facial emotion recognition in e-learning systems. Inf. Technol. Learn. Tools 60(4), 95–104 (2017)Google Scholar
  8. 8.
    Bänziger, T., Pirker, H., Scherer, K.: GEMEP-GEneva multimodal emotion portrayals: a corpus for the study of multimodal emotional expressions. In: Proceedings of LREC, vol. 6, pp. 15–19 (2006)Google Scholar
  9. 9.
    Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S.: IEMOCAP: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42(4), 335 (2008)CrossRefGoogle Scholar
  10. 10.
    Chaw, T.V., Khor, S.W., Lau, P.Y.: Facial expression recognition using correlation of eyes regions. In: The FICT Colloquium 2016, p. 34, December 2016Google Scholar
  11. 11.
    De Silva, L.C., Miyasato, T., Nakatsu, R.: Facial emotion recognition using multi-modal information. In: Proceedings of 1997 International Conference on Information, Communications and Signal Processing, ICICS 1997, vol. 1, pp. 397–401. IEEE (1997)Google Scholar
  12. 12.
    D’mello, S.K., Kory, J.: A review and meta-analysis of multimodal affect detection systems. ACM Comput. Surv. 47(3), 43:1–43:36 (2015). http://doi.acm.org/10.1145/2682899Google Scholar
  13. 13.
    Douglas, M.: Purity and danger: an analysis of pollution and taboo London (1966)Google Scholar
  14. 14.
    El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recogn. 44(3), 572–587 (2011)CrossRefGoogle Scholar
  15. 15.
    Gouizi, K., Bereksi Reguig, F., Maaoui, C.: Emotion recognition from physiological signals. J. Med. Eng. Technol. 35(6–7), 300–307 (2011)CrossRefGoogle Scholar
  16. 16.
    Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007)CrossRefGoogle Scholar
  17. 17.
    Karg, M., Samadani, A.A., Gorbet, R., Kühnlenz, K., Hoey, J., Kulić, D.: Body movements for affective expression: a survey of automatic recognition and generation. IEEE Trans. Affect. Comput. 4(4), 341–359 (2013)CrossRefGoogle Scholar
  18. 18.
    Krippendorff, K.: Estimating the reliability, systematic error and random error of interval data. Educ. Psychol. Meas. 30(1), 61–70 (1970)CrossRefGoogle Scholar
  19. 19.
    Mayer, J.D., Salovey, P., Caruso, D.R., Sitarenios, G.: Measuring emotional intelligence with the MSCEIT V2. 0. Emotion 3(1), 97 (2003)CrossRefGoogle Scholar
  20. 20.
    Metallinou, A., Lee, C.C., Busso, C., Carnicke, S., Narayanan, S.: The USC creativeIT database: a multimodal database of theatrical improvisation. Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, p. 55 (2010)Google Scholar
  21. 21.
    Rachman, S.: Anxiety. Psychology Press Ltd., Publishers, East Sussex (1998)Google Scholar
  22. 22.
    Ranganathan, H., Chakraborty, S., Panchanathan, S.: Multimodal emotion recognition using deep learning architectures. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9, March 2016.  https://doi.org/10.1109/WACV.2016.7477679
  23. 23.
    Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2013)Google Scholar
  24. 24.
    Russell, J.A., Fernández-Dols, J.M.: The Psychology of Facial Expression. Cambridge University Press, Cambridge (1997)CrossRefGoogle Scholar
  25. 25.
    Sergienko, E.G., Vetrova, I.I., Volochkov, A.A., Popov, A.Y.: Adaptation of J. Mayer P. Salovey and D. Caruso emotional intelligence test on russian-speaking sample. Psikhologicheskii Zhurnal 31(1), 55–73 (2010)Google Scholar
  26. 26.
    Sloetjes, H., Wittenburg, P.: Annotation by category: ELAN and ISO DCR. In: LREC (2008)Google Scholar
  27. 27.
    Tarnowski, P., Kołodziej, M., Majkowski, A., Rak, R.J.: Emotion recognition using facial expressions. Procedia Comput. Sci. 108, 1175–1184 (2017)CrossRefGoogle Scholar
  28. 28.
    Tomkins, S.: Affect Imagery Consciousness: Volume II: The Negative Affects. Springer, New York (1963)Google Scholar
  29. 29.
    Volkova, E., De La Rosa, S., Bülthoff, H.H., Mohler, B.: The MPI emotional body expressions database for narrative scenarios. PloS one 9(12), e113647 (2014)CrossRefGoogle Scholar
  30. 30.
    Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 831–834. ACM (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Neurodata Lab LLCMiamiUSA
  2. 2.Lomonosov Moscow State UniversityMoscowRussia

Personalised recommendations