Journal on Multimodal User Interfaces

, Volume 7, Issue 1–2, pp 111–119 | Cite as

The influence of facial interface design on dynamic emotional recognition

  • Michel Dubois
  • Damien Dupré
  • Jean-Michel Adam
  • Anna Tcherkassof
  • Nadine Mandran
  • Brigitte Meillon
Original Paper


The use of facial interfaces in distant communications highlights the relevance of emotional recognition. However researches on emotional facial expression (EFE) recognition are mainly based on static and posed stimuli and their results are not much transferable to daily interactions. The purpose of the present study is to compare emotional recognition of authentic EFEs with 11 different interface designs. A widget allowing participants both to recognize an emotion and to assess it on-line was used. Divided-face and compound-face interfaces are compared with a common full frontal interface. Analytic and descriptive on-line results reveal that some interfaces facilitate emotional recognition whereas others would decrease it. This study suggests that relevant interfaces could improve emotional recognition and thus facilitate distant communications.


Facial expression Interface  Dynamic recognition Emotional recognition  Spontaneous emotion 


  1. 1.
    O’Malley C, Langton S, Anderson A, Doherty-Sneddon G, Bruce V (1996) Comparison of face-to-face and video-mediated interaction. Interact Comput 8:177–192CrossRefGoogle Scholar
  2. 2.
    Tajariol F, Dubois M, Adam JM (2002) Mediated-communication in tutoring: When the face isn’t enough. Presented at international conference on computer and advanced technology in education, CancunGoogle Scholar
  3. 3.
    Tajariol F, Adam J-M, Dubois M (2008) Seeing the face and observing the actions: the effects of nonverbal cues on mediated tutoring dialogue. In: Woolf B, Aïmeur E, Nkambou R, Lajoie S (eds) Intelligent tutoring systems, vol. 5091. Lecture Notes in Computer Science. Springer, Berlin, pp 480–489Google Scholar
  4. 4.
    Anderson A, Garrod SC, Clark A, Boyle E, Mullin J (1992) The human communication research centre dialogue database. J Child Lang. 19:711–711CrossRefGoogle Scholar
  5. 5.
    Doherty-Sneddon G, Kent G (1996) Visual signals and the communication abilities of children. J Child Psychol Psychiatry 37:949–959CrossRefGoogle Scholar
  6. 6.
    Doherty-Sneddon G, Anderson A, O’Malley C, Langton S, Garrod S, Bruce V (1997) Face-to-face and video-mediated communication: a comparison of dialogue structure and task performance. J Exp Psychol Appl 3:105–125CrossRefGoogle Scholar
  7. 7.
    Nardi BA, Kuchinsky A, Whittaker S, Leichner R, Schwarz H (1995) Video-as-data: Technical and social aspects of a collaborative multimedia application. Comput Support Coop Work (CSCW) 4:73–100CrossRefGoogle Scholar
  8. 8.
    Gaver WW, Sellen A, Heath C, Luff P (1993) One is not enough: Multiple views in a media space. Presented at conference on human factors in computing systems, AmsterdamGoogle Scholar
  9. 9.
    Carletta J, McEwan R, Anderson A (1998) Communication in virtual supply chain teams. Globalization of manufacturing in the digital communications era of the 21st century: innovation, agility, and the virtual enterprise, pp 55–67Google Scholar
  10. 10.
    Carles L (2001) Benefits and limits of video channel on mediated interactions. Technical report 1, Geneva Interaction LabGoogle Scholar
  11. 11.
    Ekman P (1992) An argument for basic emotions. Cogn Emotion 6:169–200Google Scholar
  12. 12.
    Izard CE (1994) Innate and universal facial expressions: evidence from developmental and cross-cultural research. Psychol Bull 115:288–299CrossRefGoogle Scholar
  13. 13.
    Hess U, Thibault P (2009) Darwin and emotion expression. Am Psychol 64:120CrossRefGoogle Scholar
  14. 14.
    Ekman P, Rosenberg EL (1997) What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS). Oxford University Press, New YorkGoogle Scholar
  15. 15.
    Wagner HL, MacDonald CJ, Manstead AS (1986) Communication of individual emotions by spontaneous facial expressions. J Pers Soc Psychol 50:737CrossRefGoogle Scholar
  16. 16.
    Hess U, Banse R, Kappas A (1995) The intensity of facial expression is determined by underlying affective state and social situation. J Pers Soc Psychol 69:280CrossRefGoogle Scholar
  17. 17.
    Fernandez-Dols JM, Sanchez F, Carrera P, Ruiz-Belda MA (1997) Are spontaneous expressions and emotions linked? An experimental test of coherence. J Nonverbal Behav 21:163–177CrossRefGoogle Scholar
  18. 18.
    Nummenmaa T (1992) Pure and blended emotion in the human face: psychometric experiments. Suomalainen tiedeakatemia, HelsinkiGoogle Scholar
  19. 19.
    Wallbott HG, Scherer KR (1986) Cues and channels in emotion recognition. J Pers Soc Psychol 51:690–699CrossRefGoogle Scholar
  20. 20.
    Russell JA (1997) Reading emotions from and into faces: Resurrecting a dimensional–contextual perspective. In: Russell JA, Fernández-Dols JM (eds) The psychology of facial expression. Cambridge University Press, New York, p 320CrossRefGoogle Scholar
  21. 21.
    Tcherkassof A, Bollon T, Dubois M, Pansu P, Adam JM (2007) Facial expressions of emotions: a methodological contribution to the study of spontaneous and dynamic emotional faces. Eur J Soc Psychol 37:1325–1345CrossRefGoogle Scholar
  22. 22.
    Bassili JN (1978) Facial motion in the perception of faces and of emotional expression. J Exp Psychol Hum Percept Perform 4:373–379CrossRefGoogle Scholar
  23. 23.
    Bassili JN (1979) Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. J Pers Soc Psychol 37:2049–2058CrossRefGoogle Scholar
  24. 24.
    Hess U, Kleck RE (1990) Differentiating emotion elicited and deliberate emotional facial expressions. Eur J Soc Psychol 20:369–385CrossRefGoogle Scholar
  25. 25.
    Kamachi M, Bruce V, Mukaida S, Gyoba J, Yoshikawa S, Akamatsu S (2001) Dynamic properties influence the perception of facial expressions. Perception 30:875–887CrossRefGoogle Scholar
  26. 26.
    Manstead ASR, Fischer AH, Jakobs EB (1999) The social and emotional functions of facial displays. In: Philippot P, Feldman RS, Coats EJ (eds) The social context of nonverbal behavior. Cambridge University Press, New York, p 313Google Scholar
  27. 27.
    Ekman P, Friesen WV, Press CP (1975) Pictures of facial affect. Consulting Psychologists Press, Palo AltoGoogle Scholar
  28. 28.
    Matsumoto D, Ekman P (1995) Japanese and Caucasian facial expressions of emotion (JACFEE) and Japanese and Caucasian neutral faces (JACNeuF), intercultural and emotion research laboratory, department of psychology. San Francisco State University, San FranciscoGoogle Scholar
  29. 29.
    Bould E, Morris N (2008) Role of motion signals in recognizing subtle facial expressions of emotion. Br J Psychol 99:167–189CrossRefGoogle Scholar
  30. 30.
    Motley MT, Camden CT (1988) Facial expression of emotion: a comparison of posed expressions versus spontaneous expressions in an interpersonal communication setting. West J Commun (includes communication reports) 52:1–22Google Scholar
  31. 31.
    Heath C, Luff P (1991) Disembodied conduct: communication through video in a multi-media office environment. Presented at conference on human factors in computing systems: reaching through technology. New OrleansGoogle Scholar
  32. 32.
    Lemay G, Kirouac G, Lacouture Y (1995) Expressions faciales émotionnelles spontanées dynamiques et statiques: comparaison d’études de jugement catégoriel et dimensionnel. Dynamic and static spontaneous facial expressions of emotion: A comparison of categorical and dimensional judgments. Can J Behav Sci (Revue canadienne des sciences du comportement) 27:125–139CrossRefGoogle Scholar
  33. 33.
    Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. Presented at international conference on automatic face and gesture recognition, GrenobleGoogle Scholar
  34. 34.
    Ambadar Z, Schooler JW, Cohn JF (2005) Deciphering the enigmatic face. Psychol Sci 16:403–410CrossRefGoogle Scholar
  35. 35.
    Back DE, Jordan TR, Thomas SM (2009) The recognition of mental states from dynamic and static facial expressions. Visual Cogn 17:1271–1286CrossRefGoogle Scholar
  36. 36.
    Short J, Williams E, Christie B (1976) The social psychology of telecommunications. Wiley, LondonGoogle Scholar
  37. 37.
    McKeown G, Valstar MF, Cowie R, Pantic M (2010) The SEMAINE corpus of emotionally coloured character interactions. Presented at international conference on multimedia and expo (ICME), Suntec Google Scholar
  38. 38.
    Sneddon I, McRorie M, McKeown G, Hanratty J (2012) The Belfast induced natural emotion database. IEEE Trans Affect Comput 3:32–41CrossRefGoogle Scholar
  39. 39.
    Tcherkassof A, Dupré D, Meillon B, Mandran N, Dubois M, Adam JM (2012) DynEmo: a video database of natural facial expressions of emotions. IEEE Trans Affect Comput (submitted)Google Scholar
  40. 40.
    Sweller J, Chandler P (1994) Why some material is difficult to learn. Cogn Instr 12:185–233CrossRefGoogle Scholar
  41. 41.
    Anderson AH, Smallwood L, MacDonald R, Mullin J, Fleming AM, O’Malley C (2000) Video data and video links in mediated communication: what do users value? Int J Hum Comput Stud 52:165–187CrossRefGoogle Scholar
  42. 42.
    Roy C, Blais C, Fiset D, Gosselin F (2010) Visual information extraction for static and dynamic facial expression of emotions: an eye-tracking experiment. J Vision 10:531–531CrossRefGoogle Scholar
  43. 43.
    Dupré D, Dubois M, Mandran N, Tcherkassof A (2010) Emotional facial expression interface: effects of displayed facial designs. Presented at proceedings of ErgoIA 2010 conference, BiarritzGoogle Scholar
  44. 44.
    Douglas-Cowie E, Cowie R, Sneddon I, Cox C, Lowry O, McRorie M, Martin JC, Devillers L, Abrilian S, Batliner A (2007) The humaine database: addressing the collection and annotation of naturalistic and induced emotional data. Presented at affective computing and intelligent interaction, BerlinGoogle Scholar
  45. 45.
    Mayer RE, Anderson RB (1992) The instructive animation: helping students build connections between words and pictures in multimedia learning. J Educ Psychol 84:444–452CrossRefGoogle Scholar
  46. 46.
    Meillon B, Tcherkassov A, Mandran N, Adam JM, Dubois M, Dupré D, Benoit AM, Guérin-Dugué A, Caplier A (2010) Dynemo: a corpus of dynamic and spontaneous emotional facial expressions. Presented at international workshop series on multimodal corpora, tools and resources: advances in capturing. Coding and analyzing multimodality, MalteGoogle Scholar

Copyright information

© OpenInterface Association 2012

Authors and Affiliations

  • Michel Dubois
    • 1
  • Damien Dupré
    • 1
  • Jean-Michel Adam
    • 2
  • Anna Tcherkassof
    • 1
  • Nadine Mandran
    • 2
  • Brigitte Meillon
    • 2
  1. 1.Laboratoire Inter-Universitaire de Psychologie (LIP)University of GrenobleGrenobleFrance
  2. 2.Laboratoire d’Informatique de Grenoble (LIG)University of GrenobleGrenobleFrance

Personalised recommendations