Laugh When You’re Winning

  • Maurizio Mancini
  • Laurent Ach
  • Emeline Bantegnie
  • Tobias Baur
  • Nadia Berthouze
  • Debajyoti Datta
  • Yu Ding
  • Stéphane Dupont
  • Harry J. Griffin
  • Florian Lingenfelser
  • Radoslaw Niewiadomski
  • Catherine Pelachaud
  • Olivier Pietquin
  • Bilal Piot
  • Jérôme Urbain
  • Gualtiero Volpe
  • Johannes Wagner
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 425)

Abstract

Developing virtual characters with naturalistic game playing capabilities is an increasingly researched topic in Human-Computer Interaction. Possible roles for such characters include virtual teachers, personal care assistants, and companions for children. Laughter is an under-investigated emotional expression both in Human-Human and Human-Computer Interaction. The EU Project ILHAIRE, aims to study this phenomena and endow machines with laughter detection and synthesis capabilities. The Laugh when you’re winning project, developed during the eNTERFACE 2013 Workshop in Lisbon, Portugal, aimed to set up and test a game scenario involving two human participants and one such virtual character. The game chosen, the yes/no game, induces natural verbal and non-verbal interaction between participants, including frequent hilarious events, e.g., one of the participants saying “yes” or “no” and so losing the game. The setup includes software platforms, developed by the ILHAIRE partners, allowing automatic analysis and fusion of human participants’ multimodal data (voice, facial expression, body movements, respiration) in real-time to detect laughter. Further, virtual characters endowed with multimodal skills were synthesised in order to interact with the participants by producing laughter in a natural way.

Keywords

HCI laughter virtual characters game detection fusion multimodal 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bachorowski, J., Smoski, M.J., Owren, M.J.: Automatic discrimination between laughter and speech. Journal of the Acoustical Society of America 110, 1581–1597 (2001)CrossRefGoogle Scholar
  2. 2.
    Becker-Asano, C., Ishiguro, H.: Laughter in social robotics - no laughing matter. In: International Workshop on Social Intelligence Design (SID 2009), pp. 287–300 (2009)Google Scholar
  3. 3.
    Becker-Asano, C., Kanda, T., Ishi, C., Ishiguro, H.: How about laughter? perceived naturalness of two laughing humanoid robots. In: 3rd International Conference on Affective Computing & Intelligent Interaction, Affective Computing and Intelligent Interaction and Workshops, pp. 1–6 (2009)Google Scholar
  4. 4.
    Bernhardt, D., Robinson, P.: Detecting affect from non-stylised body motions. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 59–70. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Bourgeois, P., Hess, U.: The impact of social context on mimicry. Biological Psychology 77(3), 343–352 (2008)CrossRefGoogle Scholar
  6. 6.
    Brand, M.: Voice puppetry. In: Proceedings of Conference on Computer Graphics and Interactive Techniques, pp. 21–28 (1999)Google Scholar
  7. 7.
    Bregler, C., Covell, M., Slaney, M.: Video rewrite: Driving visual speech with audio. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, pp. 353–360. ACM Press/Addison-Wesley Publishing Co., New York (1997), http://dx.org/10.1145/258734.258880 CrossRefGoogle Scholar
  8. 8.
    Cai, R., Lu, L., Zhang, H., Cai, L.: Highlight sound effects detection in audio stream. In: Proceedings of the 2003 IEEE International Conference on Multimedia and Expo (ICME). Baltimore, USA, pp. 37–40 (2003)Google Scholar
  9. 9.
    Castellano, G., Villalba, S.D., Camurri, A.: Recognising human emotions from body movement and gesture dynamics. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 71–82. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  10. 10.
    Cohen, M.M., Massaro, D.W.: Modeling coarticulation in synthetic visual speech. In: Models and Techniques in Computer Animation, pp. 139–156. Springer (1993)Google Scholar
  11. 11.
    Cosker, D., Edge, J.: Laughing, crying, sneezing and yawning: Automatic voice driven animation of non-speech articulations. In: Proceedings of Computer Animation and Social Agents (CASA 2009), pp. 21–24 (2009)Google Scholar
  12. 12.
    Deng, Z., Lewis, J., Neumann, U.: Synthesizing speech animation by learning compact speech co-articulation models. In: Computer Graphics International 2005, pp. 19–25 (2005)Google Scholar
  13. 13.
    DiLorenzo, P.C., Zordan, V.B., Sanders, B.L.: Laughing out loud: Control for modeling anatomically inspired laughter using audio. ACM Transactions on Graphics (TOG) 27(5), 125 (2008)CrossRefGoogle Scholar
  14. 14.
    Ezzat, T., Geiger, G., Poggio, T.: Trainable videorealistic speech animation. In: Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 57–64 (2004)Google Scholar
  15. 15.
    Fukushima, S., Hashimoto, Y., Nozawa, T., Kajimoto, H.: Laugh enhancer using laugh track synchronized with the user’s laugh motion. In: CHI 2010 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2010, pp. 3613–3618. ACM, New York (2010)Google Scholar
  16. 16.
    Gilroy, S.W., Cavazza, M., Niranen, M., Andre, E., Vogt, T., Urbain, J., Benayoun, M., Seichter, H., Billinghurst, M.: Pad-based multimodal affective fusion. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII 2009 (2009)Google Scholar
  17. 17.
    Gosling, S., Rentfrow, P.J., Swann, W.B.: A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6), 504–528 (2003)CrossRefGoogle Scholar
  18. 18.
    Hofmann, J., Platt, T., Urbain, J., Niewiadomski, R., Ruch, W.: Laughing avatar interaction evaluation form. Unpublished Research Instrument (2012)Google Scholar
  19. 19.
    Kennedy, L., Ellis, D.: Laughter detection in meetings. In: NIST ICASSP 2004 Meeting Recognition Workshop, pp. 118–121. Montreal (May 2004)Google Scholar
  20. 20.
    Kleinsmith, A., Bianchi-Berthouze, N.: Affective body expression perception and recognition: A survey (2012)Google Scholar
  21. 21.
    Kleinsmith, A., Bianchi-Berthouze, N., Steed, A.: Automatic recognition of non-acted affective postures. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41(4), 1027–1038 (2011)CrossRefGoogle Scholar
  22. 22.
    Knox, M.T., Mirghafori, N.: Automatic laughter detection using neural networks. In: Proceedings of Interspeech 2007, pp. 2973–2976. Antwerp, Belgium (2007)Google Scholar
  23. 23.
    Kshirsagar, S., Magnenat-Thalmann, N.: Visyllable based speech animation. Comput. Graph. Forum 22(3), 632–640 (2003)CrossRefGoogle Scholar
  24. 24.
    Lasarcyk, E., Trouvain, J.: Imitating conversational laughter with an articulatory speech synthesis. In: Proceedings of the Interdisciplinary Workshop on The Phonetics of Laughter, pp. 43–48 (2007)Google Scholar
  25. 25.
    Leite, I., Castellano, G., Pereira, A., Martinho, C., Paiva, A.: Modelling empathic behaviour in a robotic game companion for children: An ethnographic study in real-world settings. In: Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 367–374. ACM (2012)Google Scholar
  26. 26.
    Liu, W., Yin, B., Jia, X., Kong, D.: Audio to visual signal mappings with hmm. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2004 (2004)Google Scholar
  27. 27.
    Mancini, M., Glowinski, D., Massari, A.: Realtime expressive movement detection using the eyesweb xmi platform. In: Camurri, A., Costa, C. (eds.) INTETAIN. LNICST, vol. 78, pp. 221–222. Springer (2011)Google Scholar
  28. 28.
    Mancini, M., Hofmann, J., Platt, T., Volpe, G., Varni, G., Glowinski, D., Ruch, W., Camurri, A.: Towards automated full body detection of laughter driven by human expert annotation. In: Proceedings of the Fifth Biannual Humaine Association Conference on Affective Computing and Intelligent Interaction, Affective Interaction in Natural Environments (AFFINE) Workshop, Geneva, Switzerland, pp. 757–762 (2013)Google Scholar
  29. 29.
    Mancini, M., Varni, G., Glowinski, D., Volpe, G.: Computing and evaluating the body laughter index. Human Behavior Understanding, 90–98 (2012)Google Scholar
  30. 30.
    Meng, H., Kleinsmith, A., Bianchi-Berthouze, N.: Multi-score learning for affect recognition: The case of body postures. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part I. LNCS, vol. 6974, pp. 225–234. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  31. 31.
    Niewiadomski, R., Mancini, M., Baur, T., Varni, G., Griffin, H., Aung, M.: MMLI: Multimodal multiperson corpus of laughter in interaction. In: Fourth Int. Workshop on Human Behavior Understanding, in Conjunction with ACM Multimedia 2013 (2013)Google Scholar
  32. 32.
    Niewiadomski, R., Pelachaud, C.: Towards multimodal expression of laughter. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds.) IVA 2012. LNCS, vol. 7502, pp. 231–244. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  33. 33.
    Niewiadomski, R., Hofmann, J., Urbain, J., Platt, T., Wagner, J., Piot, B., Cakmak, H., Pammi, S., Baur, T., Dupont, S., Geist, M., Lingenfelser, F., McKeown, G., Pietquin, O., Ruch, W.: Laugh-aware virtual agent and its impact on user amusement. In: Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS 2013. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, pp. 619–626 (2013)Google Scholar
  34. 34.
    Oura, K.: HMM-based speech synthesis system (HTS) (computer program webpage), http://hts.sp.nitech.ac.jp/ (consulted on June 22, 2011)
  35. 35.
    Owens, M.D.: It’s all in the game: Gamification, games, and gambling. Gaming Law Review and Economics 16 (2012)Google Scholar
  36. 36.
    Petridis, S., Pantic, M.: Audiovisual discrimination between laughter and speech. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Las Vegas, Nevada, pp. 5117–5120 (2008)Google Scholar
  37. 37.
    Petridis, S., Pantic, M.: Is this joke really funny? Judging the mirth by audiovisual laughter analysis. In: Proceedings of the IEEE International Conference on Multimedia and Expo, New York, USA, pp. 1444–1447 (2009)Google Scholar
  38. 38.
    Poe, E.A.: Maelzel’s chess-player. In: Southern Literary Messenger, vol. 2, pp. 318–326 (1836) Google Scholar
  39. 39.
    Qu, B., Pammi, S., Niewiadomski, R., Chollet, G.: Estimation of FAPs and intensities of AUs based on real-time face tracking. In: Pucher, M., Cosker, D., Hofer, G., Berger, M., Smith, W. (eds.) The 3rd International Symposium on Facial Analysis and Animation. ACM (2012)Google Scholar
  40. 40.
    Ruch, W., Ekman, P.: The expressive pattern of laughter. In: Kaszniak, A. (ed.) Emotion, Qualia and Consciousness, pp. 426–443. World Scientific Publishers, Tokyo (2001)CrossRefGoogle Scholar
  41. 41.
    Ruch, W., Proyer, R.: Extending the study of gelotophobia: On gelotophiles and katagelasticists. Humor-International Journal of Humor Research 22(1/2), 183–212 (2009)Google Scholar
  42. 42.
    Ruf, T., Ernst, A., Küblbeck, C.: Face detection with the sophisticated high-speed object recognition engine (shore). In: Heuberger, A., Elst, G., Hanke, R. (eds.) Microelectronic Systems, pp. 243–252. Springer, Heidelberg (2011), http://dx.org/10.1007/978-3-642-23071-4_23 CrossRefGoogle Scholar
  43. 43.
    Scherer, S., Glodek, M., Schwenker, F., Campbell, N., Palm, G.: Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data. ACM Trans. Interact. Intell. Syst. 2(1), 4:1–4:31 (2012)Google Scholar
  44. 44.
    Sundaram, S., Narayanan, S.: Automatic acoustic synthesis of human-like laughter. Journal of the Acoustical Society of America 121, 527–535 (2007)CrossRefGoogle Scholar
  45. 45.
    Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., Kitamura, T.: Speech parameter generation algorithms for hmm-based speech synthesis. In: ICASSP, pp. 1315–1318 (2000)Google Scholar
  46. 46.
    Tokuda, K., Zen, H., Black, A.: An HMM-based speech synthesis system applied to English. In: Proceedings of the 2002 IEEE Speech Synthesis Workshop, Santa Monica, California, pp. 227–230 (2002)Google Scholar
  47. 47.
    Truong, K.P., van Leeuwen, D.A.: Automatic discrimination between laughter and speech. Speech Communication 49, 144–158 (2007)CrossRefGoogle Scholar
  48. 48.
    Urbain, J., Çakmak, H., Dutoit, T.: Arousal-driven synthesis of laughter. Submitted to the IEEE Journal of Selected Topics in Signal Processing, Special Issue on Statistical Parametric Speech Synthesis (2014)Google Scholar
  49. 49.
    Urbain, J., Cakmak, H., Dutoit, T.: Development of hmm-based acoustic laughter synthesis. In: Interdisciplinary Workshop on Laughter and other Non-Verbal Vocalisations in Speech, Dublin, Ireland, pp. 26–27 (2012)Google Scholar
  50. 50.
    Urbain, J., Cakmak, H., Dutoit, T.: Evaluation of hmm-based laughter synthesis. In: International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vancouver, Canada (2013)Google Scholar
  51. 51.
    Urbain, J., Dutoit, T.: A phonetic analysis of natural laughter, for use in automatic laughter processing systems. In: Proceedings of the Fourth Bi-annual International Conference of the HUMAINE Association on Affective Computing and Intelligent Interaction (ACII 2011), Memphis, Tennesse, pp. 397–406 (2011)Google Scholar
  52. 52.
    Urbain, J., Niewiadomski, R., Bevacqua, E., Dutoit, T., Moinet, A., Pelachaud, C., Picart, B., Tilmanne, J., Wagner, J.: Avlaughtercycle: Enabling a virtual agent to join in laughing with a conversational partner using a similarity-driven audiovisual laughter animation. Journal on Multimodal User Interfaces 4(1), 47–58 (2010); special Issue: eNTERFACE 2009Google Scholar
  53. 53.
    Urbain, J., Niewiadomski, R., Mancini, M., Griffin, H., Çakmak, H., Ach, L., Volpe, G.: Multimodal analysis of laughter for an interactive system. In: Mancas, M., d’ Alessandro, N., Siebert, X., Gosselin, B., Valderrama, C., Dutoit, T. (eds.) Intetain. LNICST, vol. 124, pp. 183–192. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  54. 54.
    Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (ssi) framework - multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM International Conference on Multimedia, Barcelona, Spain, October 21-25 (2013)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2014

Authors and Affiliations

  • Maurizio Mancini
    • 1
  • Laurent Ach
    • 2
  • Emeline Bantegnie
    • 2
  • Tobias Baur
    • 3
  • Nadia Berthouze
    • 4
  • Debajyoti Datta
    • 5
  • Yu Ding
    • 5
  • Stéphane Dupont
    • 6
  • Harry J. Griffin
    • 4
  • Florian Lingenfelser
    • 3
  • Radoslaw Niewiadomski
    • 1
  • Catherine Pelachaud
    • 5
  • Olivier Pietquin
    • 7
  • Bilal Piot
    • 7
  • Jérôme Urbain
    • 6
  • Gualtiero Volpe
    • 1
  • Johannes Wagner
    • 3
  1. 1.InfoMus - DIBRISUniversità Degli Studi di GenovaGenovaItaly
  2. 2.LA CANTOCHE PRODUCTIONParisFrance
  3. 3.Institut für InformatikUniversität AugsburgAugsburgGermany
  4. 4.UCLICUniversity College LondonLondonUnited Kingdom
  5. 5.CNRS - LTCI UMR 5141 - Telecom ParisTechParisFrance
  6. 6.TCTS Lab, Faculté PolytechniqueUniversité de MonsMonsBelgium
  7. 7.SUPELEC / UMI 2958 GT-CNRSMetzFrance

Personalised recommendations