Journal on Multimodal User Interfaces

, Volume 3, Issue 3, pp 227–236 | Cite as

An open source integrated framework for rapid prototyping of multimodal affective applications in digital entertainment

  • Diego ArnoneEmail author
  • Alessandro Rossi
  • Massimo Bertoncini
Original Paper


Multimodal interfaces are becoming more and more an emerging leading-edge area of interest in the Digital Art and Entertainment Domain. This paper aims at proposing a new approach to develop applications relying on integration of multimodal modules which can capture and collect the emotional states of the audience, process them and provide a final emotional output. A concrete framework has been designed and is being developed to provide application designers with a powerful and easy-to-use set of tools for building up multimodal affective interfaces. The proposed approach has been conceived inside the FP6 EU co-funded CALLAS (Conveying Affectiveness in Leading-edge Living Adaptive Systems) Project.


Multimodal Interaction Framework Multimodal affective interfaces development Software engineering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Myllys M (2002) Context Toolkit, Research Seminar on Middleware for Mobile Computing, Helsinki, University of Helsinki, Department of Computer Science 20th February, 2002 Google Scholar
  3. 3.
    Dragicevic P, Fekete JD (2001) Input device selection and interaction configuration with ICON. In: Proc IHM-HCI. Springer, Berlin, pp 543–548. Google Scholar
  4. 4.
    Sinha AK (2003) Informally prototyping multimodal, multidevice user interfaces. PhD dissertation, Electrical Engineering and Computer Sciences Department, University of California, Berkeley, CA, Fall, 2003 Google Scholar
  5. 5.
    Huot S, Dumas C, Dragicevic P, Fekete JD, Hégron G (2004) The MaggLite Post-WIMP Toolkit: draw it, connect it and run it. In: ACM proceedings: 17th ACM symposium on user interface software and technologies (UIST 2004), Santa Fe, New Mexico, USA, October, 2004, pp 257–266.
  6. 6.
    Nigay L, Coutaz J (1997) The CARE properties and their impact on software design. In: Intelligence and multimodality in multimedia interfaces Google Scholar
  7. 7.
    Alexandersson J, Becker T (2001) Overlay as the basic operation for discourse processing in a multimodal dialogue system. In: Proc of the 2nd IJCAI workshop on knowledge and reasoning in practical dialogue systems, Seattle, August, 2001 Google Scholar
  8. 8.
    Heider T, Jantz D, Kirste T (2001) Embassi. Comput Graph Top 13(2):15–17 Google Scholar
  9. 9.
    Elting C, Rapp S, Möhler G, Strube M (2003) Architecture and implementation of multimodal plug and play. In: Proceedings of the 5th international conference on multimodal interfaces, Vancouver, British Columbia, Canada, November 5–7, 2003 Google Scholar
  10. 10.
  11. 11.
    Turunen M, Hakulinen J, Räihä K-J, Salonen E-P, Kainulainen A, Prusi P (2005) An architecture and applications for speech-based accessibility systems, IBM Syst J 44(3) Google Scholar
  12. 12.
  13. 13.
    Reas B, Maeda J (2007) Processing: A programming handbook for visual designers and artists, 1st edn. MIT Press, Cambridge. ISBN 0262182629, Google Scholar
  14. 14.
    Greenberg I (2007) Processing creative coding and computational art (foundation), 1st edn. friends of ED, New York, ISBN 159059617X Google Scholar
  15. 15.
    Functional requirements & CHIL cooperative information system software design, deliverable D2.1 (part 1: cooperative information system software design) of the project CHIL (Computers in the human interaction loop), IP 506909, 14 July, 2004 Google Scholar
  16. 16.
    Bouchet J, Nigay L (2004) ICARE: a component-based approach for the design and development of multimodal interfaces. In: CHI 2004, Vienna, Austria, April 24–29, 2004 Google Scholar
  17. 17.
    Nicolas V, Trevisan DG, Wamy Wapet S (2004) OpenInterface-Multimodal library for multimodal applications development specification. SIMILAR network of excellence, project no FP6-507609, deliverable D4, 1st version, Communication and Remote Sensing Laboratory, Université Catholique de Louvain, 1348 Louvain-la-Neuve, Belgique, May, 2004 Google Scholar
  18. 18.
    Ghigny C, Lawson JY, Macq B, Nigay L, Trevisan DG (2005) OpenInterface-Multimodal library for multimodal applications development specification. SIMILAR network of excellence, project no. FP6-507609, deliverable D58, Final version, CLIPS-IMAG, Université Joseph Fourier, Grenoble, France TELE, Université Catholique de Louvain, Belgium, August, 2005 Google Scholar
  19. 19.
    Lawson L, Macq B, Mansoux B, Nicolas V, Nigay L, Trevisan DG (2005) OpenInterface platform, SIMILAR network of excellence, project no. FP6-507609, deliverable D60, final version, platform CLIPS-IMAG, Université Joseph Fourier, Grenoble, France, TELE, Université Catholique de Louvain, Belgium, November, 2005 Google Scholar
  20. 20.
  21. 21.
    Feigenbaum EA (1988) Book foreword. In: Engelmore R.S., Morgan A. (eds) Blackboard Systems. Addison-Wesley, Reading, pp 13–14 Google Scholar
  22. 22.
    Eugster P, Guerraoui R, Damm C (2001) On objects and events. In: Proceedings for OOPSLA 2001, Tampa Bay, Florida, October 2001 Google Scholar
  23. 23.
    Banavar G, Chandra T, Mukherjee B, Nagarajarao J, Strom RE, Sturman DC (1999) An efficient multicast protocol for content based publish-subscribe systems. In: Proceedings of the 19th international conference on distributed computing systems (ICDCS’99), 1999 Google Scholar
  24. 24.
    Carzaniga A, Rosenblum DS, Wolf AL (2000) Achieving scalability and expressiveness in an Internet-scale event notification service. In: proceedings nineteenth annual ACM symposium on principles of distributed computing (PODC 2000), Portland, Oregon, July 2000, pp. 219–227 Google Scholar
  25. 25.
    Segall B, Arnold D, Boot J, Henderson M, Phelps T (2000) Content based routing with Elvin4. In: AUUG2K, Canberra, Australia, June 2000 Google Scholar
  26. 26.
    Cugola G, Di Nitto E, Fugetta A (2001) The Jedi event-based infrastructure and its application to the development of the opss wfms. IEEE Trans Softw Eng 27(9):827–850 CrossRefGoogle Scholar
  27. 27.
    Hapner M, Burridge R, Sharma R, Fialli J, Stout K (2000) Java message service. Sun Microsystems Inc, April 2002 Google Scholar
  28. 28.
    Henning M (2004) Massively multiplayer middleware. ACM Queue 1(10):40–45 CrossRefGoogle Scholar
  29. 29.
    Henning M, Spruiell M Distributed programming with Ice,
  30. 30.
    Sleepycat Software, Inc (2003) Berkeley DB technical articles. Lincoln, MA: Sleepycat Software, Inc.
  31. 31.
    Henning M (2004) A new approach to object-oriented middleware. IEEE Internet Comput 8(1):66–75 CrossRefGoogle Scholar
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
    Charles F, Lemercier S, Cavazza M, Vogt T, Bee N, André E, Mancini M, Pélachaud C, Urbain J, Price M (2007) Affective interactive narrative in the CALLAS project. In: ICVS2007, Saint-Malo, France, 5–7, Dec, 2007 Google Scholar
  39. 39.
    Gilroy SW, Cavazza M, Chaignon R, Makela SM, Niranen M, Andre E, Vogt T, Urbain J, Billinghurst M, Seichter H, Benayoun M E-tree: emotionally driven augmented reality art. In: Proc ACM multimedia, Vancouver, BC, Canada, 2008, pp 945–948 Google Scholar
  40. 40.
    Cavazza M, Gilroy SW, Niiranen M, André E, Vogt T, Urbain J, Benayoun M, Seichter H, Billinghurst M (2009) PAD-based multimodal affective fusion. In: Proc. ACII 2009, Amsterdam, The Netherlands, September 2009 Google Scholar
  41. 41.
    Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2007) Non-verbal feedback on user interest based on gaze direction and head pose. In: 2nd international workshop on semantic media adaptation and personalization (SMAP 2007), London, United Kingdom, December, 2007 Google Scholar
  42. 42.
    Niiranen M, Vehkaperä J, Mäkelä S-M, Peltola J, Räty T (2008) Fusion of sound source localization and face detection for supporting human behavior analysis. In: Proc 4th international mobile multimedia communications conference, MobiMedia 2008, ICST, Oulu, Finland, Jul 7–9, 2008 Google Scholar
  43. 43.
    Vogt T, André E (2006) Improving automatic emotion recognition from speech via gender differentiation. In: Proc language resources and evaluation conference (LREC 2006), Genoa, Italy, 2006 Google Scholar
  44. 44.
    Guile 3D Studio—Nicole Visual Character,

Copyright information

© OpenInterface Association 2010

Authors and Affiliations

  • Diego Arnone
    • 1
    Email author
  • Alessandro Rossi
    • 1
  • Massimo Bertoncini
    • 2
  1. 1.Engineering Ingegneria InformaticaPalermoItaly
  2. 2.Engineering Ingegneria InformaticaRomeItaly

Personalised recommendations