Skip to main content
SpringerLink
Log in

Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

This article introduces the problem of modeling multimodal interaction, in the form of markup languages. After an analysis of the current state of the art in multimodal interaction description languages, nine guidelines for languages dedicated at multimodal interaction description are introduced, as well as four different roles that such language should target: communication, configuration, teaching and modeling. The article further presents the SMUIML language, our proposed solution to improve the time synchronicity aspect while still fulfilling other guidelines. SMUIML is finally mapped to these guidelines as a way to evaluate their spectrum and to sketch future works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Araki M, Tachibana K (2006) Multimodal dialog description language for rapid system development. In: Proceedings of the 7th SIGdial workshop on discourse and dialogue, Sydney, Australia, July 2006

  2. Bouchet J, Nigay L, Ganille T (2004) ICARE software components for rapidly developing multimodal interfaces. In: Conference proceedings of ICMI’2004, State College, PA, USA, October 2004. ACM, New York, pp 251–258

    Google Scholar 

  3. Bourguet ML (2002) A toolkit for creating and testing multimodal interface designs. In: Companion proceedings of UIST’02, Paris, October 2002, pp 29–30

  4. Bourguet ML (2006) Towards a taxonomy of error-handling strategies in recognition-based multi-modal human-computer interfaces. Signal Process 86(12)

  5. Cohen Pr, Johnston M, Mcgee D, Oviatt S, Pittman J, Smith I, Chen L, Clow J (1997) QuickSet: multimodal interaction for distributed applications. In: Proceedings of the fifth ACM international conference on multimedia, Seattle, USA

  6. Coutaz J, Nigay L, Salber D, Blandford A, May J, Young R (1995) Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Arnesen SA, Gilmore D (eds) Proceedings of the INTERACT’95 conference, Lillehammer, Norway, June 1995. Chapman & Hall, London, pp 115–120

    Google Scholar 

  7. De Boeck J, Vanacken D, Raymaekers C, Coninx K (2007) High-level modeling of multimodal interaction techniques using NiMMiT 4(2007), no 2, September 2007, urn:nbn:de:0009-6-11615, ISSN 1860-2037

  8. Dumas B (2009) SMUIML XML Schema reference. http://diuf.unifr.ch/diva/web/downloads/documents/other/2009/smuiml_v04.xsd Accessed 28 September 2009

  9. Dumas B, Lalanne D, Ingold R (2008) Prototyping multimodal interfaces with smuiml modeling language. In: Workshop user interface description languages for next generation user interfaces, CHI 2008, Firenze, Italy, pp 63–66

  10. Dumas B, Lalanne D, Oviatt S (2009) Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne D., Kohlas J (eds) Human machine interaction, state-of-the-art survey. LNCS, vol 5440. Springer, Berlin, pp 1–25

    Google Scholar 

  11. Garofolo J (2008) Overcoming barriers to progress in multimodal fusion research. In: AAAI Fall 2008 symposium proceedings

  12. Greenberg S, Fitchett C (2001) Phidgets: easy development of physical interfaces through physical widgets. In: User interface software & technology, CHI letters 2001

  13. Flippo F, Krebs A, Marsic I (2003) A framework for rapid development of multimodal interfaces. In: Proceedings of ICMI’03, Vancouver, BC, 5–7 November 2003

  14. Kaltenbrunner M, Bencina R (2007) ReacTIVision: a computer-vision framework for table-based tangible interaction. In: Proceedings of the first international conference on tangible and embedded interaction (TEI07), Baton Rouge, LO

  15. Katsurada K, Nakamura Y, Yamada H, Nitta T (2003) XISL: a language for describing multimodal interaction scenarios In: Proceedings of the 5th international conference on multimodal interfaces (ICMI 2003), Vancouver, BC, Canada, 5–7 November 2003

  16. Ladry JF, Palanque P, Basnyat S, Barboni E, Navarre D (2008) Dealing with reliability and evolvability in description techniques for next generation user interfaces. In: Proceedings of user interface description languages for next generation user interface workshop at CHI’08, Florence, Italy

  17. Oviatt SL (2008) Multimodal interfaces. In: Jacko J, Sears A (eds) The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, 2nd edn. CRC Press, Boca Raton, pp 286–304

    Google Scholar 

  18. Paternò F, Santoro C, Mäntyjärvi J, Mori G, Sansone S (2008) Authoring pervasive multimodal user interfaces. Int J Eng Technol 4(2):235–261

    Article  Google Scholar 

  19. Serrano M, Nigay L, Lawson JYL, Ramsay A, Murray-Smith R, Denef S (2008) The openinterface framework: a tool for multimodal interaction. In: Adjunct proceedings of CHI’2008, Florence, Italy, 5–10 April 2008. ACM, New York, pp 3501–3506

    Google Scholar 

  20. Shaer O, Jacob RJ, Green M, Luyten K (2008) User interface description languages for next generation user interfaces. In: CHI ’08 extended abstracts on human factors in computing systems, CHI ’08, Florence, Italy, 5–10 April 2008. ACM, New York, pp 3949–3952

    Google Scholar 

  21. Sire S, Chatty C (2002) The markup way to multimodal toolkits, In: W3C multimodal interaction workshop

  22. Stanciulescu A, Limbourg Q, Vanderdonckt J, Michotte B, Montero F (2005) A transformational approach for multimodal web user interfaces based on UsiXML. In: Proceedings of the 7th international conference on multimodal interfaces (ICMI 2005), Torento, Italy, 4–6 October 2005

  23. Windgrave C (2008) Chasm: a tiered developer-inspired 3D interface representation. In: Proceedings of user interface description languages for next generation user interface workshop at CHI’08, Florence, Italy

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, University of Fribourg, Boulevard de Pérolles 90, 1700, Fribourg, Switzerland

    Bruno Dumas, Denis Lalanne & Rolf Ingold

Authors
  1. Bruno Dumas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Denis Lalanne
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rolf Ingold
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruno Dumas.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dumas, B., Lalanne, D. & Ingold, R. Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML. J Multimodal User Interfaces 3, 237–247 (2010). https://doi.org/10.1007/s12193-010-0043-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-010-0043-3

Keywords

Search

Navigation