Journal on Multimodal User Interfaces

, Volume 2, Issue 1, pp 53–60 | Cite as

Design and usability evaluation of multimodal interaction with finite state machines: a conceptual framework

Original Paper
First Online:
Received:
Accepted:

Abstract

Designing multimodal systems that take the best advantage of multiple error prone recognition-based technologies, such as speech and gesture recognition, is difficult. To guarantee a robust and usable interaction, careful consideration must be given to the choice of modalities of interaction made available, their allocation to tasks, and the range of modality combinations allowed. In this paper, we present a conceptual framework for evaluating the usability and robustness of different interaction modality combinations early in the process of designing a multimodal system. First, models of multimodal elementary commands are built using Finite State Machine (FSM) modelling. Second, the most usable representations of multimodal commands are flagged up in the model by activating the FSM-based model with real or simulated user inputs. The output of this step is a collection of FSMs that more closely represent user preferences and natural behaviour. Third, the disambiguating potential of sets of multimodal commands is evaluated by observing the models response to simulated recognition errors.

Keywords

Multimodal interaction Recognition-based technologies Recognition errors Modality combinations Interaction models Finite state machines Usability evaluation 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bouchet J, Nigay L, Ganille T (2004) Icare software components for rapidly developing multimodal interfaces. In: ICMI, October 2004, pp 251–258 Google Scholar
  2. 2.
    Bourguet M-L (2002) A toolkit for creating and testing multimodal interface designs. In: Posters and demos from the 15th annual ACM symposium on user interface software and technology, October 2002, pp 29–30 Google Scholar
  3. 3.
    Bourguet M-L (2003) Designing and prototyping multimodal commands. In: INTERACT, September 2003, pp 717–720 Google Scholar
  4. 4.
    Bourguet M-L (2003) How finite state machines can be used to build error free multimodal interaction systems. In: The 17th British HCI group annual conference, vol 2, September 2003, pp 81–84 Google Scholar
  5. 5.
    Bourguet M-L (2006) A study into multimodal behaviour in error correction. In: Workshop on multimodal corpora: from multimodal behaviour theories to usable models. In association with the 5th international conference on language resources and evaluation, May 2006 Google Scholar
  6. 6.
    Bourguet M-L, Ando A (1998) Synchronisation of speech and hand gestures during multimodal human computer interaction. In: ACM CHI’98, April 1998, pp 241–242 Google Scholar
  7. 7.
    Bernsen NO, (2002) In: Multimodality in language and speech systems. From theory to design support tool. Kluwer Academic, Dordrecht, pp 93–148 Google Scholar
  8. 8.
    Clow J, Oviatt SL (1998) Stamp: an automated tool for analysis of multimodal system performance. In: International conference on spoken language processing, December 1998 Google Scholar
  9. 9.
    Coutaz GFC, Paterno F, Nigay L (1993) A comparison of approaches for specifying multimodal interactive systems. In: ERCIM workshop on multimodal human-computer interaction, November 1993 Google Scholar
  10. 10.
    Coutaz C, Nigay L, Salber D, Blandford A, May J, Young R (1995) Four easy pieces for assessing the usability of multimodal interaction: the care properties. In: INTERACT 95, June 1995, pp 115–120 Google Scholar
  11. 11.
    Duarte C, Carrico L (2006) A conceptual framework for developing adaptive multimodal applications. In: The 11th international conference on intelligent user interfaces, January 2006 Google Scholar
  12. 12.
    Flippo AKF, Marsic I (2003) A framework for rapid development of multimodal interfaces. In: The 5th international conference on multimodal interfaces, November 2003 Google Scholar
  13. 13.
    Johnston M, Bangalore S (2005) Finite-state multimodal integration and understanding. Nat Lang Eng 11:159–187 CrossRefGoogle Scholar
  14. 14.
    Karat CM, Halverson C, Horn D, Karat J (1999) Patterns of entry and correction in large vocabulary contentious speech recognition systems. In: ACM CHI’99, May 1999 Google Scholar
  15. 15.
    Mankoff J, Hudson S, Abowd G (2000) Interaction techniques for ambiguity resolution in recognition-based interfaces. In: ACM UIST00, November 2000, pp 11–20 Google Scholar
  16. 16.
    Navarre D, Palanque P, Dragicevic P, Bastide R (2006) An approach integrating two complementary model-based environments for the construction of multimodal interactive applications. In: Interacting with computers, vol 18, September 2006 Google Scholar
  17. 17.
    Nigay L, Coutaz J (1997) Multifeature systems: The care properties and their impact on software design. In: Intelligence and multimodality in multimedia interfaces, July 1997 Google Scholar
  18. 18.
    Oviatt S (1999) Mutual disambiguation of recognition errors in a multimodal architecture. In: ACM CHI’99, 1999, pp 576–583 Google Scholar
  19. 19.
    Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81 CrossRefGoogle Scholar
  20. 20.
    Oviatt S (2000) Taming recognition errors with a multimodal interface. Commun ACM 43(9):45–51 CrossRefGoogle Scholar
  21. 21.
    Oviatt S (2002) Breaking the robustness barrier: recent progress on the design of robust multimodal systems. Adv Comput 56:306–343 Google Scholar
  22. 22.
    Oviatt S, Coulston R, Lunsford R (2004) When do we interact multimodally? cognitive load and multimodal communication patterns, In: ICMI 2004, October 2004 Google Scholar
  23. 23.
    Palanque P, Schyn A (2003) A model-based approach for engineering multimodal interactive systems. In: INTERACT’03, September 2003 Google Scholar
  24. 24.
    Sinha AK, Landay JA (2003) Capturing user tests in a multimodal, multidevice informal prototyping tool. In: 5th international conference on multimodal interfaces, November 2003 Google Scholar
  25. 25.
    Suhm B, Myers B, Waibel A (1999) Model-based and empirical evaluation of multimodal interactive error correction. In: ACM CHI’99, May 1999, pp 584–591 Google Scholar
  26. 26.
    VanGurp J, Bosch J (1999) On the implementation of finite state machines. In: IASTED 3rd international conference on software engineering and applications, October 1999, pp 172–178 Google Scholar
  27. 27.
    Vuurpijl L, ten Bosch L, Rossignol S, Neumann A, Pflegfer N, Engel R (2004) Evaluation of multimodal dialogue systems. In: The 4th international conference on language resources and evaluation, May 2004 Google Scholar
  28. 28.
    Wasinger R (2007) In: Multi modal interaction with mobile devices: fusing a broad spectrum of modality combinations. IOS, Amsterdam Google Scholar
  29. 29.
    Wasserman A (1985) Extending state transition diagrams for the specification of human-computer interaction. IEEE Trans Softw Eng 11(8):699–713 CrossRefGoogle Scholar
  30. 30.
    W3C (2007) Emma: extensible multimodal annotation markup language. December 2007 Google Scholar

Copyright information

© OpenInterface Association 2008

Authors and Affiliations

  1. 1.Computer Science DepartmentQueen Mary University of LondonLondonUK

Personalised recommendations