A Pattern-Based Methodology for Multimodal Interaction Design

  • Andreas Ratzka
  • Christian Wolff
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4188)


This paper describes a design methodology for multimodal interactive systems. The method suggested is meant to serve as a foundation for the application of robust software engineering techniques in the field of multimodal systems. Starting from a short review of current design approaches we present a high level view of the design process for multimodal systems, highlighting design issues related to context of use factors. Our proposal is discussed in the context of a multimodal organizer which serves as our showcase application. The design of multimodal systems brings together a broad variety of analysis methods (task, context, data, user). The combination of modalities as well as the different interaction devices imply a high degree of freedom as far as design decisions are concerned. Therefore, a (simple) unification of existing approaches towards interface design like GOMS (task analysis) or Buergy’s interaction constraint model for context analysis is not sufficient. We employ the design pattern approach as a means of guiding the analysis and design process. Design patterns are discussed as a general modeling tool as well as a possible approach towards designing multimodal systems.


Design Pattern Interaction Design Interaction Style Multimodal Interaction Multimodal System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., Angel, S.: A Pattern Language. Oxford University Press, Oxford (1977)Google Scholar
  2. 2.
    Benoît, et al.: Audio-visual and Multimodal Speech Systems. Audio-visual and Multimodal Speech Systems. In: Handbook of Standards and Resources for Spoken Language Systems – Supplement Volume (2000)Google Scholar
  3. 3.
    Bernsen, N.O.: Modality Theory: Supporting Multimodal Interface Design. In: Proc. ERCIM (1993)Google Scholar
  4. 4.
    Bernsen, N.O.: A toolbox of output modalities. Representing output information in multimodal interfaces. WPCS-95-10. Centre for Cognitive Science, Roskilde University (1995)Google Scholar
  5. 5.
    Bernsen, N.O.: Towards a tool for predicting speech functionality. Speech Communication 23, 181–210 (1997)CrossRefGoogle Scholar
  6. 6.
    Bernsen, N.O.: Multimodality in language and speech systems – from theory to design support tool. In: Granström, B., House, D., Karlsson, I. (eds.) Multimodality in Language and Speech Systems, pp. 93–148. Kluwer Academic Publishers, Dordrecht (2002)Google Scholar
  7. 7.
    Borchers, J.O.: A Pattern Approach to Interaction Design. AI & Society Journal of Human-Centred Systems and Machine Intelligence 15(4), 359–376 (2001)MathSciNetGoogle Scholar
  8. 8.
    Bürgy, C.: An Interaction Constraints Model for Mobile and Wearable Computer-Aided Engineering Systems in Industrial Applications. Doctoral Dissertation, University of Pittsburgh, Pennsylvania, USA (2002)Google Scholar
  9. 9.
    Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289–308 (2003)CrossRefGoogle Scholar
  10. 10.
    Cenek, P., Melichar, M., Rajman, M.: A Framework for Rapid Multimodal Application Design. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 393–403. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  11. 11.
    Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R.M.: Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE Properties. In: Proc. Interact 1995, pp. 115–120. Chapman and Hall, London (1995)Google Scholar
  12. 12.
    Dittmar, A.: More Precise Descriptions of Temporal Relations within Task Models. In: Palanque, P., Paternó, F. (eds.) DSV-IS 2000. LNCS, vol. 1946, pp. 151–168. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  13. 13.
    Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading (1995)Google Scholar
  14. 14.
    Grasso, M.A., Ebert, D.S., Finin, T.W.: The integrality of speech in multimodal interfaces. ACM Trans. Comput.-Hum. Interact. 5(4), 303–325 (1998)CrossRefGoogle Scholar
  15. 15.
    Horrocks, I.: Constructing the User Interface with Statecharts. Addison-Wesley, Reading (1999)Google Scholar
  16. 16.
    Kiger, J.I.: The depth/breadth trade-off in the design of menu-driven user interfaces. International Journal of Man-Machine Studies 20, 201–213 (1984)CrossRefGoogle Scholar
  17. 17.
    Landauer, T.K., Nachbar, D.W.: Selection from alphabetic and numeric menu trees using a touch screen: Breath, depth, and width. In: Proc. CHI 1985, Human Factors in Computing Systems, pp. 73–78. ACM, New York (1985)CrossRefGoogle Scholar
  18. 18.
    Larsson, S.: Using a type hierarchy to characterize reliability of coding schemas for dialogue moves. Gothenburg Papers in Computational Linguistics (1998)Google Scholar
  19. 19.
    Martin, J.-C.: Towards “intelligent” cooperation between modalities. The example of a system enabling multimodal interaction with a map. In: Proc. IJCAI 1997 Workshop on Intelligent Multimodal Systems, Nagoya, Japan (1997)Google Scholar
  20. 20.
    Nigay, L., Coutaz, J.: A Design Space For Multimodal Systems: Concurrent Processing and Date Fusion. In: Proc. INTERCHI 1993. ACM Press, New York (1993)Google Scholar
  21. 21.
    Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the User Interface for Multimodal Speech and Pen-based Gesture Applications: State-of-the-Art Systems and Future Research Directions. Human Computer Interaction 15(4), 263–322 (2000)CrossRefGoogle Scholar
  22. 22.
    Pribeanu, C., Limbourg, Q., Vanderdonckt, J.: Task Modelling for Context-Sensitive User Interfaces. In: Johnson, C. (ed.) DSV-IS 2001. LNCS, vol. 2220, pp. 49–68. Springer, Heidelberg (2001)Google Scholar
  23. 23.
    Rajman, M., Bui, T.H., Rajman, A., Seydoux, F., Trutnev, A., Qurteroni, S.: Assessing the usability of a dialogue management system designed in the framework of a rapid dialogue prototyping methodology. Acta Acustica united with Acustica, the journal of the European Acoustics Association (EAA) 90, 1096–1111 (2004)Google Scholar
  24. 24.
    Ratzka, A.: Combining Modality Theory and Context Models. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS, vol. 4021, pp. 141–151. Springer, Heidelberg (to appear, 2006)CrossRefGoogle Scholar
  25. 25.
    Reeves, L.M., Lai, J., Larson, J.A., Oviatt, S., Balaji, T.S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.-C., McTear, M., Raman, T., Stanney, K.M., Su, H., Wang, Q.Y.: Guidelines for multimodal user interface design. Communications ACM 47(1), 57–59 (2004)CrossRefGoogle Scholar
  26. 26.
    Shneiderman, B.: Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley, Reading (1997)Google Scholar
  27. 27.
    da Silva, P.P.: User Interface Declarative Models and Development Environments: A Survey. In: Palanque, P., Paternó, F. (eds.) DSV-IS 2000. LNCS, vol. 1946, pp. 207–226. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  28. 28.
    Souchon, N., Limbourg, Q., Vanderdonckt, J.: Task Modelling in Multiple Contexts of Use. In: Forbrig, P., Limbourg, Q., Urban, B., Vanderdonckt, J. (eds.) DSV-IS 2002. LNCS, vol. 2545, pp. 59–73. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  29. 29.
    Trætteberg, H.: Model-based User Interface Design. Ph.D. Thesis, Department of Computer and Information Science, Norwegian University of Science and Technology (2002)Google Scholar
  30. 30.
    Trowbridge, D., Cunningham, W., et al.: Describing the Enterprise Architectural Space. Microsoft Corporation (June 2004)Google Scholar
  31. 31.
    Vernier, F., Nigay, L.: A Framework for the Combination and Characterization of Output Modalities. In: Palanque, P., Paternó, F. (eds.) DSV-IS 2000. LNCS, vol. 1946, pp. 35–50. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  32. 32.
    Van Welie, M., van der Weer, G.C.: Pattern Languages in Interaction Design: Structure and Organization. In: Proc. Interact 2003 (2003)Google Scholar
  33. 33.
    Wilson, S., Johnson, P., Kelly, C., Cunningham, J., Markopoulos, P.: Beyond Hacking: a Model Based Approach to User Interface Design. In: Proc. HCI 1993 (1993)Google Scholar
  34. 34.
    Wolff, C.: Media Design Patterns. In: Eibl, M., Womser-Hacker, C., Wolff, C. (eds.) Designing Information Systems. pp. 209–217. UVK, Constance (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Andreas Ratzka
    • 1
  • Christian Wolff
    • 1
  1. 1.Institute for Media, Information and Cultural StudiesUniversity of RegensburgRegensburgGermany

Personalised recommendations