Abstract
The context of this work is usability engineering for multimodal interaction. In contrast to other work that concentrates on prototyping toolkits or abstract guidelines, this research focuses on user interface patterns for multimodal interaction. Designing multimodal applications requires several skills ranging from design and implementation. Thus, different kinds of patterns (from architecture patterns to user interface patterns) can be applied to this field. This work focuses on user-task near user interface patterns. At first, a traditional approach of modality selection based on task- and context-based rules is presented. Next, a twofold process of pattern mining is presented. In the first phase, pattern candidates are derived top-down from proven knowledge about how multimodality enhances usability. In the second phase, literature is mined for real solutions to underpin these pattern candidates and find new ones. Along with this, relationships between patterns are depicted.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., Angel, S.: A Pattern Language. Oxford University Press, Oxford (1977)
Alexander, C.: The Timeless Way of Building. Oxford University Press, Oxford (1979)
Almeida, L., Amdal, I., Beires, N., Boualem, M., Boves, L., den Os, E., Filoche, P., Gomes, R., Knudsen, J.E., Kvale, K., Rugelbak, J., Tallec, C., Warakagoda, N.: Implementing and evaluating a multimodal and multilingual tourist guide. In: Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems (2002)
Aslan, I., Xu, F., Uszkoreit, H., Krüger, A., Steffen, J.: COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction for Mobile Tourist Guide Applications. In: Maybury, M., Stock, O., Wahlster, W. (eds.) INTETAIN 2005. LNCS (LNAI), vol. 3814, pp. 3–12. Springer, Heidelberg (2005)
Benoît, C., Martin, J.C., Pelachaud, C., Schomaker, L., Suhm, B.: Audio-Visual and Multimodal Speech Systems. In: Gibbon, D. (ed.) Handbook of Standards and Resources for Spoken Language Systems - Supplement Volume (1998)
Bernsen, N.O.: Multimodality in Language and Speech Systems – from theory to design support tool. In: Lectures at the 7th European Summer School on Language and Speech Communication (ESSLSC) (1999)
Borchers, J.: A Pattern Approach to Interaction Design. John Wiley & Sons, Inc., Chichester (2001)
Brown, W., Malveau, R., McCormick, H., Mowbray, T., Thomas, S.W.: The Software Patterns Criteria (1998), http://www.antipatterns.com/whatisapattern/
Bürgy, C.: An Interaction Constraints Model for Mobile and Wearable Computer-Aided Engineering Systems in Industrial Applications. Department of Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA (2002)
Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M.: Pattern-orientierte Softwarearchitektur. Addison-Wesley, Reading (1998)
Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: multimodal interaction for distributed applications. In: MULTIMEDIA 1997: Proceedings of the fifth ACM international conference on Multimedia, pp. 31–40. ACM Press, New York (1997)
Cohen, P., McGee, D., Clow, J.: The efficiency of multimodal interaction for a map-based task. In: Proceedings of the sixth conference on Applied natural language processing, pp. 331–338. Morgan Kaufmann, San Francisco (2000)
Comerford, L., Frank, D., Gopalakrishnan, P., Gopinath, R., Sedivy, J.: The IBM Personal Speech Assistant. In: Proc. of IEEE ICASSP 2001, pp. 319–321 (2001)
Coutaz, J.I., Nigay, L., Salber, D., Blandford, A., May, J., Young, R.M.: Four Easy Pieces for Assessing the Usability of Multimodal Interaction: the CARE Properties. In: Proceedings of INTERACT 1995 (1995)
Dragičević, P.: Un modèle d’interaction en entrée pour des systèmes interactifs multi-dispositifs hautement configurables. Université de Nantes, école doctorale sciences et technologies de l’information et des matériaux. Diss. (2004)
Duarte, C., Carriço, L.: A conceptual framework for developing adaptive multimodal applications. In: IUI 2006: Proceedings of the 11th international conference on Intelligent user interfaces. ACM Press, New York (2006)
Gabriel, D.: A Pattern Definition. web ressource (2007), http://hillside.net/patterns/definition.html
Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading (1995)
Godet-Bar, G., Dupuy-Chessa, S., Nigay, L.: Towards a System of Patterns for the Design of Multimodal Interfaces. In: Proceedings of 6th International Conference on Computer-Aided Design of User Interfaces CADUI 2006, pp. 27–40. Springer, Heidelberg (2006)
Grasso, M.A., Ebert, D.S., Finin, T.W.: The integrality of speech in multimodal interfaces. ACM Transactions on Computer-Human Interaction 5(4), 303–325 (1998)
Hastie, H.W., Johnston, M., Ehlen, P.: Context-Sensitive Help For Multimodal Dialogue. In: ICMI 2002: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces. IEEE Computer Society, Los Alamitos (2002)
Huang, X., Acero, A., Chelba, C., Deng, L., Duchene, D., Goodman, J., Hon, H., Jacoby, D., Jiang, L., Loynd, R., Mahajan, M., Mau, P., Meredith, S., Mughal, S., Neto, S., Plumpe, M., Wang, K., Wang, Y.: MiPad: A Next Generation PDA Prototype. In: ICSLP 2000 (2000)
IBM: Developing X+V Applications Using the Multimodal Toolkit and Browser. Manual (2002)
Johnston, M., Bangalore, S., Vasireddy, G., Stent, A., Ehlen, P., Walker, M., Whittaker, S., Maloor, P.: MATCH: an architecture for multimodal dialogue systems. In: Proc. of the 40th Annual Meeting on Association for Computational Linguistics, ACL 2002, pp. 376–383 (2002)
Mahemoff, M.J., Johnston, L.J.: Usability Pattern Languages: the “Language” Aspect. In: Hirose, M. (ed.) Human-Computer Interaction: Interact 2001, pp. 350–358. IOS Press, Amsterdam (2001)
Malaka, R., Haeussler, J., Aras, H.: SmartKom mobile: intelligent ubiquitous user interaction. In: IUI 2004: Proceedings of the 9th international conference on Intelligent user interface, pp. 310–312. ACM Press, New York (2004)
Marx, M., Schmandt, C.: Putting people first: specifying proper names in speech interfaces. In: UIST 1994: Proceedings of the 7th annual ACM symposium on User interface software and technology, pp. 29–37. ACM Press, New York (1994)
Milota, A.D.: Modality fusion for graphic design applications. In: ICMI 2004: Proceedings of the 6th international conference on Multimodal interfaces, pp. 167–174. ACM Press, New York (2004)
Neuss, R.: Usability Engineering als Ansatz zum Multimodalen Mensch-Maschine-Dialog. Fakultät für Elektrotechnik und Informationstechnik, Technische Universität München, Diss. (2001)
Niedermaier, F.B.: Entwicklung und Bewertung eines Rapid-Prototyping Ansatzes zur multimodalen Mensch-Maschine-Interaktion im Kraftfahrzeug. Fakultät für Elektrotechnik und Informationstechnik, Technische Universität München, Diss. (2003)
Nigay, L., Coutaz, J.: A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of Human Factors in Computing Systems, INTERCHI 1993 Conference, pp. 172–178. ACM Press, New York (1993)
Obrenović, Z., Abascal, J., Starčević, D.: Universal accessibility as a multimodal design issue. In: Commun. ACM, vol. 50, pp. 83–88. ACM Press, New York (2007)
Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the User Interface for Multimodal Speech and Pen-based Gesture Applications: State-of-the-Art Systems and Future Research Directions. Human Computer Interaction 15(4), 263–322 (2000)
Ratzka, A., Wolff, C.: A Pattern-based Methodology for Multimodal Interaction Design. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2006. LNCS (LNAI), vol. 4188, pp. 677–686. Springer, Heidelberg (2006)
Ratzka, A.: A Wizard-of-Oz Setting for Multimodal Interaction. An Approach to User-Based Elicitation of Design Patterns. In: Osswald, A., Stempfhuber, M., Wolff, C. (eds.) Open Innovation. Proc. 10th International Symposium for Information Science, Universitätsverlag Konstanz, pp. 159–170 (2007a)
Ratzka, A.: Design Patterns in the Context of Multi-modal Interaction. In: Proceedings of the 6th Nordic Conference on Pattern Languages of Programs 2007 VikingPLoP 2007 (to appear, 2007b)
Ratzka, A.: Design Patterns for Robust and Accessible Multimodal Interaction. In: Proceedings of EuroPLoP 2008 (to appear, 2008)
Sinnig, D., Gaffar, A., Reichart, D., Seffah, A., Forbrig, P.: Patterns in Model-Based Engineering. In: CADUI 2004, pp. 195–208 (2004)
Snelick, R., Indovina, M., Yen, J., Mink, A.: Multimodal biometrics: issues in design and testing. In: ICMI 2003: Proceedings of the 5th international conference on Multimodal interfaces, pp. 68–72. ACM Press, New York (2003)
Tidwell, J.: COMMON GROUND: A Pattern Language for Human-Computer Interface Design (1999), http://www.mit.edu/~jtidwell/interaction_patterns.html
Tidwell, J.: Designing Interfaces: Patterns for Effective Interaction Design. O’Reilly, Sebastopol (2005)
van Welie, M., Trætteberg, H.: Interaction Patterns in User Interfaces. In: Proceedings of the Seventh Pattern Languages of Programs Conference (2000)
van Welie, M.: Task-based User Interface Design. Dutch Graduate School for Information and Knowledge Systems, Vrije Universiteit Amsterdam, Diss. (2001)
Zeng, Z., Tu, J., Liu, M., Zhang, T., Rizzolo, N., Zhang, Z., Huang, T.S., Roth, D., Levinson, S.: Bimodal HCI-related affect recognition. In: ICMI 2004: Proceedings of the 6th international conference on Multimodal interfaces, pp. 137–143. ACM Press, New York (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 IFIP International Federation for Information Processing
About this paper
Cite this paper
Ratzka, A. (2008). Steps in Identifying Interaction Design Patterns for Multimodal Systems. In: Forbrig, P., Paternò, F. (eds) Engineering Interactive Systems. TAMODIA HCSE 2008 2008. Lecture Notes in Computer Science, vol 5247. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-85992-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-540-85992-5_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-85991-8
Online ISBN: 978-3-540-85992-5
eBook Packages: Computer ScienceComputer Science (R0)