Abstract
Output multimodal interaction involves choice and combination of relevant interaction modalities to present information to the user. In this paper, we present a framework based on reusable software components for rapidly developing output multimodal interfaces by choosing and combining interaction modalities. Such an approach enables us to quickly explore several design alternatives as part of an iterative design process. Our approach is illustrated by examples from a computer-assisted surgery system that runs in a specific environment (i.e. an operating room) and so needs adapted multimodal interaction. Our approach supports the exploration of several output multimodal interaction design alternatives with the surgeons.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Allen, J. F. [1983], Maintaining Knowledge about Temporal Intervals, Communications of the ACM 26(11), 832–43.
André, E., Finkler, W., Graf, W., Rist, T., Schauder, A., & Wahlster, W. [1993], WIP: The Automatic Synthesis of Multimodal Presentations, in M. T. Maybury (ed.), Intelligent Multimedia Interface, AAAI Press, pp.75–93.
Beaudoin-Lafon, M. [2004], Designing Interaction, Not Interfaces, in M. F. Costabile (ed.), Proceedings of the Working Conference on Advanced Visual Interface (AVI 2004), ACM Press, pp. 15–22.
Bouchet, J., Nigay, L. & Ganille, T. [2004], ICARE Software Components for Rapidly Developing Multimodal Interfaces, in R. Sharma, T. Darrell, M. Harper, G. Lazzari & M. Turk (eds.), Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI’04), ACM Press, pp.251–8.
Chalmers, M. & Galani, A. [2004], Seamful Interweaving: Heterogeneity in the Theory and Design of Interactive Systems, in D. Gruen & I. McAra-McWilliams (eds.), Proceedings of the Symposium on Designing Interactive Systems: Processes, Practices, Methods and Techniques (DIS’04), ACM Press, pp.243–52.
Dragicevic, P. & Fekete, J.-D. [2004], The Input Configurator Toolkit: Towards High Input Adaptability in Interactive Applications, in M. F. Costabile (ed.), Proceedings of the Working Conference on Advanced Visual Interface (AVI 2004), ACM Press, pp.244–7.
Dubois, E., Nigay, L. & Troccaz, J. [2002], Assessing Continuity and Compatibility in Augmented Reality Systems, Universal Access in the Information Society 4(1), 263–73.
Elting, C., Rapp, S., Mölher, G. & Strube, M. [2003], Architecture and Implementation of Multimodal Plug and Play, in S. Oviatt, T. Darrell, M. Maybury & W. Wahlster (eds.), Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI’03), ACM Press, pp.93–100.
Flippo, F., Krebs, A. & Marsic, I. [2003], A Framework for Rapid Development of Multimodal Interfaces, in S. Oviatt, T. Darrell, M. Maybury & W. Wahlster (eds.), Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI’03), ACM Press, pp. 109–16.
Glass, J., Weinstein, E., Cyphers, S., Polifroni, J., Chung, G. & Nakano, M. [2004], A Framework for Developing Conversational User Interfaces, in R. K. Jacob, Q. Limbourg & J. Vanderdonckt (eds.), Proceedings of the 5th International Conference on Computer-Aided Design of User Interfaces (CADUI 2004), Kluwer, pp.347–58.
Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L., Pittman, J. A. & Smith, I. [1997], Unification-based Multimodal Integration, in P. R. Cohen & W. Wahlster (eds.), Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and of the 8th Conference of the European Chapter of the Association for Computational Linguistics, Morgan-Kaufmann, pp.281–8.
Krahnstoever, N., Kettebekov, S., Yeasin, M. & Sharma, R. [2002], A Real-time Framework for Natural Multimodal Interaction with Large Screen Displays, in A. Waibel, W. Gao, E. Horvitz, S. Furui & J. Yang (eds.), Proceedings of the 4th IEEE International Conference on Multimodal Interfaces (ICMI’02), IEEE Computer Society Press, pp.349–54.
Mansoux, B., Nigay, L. & Troccaz, J. [2005], The Mini-Screen: An Innovative Device for Computer Assisted Surgery Systems, in J. D. Westwood, R. S. Haluck, H. M. Hoffman, G. T. Mogel, R. Phillips & R. A. R. anf K. G. Vosburgh (eds.), Medicine Meets Virtual Reality 13: The magical Next Becomes the Medical Now, Vol. 111 of Studies in Health Technology and Informatics, IOS Press, pp.314–20.
Moran, D. B., Cheyer, A. J., E., J. L., Martin, D. L. & Park, S. [1997], Multimodal User Interfaces in the Open Agent Architecture, in A. R. Puerta & E. Edmonds (eds.), Proceedings of the 2nd International Conference on Intelligent User Interfaces (IUI’97), ACM Press, pp.61–8.
Myers, B., Hudson, S. E. & Pausch, R. [2000], Past, Present, and Future of User Interface Software Tools, ACM Transactions on Computer-Human Interaction 7(1), 3–28.
Nigay, L. & Coutaz, J. [1995], A Generic Platform for Addressing the Multimodal Challenge, in I. Katz, R. Mack, L. Marks, M. B. Rosson & J. Nielsen (eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’95), ACM Press, pp.98–105.
Nigay, L. & Coutaz, J. [1997], Multifeature Systems: The CARE Properties and Their Impact on Software Design, in J. Lee (ed.), Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, AAAI Press, Chapter 9. http://iihm.imag.fr/publs/1995/IMMIChapNigay95.pdf, last accessed 2006-05-27.
Oviatt, S. L. [2000], Taming Recognition Errors with a Multimodal Interface, Communications of the ACM 43(9), 45–51. Special Issue on Spoken Language Interfaces.
Reeves, L. M., Lai, J., Larson, J. A., Oviatt, S., Balaji, T. S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J. C., McTear, M., Raman, T. V., Stanney, K. M., Su, H. & Wang, Q. Y. [2004], Guidelines for Multimodal User Interfaces Design, Communications of the ACM 47(1), 57–9.
Rousseau, C., Bellik, Y., Vernier, F. & Bazalgette, D. [2004], Architecture Framework For Output Multimodal Systems Design, in Proceedings of Australian Conference on Computer-Human Interaction (OzCHI 2004), CHISIG. http://www.limsi.fr/Individu/vernier/Publis/OZCHI04-LongPaper.pdf last accessed 2006-05-27.
Sinha, A. K. & Landay, J. A. [2003], Capturing User Tests in a Multimodal, Multidevice Informal Prototyping Tool, in S. Oviatt, T. Darrell, M. Maybury & W. Wahlster (eds.), Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI’03), ACM Press, pp.117–24.
UIMS [1992], A Metamodel for the Runtime Architecture of an Interactive System: The UIMS Tool Developers Workshop, ACM SIGCHI Bulletin 24(1), 32–7.
Vernier, F. & Nigay, L. [2000], A Framework for the Combination and Characterization of Output Modalities, in P. Palanque & F. Paternò (eds.), Interactive Systems. Design, Specification, and Verification: Proceedings of the 7th International Workshop, DSV-IS 2000, Vol. 1946 of Lecture Notes in Computer Science, Springer-Verlag, pp.35–50.
Westeyn, T., Brashear, H., Atrash, A. & Starner, T. [2003], Georgia Tech Gesture Toolkit: Supporting Experiments in Gesture Recognition, in S. Oviatt, T. Darrell, M. Maybury & W. Wahlster (eds.), Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI’03), ACM Press, pp.85–92.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag London Limited
About this paper
Cite this paper
Mansoux, B., Nigay, L., Troccaz, J. (2007). Output Multimodal Interaction: The Case of Augmented Surgery. In: Bryan-Kinns, N., Blanford, A., Curzon, P., Nigay, L. (eds) People and Computers XX — Engage. Springer, London. https://doi.org/10.1007/978-1-84628-664-3_14
Download citation
DOI: https://doi.org/10.1007/978-1-84628-664-3_14
Publisher Name: Springer, London
Print ISBN: 978-1-84628-588-2
Online ISBN: 978-1-84628-664-3
eBook Packages: Computer ScienceComputer Science (R0)