Multi-modal Interaction in Biomedicine

  • Elena V. Zudilova
  • Peter M. A. Sloot
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3345)


Everybody agrees that user tasks and preferences should play a central role in the design and development of applications oriented to non-computer experts. Nevertheless, even biomedical applications are sometimes developed in a relative vacuum from the real needs of end-users and environments where they are supposed to be used.

To provide a clinician with an intuitive environment to solve a target class of problems, a biomedical application has to be built in such a way that a user can exploit modern technologies without specialized knowledge of underlying hardware and software [18]. Unfortunately, in reality the situation is different. Many developers do not take into account the fact that their potential users are people, who are mostly inexperienced computer users, and as a result they need intuitive interaction capabilities and a relevant feedback adapted to their knowledge and skills.

User comfort is very important for the success of any software application [13]. But very often we forget that usability problems may arise not only from an ‘uncomfortable’ graphical user interface (GUI), but also from a projection modality chosen incorrectly for deploying an interactive environment [16].

Existing projection modalities have not been sufficiently investigated yet in respect to usability factors. Meanwhile, the selection of an appropriate projection modality in accordance with the user’s tasks, preferences and personal features might help in building a motivated environment for biomedical purposes. In this chapter we summarize our recent findings related to this research and introduce a new concept of multi-modal interaction based on the combination of virtual reality (VR) and desktop projection modalities within the same system. For the case study of the research we used a biomedical application simulating vascular reconstruction [2,22].


Virtual Reality Augmented Reality Virtual Object Projection Modality Interaction Style 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Belleman, R.G.: Interactive Exploration in virtual environments, University of Amsterdam, PhD thesis (2003)Google Scholar
  2. 2.
    Belleman, R.G., Sloot, P.M.A.: Simulated Vascular Reconstruction in a Virtual Operating Theatre. In: Proceedings of CARS 2001, pp. 18–27 (2001)Google Scholar
  3. 3.
    Beyer, H., Holtzblatt, K.: Contextual Design: Defining Customer-Centered Systems. Morgan Kaufmann, San Francisco (1998)Google Scholar
  4. 4.
    Dykstra, P.: X11 in virtual environments: Combining computer interaction methodologies. In: X Resource issue Nine, 8th Annual X Technical Conference. O’Reilly and Associates, Sebastopol (1994)Google Scholar
  5. 5.
    Bowman, D.G., Hodges, L.F.: User Interface Constraints for Immersive Virtual Environment Applications. Graphics, Visualization and Usability Center Technical Report GIT-GVU-95-26 (1995)Google Scholar
  6. 6.
    Chen, C., Czerwinski, M., Macredie, R.: Individual differences in virtual environments-introduction and overview. Journal of the American Society for Information Science 51(6), 499–507 (2000)CrossRefGoogle Scholar
  7. 7.
    Crabtree, A., Rodden, T., Mariani, J.: Designing Virtual Environments to Support Cooperation in the Real World. Virtual Reality 6, 63–74 (2002)CrossRefGoogle Scholar
  8. 8.
    Cramer, H.S.M., Evers, V., Zudilova, E.V., Sloot, P.M.A.: Context Analysis to Inform Virtual Reality Application Development. Int. J. Virtual Reality (in press)Google Scholar
  9. 9.
    Enright, P.L., McBurnie, M.A., Bittner, V., Tracy, R.P., McNamara, R., Arnold, A., Newman, A.B.: The 6-min walk test: a quick measure of functional status in elderly adults, Cardiovascular Health Study. Chest. J. 123(2), 387–398 (2003)Google Scholar
  10. 10.
    Forsythe, D.E.: It’s just a matter of common sense: Ethnography as invisible work. Computer Supported Cooperative Work (8), 127–145 (1999)Google Scholar
  11. 11.
    Jolesz, F.A., et al.: Image-Guided Procedures and the Operating Room of the Future. Radiology 204, 601–612 (1997)Google Scholar
  12. 12.
    Gavidia Simonetti, D.P., Zudilova, E.V., Sloot, P.M.A.: A Client-Server Engine for Parallel Computation of High-Resolution Planes. In: Bubak, M., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2004. LNCS, vol. 3038, pp. 970–977. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  13. 13.
    Nielsen, J.: Usability Engineering. Academic Press, London (2000)Google Scholar
  14. 14.
    Pierce, J., Forsberg, A., Conway, M.J., Hong, S., Zeleznik, R.: Image Plane Interaction Techniques in 3D Immersive Environments. In: Proceedings of 1997 Symposium on Interactive 3D Graphics, pp. 39–43 (1997)Google Scholar
  15. 15.
    Poston, T., Serra, L.: Dextrous virtual work. Communications of the ACM 39(5), 37–45 (1996)CrossRefGoogle Scholar
  16. 16.
    Raskin, J.: The Humane Interface: New Directions for Designing Interactive Systems. Addison-Wesley Pub Co, Reading (2000)Google Scholar
  17. 17.
    Robb, R.A.: Handbook of Medical Imaging: Processing and Analysis. Academic Press, London (2000)Google Scholar
  18. 18.
    Sloot, P.M.A., van Albada, G.D., Zudilova, E.V., Heinzlreiter, P., Kranzlmüller, D., Rosmanith, H., Volkert, J.: Grid-based Interactive Visualisation of Medical Images. In: Proceedings of the First European HealthGrid Conference, pp. 57–66 (2003)Google Scholar
  19. 19.
    Stefani, O., Hoffmann, H., Patel, H., Haselberger, F.: Extending the DesktopWorkplace by a Portable Virtual Reality System. In: CD Proceedings of the IPT 2004 Immersive Projection Technology Workshop (2004)Google Scholar
  20. 20.
    Sutherland, I.E.: The Ultimate Display. In: Proceedings of IFIP Congress, New York, pp. 506–508 (1965)Google Scholar
  21. 21.
    Yao, J.S.T., Pearce, W.H.: Current Techniques in Modern Vascular Surgery, 1st edn. McGraw-Hill Professional, New York (2000)Google Scholar
  22. 22.
    Zudilova, E.V., Sloot, P.M.A., Belleman, R.G.: A Multi-modal Interface for an Interactive Simulated Vascular Reconstruction System. In: Proceedings of the IEEE International Conference on Multimodal Interfaces, pp. 313–319 (2002)Google Scholar
  23. 23.
    Zudilova, E.V., Sloot, P.M.A.: A First Step to a User-Centered Approach to a Development of Adaptive Simulation-Visualization Complexes. In: Proceedings of the International Conference of the Systemics, Cybernetics and Informatics, Orlando, Florida, USA, vol. V, pp. 104–110 (July 2002)Google Scholar
  24. 24.
    The Section Computational Science website,
  25. 25.
    The GPS drawing project,
  26. 26.
    The Visualisation Toolkit website,

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Elena V. Zudilova
    • 1
  • Peter M. A. Sloot
    • 1
  1. 1.University of AmsterdamThe Netherlands

Personalised recommendations