Multimodal Augmented Reality in Medicine

  • Matthias Harders
  • Gerald Bianchi
  • Benjamin Knoerlein
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4555)


The driving force of our current research is the development of medical training systems using augmented reality techniques. To provide multimodal feedback for the simulation, haptic interfaces are integrated into the framework. In this setting, high accuracy and stability are a prerequisite. Misalignment of overlaid virtual objects would greatly compromise manipulative fidelity and the sense of presence, and thus reduce the overall training effect. Therefore, our work targets the precise integration of haptic devices into the augmented environment and the stabilization of the tracking process. This includes a distributed system structure which is able to handle multiple users in a collaborative augmented world. In this paper we provide an overview of related work in medical augmented reality and give an introduction to our developed system.


Augmented Reality Haptics Medicine Training 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Liu, A., Tendick, F., Cleary, K., Kaufmann, C.: A Survey of Surgical Simulation: Applications, Technology, and Education, Presence, vol. 12(6), pp. 599–614. MIT Press, Cambridge (2003)Google Scholar
  2. 2.
    Basdogan, C., Sedef, M., Harders, M., Wesarg, S.: Virtual Reality Supported Simulators for Training in Minimally Invasive Surgery. IEEE Computer Graphics and Applications (2007)Google Scholar
  3. 3.
    Azuma, R.T.: A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6(4), 355–385 (1997)Google Scholar
  4. 4.
    Edwards, P.J., King Jr., A.P., Maurer, C.R., de Cunha, D.A., Hawkes, D.J., Hill, D.L.G., Gaston, R.P., Fenlon, M.R., Chandra, S., Strong, A.J., Chandler, C.L., Richards, A., Gleeson, M.E.: Design and evaluation of a system for microscope assisted guided interventions (magi). In: Taylor, C., Colchester, A. (eds.) MICCAI 1999: Proceedings of the Second International Conference on Medical Image Computing and Computer-Assisted Intervention. LNCS, vol. 1679, pp. 842–851. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  5. 5.
    Giraldez, G.J., Talib, H., Caversaccio, M., Gonzalez, M.A.: Ballester. Multimodal augmented reality system for surgical microscopy. In: Proceedings of SPIE Medical Imaging, vol. 6141, pp. 537–544 (2006)Google Scholar
  6. 6.
    De Buck, S., Van Cleynenbreugel, J., Geys, I., Koninckx, T., Koninck, P.R., Suetens, P.: A system to support laparoscopic surgery by augmented reality visualization. In: Niessen, W.J., Viergever, M.A. (eds.) MICCAI 2001. LNCS, vol. 2208, pp. 691–698. Springer, Heidelberg (2001)Google Scholar
  7. 7.
    Feuerstein, M., Wildhirt, S., Bauernschmitt, M.R., Navab, N.: Automatic patient registration for port placement in minimally invasive endoscopic surgery. In: Duncan, J.S., Gerig, G. (eds.) MICCAI 2005. LNCS, vol. 3749, pp. 287–294. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    Mourgues, F., Vieville, T., Falk, V., Coste-Manière, E.: Interactive guidance by image overlay in robot assisted coronary artery bypass. In: Ellis, R.E., Peters, T.M. (eds.) MICCAI 2003. LNCS, vol. 2878, pp. 173–181. Springer, Heidelberg (2003)Google Scholar
  9. 9.
    Grimson, W.E.L., Lozano-Perez, T., Wells, W.M., Ettinger, I.G.J., White, S.J., Kikinis, R.: An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization. In: Transactions on Medical Imaging, pp. 430–436 (1996)Google Scholar
  10. 10.
    Sato, Y., Nakamoto, M., Tamaki, Y., Sasama, T., Sakita, I., Nakajima, Y., Monden, M., Tamura, S.: Image guidance of breast cancer surgery using 3-d ultrasound images and augmented reality visualization. IEEE Trans. on Medical Imaging 17, 681–693 (1998)CrossRefGoogle Scholar
  11. 11.
    Nicolau, S.A., Pennec, X., Soler, L., Ayache, N.: A complete augmented reality guidance system for liver punctures: First clinical evaluation. In: Duncan, J.S., Gerig, G. (eds.) MICCAI 2005. LNCS, vol. 3749, pp. 539–547. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    Peuchot, B., Tanguy, A., Eude, M.: Virtual reality as an operative tool during scoliosis surgery. In: First International Conference on Computer Vision, Virtual Reality and Robotics in Medicine (CVRMed), pp. 549–554 (1995)Google Scholar
  13. 13.
    Masamune, K., Masutani, Y., Nakajima, S., Sakuma, I., Dohi, T., Iseki, H., Takakura, K.: Three-dimensional slice image overlay system with accurate depth perception for surgery. In: Delp, S.L., DiGoia, A.M., Jaramaz, B. (eds.) MICCAI 2000. LNCS, vol. 1935, pp. 395–402. Springer, Heidelberg (2000)Google Scholar
  14. 14.
    Stetten, G., Chib, V.: Overlaying ultrasound images on direct vision. Journal of Ultrasound in Medicine 20(3), 235–240 (2001)Google Scholar
  15. 15.
    Navab, N., Bani-Kashemi, A., Mitschke, M.: Merging visible and invisible: two camera-augmented mobile c-arm (CAMC) applications. In: Augmented Reality, 1999 (IWAR 1999) Proceedings. 2nd IEEE and ACM International Workshop, pp. 134– 141 (1999)Google Scholar
  16. 16.
    Mitschke, M., Bani-Hashemi, A., Navab, N.: Interventions under video-augmented X-ray guidance: Application to needle placement. In: Delp, S.L., DiGoia, A.M., Jaramaz, B. (eds.) MICCAI 2000. LNCS, vol. 1935, pp. 858–868. Springer, Heidelberg (2000)Google Scholar
  17. 17.
    Schwald, B., Seibert, H.: Registration tasks for hybrid tracking system for medical augmented reality. Journal of WSCG 12, 411–418 (2004)Google Scholar
  18. 18.
    Bajura, M., Fuchs, H., Ohbuchi, R.: Merging virtual objects with the real world: seeing ultrasound imagery within the patient. In: SIGGRAPH 1992: Proceedings of the 19th annual conference on Computer graphics and interactive techniques, pp. 203–210 (1992)Google Scholar
  19. 19.
    State, A., Livingston, M.A., Garrett, W.F., Hirota, G., Whitton, M.C., Pisano, E.D., Fuchs, H.: Technologies for augmented reality systems: Realizing ultrasound-guided needle biopsies. In: SIGGRAPH, pp. 439–446 (1996)Google Scholar
  20. 20.
    Sauer, F., Khamene, A., Bascle, B., Schinunang, L., Wenzel, F., Vogt, S.: Augmented reality visualization of ultrasound images: system description, calibration, and features. In: Augmented Reality, 2001. In: Proceedings. IEEE and ACM International Symposium, pp. 30–39 (2001)Google Scholar
  21. 21.
    Maurer, C.R., Jr Sauer, F., Hu, B., Bascle, B., Geiger, B., Wenzel, F., Recchi, F., Rohlfing, T., Brown, C.M., Bakos, R.S., Maciunas, R.J., Bani-Hashemi, A.: Augmented reality visualization of brain structures with stereo and kinetic depth cues: system description and initial evaluation with head phantom, Medical Imaging 2001: Visualization, Display, and Image-Guided Procedures, pp. 445–456 (2001)Google Scholar
  22. 22.
    Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hanel, R., Wagner, A., Rafolt, D., Ewers, R., Bergmann, H.: The Varioscope AR - a head-mounted operating microscope for augmented reality. In: MICCAI 2000: Proceedings of the Third International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 869–877 (2000)Google Scholar
  23. 23.
    Baillot, Y., Rolland, J., Lin, K., Wright, D.: Automatic modeling of knee-joint motion for the virtual reality dynamic anatomy (vrda) tool. Presence: Teleoperators and Virtual Environments 9, 223–235 (2000)CrossRefGoogle Scholar
  24. 24.
    Lapeer, R., Chen, M.S., Villagrana, J.: An augmented reality based simulation of obstetric forceps delivery. In: Third IEEE and ACM International Symposium on Mixed and Augmented Reality, 2004. ISMAR 2004, pp. 274–275 (2004)Google Scholar
  25. 25.
    Sielhorst, T., Obst, T., Burgkart, R., Riener, R., Navab, N.: An augmented reality delivery simulator for medical training. In: International Workshop on Augmented Environments for Medical Imaging - MICCAI Satellite Workshop, pp. 11–20 (2004)Google Scholar
  26. 26.
    Bianchi, G., Wengert, C., Harders, M., Cattin, P., Székely, G.: Camera-Marker Alignment Framework and Comparison with Hand-Eye Calibration for Augmented Reality Applications, ISMAR 2005, pp. 188–189 (2005)Google Scholar
  27. 27.
    Bianchi, G., Jung, C., Knörlein, B., Harders, M., Székely, G.: High-fidelity visuo-haptic interaction with virtual objects in multi-modal AR systems. In: Proc. ISMAR (2006)Google Scholar
  28. 28.
    Bianchi, G., Knörlein, B., Székely, G., Harders, M.: High Precision Augmented Reality Haptics. In: Proc. Eurohaptics (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Matthias Harders
    • 1
  • Gerald Bianchi
    • 1
  • Benjamin Knoerlein
    • 1
  1. 1.Virtual Reality in Medicine Group, Computer Vision Lab, ETH ZurichSwitzerland

Personalised recommendations