Encyclopedia of Computer Graphics and Games

Living Edition
| Editors: Newton Lee

Augmented Reality in Image-Guided Surgery

Living reference work entry

Latest version View entry history

DOI: https://doi.org/10.1007/978-3-319-08234-9_78-2



Augmented reality visualization in image-guided surgery provides the surgeon with the ability to access the radiological images and surgical planning contextually to the anatomy of the real patient. It aims to integrate surgical navigation with virtual planning.


The general ability to see into a living human system and to transfer the three-dimensional complexity of the human body into a comprehensive and useful visual representation has historically been considered of utmost importance by physicians in their will to pass on the acquired knowledge and experience to future generations (Fig. 1). In more recent times, the growing availability of new medical imaging modalities together with the need to reduce the invasiveness of the surgical procedures has encouraged the research for new 3D visualization modalities of patient-specific virtual reconstructions of the anatomy. Those...


Augmented Reality (AR) Image-guided Surgery (IGS) Virtual Content Egocentric Viewpoint Beam Combiner 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.


  1. Abe, Y., Sato, S., Kato, K., Hyakumachi, T., Yanagibashi, Y., Ito, M., Abumi, K.: A novel 3D guidance system using augmented reality for percutaneous vertebroplasty. J. Neurosurg. Spine. 19(4), 492–501 (2013).  https://doi.org/10.3171/2013.7.Spine12917 CrossRefGoogle Scholar
  2. Badiali, G., Ferrari, V., Cutolo, F., Freschi, C., Caramella, D., Bianchi, A., Marchetti, C.: Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J. Cranio-Maxillofac. Surg. 42(8), 1970–1976 (2014).  https://doi.org/10.1016/j.jcms.2014.09.001 CrossRefGoogle Scholar
  3. Baumhauer, M., Simpfendoerfer, T., Schwarz, R., Seitel, M., Müller-Stich, B., Gutt, C., Rassweiler, J., Meinzer, H.-P., Wolf, I.: Soft tissue navigation for laparoscopic prostatectomy: evaluation of camera pose estimation for enhanced visualization. In: Medical Imaging 2007, pp. 650911-650911-650912. Society for Optics and Photonics (2007)Google Scholar
  4. Benton, S.A.: Selected Papers on Three-dimensional Displays. SPIE Optical Engineering Press, Bellingham (2001)Google Scholar
  5. Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hummel, J., Hanel, R., Greimel, W., Homolka, P., Ewers, R., Bergmann, H.: A head-mounted operating binocular for augmented reality visualization in medicine – design and initial evaluation. IEEE Trans. Med. Imaging. 21(8), 991–997 (2002).  https://doi.org/10.1109/Tmi.2002.803099 CrossRefzbMATHGoogle Scholar
  6. Blackwelll, M., Nikou, C., DiGioia, A.M., Kanade, T.: An image overlay system for medical data visualization. Med. Image Comput. Comput. Assist. Interv. Miccai’98. 1496, 232–240 (1998)Google Scholar
  7. Cabrilo, I., Bijlenga, P., Schaller, K.: Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations. Acta Neurochir. 156(9), 1769–1774 (2014).  https://doi.org/10.1007/s00701-014-2183-9 CrossRefGoogle Scholar
  8. Cabrilo, I., Schaller, K., Bijlenga, P.: Augmented reality-assisted bypass surgery: embracing minimal invasiveness. World Neurosurg. 83(4), 596–602 (2015).  https://doi.org/10.1016/j.wneu.2014.12.020 CrossRefGoogle Scholar
  9. Caversaccio, M., Giraldez, J.G., Thoranaghatte, R., Zheng, G., Eggli, P., Nolte, L.P., Ballester, M.A.G.: Augmented reality endoscopic system (ARES): preliminary results. Rhinology. 46(2), 156–158 (2008)Google Scholar
  10. Cutolo, F., Parchi, P.D., Ferrari, V.: Video see through AR head-mounted display for medical procedures. Int. Sym. Mix Augment, 393–396 (2014)Google Scholar
  11. Cutolo, F., Badiali, G., Ferrari, V.: Human-PnP: ergonomic AR interaction paradigm for manual placement of rigid bodies. Augmented Environ. Comput. Assist. Interv. Ae-Cai. 9365, 50–60 (2015).  https://doi.org/10.1007/978-3-319-24601-7_6 CrossRefGoogle Scholar
  12. Cutolo, F., Carbone, M., Parchi, P.D., Ferrari, V., Lisanti, M., Ferrari, M.: Application of a new wearable augmented reality video see-through display to aid percutaneous procedures in spine surgery. In: De Paolis, L.T., Mongelli, A. (eds.) Augmented Reality, Virtual Reality, and Computer Graphics: Third International Conference, AVR 2016, Lecce, Italy, June 15–18, 2016. Proceedings, Part II, pp. 43–54. Springer International Publishing, Cham (2016a)Google Scholar
  13. Cutolo, F., Freschi, C., Mascioli, S., Parchi, P., Ferrari, M., Ferrari, V.: Robust and accurate algorithm for wearable stereoscopic augmented reality with three indistinguishable markers. Electronics. 5(3), 59 (2016b)CrossRefGoogle Scholar
  14. Cutolo, F., Meola, A., Carbone, M., Sinceri, S., Cagnazzo, F., Denaro, E., Esposito, N., Ferrari, M., Ferrari, V.: A new head-mounted display-based augmented reality system in neurosurgical oncology: a study on phantom. Comput. Assist. Surg. 22(1), 39–53 (2017).  https://doi.org/10.1080/24699322.2017.1358400 CrossRefGoogle Scholar
  15. Deng, W., Li, F., Wang, M., Song, Z.: Easy-to-use augmented reality neuronavigation using a wireless tablet PC. Stereotact. Funct. Neurosurg. 92(1), 17–24 (2014).  https://doi.org/10.1159/000354816 CrossRefGoogle Scholar
  16. Devernay, F., Mourgues, F., Coste-Maniere, E.: Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery. In: International Workshop on Medical Imaging and Augmented Reality, Proceedings, pp. 16–20. (2001)Google Scholar
  17. Edwards, P.J., Johnson, L.G., Hawkes, D.J., Fenlon, M.R.,Strong, A., Gleeson, M.: Clinical experience and perception in stereo augmented reality surgical navigation. In: Yang, G.Z., Jiang, T. (eds.) MIAR, pp. 369–376. Springer, Berlin/Heidelberg (2004).  https://doi.org/10.1007/978-3-540-28626-4_45
  18. Elmi-Terander, A., Skulason, H., Soderman, M., Racadio, J., Homan, R., Babic, D., van der Vaart, N., Nachabe, R.: Surgical navigation technology based on augmented reality and integrated 3D intraoperative imaging: a spine cadaveric feasibility and accuracy study. Spine. 41(21), E1303–E1311 (2016).  https://doi.org/10.1097/BRS.0000000000001830 CrossRefGoogle Scholar
  19. Falk, V., Mourgues, F., Adhami, L., Jacobs, S., Thiele, H., Nitzsche, S., Mohr, F.W., Coste-Maniere, T.: Cardio navigation: planning, simulation, and augmented reality in robotic assisted endoscopic bypass grafting. Ann. Thorac. Surg. 79(6), 2040–2048 (2005).  https://doi.org/10.1016/j/athorascur.2004.11.060 CrossRefGoogle Scholar
  20. Feuerstein, M., Mussack, T., Heining, S.M., Navab, N.: Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection. IEEE Trans. Med. Imaging. 27(3), 355–369 (2008).  https://doi.org/10.1109/Tmi.2007.907327 CrossRefGoogle Scholar
  21. Freysinger, W., Gunkel, A.R., Thumfart, W.F.: Image-guided endoscopic ENT surgery. Eur. Arch. Otorhinolaryngol. 254(7), 343–346 (1997).  https://doi.org/10.1007/Bf02630726 CrossRefGoogle Scholar
  22. Fritz, J., U-Thainual, P., Ungi, T., Flammang, A.J., Kathuria, S., Fichtinger, G., Iordachita, I.I., Carrino, J.A.: MR-guided vertebroplasty with augmented reality image overlay navigation. Cardiovasc. Intervent. Radiol. 37(6), 1589–1596 (2014).  https://doi.org/10.1007/s00270-014-0885-2 CrossRefGoogle Scholar
  23. Genc, Y., Tuceryan, M., Navab, N.: Practical solutions for calibration of optical see-through devices. In: International Symposium on Mixed and Augmented Reality, Proceedings, pp. 169–175 (2002)Google Scholar
  24. Gilson, S.J., Fitzgibbon, A.W., Glennerster, A.: Spatial calibration of an optical see-through head-mounted display. J. Neurosci. Methods. 173(1), 140–146 (2008).  https://doi.org/10.1016/j.jneumeth.2008.05.015 CrossRefGoogle Scholar
  25. Grimson, W.E.L., Ettinger, G.J., White, S.J., LozanoPerez, T., Wells, W.M., Kikinis, R.: An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization. IEEE Trans. Med. Imaging. 15(2), 129–140 (1996).  https://doi.org/10.1109/42.491415 CrossRefGoogle Scholar
  26. Holliman, N.S., Dodgson, N.A., Favalora, G.E., Pockett, L.: Three-dimensional displays: a review and applications analysis. IEEE Trans. Broadcast. 57(2), 362–371 (2011).  https://doi.org/10.1109/Tbc.2011.2130930 CrossRefGoogle Scholar
  27. Inoue, D., Cho, B., Mori, M., Kikkawa, Y., Amano, T., Nakamizo, A., Yoshimoto, K., Mizoguchi, M., Tomikawa, M., Hong, J., Hashizume, M., Sasaki, T.: Preliminary study on the clinical application of augmented reality neuronavigation. J. Neurol. Surg. A. Cent. Eur. Neurosurg. 74(2), 71–76 (2013).  https://doi.org/10.1055/s-0032-1333415 CrossRefGoogle Scholar
  28. Iseki, H., Masutani, Y., Iwahara, M., Tanikawa, T., Muragaki, Y., Taira, T., Dohi, T., Takakura, K.: Volumegraph (overlaid three-dimensional image-guided navigation). Clinical application of augmented reality in neurosurgery. Stereotact. Funct. Neurosurg. 68(1–4 Pt 1), 18–24 (1997)CrossRefGoogle Scholar
  29. Kellner, F., Bolte, B., Bruder, G., Rautenberg, U., Steinicke, F., Lappe, M., Koch, R.: Geometric calibration of head-mounted displays and its effects on distance estimation. IEEE Trans. Vis. Comput. Graph. 18(4), 589–596 (2012)CrossRefGoogle Scholar
  30. Kersten-Oertel, M., Jannin, P., Collins, D.L.: DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans. Vis. Comput. Graph. 18(2), 332–352 (2012).  https://doi.org/10.1109/TVCG.2011.50 CrossRefGoogle Scholar
  31. Kersten-Oertel, M., Jannin, P., Collins, D.L.: The state of the art of visualization in mixed reality image guided surgery. Comput. Med. Imaging Graph. 37(2), 98–112 (2013).  https://doi.org/10.1016/j.compmedimag.2013.01.009 CrossRefGoogle Scholar
  32. Kersten-Oertel, M., Gerard, I., Drouin, S., Mok, K., Sirhan, D., Sinclair, D.S., Collins, D.L.: Augmented reality in neurovascular surgery: feasibility and first uses in the operating room. Int. J. Comput. Assist. Radiol. Surg. 10(11), 1823–1836 (2015).  https://doi.org/10.1007/s11548-015-1163-8 CrossRefGoogle Scholar
  33. Liao, H., Nakajima, S., Iwahara, M., Kobayashi, E., Sakuma, I., Yahagi, N., Dohi, T.: Intra-operative real-time 3-D information display system based on integral videography. In: Niessen, W., Viergever, M. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2001. Lecture Notes in Computer Science, vol. 2208, pp. 392–400. Springer Berlin, (2001)Google Scholar
  34. Liao, H., Hata, N., Nakajima, S., Iwahara, M., Sakuma, I., Dohi, T.: Surgical navigation by autostereoscopic image overlay of integral videography. IEEE Trans. Inf. Technol. Biomed. 8(2), 114–121 (2004)CrossRefGoogle Scholar
  35. Liao, H., Inomata, T., Sakuma, I., Dohi, T.: Surgical navigation of integral videography image overlay for open MRI-guided glioma surgery. Med. Imaging Augmented Real. 4091, 187–194 (2006)CrossRefGoogle Scholar
  36. Liao, H.E., Inomata, T., Sakuma, I., Dohi, T.: 3-D augmented reality for MRI-guided surgery using integral videography autostereoscopic image overlay. IEEE Trans. Biomed. Eng. 57(6), 1476–1486 (2010).  https://doi.org/10.1109/Tbme.2010.2040278 CrossRefGoogle Scholar
  37. Lovo, E.E., Quintana, J.C., Puebla, M.C., Torrealba, G., Santos, J.L., Lira, I.H., Tagle, P.: A novel, inexpensive method of image coregistration for applications in image-guided surgery using augmented reality. Neurosurgery. 60(4 Suppl 2), 366–371.; discussion 371–362 (2007).  https://doi.org/10.1227/01.NEU.0000255360.32689.FA Google Scholar
  38. Low, D., Lee, C.K., Dip, L.L., Ng, W.H., Ang, B.T., Ng, I.: Augmented reality neurosurgical planning and navigation for surgical excision of parasagittal, falcine and convexity meningiomas. Br. J. Neurosurg. 24(1), 69–74 (2010).  https://doi.org/10.3109/02688690903506093 CrossRefGoogle Scholar
  39. Marmulla, R., Hoppe, H., Muhling, J., Eggers, G.: An augmented reality system for image-guided surgery. Int. J. Oral Maxillofac. Surg. 34(6), 594–596 (2005).  https://doi.org/10.1016/j.ijom.2005.05.004 CrossRefGoogle Scholar
  40. Meola, A., Cutolo, F., Carbone, M., Cagnazzo, F., Ferrari, M., Ferrari, V.: Augmented reality in neurosurgery: a systematic review. Neurosurg. Rev. (2016).  https://doi.org/10.1007/s10143-016-0732-9
  41. Mourgues, F., Coste-Maniere, P.: Flexible calibration of actuated stereoscopic endoscope for overlay in robot assisted surgery. Med. Image Comput. Comput-Assist. Interv. Miccai. 2488(Pt 1), 25–34 (2002.) 2002zbMATHGoogle Scholar
  42. Mukawa, H., Akutsu, K., Matsumura, I., Nakano, S., Yoshida, T., Kuwahara, M., Aiki, K., Ogawa, M.: Distinguished paper: a full color eyewear display using holographic planar waveguides. In: 2008 Sid International Symposium, Digest of Technical Papers, vol Xxxix, Books I–Iii 39, pp. 89–92 (2008)Google Scholar
  43. Muller, M., Rassweiler, M.C., Klein, J., Seitel, A., Gondan, M., Baumhauer, M., Teber, D., Rassweiler, J.J., Meinzer, H.P., Maier-Hein, L.: Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int. J. Comput. Assist. Radiol. Surg. 8(4), 663–675 (2013).  https://doi.org/10.1007/s11548-013-0828-4 CrossRefGoogle Scholar
  44. Narita, Y., Tsukagoshi, S., Suzuki, M., Miyakita, Y., Ohno, M., Arita, H., Saito, Y., Kokojima, Y., Watanabe, N., Moriyama, N., Shibui, S.: Usefulness of a glass-free medical three-dimensional autostereoscopic display in neurosurgery. Int. J. Comput. Assist. Radiol. Surg. 9(5), 905–911 (2014).  https://doi.org/10.1007/s11548-014-0984-1 CrossRefGoogle Scholar
  45. Navab, N., Traub, J., Sielhorst, T., Feuerstein, M., Bichlmeier, C.: Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput. Graph. Appl. 27(5), 10–14 (2007).  https://doi.org/10.1109/Mcg.2007.117 CrossRefGoogle Scholar
  46. Navab, N., Heining, S.M., Traub, J.: Camera Augmented Mobile C-Arm (CAMC): calibration, accuracy study, and clinical applications. IEEE Trans. Med. Imaging. 29(7), 1412–1423 (2010).  https://doi.org/10.1109/Tmi.2009.2021947 CrossRefGoogle Scholar
  47. Nicolau, S., Soler, L., Mutter, D., Marescaux, J.: Augmented reality in laparoscopic surgical oncology. Surg. Oncol. Oxf. 20(3), 189–201 (2011).  https://doi.org/10.1016/j.suronc.2011.07.002 CrossRefGoogle Scholar
  48. Peters, T.M.: Image-guided surgery: from X-rays to virtual reality. Comput. Methods. Biomech. Biomed. Engin. 4(1), 27–57 (2000)MathSciNetCrossRefGoogle Scholar
  49. Peters, T.M.: Image-guidance for surgical procedures. Phys. Med. Biol. 51(14), R505–R540 (2006).  https://doi.org/10.1088/0031-9155/51/14/R01 CrossRefGoogle Scholar
  50. Plopski, A., Itoh, Y., Nitschke, C., Kiyokawa, K., Klinker, G., Takemura, H.: Corneal-imaging calibration for optical see-through head-mounted displays. IEEE Trans. Vis. Comput. Graph. 21(4), 481–490 (2015).  https://doi.org/10.1109/Tvcg.2015.2391857 CrossRefGoogle Scholar
  51. Plopski, A., Orlosky, J., Itoh, Y., Nitschke, C., Kiyokawa, K., Klinker, G.: Automated spatial calibration of HMD systems with unconstrained eye-cameras. In: Proceedings of the 2016 15th IEEE International Symposium on Mixed and Augmented Reality (Ismar), pp. 94–99 (2016).  https://doi.org/10.1109/Ismar.2016.16
  52. Rankin, T.M., Slepian, M.J., Armstrong, D.G.: Augmented reality in surgery. In: Latifi, R., Rhee, P., Gruessner, R.W.G. (eds.) Technological Advances in Surgery, Trauma and Critical Care, pp. 59–71. Springer, New York (2015)CrossRefGoogle Scholar
  53. Roberts, D.W., Strohbehn, J.W., Hatch, J.F., Murray, W., Kettenberger, H.: A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. J. Neurosurg. 65(4), 545–549 (1986).  https://doi.org/10.3171/jns.1986.65.4.0545 CrossRefGoogle Scholar
  54. Rolland, J.P., Cakmakci, O.: The past, present, and future of head-mounted display designs. In: Photonics Asia 2005, p. 10. SPIEGoogle Scholar
  55. Rolland, J.P., Holloway, R.L., Fuchs, H.: A comparison of optical and video see-through head-mounted displays. Telemanipulator Telepresence Technol. 2351, 293–307 (1994)CrossRefGoogle Scholar
  56. Sielhorst, T., Feuerstein, M., Navab, N.: Advanced medical displays: a literature review of augmented reality. J. Disp. Technol. 4(4), 451–467 (2008).  https://doi.org/10.1109/JDT.2008.2001575 CrossRefGoogle Scholar
  57. Stetten, G., Chib, V., Hildebrand, D., Bursee, J.: Real time tomographic reflection: Phantoms for calibration and biopsy. In: IEEE and ACM International Symposium on Augmented Reality, Proceedings, Los Alamitos, IEEE Press, 11–19 (2001)Google Scholar
  58. Su, L.M., Vagvoigyi, B.P., Agarwal, R., Reiley, C.E., Taylor, R.H., Hager, G.D.: Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology. 73(4), 896–900 (2009).  https://doi.org/10.1016/j.urology.2008.11.040 CrossRefGoogle Scholar
  59. Suenaga, H., Tran, H.H., Liao, H.G., Masamune, K., Dohi, T., Hoshi, K., Mori, Y., Takato, T.: Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study. Int. J. Oral Sci. 5(2), 98–102 (2013).  https://doi.org/10.1038/ijos.2013.26 CrossRefGoogle Scholar
  60. Suenaga, H., Tran, H.H., Liao, H., Masamune, K., Dohi, T., Hoshi, K., Takato, T.: Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study. BMC Med. Imaging. 15, 51 (2015).  https://doi.org/10.1186/s12880-015-0089-5. ARTN 51CrossRefGoogle Scholar
  61. Tuceryan, M., Genc, Y., Navab, N.: Single-point active alignment method (SPAAM) for optical see-through HMD calibration for augmented reality. Presence-Teleop Virt Environ. 11(3), 259–276 (2002).  https://doi.org/10.1162/105474602317473213 CrossRefGoogle Scholar
  62. Wang, J., Suenaga, H., Yang, L., Kobayashi, E., Sakuma, I.: Video see-through augmented reality for oral and maxillofacial surgery. The international journal of medical robotics + computer assisted surgery: MRCAS 13(2) (2017).  https://doi.org/10.1002/rcs.1754
  63. Wesarg, S., Firle, E.A., Schwald, B., Seibert, H., Zogal, P., Roeddiger, S.: Accuracy of needle implantation in brachytherapy using a medical AR system – a phantom study. Med. Imaging 2004: Visual. Image-Guided Proced. Display. 5367, 341–352 (2004).  https://doi.org/10.1117/12.535415 CrossRefGoogle Scholar
  64. Wu, J.R., Wang, M.L., Liu, K.C., Hu, M.H., Lee, P.Y.: Real-time advanced spinal surgery via visible patient model and augmented reality system. Comput. Methods Prog. Biomed. 113(3), 869–881 (2014).  https://doi.org/10.1016/j.cmpb.2013.12.071 CrossRefGoogle Scholar
  65. Zeng, B., Meng, F., Ding, H., Wang, G.: A surgical robot with augmented reality visualization for stereoelectroencephalography electrode implantation. Int. J. Comput. Assist. Radiol. Surg. 12(8), 1355–1368 (2017).  https://doi.org/10.1007/s11548-017-1634-1 CrossRefGoogle Scholar
  66. Zhu, M., Liu, F., Chai, G., Pan, J.J., Jiang, T., Lin, L., Xin, Y., Zhang, Y., Li, Q.: A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep. 7, 42365 (2017).  https://doi.org/10.1038/srep42365 CrossRefGoogle Scholar
  67. Zinser, M.J., Mischkowski, R.A., Dreiseidler, T., Thamm, O.C., Rothamel, D., Zoller, J.E.: Computer-assisted orthognathic surgery: waferless maxillary positioning, versatility, and accuracy of an image-guided visualisation display. Brit. J. Oral. Max. Surg. 51(8), 827–833 (2013a).  https://doi.org/10.1016/j.bjoms.2013.06.014 CrossRefGoogle Scholar
  68. Zinser, M.J., Sailer, H.F., Ritter, L., Braumann, B., Maegele, M., Zoller, J.E.: A paradigm shift in orthognathic surgery? A comparison of navigation, computer-aided designed/computer-aided manufactured splints, and “classic” intermaxillary splints to surgical transfer of virtual orthognathic planning. J. Oral. Maxillofac. Surg. 71(12.), 2151), e2151–e2121 (2013b).  https://doi.org/10.1016/j.joms.2013.07.007 CrossRefGoogle Scholar

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PisaPisaItaly