Surgery Today

, Volume 45, Issue 4, pp 397–406 | Cite as

Clinical application of navigation surgery using augmented reality in the abdominal field

  • Tomoyoshi OkamotoEmail author
  • Shinji Onda
  • Katsuhiko Yanaga
  • Naoki Suzuki
  • Asaki Hattori
Review Article


This article presents general principles and recent advancements in the clinical application of augmented reality-based navigation surgery (AR based NS) for abdominal procedures and includes a description of our clinical trial and subsequent outcomes. Moreover, current problems and future aspects are discussed. The development of AR-based NS in the abdomen is delayed compared with another field because of the problem of intraoperative organ deformations or the existence of established modalities. Although there are a few reports on the clinical use of AR-based NS for digestive surgery, sophisticated technologies in urology have often been reported. However, the rapid widespread use of video- or robot assisted surgeries requires this technology. We have worked to develop a system of AR-based NS for hepatobiliary and pancreatic surgery. Then we developed a short rigid scope that enables surgeons to obtain 3D view. We recently focused on pancreatic surgery, because intraoperative organ shifting is minimal. The position of each organ in overlaid image almost corresponded with that of the actual organ with about 5 mm of mean registration errors. Intraoperative information generated from this system provided us with useful navigation. However, AR-based NS has several problems to overcome such as organ deformity, evaluation of utility, portability or cost.


Navigation surgery Augmented reality Abdominal field Video see through system 


  1. 1.
    Marescaux J, Clement JM, Tassetti, Koehl C, Cotin S, Russier Y, et al. Virtual reality applied to hepatic surgery simulation: the next revolution. Ann Surg. 1998;228:627–34.CrossRefPubMedCentralPubMedGoogle Scholar
  2. 2.
    Lamada W, Glombitza G, Ficher L, Chiu P, Cárdenas CE Sr, Thorn M, et al. The impact of 3D reconstruction on operation planning in liver surgery. Arch Surg. 2000;135:1256–61.CrossRefGoogle Scholar
  3. 3.
    Satou S, Yamanaka J, Miura K, Nakao N, Nagao T, Sugimoto T, et al. A novel 3D hepatectomy simulation based on liver circulation: application to liver resection and transplantation. Hepatology. 2005;41:1297–304.CrossRefGoogle Scholar
  4. 4.
    Yamanaka J, Saito S, Fujimoto J. Impact of preoperative planning using virtual segmental volumetry on liver resection for hepatocellular carcinoma. World J Surg. 2007;31:1249–55.CrossRefPubMedGoogle Scholar
  5. 5.
    Endo I, Shimada H, Sugita M, Fujii Y, Morioka D, Takeda K, et al. Role of 3D imaging in operative planning for hilar cholangiocarcinoma. Surgery. 2007;142:666–75.CrossRefPubMedGoogle Scholar
  6. 6.
    Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg. 2009;96:1086–93.CrossRefPubMedGoogle Scholar
  7. 7.
    Aoki T, Mise Y, Kokudo N. Hepatic resections assisted by operation planning using a 3D navigation/simulation software (in Japanese). Tan to sui. 2013;34:27–34.Google Scholar
  8. 8.
    Lamade W, Vetter M, Hassenphlug P, Thorn M, Meinzer HP, Herfarth C. Navigation and image-guided HBP surgery: a review and preview. J Hepatobiliary Pancreat Sci. 2002;9:592–9.CrossRefGoogle Scholar
  9. 9.
    Shuhaiber JH. Augmented reality in surgery. Arch Surg. 2004;139:170–4.CrossRefPubMedGoogle Scholar
  10. 10.
    Sugimoto M. Recent advances in visualization, imaging, and navigation in hepatobiliary and pancreatic sciences. J Hepatobiliary Pancreat Sci. 2010;17:574–6.CrossRefPubMedGoogle Scholar
  11. 11.
    Nicolau S, Soler L, Mutter D, Marescaux J, et al. Augmented reality in laparoscopic surgical oncology. Surg Oncol. 2011;20:189–201.CrossRefPubMedGoogle Scholar
  12. 12.
    Das M, Sauer F, Schoepf UJ, Khamene A, Vogt SK, Scaller S, et al. Augmented reality visualization for CT-guided interventions: system description, feasibility, and initial evaluation in an abdominal phantom. Radiology. 2006;240:230–5.CrossRefPubMedGoogle Scholar
  13. 13.
    Robb RA. Vrsp: virtual reality assisted surgery program. In: 1st International symposium on computer aided surgery 1994; 18–19.Google Scholar
  14. 14.
    Mezger U, Jendrewski C, Bartels M. Navigation in surgery. Langenbecks Arch Surg. 2013;398:501–14.CrossRefPubMedCentralPubMedGoogle Scholar
  15. 15.
    Grunert P, Darabi K, Espinosa J, Filippi R. Computer-aided navigation in neurosurgery. Neurosurg Rev. 2003;26:73–99.CrossRefPubMedGoogle Scholar
  16. 16.
    MarkiewiczMR BellRB. Traditional and contemporary surgical approaches to the orbit. Oral Maxillofac Surg Clin North Am. 2012;24:573–607. doi: 10.1016/j.coms.2012.08.004.CrossRefGoogle Scholar
  17. 17.
    Hohlweg-Majert B, Schon R, Schmelzeisen R, Gellrich NC, Schramm A. Navigational maxillofacial surgery using virtual models. World J Surg. 2005;29:1530–8.CrossRefPubMedGoogle Scholar
  18. 18.
    Hamada H, Hayashi N, Asahi T, Kurimoto M, Hirashima Y, Endo S. Efficacy of a navigation system in neuro-endoscopic surgery. Minim Invasive Neurosurg. 2005;48:197–201.CrossRefPubMedGoogle Scholar
  19. 19.
    Zhou F, Shao JH, Zou SB, Huang MW, Yin XB, Yu X. Laparoscopic hepatectomy is associated with a higher incident frequency in hepatolithiasis patients. Surg Today. 2013;12:1371–81.CrossRefGoogle Scholar
  20. 20.
    Satava RM. Emerging technology for surgery in the 21st century. Arch Surg. 1999;134:1197–2002.CrossRefPubMedGoogle Scholar
  21. 21.
    Mårvik R, Langø T, Tangen GA, Andersen JO, Kaspersen JH, Ystgaard B, et al. Laparoscopic navigation pointer for 3D image-guided surgery. Surg Endosc. 2004;18:1242–8.CrossRefPubMedGoogle Scholar
  22. 22.
    Okamoto T, Onda S, Matsumoto M, Gocho T, Futagawa Y, Fujioka S, et al. Utility of augmented reality system in hepatobiliary–pancreatic surgery with laparotomy. J Jpn Soc Comput Aided Surg (in Japanese with English abstract). 2010;12:312–3.Google Scholar
  23. 23.
    Okamoto T, Onda S, Matsumoto M, Gocho T, Futagawa Y, Fujioka S, et al. Utility of augmented reality system in hepatobiliary surgery. J Hepatobiliary Pancreat Sci. 2013;20:249–53.CrossRefPubMedGoogle Scholar
  24. 24.
    Onda S, Okamoto T, Kanehira M, Fujioka S, Suzuki N, Hattori A, et al. Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci. 2013;20:448–53.CrossRefPubMedGoogle Scholar
  25. 25.
    Onda S, Okamoto T, Matsumoto M, Son K, Gocho T, Futagawa Y, et al. Utility of augmented reality system in pancreatectomy. J Jpn Soc Comput Aided Surg (in Japanese with English abstract). 2011;13:258–9.Google Scholar
  26. 26.
    Onda S, Okamoto T, Kanehira M, Ito R, Fujioka S, Yanaga K, et al. Clinical application of a stereoscopic image display system for surgical navigation in hepatobiliary and pancreatic surgery. J Jpn Soc Comput Aided Surg (in Japanese with English abstract). 2012;14:222–3.Google Scholar
  27. 27.
    Onda S, Okamoto T, Kanehira M, Suzuki F, Ito R, Fujioka S, et al. Identification of inferior pancreaticoduodenal artery during pancreaticoduodenectomy using augmented reality-based navigation system. J Hepatobiliary Pancreat Sci. 2013. doi: 10.1002/jhbp.25.Google Scholar
  28. 28.
    Ukimura O, Magi-Galluzzi C, Gill IS. Real-time transrectal ultrasound guidance during laparoscopic radical prostatectomy: impact on surgical margins. J Urol. 2006;175:1304–10.CrossRefPubMedGoogle Scholar
  29. 29.
    Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer HP, Rassweiler JJ, et al. Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol. 2011;25:1841–5.CrossRefPubMedGoogle Scholar
  30. 30.
    Ukimura O, Gill IS. Image-fusion, augmented reality, and predictive surgical navigation. Urol Clin North Am. 2009;36:115–23.CrossRefPubMedGoogle Scholar
  31. 31.
    Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, et al. Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc. 2010;24:1976–85.CrossRefPubMedGoogle Scholar
  32. 32.
    Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, et al. Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci. 2010;17:629–36.CrossRefPubMedGoogle Scholar
  33. 33.
    Liao H, Inomata T, Sakuma I, Dohi T. 3D augmented reality for MRI-guided surgery using integral videography autostereoscopic image overlay. IEEE Trans Biomed Eng. 2010;57:1476–86.CrossRefPubMedGoogle Scholar
  34. 34.
    Hansen C, Wieferich J, Ritter F, Rieder C, Peitgen HO. Illustrative visualization of 3D planning models for augmented reality in liver surgery. Int J Comput Assist Radiol Surg. 2010;5:133–41.CrossRefPubMedGoogle Scholar
  35. 35.
    Marescaux J, Rubino F, Arenas M, Mutter D, Soler L. Augmented-reality-assisted laparoscopic adrenalectomy. JAMA. 2004;292:2214–5.PubMedGoogle Scholar
  36. 36.
    Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, et al. Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatr Surg Int. 2012;28:341–6.CrossRefPubMedGoogle Scholar
  37. 37.
    Herline AJ, Stefansic JD, Debelak JP, Hartmann SL, Pinson CW, Galloway RL, et al. Image-guided surgery: preliminary feasibility studies of frameless stereotactic liver surgery. Arch Surg. 1999;134:644–9.CrossRefPubMedGoogle Scholar
  38. 38.
    Hostettler A, Nicolau SA, Rémond Y, Marescaux J, Soler L. A real-time predictive simulation of abdominal viscera positions during quiet free breathing. Prog Biophys Mol Biol. 2010;103:169–84.CrossRefPubMedGoogle Scholar
  39. 39.
    Beller S, Eulenstein S, Lange T, Hünerbein M, Schlag PM. Upgrade of an optical navigation system with a permanent electromagnetic position control: a first step towards “navigated control” for liver surgery. J Hepatobiliary Pancreat Surg. 2009;16:165–70.CrossRefPubMedGoogle Scholar
  40. 40.
    Cash DM, Miga MI, Sinha TK, Galloway RL, Chapman WC. Compensating for intraoperative soft-tissue deformations using incomplete surface data and finite elements. IEEE Trans Med Imaging. 2005;24:1479–91.CrossRefPubMedGoogle Scholar
  41. 41.
    Cash DM, Miga MI, Glasgow SC, Dawant BM, Clements LW, Cao Z, et al. Concepts and preliminary data toward the realization of image-guided liver surgery. J Gastrointest Surg. 2007;11:844–59.CrossRefPubMedGoogle Scholar
  42. 42.
    Konishi K, Hashizume M, Nakamoto M, Kakeji Y, Yoshino I, Taketomi A, et al. Augmented reality navigation system for endoscopic surgery based on three-dimensional ultrasound and computed tomography: application to 20 clinical cases. Inter Congr Ser. 2005;1281:537–42.CrossRefGoogle Scholar
  43. 43.
    Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD. Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology. 2009;73:896–900.CrossRefPubMedGoogle Scholar
  44. 44.
    Nakamoto M, Ukimura O, Faber K, Gill IS. Current progress on augmented reality visualization in endoscopic surgery. Curr Opin Urol. 2012;22:121–6.CrossRefPubMedGoogle Scholar
  45. 45.
    Nicolau SA, Vemuri A, Wu HS, Huang MH, Ho Y, Charnoz A, et al. A cost effective simulator for education of ultrasound image interpretation and probe manipulation. Stud Health Technol Inform. 2011;163:403–7.PubMedGoogle Scholar
  46. 46.
    Harders M, Bianchi G, Knoerlein B, Székely G. Calibration, registration, and synchronization for high precision augmented reality haptics. IEEE Trans Vis Comput Graph. 2009;15:138–49.CrossRefPubMedGoogle Scholar
  47. 47.
    Teber D, Guven S, Simpfendörfer T, Baumhauer M, Güven EO, Yencilek F, et al. Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol. 2009;56:332–8.CrossRefPubMedGoogle Scholar
  48. 48.
    Gavaghan KA, Peterhans M, Oliveira-Santos T, Weber S. A portable image overlay projection device for computer-aided open liver surgery. IEEE Trans Biomed Eng. 2011;58:1855–64.CrossRefPubMedGoogle Scholar
  49. 49.
    Baumhauer M, Feuerstein M, Meinzer HP, Rassweiler J. Navigation in endoscopic soft tissue surgery: perspectives and limitations. J Endourol. 2008;22:751–66.CrossRefPubMedGoogle Scholar
  50. 50.
    Gavaghan K, Oliveira-Santos T, Peterhans M, Reyes M, Kim H, et al. Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: phantom studies. Int J Comput Assist Radiol Surg. 2012;7:547–56.CrossRefPubMedGoogle Scholar
  51. 51.
    Masamune K, Fichtinger G, Deguet A, Matsuka D, Taylor R. An image overlay system with enhanced reality for percutaneous therapy performed inside CT scanner. LNCS. 2002;2489:77–84.Google Scholar
  52. 52.
    Argotti Y, Davis L, Outters V, Rolland JP. Dynamic superimposition of synthetic objects on rigid and simple deformable real objects. Comput Graph. 2002;26:919–30.CrossRefGoogle Scholar
  53. 53.
    Ferrari V, Megali G, Troia E, Pietrabissa A, Mosca F. A 3D mixed-reality system for stereoscopic visualization of medical dataset. IEEE Trans Biomed Eng. 2009;56:2627–33.CrossRefPubMedGoogle Scholar
  54. 54.
    Hattori A, Suzuki N, Hashizume M, Akahoshi T, Konishi K, Yamaguchi S, et al. A robotic surgery system (da Vinci) with image guided function system architecture and cholecystectomy. Stud Health Technol Inform. 2003;94:110–6.PubMedGoogle Scholar
  55. 55.
    Liao H, Ishihara H, Tran HH, Masamune K, Sakuma I, Dohi T. Precision-guided surgical navigation system using laser guidance and 3D autostereoscopic image overlay. Comput Med Imaging Graph. 2010;34:46–54.CrossRefPubMedGoogle Scholar
  56. 56.
    Wong DY, Leveque JC, Brumblay H, Krebsbach PH, Hollister SJ, Lamarca F. Macro-architectures in spinal cord scaffold implants influence regeneration. J Neurotrauma. 2008;25:1027–37.CrossRefPubMedCentralPubMedGoogle Scholar
  57. 57.
    Aoki T, Murakami Y, Enami Y, Fujimori T, Koizumi T, Yamada K, et al. Minimally invasive liver surgery using combination of virtual endoscopy and new ultrasound techniques with image fusion or global positioning system (GPS) tracking. J Jpn Soc Comput Aided Surg (in Japanese with English abstract). 2012;14:164–5.Google Scholar
  58. 58.
    Matsuki M, Okuda J, Kanazawa S, Kanamoto T, Inada Y, Tatsugami F, et al. Virtual CT colectomy by three-dimensional imaging using multidetector-row CT for laparoscopic colorectal surgery. Abdom Imaging. 2005;30:698–708.CrossRefPubMedGoogle Scholar
  59. 59.
    Kowalczuk J, Meyer A, Carlson J, Psota ET, Buettner S, Pérez LC, et al. Real-time 3D soft tissue reconstruction for laparoscopic surgery. Surg Endosc. 2012;26:3413–7.CrossRefPubMedGoogle Scholar
  60. 60.
    Peterhans M, vom Berg A, Dagon B, Inderbitzin D, Baur C, Candinas D, et al. A navigation system for open liver surgery: design, workflow and first clinical applications. Int J Med Robot. 2011;7:7–16.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Japan 2014

Authors and Affiliations

  • Tomoyoshi Okamoto
    • 1
    Email author
  • Shinji Onda
    • 2
  • Katsuhiko Yanaga
    • 2
  • Naoki Suzuki
    • 3
  • Asaki Hattori
    • 3
  1. 1.Department of SurgeryThe Jikei University Daisan HospitalTokyoJapan
  2. 2.Division of Digestive SurgeryThe Jikei University School of MedicineTokyoJapan
  3. 3.Institute for High Dimensional Medical ImagingThe Jikei University School of MedicineTokyoJapan

Personalised recommendations