Advertisement

Augmented reality in open surgery

  • Benish Fida
  • Fabrizio Cutolo
  • Gregorio di Franco
  • Mauro Ferrari
  • Vincenzo Ferrari
Review Article

Abstract

Augmented reality (AR) has been successfully providing surgeons an extensive visual information of surgical anatomy to assist them throughout the procedure. AR allows surgeons to view surgical field through the superimposed 3D virtual model of anatomical details. However, open surgery presents new challenges. This study provides a comprehensive overview of the available literature regarding the use of AR in open surgery, both in clinical and simulated settings. In this way, we aim to analyze the current trends and solutions to help developers and end/users discuss and understand benefits and shortcomings of these systems in open surgery. We performed a PubMed search of the available literature updated to January 2018 using the terms (1) “augmented reality” AND “open surgery”, (2) “augmented reality” AND “surgery” NOT “laparoscopic” NOT “laparoscope” NOT “robotic”, (3) “mixed reality” AND “open surgery”, (4) “mixed reality” AND “surgery” NOT “laparoscopic” NOT “laparoscope” NOT “robotic”. The aspects evaluated were the following: real data source, virtual data source, visualization processing modality, tracking modality, registration technique, and AR display type. The initial search yielded 502 studies. After removing the duplicates and by reading abstracts, a total of 13 relevant studies were chosen. In 1 out of 13 studies, in vitro experiments were performed, while the rest of the studies were carried out in a clinical setting including pancreatic, hepatobiliary, and urogenital surgeries. AR system in open surgery appears as a versatile and reliable tool in the operating room. However, some technological limitations need to be addressed before implementing it into the routine practice.

Keywords

Augmented reality Mixed reality Open surgery Image-guided surgery Surgical navigation 

Notes

Funding

This work was funded by the HORIZON2020 Project VOSTARS, Project ID: 731974. Call: ICT-29-2016—Photonics KET 2016.

Compliance with ethical standards

Conflicts of interest

The authors declare that they have no conflict of interest.

Research involving human participants and/or animals

The authors confirm that no animal nor human testing were performed in this study.

Informed consent

No informed consent is required.

References

  1. 1.
    Navab N, Traub J, Sielhorst T, Feuerstein M, Bichlmeier C (2007) Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Graph 27(5):10–14.  https://doi.org/10.1109/Mcg.2007.117 CrossRefGoogle Scholar
  2. 2.
    Cutolo F (2017) Augmented reality in image-guided surgery. In: Lee N (ed) Encyclopedia of computer graphics and games. Springer, Cham, pp 1–11.  https://doi.org/10.1007/978-3-319-08234-9_78-1 Google Scholar
  3. 3.
    Vavra P, Roman J, Zonca P, Ihnat P, Nemec M, Kumar J, Habib N, El-Gendi A (2017) Recent development of augmented reality in surgery: a review. J Healthc Eng.  https://doi.org/10.1155/2017/4574172 PubMedPubMedCentralGoogle Scholar
  4. 4.
    Cutolo F, Parchi PD, Ferrari V (2014) Video see through AR head-mounted display for medical procedures. In: 2014 IEEE international symposium on mixed and augmented reality (ISMAR), 10–12 Sept 2014, pp 393–396.  https://doi.org/10.1109/ismar.2014.6948504
  5. 5.
    Kersten-Oertel M, Jannin P, Collins DL (2012) DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Gr 18(2):332–352.  https://doi.org/10.1109/Tvcg.2011.50 CrossRefGoogle Scholar
  6. 6.
    Meola A, Cutolo F, Carbone M, Cagnazzo F, Ferrari M, Ferrari V (2017) Augmented reality in neurosurgery: a systematic review. Neurosurg Rev 40(4):537–548.  https://doi.org/10.1007/s10143-016-0732-9 CrossRefPubMedGoogle Scholar
  7. 7.
    Azuma RT (1997) A survey of augmented reality. Presence Teleop Virt 6(4):355–385CrossRefGoogle Scholar
  8. 8.
    Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph 21(6):34–47.  https://doi.org/10.1109/38.963459 CrossRefGoogle Scholar
  9. 9.
    Bimber O, Raskar R (2006) Modern approaches to augmented reality. Paper presented at the ACM SIGGRAPH 2006 Courses, Boston, MassachusettsGoogle Scholar
  10. 10.
    Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Found Trends Hum Comput Interact 8(2–3):73–272.  https://doi.org/10.1561/1100000049 CrossRefGoogle Scholar
  11. 11.
    Borgmann H, Rodriguez Socarras M, Salem J, Tsaur I, Gomez Rivas J, Barret E, Tortolero L (2017) Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World J Urol 35(6):967–972.  https://doi.org/10.1007/s00345-016-1956-6 CrossRefPubMedGoogle Scholar
  12. 12.
    Sauer IM, Queisner M, Tang P, Moosburner S, Hoepfner O, Horner R, Lohmann R, Pratschke J (2017) Mixed reality in visceral surgery: development of a suitable workflow and evaluation of intraoperative use-cases. Ann Surg 266(5):706–712.  https://doi.org/10.1097/SLA.0000000000002448 CrossRefPubMedGoogle Scholar
  13. 13.
    Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P (2016) Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience. World J Surg 40(2):419–426.  https://doi.org/10.1007/s00268-015-3229-8 CrossRefPubMedGoogle Scholar
  14. 14.
    Marzano E, Piardi T, Soler L, Diana M, Mutter D, Marescaux J, Pessaux P (2013) Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg 17(11):1980–1983.  https://doi.org/10.1007/s11605-013-2307-1 CrossRefPubMedGoogle Scholar
  15. 15.
    Okamoto T, Onda S, Yasuda J, Yanaga K, Suzuki N, Hattori A (2015) Navigation surgery using an augmented reality for pancreatectomy. Dig Surg 32(2):117–123.  https://doi.org/10.1159/000371860 CrossRefPubMedGoogle Scholar
  16. 16.
    Onda S, Okamoto T, Kanehira M, Fujioka S, Suzuki N, Hattori A, Yanaga K (2013) Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci 20(4):448–453.  https://doi.org/10.1007/s00534-012-0582-y CrossRefPubMedGoogle Scholar
  17. 17.
    Onda S, Okamoto T, Kanehira M, Suzuki F, Ito R, Fujioka S, Suzuki N, Hattori A, Yanaga K (2014) Identification of inferior pancreaticoduodenal artery during pancreaticoduodenectomy using augmented reality-based navigation system. J Hepatobiliary Pancreat Sci 21(4):281–287.  https://doi.org/10.1002/jhbp.25 CrossRefPubMedGoogle Scholar
  18. 18.
    KleinJan GH, van den Berg NS, van Oosterom MN, Wendler T, Miwa M, Bex A, Hendricksen K, Horenblas S, van Leeuwen FW (2016) Toward (hybrid) navigation of a fluorescence camera in an open surgery setting. J Nucl Med 57(10):1650–1653.  https://doi.org/10.2967/jnumed.115.171645 CrossRefPubMedGoogle Scholar
  19. 19.
    van Oosterom MN, Meershoek P, KleinJan GH, Hendricksen K, Navab N, van de Velde CJH, van der Poel HG, van Leeuwen FWB (2018) Navigation of fluorescence cameras during soft tissue surgery—is it possible to use a single navigation setup for various open and laparoscopic urological surgery applications? J Urol 199(4):1061–1068.  https://doi.org/10.1016/j.juro.2017.09.160 CrossRefPubMedGoogle Scholar
  20. 20.
    Okamoto T, Onda S, Matsumoto M, Gocho T, Futagawa Y, Fujioka S, Yanaga K, Suzuki N, Hattori A (2013) Utility of augmented reality system in hepatobiliary surgery. J Hepatobiliary Pancreat Sci 20(2):249–253.  https://doi.org/10.1007/s00534-012-0504-z CrossRefPubMedGoogle Scholar
  21. 21.
    Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, Dong J (2017) Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: a case report. Medicine (Baltimore) 96(37):e8083.  https://doi.org/10.1097/MD.0000000000008083 CrossRefGoogle Scholar
  22. 22.
    Ferrari V, Megali G, Troia E, Pietrabissa A, Mosca F (2009) A 3-D mixed-reality system for stereoscopic visualization of medical dataset. IEEE T Bio Med Eng 56(11):2627–2633.  https://doi.org/10.1109/Tbme.2009.2028013 CrossRefGoogle Scholar
  23. 23.
    Gavaghan KA, Peterhans M, Oliveira-Santos T, Weber S (2011) A portable image overlay projection device for computer-aided open liver surgery. IEEE Trans Biomed Eng 58(6):1855–1864.  https://doi.org/10.1109/TBME.2011.2126572 CrossRefPubMedGoogle Scholar
  24. 24.
    Kersten-Oertel M, Jannin P, Collins DL (2010) DVV: towards a taxonomy for mixed reality visualization in image guided surgery. Med Imaging Augmented Reality 6326:334–343CrossRefGoogle Scholar
  25. 25.
    Cutolo F, Freschi C, Mascioli S, Parchi P, Ferrari M, Ferrari V (2016) Robust and accurate algorithm for wearable stereoscopic augmented reality with three indistinguishable markers. Electronics 5(3):59CrossRefGoogle Scholar
  26. 26.
    Navab N, Heining SM, Traub J (2010) Camera augmented mobile C-Arm (CAMC): calibration, accuracy study, and clinical applications. IEEE T Med Imaging 29(7):1412–1423.  https://doi.org/10.1109/Tmi.2009.2021947 CrossRefGoogle Scholar
  27. 27.
    Marmulla R, Hoppe H, Muhling J, Eggers G (2005) An augmented reality system for image-guided surgery. Int J Oral Maxillofac Surg 34(6):594–596.  https://doi.org/10.1016/j.ijom.2005.05.004 CrossRefPubMedGoogle Scholar
  28. 28.
    Haouchine N, Dequidt J, Berger MO, Cotin S (2013) Deformation-based augmented reality for hepatic surgery. Stud Health Technol Inf 184:182–188Google Scholar
  29. 29.
    Peterhans M, vom Berg A, Dagon B, Inderbitzin D, Baur C, Candinas D, Weber S (2011) A navigation system for open liver surgery: design, workflow and first clinical applications. Int J Med Robot Comput Assist Surg MRCAS 7(1):7–16.  https://doi.org/10.1002/rcs.360 CrossRefGoogle Scholar
  30. 30.
    Rolland JP, Holloway RL, Fuchs H (1994) A comparison of optical and video see-through head-mounted displays. P Soc Photo Opt Ins 2351:293–307Google Scholar
  31. 31.
    Cutolo F, Fontana U, Ferrari V (2018) Perspective Preserving Solution for Quasi-Orthoscopic Video See-Through HMDs. Technologies.  https://doi.org/10.3390/technologies6010009 Google Scholar
  32. 32.
    Ferrari V, Viglialoro RM, Nicoli P, Cutolo F, Condino S, Carbone M, Siesto M, Ferrari M (2016) Augmented reality visualization of deformable tubular structures for surgical simulation. Int J Med Robot Comput Assist Surg MRCAS 12(2):231–240.  https://doi.org/10.1002/rcs.1681 CrossRefGoogle Scholar
  33. 33.
    Viglialoro RM, Condino S, Gesi M, Ferrari M, Ferrari V (2014) Augmented reality simulator for laparoscopic cholecystectomy training. In: De Paolis LT, Mongelli A (eds) Augmented and virtual reality. Springer, Cham, pp 428–433.  https://doi.org/10.1007/978-3-319-13969-2_33 Google Scholar
  34. 34.
    Suenaga H, Tran HH, Liao H, Masamune K, Dohi T, Hoshi K, Takato T (2015) Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study. BMC Med Imaging.  https://doi.org/10.1186/s12880-015-0089-5 PubMedPubMedCentralGoogle Scholar
  35. 35.
    Wang JC, Suenaga H, Yang LJ, Kobayashi E, Sakuma I (2017) Video see-through augmented reality for oral and maxillofacial surgery. Int J Med Robot Comp.  https://doi.org/10.1002/Rcs.1754 Google Scholar
  36. 36.
    Kilgus T, Heim E, Haase S, Prufer S, Muller M, Seitel A, Fangerau M, Wiebe T, Iszatt J, Schlemmer HP, Hornegger J, Yen K, Maier-Hein L (2015) Mobile markerless augmented reality and its application in forensic medicine. Int J Comput Ass Rad 10(5):573–586.  https://doi.org/10.1007/s11548-014-1106-9 Google Scholar

Copyright information

© Italian Society of Surgery (SIC) 2018

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PisaPisaItaly
  2. 2.Department of Translational Research and New Technologies in Medicine and Surgery, EndoCAS CenterUniversity of PisaPisaItaly
  3. 3.General Surgery Unit, Department of Surgery, Translational and New TechnologiesUniversity of PisaPisaItaly
  4. 4.Vascular Surgery UnitCisanello University Hospital AOUPPisaItaly

Personalised recommendations