Abstract
To provide an overview of the novel applications of Augmented Reality (AR) and Virtual Reality (VR) technologies within the realm of Surgery. This chapter will clarify the concepts associated with AR and VR in the context of surgery, and address implementation approaches of AR and VR in surgical pre-operative planning and surgical training. Some examples of state-of-the-art solutions and system architecture developed by the editors will be described and discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
AlZubi S, Islam N, Abbod M (2011) Multiresolution analysis using wavelet, ridgelet, and curvelet transforms for medical image segmentation. Int J Biomed Imaging. https://doi.org/10.1155/2011/136034
Badiali G, Ferrari V, Cutolo F et al (2014) Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J Craniomaxillofac Surg 42: 1970–1976. https://doi.org/10.1016/j.jcms.2014.09.001
Benninger B (2015) Google glass, ultrasound and palpation: the anatomy teacher of the future? Clin Anat 28: 152–155. https://doi.org/10.1002/ca.22480
Borgmann H, Rodriguez Socarras M, Salem J et al (2017) Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World J Urol 35: 967–972. https://doi.org/10.1007/s00345-016-1956-6
Bruckheimer E, Rotschild C (2016) Holography for imaging in structural heart disease. EuroIntervention 12:X81–X84. https://doi.org/10.4244/EIJV12SXA15
Chan F, Aguirre S, Bauser-Heaton H et al (2013) Head tracked stereoscopic pre-surgical evaluation of major Aortopulmonary collateral arteries in the newborns
Chartrand G, Cresson T, Chav R et al (2017) Liver segmentation on CT and MR using Laplacian mesh optimization. IEEE Trans Biomed Eng. https://doi.org/10.1109/TBME.2016.2631139
Chen X, Xu L, Wang Y et al (2015) Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J Biomed Inform 55:124–131. https://doi.org/10.1016/j.jbi.2015.04.003
Cheng I, Shen R, Moreau R et al (2014) An augmented reality framework for optimization of computer assisted navigation in endovascular surgery. Conf Proc. Annual Int Conf IEEE Eng Med Biol Soc Eng Med Biol Soc Conf 2014: 5647–5650. https://doi.org/10.1109/EMBC.2014.6944908
Dakua SP (2013) Performance divergence with data discrepancy: a review. Artif Intell Rev. https://doi.org/10.1007/s10462-011-9289-8
Yang M Der, Chao CF, Huang KS et al (2013) Image-based 3D scene reconstruction and exploration in augmented reality. Autom Constr. https://doi.org/10.1016/j.autcon.2012.09.017
Dickey RM, Srikishen N, Lipshultz LI et al (2016) Augmented reality assisted surgery: a urologic training tool. Asian J Androl 18: 732–734. https://doi.org/10.4103/1008-682X.166436
Docquier P-L, Paul L, TranDuy K (2016) Surgical navigation in paediatric orthopaedics. EFORT Open Rev. https://doi.org/10.1302/2058-5241.1.000009
Du B, Sun Y, Cai S, et al (2018) Object tracking in satellite videos by fusing the Kernel correlation filter and the three-frame-difference algorithm. IEEE Geosci Remote Sens Lett. https://doi.org/10.1109/LGRS.2017.2776899
Du D, Wen L, Qi H et al (2018) Iterative graph seeking for object tracking. IEEE Trans Image Process. https://doi.org/10.1109/TIP.2017.2785626
Duffner S, Garcia C (2017) Fast pixelwise adaptive visual tracking of non-rigid objects. IEEE Trans Image Process. https://doi.org/10.1109/TIP.2017.2676346
El-Hariri H, Pandey P, Hodgson AJ, Garbi R (2018) Augmented reality visualisation for orthopaedic surgical guidance with pre- and intra-operative multimodal image data fusion. Healthc Technol Lett 5:189–193. https://doi.org/10.1049/htl.2018.5061
Engelhardt S, De Simone R, Al-Maisary S et al (2016) Accuracy evaluation of a mitral valve surgery assistance system based on optical tracking. Int J Comput Assist Radiol Surg 11:1891–1904. https://doi.org/10.1007/s11548-016-1353-z
Esser M, Schneeweiss S, Nikolaou K, Horger M (2016) Complexity of ultrasound and CT fusion examinations: are they feasible in the daily routine? Am J Roentgenol. https://doi.org/10.2214/AJR.16.16246
Fida B, Cutolo F, di Franco G et al (2018) Augmented reality in open surgery. Updates Surg. https://doi.org/10.1007/s13304-018-0567-8 [doi]
Gavaghan KA, Peterhans M, Oliveira-Santos T, Weber S (2011) A portable image overlay projection device for computer-aided open liver surgery. IEEE Trans Biomed Eng 58: 1855–1864. https://doi.org/10.1109/TBME.2011.2126572
Grady L (2006) Random walks for image segmentation. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2006.233
Guo YY, He DJ, Liu C (2018) Target tracking and 3D trajectory acquisition of cabbage butterfly (P. rapae) based on the KCF-BS algorithm. Sci Rep. https://doi.org/10.1038/s41598-018-27520-z
Ha H-G, Hong J (2016) Augmented reality in medicine. Hanyang Med Rev 36:242. https://doi.org/10.7599/hmr.2016.36.4.242
Halabi O, Halwani Y (2018) Design and implementation of haptic virtual fixtures for preoperative surgical planning. Displays 54:9–19. https://doi.org/10.1016/j.displa.2018.07.004
Hamacher A, Kim SJ, Cho ST et al (2016) Application of virtual, augmented, and mixed reality to urology. Int Neurourol J 20: 172–181. https://doi.org/10.5213/inj.1632714.357
Hanna MG, Ahmed I, Nine J et al (2018) Augmented reality technology using microsoft HoloLens in anatomic pathology. Arch Pathol Lab Med 142:638–644. https://doi.org/10.5858/arpa.2017-0189-OA
Heimann T, Van Ginneken B, Styner MA et al (2009) Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2009.2013851
Heine C (2016) Pokémon go is inspiring small retailers. So has augmented reality gone mainstream? Adweek
Hooten KG, Lister JR, Lombard G et al (2014) Mixed reality ventriculostomy simulation: experience in neurosurgical residency. Neurosurgery 10(Suppl 4): 576–581; discussion 581. https://doi.org/10.1227/NEU.0000000000000503
Jeon Y, Choi S, Kim H (2014) Evaluation of a simplified augmented reality device for ultrasound-guided vascular access in a vascular phantom. J Clin Anesth 26: 485–489. https://doi.org/10.1016/j.jclinane.2014.02.010
Jeon S, Hwangbo S, Hong J (2016) A surgical navigation system to assist in chronic total occlusion intervention. In: 2016 13th international conference on ubiquitous robots and ambient intelligence. URAI
Kadoury S, Fecamp V, Gerard M, Soulez G (2016) Fiducial-less fusion imaging of 3D ultrasound with prior CT/MRI in liver interventions. J Vasc Interv Radiol. https://doi.org/10.1016/j.jvir.2015.12.080
Kennedy DN, Filipek PA, Caviness VS (1989) Anatomic segmentation and volumetric calculations in nuclear magnetic resonance imaging. IEEE Trans Med Imaging
Kim BG, Park DJ (2004) Unsupervised video object segmentation and tracking based on new edge features. Pattern Recognit Lett. https://doi.org/10.1016/j.patrec.2004.07.009
Kirbas C, Quek F (2005) A review of vessel extraction techniques and algorithms. ACM Comput Surv. https://doi.org/10.1145/1031120.1031121
Kobayashi L, Zhang XC, Collins SA et al (2018) Exploratory application of augmented reality/mixed reality devices for acute care procedure training. West J Emerg Med 19: 158–164. https://doi.org/10.5811/westjem.2017.10.35026
Lee MW (2014) Fusion imaging of real-time ultrasonography with CT or MRI for hepatic intervention. Ultrasonography. https://doi.org/10.14366/usg.14021
Li Y, Liang W, An H, et al (2017a) C-arm based image-guided percutaneous puncture of minimally invasive spine surgery
Li Z, Cui L, Wang J, Li S (2017b) Fusion of real-time us with CT images to guide selective lumbosacral nerve root interventional therapy. Ultrasound Med Biol. https://doi.org/10.1016/j.ultrasmedbio.2017.08.1624
Li J, Zhou X, Chan S, Chen S (2017c) Robust object tracking via large margin and scale-adaptive correlation filter. IEEE Access. https://doi.org/10.1109/ACCESS.2017.2778740
Liao H, Inomata T, Sakuma I, Dohi T (2010) 3-D augmented reality for MRI-guided surgery using integral videography autostereoscopic image overlay. IEEE Trans Biomed Eng 57:1476–1486. https://doi.org/10.1109/TBME.2010.2040278
Linguraru MG, Richbourg WJ, Liu J et al (2012) Tumor burden analysis on computed tomography by automated liver and tumor segmentation. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2012.2211887
Liu S, Feng Y (2018) Real-time fast moving object tracking in severely degraded videos captured by unmanned aerial vehicle. Int J Adv Robot Syst. https://doi.org/10.1177/1729881418759108
Liu TJ, Ko AT, Tang YB et al (2016) Clinical application of different surgical navigation systems in complex craniomaxillofacial surgery: the use of multisurface 3-dimensional images and a 2-plane reference system. Ann Plast Surg 76:411–419. https://doi.org/10.1097/sap.0000000000000429
Liu G, Liu S, Muhammad K et al (2018) Object tracking in vary lighting conditions for fog based intelligent surveillance of public spaces. IEEE Access. https://doi.org/10.1109/ACCESS.2018.2834916
Magic Leap Magic Leap One (2005). https://www.magicleap.com. Accessed 20 Sept 2005
Mahalingam T, Subramoniam M (2018) A robust single and multiple moving object detection, tracking and classification. Appl Comput Inform. https://doi.org/10.1016/j.aci.2018.01.001
Mahr A, Levegrün S, Bahner ML et al (1999) Usability of semiautomatic segmentation algorithms for tumor volume determination. Invest Radiol. https://doi.org/10.1097/00004424-199902000-00007
Mat Isham MI, Mohamed F, Siang CV, et al (2019) A framework of ultrasounds image slice positioning and orientation in 3D augmented reality environment using hybrid tracking method. In: 2018 IEEE Conference on Big Data Analysis ICBDA 2018, pp. 105–110. https://doi.org/10.1109/ICBDAA.2018.8629589
Mauri G, Cova L, De Beni S et al (2015) Real-time US-CT/MRI image fusion for guidance of thermal ablation of liver tumors undetectable with US: results in 295 cases. Cardiovasc Intervent Radiol. https://doi.org/10.1007/s00270-014-0897-y
Mharib AM, Ramli AR, Mashohor S, Mahmood RB (2012) Survey on liver CT image segmentation methods. Artif Intell Rev
Microsoft HoloLens (2019) HoloLens 2. In: Microsoft HoloLens. https://www.microsoft.com/en-us/hololens. Accessed 20 Sept 2005
Milgram P, Kishimo F (1994) A taxonomy of mixed reality. IEICE Trans Inf Syst 77:1321–1329
Milgram P, Takemura H, Utsumi A, Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. In: Das H (ed). pp 282–292
Monsky WL, James R, Seslar SS (2019) Virtual and augmented reality applications in medicine and surgery-the fantastic voyage is here. Anat Physiol Curr Res 9:1–5. https://doi.org/10.4172/2161-0940.1000313
Navab N, Traub J, Sielhorst T et al (2007) Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Graph Appl 27:10–14. https://doi.org/10.1109/MCG.2007.117
Niehaus R, Schilter D, Fornaciari P et al (2017) Experience of total knee arthroplasty using a novel navigation system within the surgical field. Knee 24:518–524. https://doi.org/10.1016/j.knee.2016.10.021
Nuzillard D, Lazar C (2007) Partitional clustering techniques for multi-spectral image segmentation. J Comput. https://doi.org/10.4304/jcp.2.10.1-8
Ozkan E, Tanner C, Kastelic M et al (2017) Robust motion tracking in liver from 2D ultrasound images using supporters. Int J Comput Assist Radiol Surg 12:941–950. https://doi.org/10.1007/s11548-017-1559-8
Pandya A (2004) Medical augmented reality system for image-guided and robotic surgery: development and surgeon factors analysis. Wayne State University
Parkhomenko E, Safiullah S, Walia S et al (2018) Mp26-20 virtual-reality projected renal models with urolithiasis as an educational and preoperative planning tool for nephrolithotomy: a pilot study. J Urol 199. https://doi.org/10.1016/j.juro.2018.02.878
Patwardhan SV, Dai S, Dhawan AP (2005) Multi-spectral image analysis and classification of melanoma using fuzzy membership based partitions. Comput Med Imaging Graph. https://doi.org/10.1016/j.compmedimag.2004.11.001
Piromchai P, Avery A, Laopaiboon M et al (2015) Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev (9). https://doi.org/10.1002/14651858.CD010198.pub2
Ponce BA, Menendez ME, Oladeji LO et al (2014) Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Orthopedics 37: 751–757. https://doi.org/10.3928/01477447-20141023-05
Pratt P, Ives M, Lawton G et al (2018) Through the HoloLens looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. Eur Radiol Exp 2. https://doi.org/10.1186/s41747-017-0033-2
Pruliere-Escabasse V, Coste A (2010) Image-guided sinus surgery. Eur Ann Otorhinolaryngol Head Neck Dis 127:33–39. https://doi.org/10.1016/j.anorl.2010.02.009
Raconteur Reports (2018) XR For business
Ruan S, Moretti B, Fadili J, Bloyet D (2002) Fuzzy Markovian segmentation in application of magnetic resonance images. Comput Vis Image Underst. https://doi.org/10.1006/cviu.2002.0957
Sato Y, Nakajima S, Shiraga N et al (1998) Three-dimensional multi-scale line filter for segmentation and visualization of curvilinear structures in medical images. Med Image Anal. https://doi.org/10.1016/S1361-8415(98)80009-1
Sauer IM, Queisner M, Tang P et al (2017) Mixed reality in visceral surgery: development of a suitable workflow and evaluation of intraoperative use-cases. Ann Surg 266: 706–712. https://doi.org/10.1097/SLA.0000000000002448
Seo KS (2005) Automatic hepatic tumor segmentation using composite hypotheses. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Seror O (2017) Novel guidance for percutaneous treatment of hepatic tumors: fusion images and cone-beam CT. Ultrasound Med Biol. https://doi.org/10.1016/j.ultrasmedbio.2017.08.1473
Shang Y, Deklerck R, Nyssen E et al (2011) Vascular active contour for vessel tree segmentation. IEEE Trans Biomed Eng. https://doi.org/10.1109/TBME.2010.2097596
Silva JJNA, Southworth M, Raptis C, Silva JJNA (2018) Emerging applications of virtual reality in cardiovascular medicine. JACC Basic Transl Sci 3:420–430. https://doi.org/10.1016/j.jacbts.2017.11.009
Subudhi BN, Nanda PK, Ghosh A (2011) A change information based fast algorithm for video object detection and tracking. IEEE Trans Circ Syst Video Technol. https://doi.org/10.1109/TCSVT.2011.2133870
Sugano N (2003) Computer-assisted orthopedic surgery. J Orthop Sci 8:442–448. https://doi.org/10.1007/s10776-002-0623-6
Tadayyon H, Lasso A, Kaushal A et al (2011) Target motion tracking in MRI-guided transrectal robotic prostate biopsy. IEEE Trans Biomed Eng 58:3135–3142. https://doi.org/10.1109/tbme.2011.2163633
Venkatesan AM, Kadoury S, Abi-Jaoudeh N et al (2011) Real-time FDG PET guidance during biopsies and radiofrequency ablation using multimodality fusion with electromagnetic navigation. Radiology. https://doi.org/10.1148/radiol.11101985
Volonté F, Pugin F, Bucher P et al (2011) Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. J Hepatobiliary Pancreat Sci 18:506–509. https://doi.org/10.1007/s00534-011-0385-6
Wein W, Brunke S, Khamene A et al (2008) Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. Med Image Anal. https://doi.org/10.1016/j.media.2008.06.006
Wein W, Ladikos A, Fuerst B et al (2013) Global registration of ultrasound to MRI using the LC2 metric for enabling neurosurgical guidance. In: Lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Yim PJ, Foran DJ (2003) Volumetry of hepatic metastases in computed tomography using the watershed and active contour algorithms
Zhang X, Tian J, Deng K et al (2010) Automatic liver segmentation using a statistical shape model with optimal surface detection. IEEE Trans Biomed Eng. https://doi.org/10.1109/TBME.2010.2056369
Zhang Q, Fan Y, Wan J, Liu Y (2017) An efficient and clinical-oriented 3D liver segmentation method. IEEE Access. https://doi.org/10.1109/ACCESS.2017.2754298
Zhang X, Chen G, Liao H (2017b) High-quality see-through surgical guidance system using enhanced 3-D autostereoscopic augmented reality. IEEE Trans Biomed Eng 64:1815–1825. https://doi.org/10.1109/TBME.2016.2624632
Zhang T, Liu S, Xu C et al (2018) Correlation particle filter for visual tracking. IEEE Trans Image Process. https://doi.org/10.1109/TIP.2017.2781304
Zhao B, Schwartz LH, Jiang L et al (2006) Shape-constraint region growing for delineation of hepatic metastases on contrast-enhanced computed tomograph scans. Invest Radiol. https://doi.org/10.1097/01.rli.0000236907.81400.18
Acknowledgements
This work was partly supported by NPRP Grant #NPRP 11S-1219-170106 from the Qatar National Research Fund (a member of Qatar Foundation).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Halabi, O., Balakrishnan, S., Dakua, S.P., Navab, N., Warfa, M. (2020). Virtual and Augmented Reality in Surgery. In: Doorsamy, W., Paul, B., Marwala, T. (eds) The Disruptive Fourth Industrial Revolution. Lecture Notes in Electrical Engineering, vol 674. Springer, Cham. https://doi.org/10.1007/978-3-030-48230-5_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-48230-5_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-48229-9
Online ISBN: 978-3-030-48230-5
eBook Packages: Computer ScienceComputer Science (R0)