Skip to main content

Applying Modern Virtual and Augmented Reality Technologies to Medical Images and Models

Journal of Digital Imaging volume 32, pages 38–53 (2019)Cite this article

Abstract

Recent technological innovations have created new opportunities for the increased adoption of virtual reality (VR) and augmented reality (AR) applications in medicine. While medical applications of VR have historically seen greater adoption from patient-as-user applications, the new era of VR/AR technology has created the conditions for wider adoption of clinician-as-user applications. Historically, adoption to clinical use has been limited in part by the ability of the technology to achieve a sufficient quality of experience. This article reviews the definitions of virtual and augmented reality and briefly covers the history of their development. Currently available options for consumer-level virtual and augmented reality systems are presented, along with a discussion of technical considerations for their adoption in the clinical environment. Finally, a brief review of the literature of medical VR/AR applications is presented prior to introducing a comprehensive conceptual framework for the viewing and manipulation of medical images in virtual and augmented reality. Using this framework, we outline considerations for placing these methods directly into a radiology-based workflow and show how it can be applied to a variety of clinical scenarios.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Sutherland J, La Russa D: Virtual reality. In: Rybicki FJ, Grant GT Eds. 3D Printing in Medicine: A Practical Guide for Medical Professionals. Cham: Springer International Publishing, 2017, pp. 125–133

    Chapter  Google Scholar 

  2. Satava RM: Virtual reality surgical simulator. Surg Endosc 7:203–205, 1993

    CAS  Article  Google Scholar 

  3. Phillips JR: Virtual reality: A new vista for nurse researchers? Nurs Sci Q 6:5–7, 1993

    CAS  Article  Google Scholar 

  4. Chinnock C: Virtual reality in surgery and medicine. Hosp Technol Ser 13:1–48, 1994

    CAS  PubMed  Google Scholar 

  5. Rosen JM, Soltanian H, Redett RJ, Laub DR: Evolution of virtual reality. IEEE Eng Med Biol Mag 15:16–22, 1996

    Article  Google Scholar 

  6. Pensieri C, Pennacchini M, Sivan Y, Gefen D, Boulos MK, Claudio P, Maddalena P: Overview: Virtual reality in medicine. J Virtual Worlds Res 7:1–34, 2014

    Article  Google Scholar 

  7. Worldwide Spending on Augmented and Virtual Reality. Available at https://www.idc.com/getdoc.jsp?containerId=prUS42959717. Accessed January 2018.

  8. Healthcare Augmented & Virtual Reality Market Worth $5.1 Billion By 2025. Available at https://www.grandviewresearch.com/press-release/global-augmented-reality-ar-virtual-reality-vr-in-healthcare-market. Accessed January 2018.

  9. Rybicki FJ, Grant GT: 3D Printing in Medicine. Cham: Springer International Publishing, 2017

    Book  Google Scholar 

  10. Burdea GC, Coiffet P: Virtual Reality Technology. New York: Wiley, 2003

    Book  Google Scholar 

  11. Heilig ML: Stereoscopic-television apparatus for individual use. US Patent 2:955–156, 1960

    Google Scholar 

  12. Linte CA, Davenport KP, Cleary K, Peters C, Vosburgh KG, Navab N, Eddie P, Jannin P, Peters TM, Holmes DR, Robb RA: On mixed reality environments for minimally invasive therapy guidance: Systems architecture, successes and challenges in their implementation from laboratory to clinic. Comput Med Imaging Graph 37:83–97, 2013

    Article  Google Scholar 

  13. Yaniv Z, Linte CA: Applications of augmented reality in the operating room. Fundam Wearable Comput Augment Real 485–510, 2016

  14. Moro C, Štromberga Z, Raikos A, Stirling A: The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ 1–11, 2017

  15. Plasencia DM: One step beyond virtual reality: Connecting past and future developments. Crossroads ACM Mag Stud 22:18–23, 2015

    Article  Google Scholar 

  16. Abrash M: Why You Won’t See Hard AR Anytime Soon. Available at http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-ar-anytime-soon/. Accessed May 2017

  17. Milgram P, Takemura H, Utsumi A, Kishino F: Augmented reality: A class of display on the reality-virtuality continuum. Proc SPIE 2351:282–292, 1995

    Article  Google Scholar 

  18. Milgram P, Kishino F: A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77:1–15, 1994

    Google Scholar 

  19. Foxlin E, Harrington M, Pfeifer G: Constellation: A Wide-Range Wireless Motion-tracking System for Augmented Reality and Virtual Set Applications. Proc. 25th Annu. Conf. Comput. Graph. Interact. Tech. - SIGGRAPH ‘98 98, pp. 371–378, 1998

  20. Valve: SteamVR Tracking. Available at https://partner.steamgames.com/vrtracking. Accessed May 2017

  21. Valve Reveals Timeline of Vive Prototypes, We Chart it For You. Available at http://www.roadtovr.com/valve-reveals-timeline-of-vive-prototypes-we-chart-it-for-you/. Accessed May 2017

  22. Tazartes D: An historical perspective on inertial navigation systems. In: Proc. Int. Symp. Inertial Sensors Syst, pp. 1–5, 2014

  23. Riva G, Wiederhold BK: Gaggi: Being different: The transformative potential of virtual reality. Annu Rev Cybern Therapy Telemed 14:3–6, 2016

    Google Scholar 

  24. Tepper OM, Rudy HL, Lefkowitz A, Weimer KA, Marks SM, Stern CS, Garfein ES: Mixed reality with HoloLens: Where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg 140:1066–1070, 2017

    CAS  Article  Google Scholar 

  25. Wang S, Parsons M, Stone-McLean J, Rogers P, Boyd S, Hoover K, Meruvia-pastor O, Gong M, Smith A: Augmented reality as a telemedicine platform for remote procedural training. Sensors 17:1–21, 2017

    CAS  Article  Google Scholar 

  26. Lee N: LG’s SteamVR headset is a bulky yet promising HTC Vive alternative. Available at https://www.engadget.com/2017/03/02/lg-steamvr-headset/. Accessed May 2017

  27. Bharathan R, Vali S, Setchell T, Miskry T, Darzi A, Aggarwal R: Psychomotor skills and cognitive load training on a virtual reality laparoscopic simulator for tubal surgery is effective. Eur J Obstet Gynecol Reprod Biol 169:347–352, 2013

    Article  Google Scholar 

  28. 3D Systems Extends Leadership in Precision Healthcare with Technology Advancements to Quickly Create 3D Models. Available at https://www.3dsystems.com/press-releases/3d-systems-extends-leadership-precision-healthcare-technology-advancements-quickly. Accessed April 2018

  29. 3D Systems Leverages Virtual Reality to Advance Surgical Training. Available at https://www.3dsystems.com/press-releases/3d-systems-leverages-virtual-reality-advance-surgical-training. Accessed April 2018

  30. Interactive Virtual Reality. Available at http://www.echopixeltech.com/interactive-virtual-reality/. Accessed April 2018

  31. TeraRecon Launches Augmented Reality 3-D Imaging at HIMSS17. Available at https://www.itnonline.com/content/terarecon-launches-augmented-reality-3-d-imaging-himss17. Accessed April 2018

  32. Rebenitsch L, Owen C: Review on cybersickness in applications and visual displays. Virtual Real 20:101–125, 2016

    Article  Google Scholar 

  33. Rebenitsch L: Managing cybersickness in virtual reality. Crossroads ACM Mag Stud 22:46–51, 2015

    Article  Google Scholar 

  34. Satava RM, Member A, Jones SB: Current and future applications of virtual reality for medicine. Proc IEEE 86:484–489, 1998

    Article  Google Scholar 

  35. Cates CU, Lönn L, Gallagher AG: Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Simul Technol Enhanc Learn 2:1–5, 2016

    Article  Google Scholar 

  36. Ali S, Qandeel M, Ramakrishna R, Yang CW: Virtual simulation in enhancing procedural training for fluoroscopy-guided lumbar puncture: A Pilot Study. Acad Radiol 25:235–239, 2018

    Article  Google Scholar 

  37. Ramlogan R, Niazi AU, Jin R, Johnson J, Chan VW, Perlas A: A virtual reality simulation model of spinal ultrasound: Role in teaching spinal sonoanatomy. Reg Anesth Pain Med 42:217–222, 2017

    Article  Google Scholar 

  38. Azagury DE, Ryou M, Shaikh SN, San José Estépar R, Lengyel BI, Jagadeesan J, Vosburgh KG, Thompson CC: Real-time computed tomography-based augmented reality for natural orifice transluminal endoscopic surgery navigation. Br J Surg 99:1246–1253, 2012

    CAS  Article  Google Scholar 

  39. Johnston APR, Rae J, Ariotti N, Bailey B, Lija A, Webb R, Ferguson C, Maher S, Davis TP, Webb RI, Mcghee J, Parton RG: Journey to the centre of the cell: Virtual reality immersion into scientific data. Traffic 105–110, 2017.

  40. Shanahan M: Use of a virtual radiography simulation enhances student learning. J Med Radiat Sci 63:27, 2016

    Google Scholar 

  41. Syed AZ, Zakaria A, Lozanoff S: Dark room to augmented reality: Application of HoloLens technology for oral radiological diagnosis. Oral Surg Oral Med Oral Pathol Oral Radiol 124:e33, 2017

    Article  Google Scholar 

  42. Mitsouras D, Liacouras P, Imanzadeh A, Giannopoulos AA, Cai T, Kumamaru KK, George E, Wake N, Caterson EJ, Pomahac B, Ho VB, Grant GT, Rybicki FJ: Medical 3D printing for the radiologist. RadioGraphics 35:1965–1988, 2015

    Article  Google Scholar 

  43. Giannopoulos AA, Mitsouras D, Yoo S-J, Liu PP, Chatzizisis YS, Rybicki FJ: Applications of 3D printing in cardiovascular diseases. Nat Rev Cardiol 13:701–718, 2016

    CAS  Article  Google Scholar 

  44. Christensen A, Rybicki FJ: Maintaining safety and efficacy for 3D printing in medicine. 3D Print Med 3:1, 2017

    Article  Google Scholar 

  45. Di Prima M, Coburn J, Hwang D, Kelly J, Khairuzzaman A, Ricles L: Additively manufactured medical products—The FDA perspective. 3D Print Med 2(1), 2015

  46. Rybicki FJ: Medical 3D printing and the physician-artist. Lancet 391:651–652, 2018

    Article  Google Scholar 

  47. Rybicki FJ: Message from Frank J. Rybicki, MD, incoming chair of ACR appropriateness criteria. J Am Coll Radiol 14:723–724, 2017

    Article  Google Scholar 

  48. Douglas DB, Wilke CA, Gibson D, Petricoin EF, Liotta L: Virtual reality and augmented reality: Advances in surgery. Biol Eng Med 2:1–8, 2017

    Google Scholar 

  49. Gallagher K, Jain S, Okhravi N: Making and viewing stereoscopic surgical videos with smartphones and virtual reality headset. Eye 30:503–504, 2016

    CAS  Article  Google Scholar 

  50. Huber T, Wunderling T, Paschold M, Lang H, Kneist W, Hansen C: Highly immersive virtual reality laparoscopy simulation: Development and future aspects. Int J Comput Assist Radiol Surg 13:281–290, 2018

    Article  Google Scholar 

  51. Kersten-Oertel M, Gerard I, Drouin S, Mok K, Sirhan D, Sinclair DS, Collins DL: Augmented reality in neurovascular surgery: Feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg 10:1823–1836, 2015

    Article  Google Scholar 

  52. Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A: A systematic review of virtual reality simulators for robot-assisted surgery. Eur Urol 69:1065–1080, 2016

    Article  Google Scholar 

  53. Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, Navab N, Fallavollita P: Intra-operative augmented reality in distal locking. Int J Comput Assist Radiol Surg 10:1395–1403, 2015

    Article  Google Scholar 

  54. King F, Jayender J, Bhagavatula SK, Shyn PB, Pieper S, Kapur T, Lasso A, Fichtinger G: An immersive virtual reality environment for diagnostic imaging. J Med Robot Res 01:1640003, 2016

    Article  Google Scholar 

  55. Douglas DB, Petricoin EF, Liotta L, Wilson E: D3D augmented reality imaging system: Proof of concept in mammography. Med Devices Evid Res 9:277–283, 2016

    Article  Google Scholar 

  56. Hanna MG, Ahmed I, Nine J, Prajapati S, Pantanowitz L: Augmented reality technology using Microsoft HoloLens in anatomic pathology. Arch Pathol Lab Med, 2018. https://doi.org/10.5858/arpa.2017-0189-OA

  57. Calì C, Baghabra J, Boges DJ, Holst GR, Kreshuk A, Hamprecht FA, Srinivasan M, Lehväslaiho H, Magistretti PJ: Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. J Comp Neurol 524:23–38, 2016

    Article  Google Scholar 

  58. Zheng LL, He L, Yu CQ: Mobile virtual reality for ophthalmic image display and diagnosis. J Mob Technol Med 4:35–38, 2015

    Article  Google Scholar 

  59. Dascal J, Reid M, Ishak WW, Spiegel B, Recacho J, Rosen B, Danovitch I: Virtual reality and medical inpatients: A systematic review of randomized, controlled trials. Innov Clin Neurosci 14:14–21, 2017

    PubMed  PubMed Central  Google Scholar 

  60. Schmitt YS, Hoffman HG, Blough DK, David R, Jensen MP, Soltani M, Carrougher GJ, Mn RN, Nakamura D, Otr L, Sharar SR: A randomized, controlled trial of immersive virtual reality analgesia during physical therapy for pediatric burn injuries. Burns 37:61–68, 2011

    Article  Google Scholar 

  61. Carrougher GJ, Hoffman HG, Nakamura D, Lezotte D, Soltani M, Leahy L, Engrav LH, Patterson DR: The effect of virtual reality on pain and range of motion in adults with burn injuries. J Burn Care Res 30:785–791, 2009

    Article  Google Scholar 

  62. Kipping B, Rodger S, Miller K, Kimble RM: Virtual reality for acute pain reduction in adolescents undergoing burn wound care: A prospective randomized controlled trial. Burns 38:650–657, 2012

    Article  Google Scholar 

  63. Morris LD, Louw QA, Crous LC: Feasibility and potential effect of a low-cost virtual reality system on reducing pain and anxiety in adult burn injury patients during physiotherapy in a developing country. Burns 36:659–664, 2010

    Article  Google Scholar 

  64. Hoffman HG, Patterson DR, Seibel E, Soltani M, Jewett-Leahy L, Sharar SR: Virtual reality pain control during burn wound debridement in the hydrotank. Clin J Pain 24:299–304, 2008

    Article  Google Scholar 

  65. Patterson DR, Jensen MP, Wiechman SA, Sharar SR: Virtual reality hypnosis for pain associated with recovery from physical trauma. Int J Clin Exp Hypn 58:288–300, 2010

    Article  Google Scholar 

  66. Li WH, Chung JO, Ho EK: The effectiveness of therapeutic play, using virtual reality computer games, in promoting the psychological well-being of children hospitalised with cancer. J Clin Nurs 20:2135–2143, 2011

    Article  Google Scholar 

  67. Cesa GL, Manzoni GM, Bacchetta M, Castelnuovo G, Conti S, Gaggioli A, Mantovani F, Molinari E, Cárdenas-López G, Riva G: Virtual reality for enhancing the cognitive behavioral treatment of obesity with binge eating disorder: Randomized controlled study with one-year follow-up. J Med Internet Res 15:1–13, 2013

    Article  Google Scholar 

  68. Manzoni GM, Pagnini F, Gorini A, Preziosa A, Castelnuovo G, Molinari E, Riva G: Can relaxation training reduce emotional eating in women with obesity? An exploratory study with 3 months of follow-up. J Am Diet Assoc 109:1427–1432, 2009

    Article  Google Scholar 

  69. Larson EB, Ramaiya M, Zollman FS, Pacini S, Hsu N, Patton JL, Dvorkin AY: Tolerance of a virtual reality intervention for attention remediation in persons with severe TBI. Brain Inj 25:274–281, 2011

    Article  Google Scholar 

  70. Saposnik G, Cohen LG, Mamdani M, Pooyania S, Ploughman M, Cheung D, Shaw J, Hall J, Nord P, Dukelow S, Nilanont Y, De los Rios F, Olmos L, Levin M, Teasell R, Cohen A, Thorpe K, Laupacis A, Bayley M: Efficacy and safety of non-immersive virtual reality exercising in stroke rehabilitation (EVREST): A randomised, multicentre, single-blind, controlled trial. Lancet Neurol 15:1019–1027, 2016

    Article  Google Scholar 

  71. Silver B: Virtual reality versus reality in post-stroke rehabilitation. Lancet Neurol 15:996–997, 2016

    Article  Google Scholar 

  72. Ambron E, Miller A, Kuchenbecker KJ, Laurel B, Coslett HB: Immersive low-cost virtual reality treatment for phantom limb pain: Evidence from two cases. Front Neurol 9:1–7, 2018

    Article  Google Scholar 

  73. Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, Slater M: Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med 47:2393–2400, 2017

    CAS  Article  Google Scholar 

  74. Maples-Keller JL, Price M, Rauch S, Gerardi M, Rothbaum BO: Investigating relationships between PTSD symptom clusters within virtual reality exposure therapy for OEF/OIF veterans. Behav Ther 48:147–155, 2017

    Article  Google Scholar 

  75. Miloff A, Lindner P, Hamilton W, Reuterskiöld L, Andersson G, Carlbring P: Single-session gamified virtual reality exposure therapy for spider phobia vs. traditional exposure therapy: Study protocol for a randomized controlled non-inferiority trial. Trials 17:1–8, 2016

    Article  Google Scholar 

  76. Nararro-Haro MV, Hoffman HG, Garcia-Palacios A, Sampaio M, Alhalabi W, Hall K, Linehan M: The use of virtual reality to facilitate mindfulness skills training in dialectical behavioral therapy for borderline personality disorder: A case study. Front Psychol 7:1–9, 2016

    Article  Google Scholar 

  77. Wiederhold BK, Wiederhold MD: Virtual Reality Therapy for Anxiety Disorders: Advances in Evaluation and Treatment. Washington, DC: American Psychological Association, 2005

    Book  Google Scholar 

  78. Chepelev L, Giannopoulos A, Tang A, Mitsouras D, Rybicki FJ: Medical 3D printing: Methods to standardize terminology and report trends. 3D Print Med 3:1–9, 2017

    Article  Google Scholar 

  79. Rybicki FJ: 3D printing in medicine: An introductory message from the editor-in-chief. 3D Print Med 1(1):1, 2015

    Article  Google Scholar 

  80. Mitsouras D, Liacouras PC: 3D printing in medicine: 3D printing technologies. In: Rybicki FJ, Grant GT Eds. 3D Printing in Medicine: A Practical Guide for Medical Professionals. Cham: Springer International Publishing, 2017, pp. 5–22

    Chapter  Google Scholar 

  81. American Association of Physicists in Medicine: Assessment of display performance for medical imaging systems. Med Phys:1–156, 2005

  82. Norweck JT, Seibert JA, Andriole KP, Clunie DA, Curran BH, Flynn MJ, Krupinski E, Lieto RP, Peck DJ, Mian TA, Wyatt M: ACR-AAPM-SIIM technical standard for electronic practice of medical imaging. J Digit Imaging 26:38–52, 2013

    Article  Google Scholar 

  83. Drebin R A, Carpenter L, Hanrahan P: Volume rendering. SIGGRAPH ’88. In: Proc. 15th Annu. Conf. Comput. Graph. Interact. Tech. 22, pp. 65–74, 1988

  84. Levoy M: Volume rendering: Display of surface from volume data. IEEE Comput Graph Appl 8:29–37, 1988

    Article  Google Scholar 

  85. Stegmaiert S, Kieint T, Stegmaier S, Strengert M, Klein T, Ertl T: A simple and flexible volume rendering framework for graphics-hardware-based raycasting. Fourth Int. Work. Vol. Graph, pp. 187–241, 2005

  86. Ljung P, Krüger J, Groller E, Hadwiger M, Hansen CD, Ynnerman A, Gröller E, Hadwiger M, Hansen CD, Ynnerman A: State of the art in transfer functions for direct volume rendering. Comput Graph Forum 35:669–691, 2016

    Article  Google Scholar 

  87. Arens S, Domik G: A survey of transfer functions suitable for volume rendering. IEEE/EG Int. Symp. Vol. Graph., pp. 77–83, 2010

  88. Krüger J, Westermann R: Acceleration techniques for GPU-based volume rendering. IEEE Vis, pp. 287–292, 2003.

  89. Vizua3D: https://vizua3d.com. Available at https://vizua3d.com. Accessed February 2018

Download references

Author information

Affiliations

  1. Department of Radiology, University of Ottawa, 501 Smyth Road, Box 232, Ottawa, ON, K1H 8L6, Canada

    Justin Sutherland, Jason Belec, Adnan Sheikh, Leonid Chepelev, Waleed Althobaity, Benjamin J. W. Chow, Dimitrios Mitsouras, Andy Christensen, Frank J. Rybicki & Daniel J. La Russa

Authors
  1. Justin Sutherland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jason Belec
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adnan Sheikh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leonid Chepelev
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Waleed Althobaity
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Benjamin J. W. Chow
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dimitrios Mitsouras
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Andy Christensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Frank J. Rybicki
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Daniel J. La Russa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Justin Sutherland.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sutherland, J., Belec, J., Sheikh, A. et al. Applying Modern Virtual and Augmented Reality Technologies to Medical Images and Models. J Digit Imaging 32, 38–53 (2019). https://doi.org/10.1007/s10278-018-0122-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-018-0122-7

Keywords

  • Virtual reality
  • Augmented reality
  • Radiology
  • Visualization