Journal of Digital Imaging

, Volume 32, Issue 1, pp 38–53 | Cite as

Applying Modern Virtual and Augmented Reality Technologies to Medical Images and Models

  • Justin SutherlandEmail author
  • Jason Belec
  • Adnan Sheikh
  • Leonid Chepelev
  • Waleed Althobaity
  • Benjamin J. W. Chow
  • Dimitrios Mitsouras
  • Andy Christensen
  • Frank J. Rybicki
  • Daniel J. La Russa


Recent technological innovations have created new opportunities for the increased adoption of virtual reality (VR) and augmented reality (AR) applications in medicine. While medical applications of VR have historically seen greater adoption from patient-as-user applications, the new era of VR/AR technology has created the conditions for wider adoption of clinician-as-user applications. Historically, adoption to clinical use has been limited in part by the ability of the technology to achieve a sufficient quality of experience. This article reviews the definitions of virtual and augmented reality and briefly covers the history of their development. Currently available options for consumer-level virtual and augmented reality systems are presented, along with a discussion of technical considerations for their adoption in the clinical environment. Finally, a brief review of the literature of medical VR/AR applications is presented prior to introducing a comprehensive conceptual framework for the viewing and manipulation of medical images in virtual and augmented reality. Using this framework, we outline considerations for placing these methods directly into a radiology-based workflow and show how it can be applied to a variety of clinical scenarios.


Virtual reality Augmented reality Radiology Visualization 


  1. 1.
    Sutherland J, La Russa D: Virtual reality. In: Rybicki FJ, Grant GT Eds. 3D Printing in Medicine: A Practical Guide for Medical Professionals. Cham: Springer International Publishing, 2017, pp. 125–133CrossRefGoogle Scholar
  2. 2.
    Satava RM: Virtual reality surgical simulator. Surg Endosc 7:203–205, 1993CrossRefGoogle Scholar
  3. 3.
    Phillips JR: Virtual reality: A new vista for nurse researchers? Nurs Sci Q 6:5–7, 1993CrossRefGoogle Scholar
  4. 4.
    Chinnock C: Virtual reality in surgery and medicine. Hosp Technol Ser 13:1–48, 1994PubMedGoogle Scholar
  5. 5.
    Rosen JM, Soltanian H, Redett RJ, Laub DR: Evolution of virtual reality. IEEE Eng Med Biol Mag 15:16–22, 1996CrossRefGoogle Scholar
  6. 6.
    Pensieri C, Pennacchini M, Sivan Y, Gefen D, Boulos MK, Claudio P, Maddalena P: Overview: Virtual reality in medicine. J Virtual Worlds Res 7:1–34, 2014CrossRefGoogle Scholar
  7. 7.
    Worldwide Spending on Augmented and Virtual Reality. Available at Accessed January 2018.
  8. 8.
    Healthcare Augmented & Virtual Reality Market Worth $5.1 Billion By 2025. Available at Accessed January 2018.
  9. 9.
    Rybicki FJ, Grant GT: 3D Printing in Medicine. Cham: Springer International Publishing, 2017CrossRefGoogle Scholar
  10. 10.
    Burdea GC, Coiffet P: Virtual Reality Technology. New York: Wiley, 2003CrossRefGoogle Scholar
  11. 11.
    Heilig ML: Stereoscopic-television apparatus for individual use. US Patent 2:955–156, 1960Google Scholar
  12. 12.
    Linte CA, Davenport KP, Cleary K, Peters C, Vosburgh KG, Navab N, Eddie P, Jannin P, Peters TM, Holmes DR, Robb RA: On mixed reality environments for minimally invasive therapy guidance: Systems architecture, successes and challenges in their implementation from laboratory to clinic. Comput Med Imaging Graph 37:83–97, 2013CrossRefGoogle Scholar
  13. 13.
    Yaniv Z, Linte CA: Applications of augmented reality in the operating room. Fundam Wearable Comput Augment Real 485–510, 2016Google Scholar
  14. 14.
    Moro C, Štromberga Z, Raikos A, Stirling A: The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ 1–11, 2017Google Scholar
  15. 15.
    Plasencia DM: One step beyond virtual reality: Connecting past and future developments. Crossroads ACM Mag Stud 22:18–23, 2015CrossRefGoogle Scholar
  16. 16.
    Abrash M: Why You Won’t See Hard AR Anytime Soon. Available at Accessed May 2017
  17. 17.
    Milgram P, Takemura H, Utsumi A, Kishino F: Augmented reality: A class of display on the reality-virtuality continuum. Proc SPIE 2351:282–292, 1995CrossRefGoogle Scholar
  18. 18.
    Milgram P, Kishino F: A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77:1–15, 1994Google Scholar
  19. 19.
    Foxlin E, Harrington M, Pfeifer G: Constellation: A Wide-Range Wireless Motion-tracking System for Augmented Reality and Virtual Set Applications. Proc. 25th Annu. Conf. Comput. Graph. Interact. Tech. - SIGGRAPH ‘98 98, pp. 371–378, 1998Google Scholar
  20. 20.
    Valve: SteamVR Tracking. Available at Accessed May 2017
  21. 21.
    Valve Reveals Timeline of Vive Prototypes, We Chart it For You. Available at Accessed May 2017
  22. 22.
    Tazartes D: An historical perspective on inertial navigation systems. In: Proc. Int. Symp. Inertial Sensors Syst, pp. 1–5, 2014Google Scholar
  23. 23.
    Riva G, Wiederhold BK: Gaggi: Being different: The transformative potential of virtual reality. Annu Rev Cybern Therapy Telemed 14:3–6, 2016Google Scholar
  24. 24.
    Tepper OM, Rudy HL, Lefkowitz A, Weimer KA, Marks SM, Stern CS, Garfein ES: Mixed reality with HoloLens: Where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg 140:1066–1070, 2017CrossRefGoogle Scholar
  25. 25.
    Wang S, Parsons M, Stone-McLean J, Rogers P, Boyd S, Hoover K, Meruvia-pastor O, Gong M, Smith A: Augmented reality as a telemedicine platform for remote procedural training. Sensors 17:1–21, 2017CrossRefGoogle Scholar
  26. 26.
    Lee N: LG’s SteamVR headset is a bulky yet promising HTC Vive alternative. Available at Accessed May 2017
  27. 27.
    Bharathan R, Vali S, Setchell T, Miskry T, Darzi A, Aggarwal R: Psychomotor skills and cognitive load training on a virtual reality laparoscopic simulator for tubal surgery is effective. Eur J Obstet Gynecol Reprod Biol 169:347–352, 2013CrossRefGoogle Scholar
  28. 28.
    3D Systems Extends Leadership in Precision Healthcare with Technology Advancements to Quickly Create 3D Models. Available at Accessed April 2018
  29. 29.
    3D Systems Leverages Virtual Reality to Advance Surgical Training. Available at Accessed April 2018
  30. 30.
    Interactive Virtual Reality. Available at Accessed April 2018
  31. 31.
    TeraRecon Launches Augmented Reality 3-D Imaging at HIMSS17. Available at Accessed April 2018
  32. 32.
    Rebenitsch L, Owen C: Review on cybersickness in applications and visual displays. Virtual Real 20:101–125, 2016CrossRefGoogle Scholar
  33. 33.
    Rebenitsch L: Managing cybersickness in virtual reality. Crossroads ACM Mag Stud 22:46–51, 2015CrossRefGoogle Scholar
  34. 34.
    Satava RM, Member A, Jones SB: Current and future applications of virtual reality for medicine. Proc IEEE 86:484–489, 1998CrossRefGoogle Scholar
  35. 35.
    Cates CU, Lönn L, Gallagher AG: Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Simul Technol Enhanc Learn 2:1–5, 2016CrossRefGoogle Scholar
  36. 36.
    Ali S, Qandeel M, Ramakrishna R, Yang CW: Virtual simulation in enhancing procedural training for fluoroscopy-guided lumbar puncture: A Pilot Study. Acad Radiol 25:235–239, 2018CrossRefGoogle Scholar
  37. 37.
    Ramlogan R, Niazi AU, Jin R, Johnson J, Chan VW, Perlas A: A virtual reality simulation model of spinal ultrasound: Role in teaching spinal sonoanatomy. Reg Anesth Pain Med 42:217–222, 2017CrossRefGoogle Scholar
  38. 38.
    Azagury DE, Ryou M, Shaikh SN, San José Estépar R, Lengyel BI, Jagadeesan J, Vosburgh KG, Thompson CC: Real-time computed tomography-based augmented reality for natural orifice transluminal endoscopic surgery navigation. Br J Surg 99:1246–1253, 2012CrossRefGoogle Scholar
  39. 39.
    Johnston APR, Rae J, Ariotti N, Bailey B, Lija A, Webb R, Ferguson C, Maher S, Davis TP, Webb RI, Mcghee J, Parton RG: Journey to the centre of the cell: Virtual reality immersion into scientific data. Traffic 105–110, 2017.Google Scholar
  40. 40.
    Shanahan M: Use of a virtual radiography simulation enhances student learning. J Med Radiat Sci 63:27, 2016Google Scholar
  41. 41.
    Syed AZ, Zakaria A, Lozanoff S: Dark room to augmented reality: Application of HoloLens technology for oral radiological diagnosis. Oral Surg Oral Med Oral Pathol Oral Radiol 124:e33, 2017CrossRefGoogle Scholar
  42. 42.
    Mitsouras D, Liacouras P, Imanzadeh A, Giannopoulos AA, Cai T, Kumamaru KK, George E, Wake N, Caterson EJ, Pomahac B, Ho VB, Grant GT, Rybicki FJ: Medical 3D printing for the radiologist. RadioGraphics 35:1965–1988, 2015CrossRefGoogle Scholar
  43. 43.
    Giannopoulos AA, Mitsouras D, Yoo S-J, Liu PP, Chatzizisis YS, Rybicki FJ: Applications of 3D printing in cardiovascular diseases. Nat Rev Cardiol 13:701–718, 2016CrossRefGoogle Scholar
  44. 44.
    Christensen A, Rybicki FJ: Maintaining safety and efficacy for 3D printing in medicine. 3D Print Med 3:1, 2017CrossRefGoogle Scholar
  45. 45.
    Di Prima M, Coburn J, Hwang D, Kelly J, Khairuzzaman A, Ricles L: Additively manufactured medical products—The FDA perspective. 3D Print Med 2(1), 2015Google Scholar
  46. 46.
    Rybicki FJ: Medical 3D printing and the physician-artist. Lancet 391:651–652, 2018CrossRefGoogle Scholar
  47. 47.
    Rybicki FJ: Message from Frank J. Rybicki, MD, incoming chair of ACR appropriateness criteria. J Am Coll Radiol 14:723–724, 2017CrossRefGoogle Scholar
  48. 48.
    Douglas DB, Wilke CA, Gibson D, Petricoin EF, Liotta L: Virtual reality and augmented reality: Advances in surgery. Biol Eng Med 2:1–8, 2017Google Scholar
  49. 49.
    Gallagher K, Jain S, Okhravi N: Making and viewing stereoscopic surgical videos with smartphones and virtual reality headset. Eye 30:503–504, 2016CrossRefGoogle Scholar
  50. 50.
    Huber T, Wunderling T, Paschold M, Lang H, Kneist W, Hansen C: Highly immersive virtual reality laparoscopy simulation: Development and future aspects. Int J Comput Assist Radiol Surg 13:281–290, 2018CrossRefGoogle Scholar
  51. 51.
    Kersten-Oertel M, Gerard I, Drouin S, Mok K, Sirhan D, Sinclair DS, Collins DL: Augmented reality in neurovascular surgery: Feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg 10:1823–1836, 2015CrossRefGoogle Scholar
  52. 52.
    Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A: A systematic review of virtual reality simulators for robot-assisted surgery. Eur Urol 69:1065–1080, 2016CrossRefGoogle Scholar
  53. 53.
    Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, Navab N, Fallavollita P: Intra-operative augmented reality in distal locking. Int J Comput Assist Radiol Surg 10:1395–1403, 2015CrossRefGoogle Scholar
  54. 54.
    King F, Jayender J, Bhagavatula SK, Shyn PB, Pieper S, Kapur T, Lasso A, Fichtinger G: An immersive virtual reality environment for diagnostic imaging. J Med Robot Res 01:1640003, 2016CrossRefGoogle Scholar
  55. 55.
    Douglas DB, Petricoin EF, Liotta L, Wilson E: D3D augmented reality imaging system: Proof of concept in mammography. Med Devices Evid Res 9:277–283, 2016CrossRefGoogle Scholar
  56. 56.
    Hanna MG, Ahmed I, Nine J, Prajapati S, Pantanowitz L: Augmented reality technology using Microsoft HoloLens in anatomic pathology. Arch Pathol Lab Med, 2018.
  57. 57.
    Calì C, Baghabra J, Boges DJ, Holst GR, Kreshuk A, Hamprecht FA, Srinivasan M, Lehväslaiho H, Magistretti PJ: Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. J Comp Neurol 524:23–38, 2016CrossRefGoogle Scholar
  58. 58.
    Zheng LL, He L, Yu CQ: Mobile virtual reality for ophthalmic image display and diagnosis. J Mob Technol Med 4:35–38, 2015CrossRefGoogle Scholar
  59. 59.
    Dascal J, Reid M, Ishak WW, Spiegel B, Recacho J, Rosen B, Danovitch I: Virtual reality and medical inpatients: A systematic review of randomized, controlled trials. Innov Clin Neurosci 14:14–21, 2017PubMedPubMedCentralGoogle Scholar
  60. 60.
    Schmitt YS, Hoffman HG, Blough DK, David R, Jensen MP, Soltani M, Carrougher GJ, Mn RN, Nakamura D, Otr L, Sharar SR: A randomized, controlled trial of immersive virtual reality analgesia during physical therapy for pediatric burn injuries. Burns 37:61–68, 2011CrossRefGoogle Scholar
  61. 61.
    Carrougher GJ, Hoffman HG, Nakamura D, Lezotte D, Soltani M, Leahy L, Engrav LH, Patterson DR: The effect of virtual reality on pain and range of motion in adults with burn injuries. J Burn Care Res 30:785–791, 2009CrossRefGoogle Scholar
  62. 62.
    Kipping B, Rodger S, Miller K, Kimble RM: Virtual reality for acute pain reduction in adolescents undergoing burn wound care: A prospective randomized controlled trial. Burns 38:650–657, 2012CrossRefGoogle Scholar
  63. 63.
    Morris LD, Louw QA, Crous LC: Feasibility and potential effect of a low-cost virtual reality system on reducing pain and anxiety in adult burn injury patients during physiotherapy in a developing country. Burns 36:659–664, 2010CrossRefGoogle Scholar
  64. 64.
    Hoffman HG, Patterson DR, Seibel E, Soltani M, Jewett-Leahy L, Sharar SR: Virtual reality pain control during burn wound debridement in the hydrotank. Clin J Pain 24:299–304, 2008CrossRefGoogle Scholar
  65. 65.
    Patterson DR, Jensen MP, Wiechman SA, Sharar SR: Virtual reality hypnosis for pain associated with recovery from physical trauma. Int J Clin Exp Hypn 58:288–300, 2010CrossRefGoogle Scholar
  66. 66.
    Li WH, Chung JO, Ho EK: The effectiveness of therapeutic play, using virtual reality computer games, in promoting the psychological well-being of children hospitalised with cancer. J Clin Nurs 20:2135–2143, 2011CrossRefGoogle Scholar
  67. 67.
    Cesa GL, Manzoni GM, Bacchetta M, Castelnuovo G, Conti S, Gaggioli A, Mantovani F, Molinari E, Cárdenas-López G, Riva G: Virtual reality for enhancing the cognitive behavioral treatment of obesity with binge eating disorder: Randomized controlled study with one-year follow-up. J Med Internet Res 15:1–13, 2013CrossRefGoogle Scholar
  68. 68.
    Manzoni GM, Pagnini F, Gorini A, Preziosa A, Castelnuovo G, Molinari E, Riva G: Can relaxation training reduce emotional eating in women with obesity? An exploratory study with 3 months of follow-up. J Am Diet Assoc 109:1427–1432, 2009CrossRefGoogle Scholar
  69. 69.
    Larson EB, Ramaiya M, Zollman FS, Pacini S, Hsu N, Patton JL, Dvorkin AY: Tolerance of a virtual reality intervention for attention remediation in persons with severe TBI. Brain Inj 25:274–281, 2011CrossRefGoogle Scholar
  70. 70.
    Saposnik G, Cohen LG, Mamdani M, Pooyania S, Ploughman M, Cheung D, Shaw J, Hall J, Nord P, Dukelow S, Nilanont Y, De los Rios F, Olmos L, Levin M, Teasell R, Cohen A, Thorpe K, Laupacis A, Bayley M: Efficacy and safety of non-immersive virtual reality exercising in stroke rehabilitation (EVREST): A randomised, multicentre, single-blind, controlled trial. Lancet Neurol 15:1019–1027, 2016CrossRefGoogle Scholar
  71. 71.
    Silver B: Virtual reality versus reality in post-stroke rehabilitation. Lancet Neurol 15:996–997, 2016CrossRefGoogle Scholar
  72. 72.
    Ambron E, Miller A, Kuchenbecker KJ, Laurel B, Coslett HB: Immersive low-cost virtual reality treatment for phantom limb pain: Evidence from two cases. Front Neurol 9:1–7, 2018CrossRefGoogle Scholar
  73. 73.
    Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, Slater M: Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med 47:2393–2400, 2017CrossRefGoogle Scholar
  74. 74.
    Maples-Keller JL, Price M, Rauch S, Gerardi M, Rothbaum BO: Investigating relationships between PTSD symptom clusters within virtual reality exposure therapy for OEF/OIF veterans. Behav Ther 48:147–155, 2017CrossRefGoogle Scholar
  75. 75.
    Miloff A, Lindner P, Hamilton W, Reuterskiöld L, Andersson G, Carlbring P: Single-session gamified virtual reality exposure therapy for spider phobia vs. traditional exposure therapy: Study protocol for a randomized controlled non-inferiority trial. Trials 17:1–8, 2016CrossRefGoogle Scholar
  76. 76.
    Nararro-Haro MV, Hoffman HG, Garcia-Palacios A, Sampaio M, Alhalabi W, Hall K, Linehan M: The use of virtual reality to facilitate mindfulness skills training in dialectical behavioral therapy for borderline personality disorder: A case study. Front Psychol 7:1–9, 2016CrossRefGoogle Scholar
  77. 77.
    Wiederhold BK, Wiederhold MD: Virtual Reality Therapy for Anxiety Disorders: Advances in Evaluation and Treatment. Washington, DC: American Psychological Association, 2005CrossRefGoogle Scholar
  78. 78.
    Chepelev L, Giannopoulos A, Tang A, Mitsouras D, Rybicki FJ: Medical 3D printing: Methods to standardize terminology and report trends. 3D Print Med 3:1–9, 2017CrossRefGoogle Scholar
  79. 79.
    Rybicki FJ: 3D printing in medicine: An introductory message from the editor-in-chief. 3D Print Med 1(1):1, 2015CrossRefGoogle Scholar
  80. 80.
    Mitsouras D, Liacouras PC: 3D printing in medicine: 3D printing technologies. In: Rybicki FJ, Grant GT Eds. 3D Printing in Medicine: A Practical Guide for Medical Professionals. Cham: Springer International Publishing, 2017, pp. 5–22CrossRefGoogle Scholar
  81. 81.
    American Association of Physicists in Medicine: Assessment of display performance for medical imaging systems. Med Phys:1–156, 2005Google Scholar
  82. 82.
    Norweck JT, Seibert JA, Andriole KP, Clunie DA, Curran BH, Flynn MJ, Krupinski E, Lieto RP, Peck DJ, Mian TA, Wyatt M: ACR-AAPM-SIIM technical standard for electronic practice of medical imaging. J Digit Imaging 26:38–52, 2013CrossRefGoogle Scholar
  83. 83.
    Drebin R A, Carpenter L, Hanrahan P: Volume rendering. SIGGRAPH ’88. In: Proc. 15th Annu. Conf. Comput. Graph. Interact. Tech. 22, pp. 65–74, 1988Google Scholar
  84. 84.
    Levoy M: Volume rendering: Display of surface from volume data. IEEE Comput Graph Appl 8:29–37, 1988CrossRefGoogle Scholar
  85. 85.
    Stegmaiert S, Kieint T, Stegmaier S, Strengert M, Klein T, Ertl T: A simple and flexible volume rendering framework for graphics-hardware-based raycasting. Fourth Int. Work. Vol. Graph, pp. 187–241, 2005Google Scholar
  86. 86.
    Ljung P, Krüger J, Groller E, Hadwiger M, Hansen CD, Ynnerman A, Gröller E, Hadwiger M, Hansen CD, Ynnerman A: State of the art in transfer functions for direct volume rendering. Comput Graph Forum 35:669–691, 2016CrossRefGoogle Scholar
  87. 87.
    Arens S, Domik G: A survey of transfer functions suitable for volume rendering. IEEE/EG Int. Symp. Vol. Graph., pp. 77–83, 2010Google Scholar
  88. 88.
    Krüger J, Westermann R: Acceleration techniques for GPU-based volume rendering. IEEE Vis, pp. 287–292, 2003.Google Scholar
  89. 89.
    Vizua3D: Available at Accessed February 2018

Copyright information

© Society for Imaging Informatics in Medicine 2018

Authors and Affiliations

  • Justin Sutherland
    • 1
    Email author
  • Jason Belec
    • 1
  • Adnan Sheikh
    • 1
  • Leonid Chepelev
    • 1
  • Waleed Althobaity
    • 1
  • Benjamin J. W. Chow
    • 1
  • Dimitrios Mitsouras
    • 1
  • Andy Christensen
    • 1
  • Frank J. Rybicki
    • 1
  • Daniel J. La Russa
    • 1
  1. 1.Department of RadiologyUniversity of OttawaOttawaCanada

Personalised recommendations