Skip to main content

Advertisement

SpringerLink
Log in

On the Use of Virtual Reality for Medical Imaging Visualization

  • Original Paper
  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Advanced visualization of medical imaging has been a motive for research due to its value for disease analysis, surgical planning, and academical training. More recently, attention has been turning toward mixed reality as a means to deliver more interactive and realistic medical experiences. However, there are still many limitations to the use of virtual reality for specific scenarios. Our intent is to study the current usage of this technology and assess the potential of related development tools for clinical contexts. This paper focuses on virtual reality as an alternative to today’s majority of slice-based medical analysis workstations, bringing more immersive three-dimensional experiences that could help in cross-slice analysis. We determine the key features a virtual reality software should support and present today’s software tools and frameworks for researchers that intend to work on immersive medical imaging visualization. Such solutions are assessed to understand their ability to address existing challenges of the field. It was understood that most development frameworks rely on well-established toolkits specialized for healthcare and standard data formats such as DICOM. Also, game engines prove to be adequate means of combining software modules for improved results. Virtual reality seems to remain a promising technology for medical analysis but has not yet achieved its true potential. Our results suggest that prerequisites such as real-time performance and minimum latency pose the greatest limitations for clinical adoption and need to be addressed. There is also a need for further research comparing mixed realities and currently used technologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://gitlab.com/3dheart_public/vtktounity

  2. MeVisLab publications: https://www.mevislab.de/mevislab/publications

  3. https://github.com/IMHOTEP-Medical/imhotep

  4. An open-source 3D creation suite software, supporting modeling, rigging, animation, rendering and more, that can import many common file formats.

  5. https://www.dicomlibrary.com/

  6. https://www.dclunie.com

  7. https://oasis-brains.org/

  8. https://www.sci.utah.edu/cibc-software/cibc-datasets.html

  9. http://www.janegger.de/Opportunities/MeVisLab-Unity%203D.pdf

References

  1. HTC Corporation, HTC VIVE Cosmos: own your VR world, in https://vive.com. Retrieved April 2020.

  2. Oculus Go, Standalone VR Headset: Our all-in-one headset made for entertainment., in oculus.com. Retrieved April 2020.

  3. Perkins Coie & XR Association. (2019). Augmented and Virtual Reality Survey Report. Industry Insights into the Future of Immersive Technology, vol. 3.

  4. Lorensen, William & Cline, Harvey. (1987). Marching Cubes: A High Resolution 3D Surface Construction Algorithm. ACM SIGGRAPH Computer Graphics. 21. 163–169. https://doi.org/10.1145/37401.37422.

  5. Chen, Long & Day, Thomas & Tang, Wen & John, Nigel. (2017). Recent Developments and Future Challenges in Medical Mixed Reality. 10.1109/ISMAR.2017.29.

  6. Aouam, Djamel & Zenati-Henda, Nadia & Benbelkacem, Samir & Hamitouche, Chafiaa. (2020). An Interactive VR System for Anatomy Training. IntechOpen, https://doi.org/10.5772/intechopen.91358.

  7. 4D Anatomy: AR Mobile Application, by 4D Interactive Anatomy. Retrieved April 2020.

  8. Xin Wang and Xiuyue Wang. 2018. Virtual Reality Training System for Surgical Anatomy. In Proceedings of the 2018 International Conference on Artificial Intelligence and Virtual Reality (AIVR 2018). Association for Computing Machinery, New York, NY, USA, 30-34. https://doi.org/10.1145/3293663.3293670.

  9. Cabrera, Mildred & Carrillo, Jos & Nigenda, Juan & Gonzalez, Ricardo & Valdez, Jorge & ChavarRa, Belinda. (2019). Assessing the Effectiveness of Teaching Anatomy with Virtual Reality. 43-46. https://doi.org/10.1145/3369255.3369260.

  10. Makled, Elhassan & Yassien, Amal & Elagroudy, Passant & Magdy, Mohamed & Abdennadher, Slim & Hamdi, Nabila. (2019). PathoGenius VR: VR medical training. 1-2. https://doi.org/10.1145/3321335.3329694.

  11. Papagiannakis, George & Lydatakis, Nick & Kateros, Steve & Georgiou, Stelios & Zikas, Paul. (2018). Transforming medical education and training with VR using M.A.G.E.S. 1-2. https://doi.org/10.1145/3283289.3283291.

  12. Huber, Tobias & Wunderling, Tom & Paschold, Markus & Lang, Hauke & Kneist, Werner & Hansen, Christian. (2017). Highly Immersive Virtual Reality Laparoscopy Simulation: Development and Future Aspects. International Journal of Computer Assisted Radiology and Surgery. 13. https://doi.org/10.1007/s11548-017-1686-2.

  13. Gonzalez Izard, Santiago & Juanes, Juan. (2016). Virtual reality medical training system. Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM 16). 479-485. https://doi.org/10.1145/3012430.3012560.

  14. Paiva, Paulo & Machado, Liliane & Valena, Ana & Batista, Thiago & Moraes, Ronei. (2018). SimCEC: A Collaborative VR-Based Simulator for Surgical Teamwork Education. Computers in Entertainment. 16. 1-26. https://doi.org/10.1145/3177747.

  15. Javaux, Allan & Bouget, David & Gruijthuijsen, Caspar & Stoyanov, Danail & Vercauteren, Tom & Ourselin, Sebastien & Deprest, Jan & Denis, Kathleen & Vander Poorten, Emmanuel. (2018). A mixed-reality surgical trainer with comprehensive sensing for fetal laser minimally invasive surgery. International Journal of Computer Assisted Radiology and Surgery. 13. https://doi.org/10.1007/s11548-018-1822-7.

  16. Holland, Mark & Pop, Serban & John, Nigel. (2017). VR Cardiovascular Blood Simulation as Decision Support for the Future Cyber Hospital. 233-236. https://doi.org/10.1109/CW.2017.49

  17. Tan, Wenjun & Ge, Wen & Hang, Yucheng & Wu, Simeng & Liu, Sixing & Liu Ming (2018). Computer assisted system for precise lung surgery based on medical image computing and mixed reality. Health Inf Sci Syst 6, 10. https://doi.org/10.1007/s13755-018-0053-1.

  18. Ayoub, Ashraf & Pulijala, Yeshwanth. (2019). The application of virtual reality and augmented reality in Oral & Maxillofacial Surgery. BMC Oral Health. 19. https://doi.org/10.1186/s12903-019-0937-8.

  19. Li, Feiyan & Yonghang, Tai & Li, Qiong & Peng, Jun & Xiaoqiao, Huang & Chen, Zaiqing & Shi, Junsheng. (2019). Real-Time Needle Force Modeling for VR-Based Renal Biopsy Training with Respiratory Motion Using Direct Clinical Data. Applied Bionics and Biomechanics. 2019. 1-14. https://doi.org/10.1155/2019/9756842.

    Article  Google Scholar 

  20. Malpani, Anand & Vedula, Sunil & Lin, Henry & Hager, Gregory & Taylor, Russell. (2020). Effect of real-time virtual reality-based teaching cues on learning needle passing for robot-assisted minimally invasive surgery: a randomized controlled trial. International Journal of Computer Assisted Radiology and Surgery. 15. https://doi.org/10.1007/s11548-020-02156-5.

  21. Mehralivand, Sherif & Kolagunda, Abhishek & Hammerich, Kai & Sabarwal, Vikram & Harmon, Stephanie & Sanford, Thomas & Gold, Samuel & Hale, Graham & Romero, Vladimir & Bloom, Jonathan & Merino, Maria & Wood, Bradford & Kambhamettu, Chandra & Choyke, Peter & Pinto, Peter & Turkbey, Baris. (2019). A multiparametric magnetic resonance imaging-based virtual reality surgical navigation tool for robotic-assisted radical prostatectomy. Trk roloji Dergisi/Turkish Journal of Urology. 45. 357-365. https://doi.org/10.5152/tud.2019.19133.

  22. Pan, Zhaoxi & Tian, Song & Guo, Mengzhao & Zhang, Jianxun & Yu, Ningbo & Xin, Yunwei. (2017). Comparison of medical image 3D reconstruction rendering methods for robot-assisted surgery. 94-99. https://doi.org/10.1109/ICARM.2017.8273141.

  23. Ruthenbeck, Greg & Reynolds, Karen. (2014). Virtual Reality for Medical Training: The State of the Art. Journal of Simulation. 9. https://doi.org/10.1057/jos.2014.14.

  24. Sharkey, P.M. & Merrick, J.. (2014). Virtual reality: Rehabilitation in motor, cognitive and sensorial disorders. http://site.ebrary.com/id/10929399.

  25. Mitrousia, V. & Giotakos, O. (2016). Virtual reality therapy in anxiety disorders. Psychiatriki. 27. 276-286. https://doi.org/10.22365/jpsych.2016.274.276.

  26. Wiederhold, B.. (2018). Are We Ready for Online Virtual Reality Therapy?. Cyberpsychology, Behavior, and Social Networking, 21: 341-342. https://doi.org/10.1089/cyber.2018.29114.bkw.

  27. Dias, Paulo & Silva, Ricardo & Amorim, Paula & Lains, Jorge & Roque, Eulalia & Pereira, Ines & Pereira, Fatima & Santos, Beatriz & Potel, Mike. (2019). Using Virtual Reality to Increase Motivation in Poststroke Rehabilitation. IEEE Computer Graphics and Applications. 39. 64-70. https://doi.org/10.1109/MCG.2018.2875630.

  28. Moreira, Gabrielly & Machado, Ingrid & Lima, Elis & Loureiro, Ana Paula & Manffra, Elisangela. (2020). The Use of Virtual Reality Rehabilitation for Individuals Post Stroke. 1. 21-27.

  29. August, K. & Sellathurai, M. & Bleichenbacher, D. & Adamovich, S., VESLI Virtual Reality Rehabilitation for the Hand, 2020.

  30. Weiss, P.L.T. & Weintraub, N. & Laufer, Y., Virtual reality therapy in paediatric rehabilitation, 2016.

  31. W. Dean Bidgood, Jr & Steven C. Horii & Fred W. Prior & Donald E. Van Syckle. (1997). Understanding and Using DICOM, the Data Interchange Standard for Biomedical Imaging, Journal of the American Medical Informatics Association, Volume 4, Issue 3, Pages 199-212. https://doi.org/10.1136/jamia.1997.0040199.

  32. R.A. Robb & D.P. Hanson & R.A. Karwoski & A.G. Larson & E.L. Workman & M.C. Stacy. (1989). Analyze: A Comprehensive, operator-interactive software package for multidimensional medical image display and analysis. Computerized Medical Imaging and Graphics, 13 (6): 433-454. https://doi.org/10.1016/0895-6111(89)90285-1.

  33. NIfTI Documentation, in https://nifti.nimh.nih.gov. Retrieved April 2020.

  34. The McConnle Brain Imaging Centre, MINC software library and tools, in https://bic.mni.mcgill.ca. Retrieved April 2020.

  35. Kahn, Jr, Charles & Carrino, John & Flynn, Michael & Peck, Donald & Horii, Steven. (2007). DICOM and radiology: Past, present, and future. Journal of the American College of Radiology : JACR. 4. 652-7. https://doi.org/10.1016/j.jacr.2007.06.004.

  36. National Electrical Manufacturers Association, Digital Imaging and Communications in Medicine, in https://dicomstandard.org. Retrieved April 2020.

  37. National Electrical Manufacturers Association, WG-17 3D, in https://dicomstandard.org. Retrieved April 2020.

  38. Realize Medical, Elucis: The Future of Medical Modeling, in https://realizemed.com. Retrieved April 2020.

  39. Diffuse, Specto, in https://diffuse.ch. Retrieved April 2020.

  40. C. Luciano & P. Banerjee & L. Florea & G. Dawe, Design of the ImmersiveTouch: a high-performance haptic augmented virtual reality system, 2014.

  41. ImmersiveTouch, ImmersiveView VR, in https://immersivetouch.com. Retrieved April 2020.

  42. Imaging Reality, About Imaging Reality, in https://imagingreality.com. Retrieved April 2020.

  43. DICOM VR, DICOM VR: a new way of viewing volumetric medical imaging and planning targeted radiation treatment in virtual reality, in https://dicomvr.com. Retrieved April 2020.

  44. Surgical Theater, Precision VR, in https://surgicaltheater.net. Retrieved April 2020.

  45. HoloDICOM, HoloDICOM: diagnostic medical image in augmented reality, in https://holodicom.com. Retrieved April 2020.

  46. Seal, Arindrajit & Das, Arunava & Sen, Prasad. (2015). Watershed: An Image Segmentation Approach. International Journal of Computer Science and Information Technologies, 6(3):2295-2297. 0975-0946.

  47. Kemmling, Andre & Minnerup, Heike & Berger, Karsten & Knecht, Stefan & Groden, Christoph & Nlte, Ingo. (2012). Decomposing the Hounsfield Unit Probabilistic Segmentation of Brain Tissue in Computed Tomography. Clinical neuroradiology. 22. 79-91. https://doi.org/10.1007/s00062-011-0123-0.

  48. EchoPixel, True3D, in https://echopixeltech.com. Retrieved April 2020.

  49. Mohammed, Mohammed & Khalaf, Mosbah & Kesselman, Andrew & Wang, David & Kothary, Nishita. (2018). A Role for Virtual Reality in Planning Endovascular Procedures. Journal of vascular and interventional radiology : JVIR. 29. 971-974. 10.1016/j.jvir.2018.02.018.

    Article  PubMed  Google Scholar 

  50. Kaley, Vishal & Aregullin, Enrique & Samuel, Bennett & Vettukattil, Joseph. (2018). Transcatheter Intervention for Paravalvular Leak in Mitroflow Bioprosthetic Pulmonary Valve. https://doi.org/10.12945/j.jshd.2018.006.18.

  51. Huang, Chenxi & Zhou, Wen & Lan, Yisha & Chen, Fei & Hao, Yongtao & Cheng, Yongqiang & Peng, Yonghong. (2018). A Novel WebVR-based Lightweight Framework for Virtual Visualization of Blood Vasculum. IEEE Access. PP. 1-1. 10.1109/ACCESS.2018.2840494.

  52. Adochiei, Felix & Ciucu, Radu & Adochiei, Ioana & Grigorescu, Dan & Seritan, George & Miron, Casian. (2019). A WEB Platform for Rendring and Viewing MRI Volumes using Real-Time Raytracing Principles. 1-4. https://doi.org/10.1109/ATEE.2019.8724963.

  53. Gonzalez Izard, Santiago & Plaza, Oscar & Torres, Ramiro & Juanes, Juan & Garcia-Pealvo, Francisco. (2019). NextMed, Augmented and Virtual Reality platform for 3D medical imaging visualization: Explanation of the software platform developed for 3D models visualization related with medical images using Augmented and Virtual Reality technology. 459-467. https://doi.org/10.1145/3362789.3362936.

  54. Reddivari, Sandeep & Smith, Jason & Pabalate, Jonathan. (2017). VRvisu: A Tool for Virtual Reality Based Visualization of Medical Data. 280-281. https://doi.org/10.1109/CHASE.2017.102.

  55. Ard, Tyler & Krum, David & Phan, Thai & Duncan, Dominique & Essex, Ryan & Bolas, Mark & Toga, Arthur. (2017). NIVR: Neuro imaging in virtual reality. 465-466. https://doi.org/10.1109/VR.2017.7892381. Sandeep Reddivari and Jason Smith and Jonathan Pabalate. (2017). V Rvisu: a tool for virtual reality based visualization of medical data. Proceedings of the Second IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE 17). IEEE Press, 280-281. https://doi.org/10.1109/CHASE.2017.102

  56. The Visualization Toolkit, in https://vtk.org. Retrieved April 2020.

  57. Schroeder, William & Martin, K. & Lorensen, William. (2006). The Visualization Toolkit, An Object-Oriented Approach To 3D Graphics. ISBN 978-1-930934-19-1.

  58. Ibanez, L. & Schroeder, W. & Ng, L. & Cates, J., The ITK Software Guide: The Insight Segmentation and Registration Toolkit, 2005. Kitware Inc.

  59. Enquobahrie, Andinet & Cheng, Patrick & Gary, Kevin & Ibanez, Luis & Gobbi, David & Lindseth, Frank & Yaniv, Ziv & Aylward, Stephen & Jomier, Julien & Cleary, Kevin. (2007). The Image-Guided Surgery Toolkit IGSTK: An open source C++ software toolkit. Journal of digital imaging : the official journal of the Society for Computer Applications in Radiology. 20 Suppl 1. 21-33. https://doi.org/10.1007/s10278-007-9054-3.

  60. 3DSlicer: a multi-platform, free and open source software package for visualization and medical image computing, in https://slicer.org, by Kitware, Inc. Retrieved April 2020.

  61. ParaView, in https://paraview.org, by Kitware, Inc. Retrieved April 2020.

  62. Amorim, Paulo & Franco de Moraes, Thiago & Pedrini, Helio & Silva, Jorge. (2015). InVesalius: An Interactive Rendering Framework for Health Care Support. https://doi.org/10. 10.1007/978-3-319-27857-5_5.

  63. B. Fischl. (2012). FreeSurfer. NeuroImage. 62. 774-81. https://doi.org/10.1016/j.neuroimage.2012.01.021.

  64. OsiriX DICOM Viewer, in https://osirix-viewer.com, by Pixmeo. Retrieved April 2020.

  65. Nicolas Baghdadi & Clment Mallet & Mehrez Zribi. (2018). QGIS and Generic Tools. https://doi.org/10.1002/9781119457091.

  66. O’Leary, Patrick & Jhaveri, Sankhesh & Chaudhary, Aashish & Sherman, William & Martin, Ken & Lonie, David & Whiting, Eric & Money, James & McKenzie, Sandy. (2017). Enhancements to VTK enabling scientific visualization in immersive environments. 186-194. https://doi.org/10.1109/VR.2017.7892246.

  67. S. Arikatla & J. Fillion-Robin & B. Paniagua & M. Holden & Z. Keri & M. Jolley & A. Lasso & A. Enquobahrie, Bringing Virtual Reality to 3D Slicer, in https://blog.kitware.com, 2018. Retrieved April 2020.

  68. K. Martin & D. DeMarle & S. Jhaveri & U. Ayachit, Taking ParaView into Virtual Reality, in https://blog.kitware.com, 2018. Retrieved April 2020.

  69. Unity Technologies, Unity for All, in https://unity.com. Retrieved April 2020.

  70. Epic Games, Unreal Engine: Make something Unreal, in https://unrealengine.com. Retrieved April 2020.

  71. Wheeler, Gavin & Deng, Shujie & Toussaint, Nicolas & Pushparajah, Kuberan & Schnabel, Julia & Simpson, John & Gomez, Alberto. (2018). Virtual Interaction and Visualisation of 3D Medical Imaging Data with VTK and Unity. Healthcare Technology Letters. 5. https://doi.org/10.1049/htl.2018.5064.

  72. MeVis Medical Solutions AG, MeVisLab: A powerful modular framework for image processing research and development, in https://mevislab.de. Retrieved April 2020.

  73. Muhler, Konrad & Preim, Bernhard. (2010). Reusable Visualizations and Animations for Surgery Planning. Computer Graphics Forum. 29. 1103-1112. https://doi.org/10.1111/j.1467-8659.2009.01669.x.

  74. Gunacker, Simon & Gall, Markus & Schmalstieg, Dieter & Egger, Jan. (2018). Multi-Threaded Integration of HTC-Vive and MeVisLab. https://doi.org/10.13140/RG.2.2.18864.05121.

  75. National Center for Tumor Disease, Dresden, University Hospital, Immersive Medical Hands-On Operation Teaching and Planning System, in https://imhotep-medical.org. Retrieved April 2020.

  76. Pfeiffer, Micha & Kenngott, Hannes & Preukschas, Anas & Huber, Matthias & Bettscheider, Lisa & Muller, Beat & Speidel, Stefanie. (2018). IMHOTEP - Virtual Reality Framework for Surgical Applications. International Journal of Computer Assisted Radiology and Surgery. 13. https://doi.org/10.1007/s11548-018-1730-x.

  77. Olveres, Jimena & Nava, Rodrigo & Escalante-Ramirez, Boris & Cristobal, Gabriel & Mara, Carla. (2013). Midbrain volume segmentation using Active Shape Models and LBPs. Proceedings of SPIE - The International Society for Optical Engineering. 8856. https://doi.org/10.1117/12.2024396.

  78. Loewe, Axel & Poremba, Emanuel & Oesterlein, Tobias & Pilia, Nicolas & Pfeiffer, Micha & Doessel, Olaf & Speidel, Stefanie. (2017). An Interactive Virtual Reality Environment for Analysis of Clinical Atrial Arrhythmias and Ablation Planning. https://doi.org/10.22489/CinC.2017.125-118.

Download references

Funding

This work was partially funded by FCT - Foundation for Science and Technology, in the context of the projects EHDEN-H2020/806968 and UIDB/00127/2020.

Author information

Authors and Affiliations

  1. Department of Electronics Telecommunications and Informatics, University of Aveiro, Aveiro, Portugal

    Filipi Pires, Carlos Costa & Paulo Dias

Authors
  1. Filipi Pires
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlos Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paulo Dias
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filipi Pires.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

This article does not contain patient data.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pires, F., Costa, C. & Dias, P. On the Use of Virtual Reality for Medical Imaging Visualization. J Digit Imaging 34, 1034–1048 (2021). https://doi.org/10.1007/s10278-021-00480-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-021-00480-z

Keywords

Search

Navigation