Abstract
Preoperative medical imaging is an essential part of surgical planning. The data from medical imaging devices, such as CT and MRI scanners, consist of stacks of 2D images in DICOM format. Conversely, advances in 3D data visualization provide further information by assembling cross-sections into 3D volumetric datasets. As Microsoft unveiled the HoloLens 2 (HL2), which is considered one of the best Mixed Reality (XR) headsets in the market, it promised to enhance visualization in 3D by providing an immersive experience to users. This paper introduces a prototype holographic XR DICOM Viewer for the 3D visualization of DICOM image sets on HL2 for surgical planning. We first developed a standalone graphical C++ engine using the native DirectX11 API and HLSL shaders. Based on that, the prototype further applies the OpenXR API for potential deployment on a wide range of devices from vendors across the XR spectrum. With native access to the device, our prototype unravels the limitation of hardware capabilities on HL2 for 3D volume rendering and interaction. Moreover, smartphones can act as input devices to provide another user interaction method by connecting to our server. In this paper, we present a holographic DICOM viewer for the HoloLens 2 and contribute (i) a prototype that renders the DICOM image stacks in real-time on HL2, (ii) three types of user interactions in XR, and (iii) a preliminary qualitative evaluation of our prototype.
This research was funded in part by a grant from The Leona M. and Harry B. Helmsley Charitable Trust. The authors would also like to thank the volunteers who provided their medical data for us to work with under UCSD IRB protocol \(\#182146\).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Büschel, W., Mitschick, A., Meyer, T., Dachselt, R.: Investigating smartphone-based pan and zoom in 3d data spaces in augmented reality. In: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 1–13 (2019)
Fröhlich, M., et al.: Holographic visualisation and interaction of fused CT, PET and MRI volumetric medical imaging data using dedicated remote GPGPU Ray casting. In: Stoyanov, D., et al. (eds.) POCUS/BIVPCS/CuRIOUS/CPM -2018. LNCS, vol. 11042, pp. 102–110. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01045-4_12
HOLODICOM: Diagnostic medical image in augmented reality. http://www.holodicom.com/
Jang, S., Stuerzlinger, W., Ambike, S., Ramani, K.: Modeling cumulative arm fatigue in mid-air interaction based on perceived exertion and kinetics of arm motion. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3328–3339 (2017)
Karan, M.S., Berkman, M.İ, Çatak, G.: Smartphone as a paired game input device: an application on hololens head mounted augmented reality system. In: Cordan, Ö., Dinçay, D.A., Yurdakul Toker, Ç., Öksüz, E.B., Semizoğlu, S. (eds.) Game + Design Education. SSDI, vol. 13, pp. 265–277. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-65060-5_22
Kovesi, P.: Good colour maps: How to design them. arXiv preprint arXiv:1509.03700 (2015)
Langner, R., Satkowski, M., Büschel, W., Dachselt, R.: Marvis: Combining mobile devices and augmented reality for visual data analysis. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–17 (2021)
Lucknavalai, K., Schulze, J.P.: Real-time contrast enhancement for 3d medical images using histogram equalization. In: Bebis, G., et al. (eds.) ISVC 2020. LNCS, vol. 12509, pp. 224–235. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64556-4_18
Mojica, C.M.M., et al.: Holographic interface for three-dimensional visualization of mri on hololens: a prototype platform for mri guided neurosurgeries. In: 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE), pp. 21–27. IEEE (2017)
Morales Mojica, C.M., et al.: A holographic augmented reality interface for visualizing of mri data and planning of neurosurgical procedures. J. Digit. Imaging 34(4), 1014–1025 (2021)
Munoz-Salinas, R.: Aruco: a minimal library for augmented reality applications based on opencv. Universidad de Córdoba (2012)
OpenCV: Opencv (May 2020). https://opencv.org/
Ungureanu, D., et al.: Hololens 2 research mode as a tool for computer vision research. CoRR abs/ arXiv: 2008.11239 (2020)
Zhang, M., Lucknavalai, K., Liu, W., Alipour, K., Schulze, J.P.: Arcalvr: augmented reality playground on mobile devices. In: ACM SIGGRAPH 2019 Appy Hour, pp. 1–2 (2019)
Zhang, M., Lucknavalai, K., Liu, W., Schulze, J.P.: Calar: A c++ engine for augmented reality applications on android mobile devices. Elect. Imaging 2020(13), 1–364 (2020)
Zhang, M., Schulze, J.P.: Server-aided 3d dicom viewer for mobile platforms. Elect. Imaging 2021(13), 1–179 (2021)
Ziyang, Z., et al.: Virtualforce: Simulating writing and sketching on a 2d-surface in virtual reality. Elect. Imaging 34, 1–5 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, M., Liu, W., Weibel, N., Schulze, J.P. (2022). A DirectX-Based DICOM Viewer for Multi-user Surgical Planning in Augmented Reality. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13599. Springer, Cham. https://doi.org/10.1007/978-3-031-20716-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-20716-7_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20715-0
Online ISBN: 978-3-031-20716-7
eBook Packages: Computer ScienceComputer Science (R0)