Abstract
In 3D echocardiography (3D echo), the image orientation varies depending on the position and direction of the transducer during examination. As a result, when reviewing images the user must initially identify anatomical landmarks to understand image orientation – a potentially challenging and time-consuming task. We automated this initial step by training a deep residual neural network (ResNet) to predict the rotation required to re-orient an image to the standard apical four-chamber view). Three data pre-processing strategies were explored: 2D, 2.5D and 3D. Three different loss function strategies were investigated: classification of discrete integer angles, regression with mean absolute angle error loss, and regression with geodesic loss. We then integrated the model into a virtual reality application and aligned the re-oriented 3D echo images with a standard anatomical heart model. The deep learning strategy with the highest accuracy – 2.5D classification of discrete integer angles – achieved a mean absolute angle error on the test set of 9.0\(^\circ \). This work demonstrates the potential of artificial intelligence to support visualisation and interaction in virtual reality.
L. Munroe—This work is independent research funded by the National Institute for Health Research (NIHRi4i, 3D Heart Project, II-LA-0716-20001, https://www.3dheart.co.uk/). This work was also supported by the Wellcome/EPSRC Centre for Medical Engineering (WT203148/Z/16/Z). Lindsay Munroe and Suryava Bhattacharya would like to acknowledge funding from the EPSRC Centre for Doctoral Training in Smart Medical Imaging (EP/S022104/1). Authors also acknowledge financial support from the Department of Health via the National Institute for Health Research (NIHR) comprehensive Biomedical Research Centre award to Guy’s and St Thomas’ NHS Foundation Trust in partnership with King’s College London and King’s College Hospital NHS Foundation Trust.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at https://www.thingiverse.com/thing:852939.
References
Balduzzi, D., et al.: The shattered gradients problem: if resnets are the answer, then what is the question? In: International Conference on Machine Learning. PMLR, pp. 342–350 (2017)
Cheng, K., et al.: 3D echocardiography: benefits and steps to wider implementation. Br. J. Cardiol. 25, 63–68 (2018)
Dolk, H., et al.: Congenital heart defects in Europe: prevalence and perinatal mortality, 2000 to 2005. Circulation 123(8), 841–849 (2011)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised rep- resentation learning by predicting image rotations. In: arXiv preprint arXiv:1803.07728 (2018)
Gomez, A.: MIPROT: A Medical Image Processing Toolbox for MAT-LAB. In: arXiv preprint arXiv:2104.04771 (2021)
He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
CREATIS Lab.-France. CETUS datasets. https://www.creatis.insa-lyon.fr/Challenge/CETUS/index.html. https://www.creatis.insa-lyon.fr/Challenge/CETUS/databases.html
Lau, I., Sun, Z.: Three-dimensional printing in congenital heart disease: a systematic review. J. Med. Radiat. Sci. 65(3), 226–236 (2018)
Leclerc, S., et al.: Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Trans. Med. Imaging 38(9), 2198–2210 (2019)
Mahendran, S., Ali, H., Vidal, R.: 3D pose regression using convolutional neural networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2174–2182 (2017)
Pushparajah, K., et al.: Virtual reality three dimensional echocardiographic imaging for planning surgical atrioventricular valve repair. JTCVS Tech. 7, 269–277 (2021)
Saez, D.: Correcting Image Orientation Using Convolutional Neural Networks. http://d4nst.github.io/2017/01/12/image-orientation
Simpson, J., et al.: Image orientation for three-dimensional echocardiography of congenital heart disease. The Int. J. Cardiovasc. Imaging 28(4), 743–753 (2012)
Simpson, J.M., Miller, O.: Three-dimensional echocardiography in congenital heart disease. Arch. Cardiovasc. Dis. 104(1), 45–56 (2011)
Valverde, I., et al.: Three-dimensional printed models for surgical planning of complex congenital heart defects: an international multicentre study. Eur. J. Cardiothorac. Surg. 52(6), 1139–1148 (2017)
Wheeler, G., et al.: Virtual interaction and visualisation of 3D medical imaging data with VTK and unity. Healthc. Technol. Lett. 5(5), 148–153 (2018)
Wright, R., et al.: LSTM spatial co-transformer networks for registration of 3D fetal US and MR brain images. In: Melbourne, A., et al. (eds.) PIPPI/DATRA -2018. LNCS, vol. 11076, pp. 149–159. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00807-9_15
Jia-Jun, X., et al.: Patient-specific three-dimensional printed heart models benefit preoperative planning for complex congenital heart disease. World J. Pediatr. 15(3), 246–254 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Munroe, L. et al. (2021). Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning. In: Papież, B.W., Yaqub, M., Jiao, J., Namburete, A.I.L., Noble, J.A. (eds) Medical Image Understanding and Analysis. MIUA 2021. Lecture Notes in Computer Science(), vol 12722. Springer, Cham. https://doi.org/10.1007/978-3-030-80432-9_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-80432-9_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-80431-2
Online ISBN: 978-3-030-80432-9
eBook Packages: Computer ScienceComputer Science (R0)