Abstract
In this paper, we introduce a completion framework to reconstruct the geometric shapes of various anatomies, including organs, vessels and muscles. Our work targets a scenario where one or multiple anatomies are missing in the imaging data due to surgical, pathological or traumatic factors, or simply because these anatomies are not covered by image acquisition. Automatic reconstruction of the missing anatomies benefits many applications, such as organ 3D bio-printing, whole-body segmentation, animation realism, paleoradiology and forensic imaging. We propose two paradigms based on a 3D denoising auto-encoder (DAE) to solve the anatomy reconstruction problem: (i) the DAE learns a many-to-one mapping between incomplete and complete instances; (ii) the DAE learns directly a one-to-one residual mapping between the incomplete instances and the target anatomies. We apply a loss aggregation scheme that enables the DAE to learn the many-to-one mapping more effectively and further enhances the learning of the residual mapping. On top of this, we extend the DAE to a multiclass completor by assigning a unique label to each anatomy involved. We evaluate our method using a CT dataset with whole-body segmentations. Results show that our method produces reasonable anatomy reconstructions given instances with different levels of incompleteness (i.e., one or multiple random anatomies are missing). Codes and pretrained models are publicly available at https://github.com/Jianningli/medshapenet-feedback/tree/main/anatomy-completor.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abadi, M., et al.: Tensorflow: large-scale machine learning on heterogeneous systems. https://www.tensorflow.org (2015)
Ali-Hamadi, D., et al.: Anatomy transfer. ACM Trans. Graph. (TOG) 32(6), 1–8 (2013)
Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3D shape reconstruction and completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6970–6981 (2020)
Egger, J., et al.: Medical deep learning-a systematic meta-review. Comput. Methods Programs Biomed. 221, 106874 (2022)
Goparaju, A., et al.: Benchmarking off-the-shelf statistical shape modeling tools in clinical applications. Med. Image Anal. 76, 102271 (2022)
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Jaus, A., et al.: Towards unifying anatomy segmentation: automated generation of a full-body CT dataset via knowledge aggregation and anatomical guidelines. arXiv preprint arXiv:2307.13375 (2023)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kodym, O., Španěl, M., Herout, A.: Skull shape reconstruction using cascaded convolutional networks. Comput. Biol. Med. 123, 103886 (2020)
Kodym, O., Španěl, M., Herout, A.: Deep learning for cranioplasty in clinical practice: going from synthetic to real patient data. Comput. Biol. Med. 137, 104766 (2021)
Kroviakov, A., Li, J., Egger, J.: Sparse convolutional neural network for skull reconstruction. In: Li, J., Egger, J. (eds.) AutoImplant 2021. LNCS, vol. 13123, pp. 80–94. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-92652-6_7
La Cava, S.M., Orrù, G., Goldmann, T., Drahansky, M., Marcialis, G.L.: 3D face reconstruction for forensic recognition-a survey. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 930–937. IEEE (2022)
Li, J., et al.: Towards clinical applicability and computational efficiency in automatic cranial implant design: an overview of the autoimplant 2021 cranial implant design challenge. Med. Image Anal. 102865 (2023)
Li, J., Pepe, A., Gsaxner, C., et al.: Medshapenet - a large-scale dataset of 3D medical shapes for computer vision. arXiv preprint arXiv:2308.16139 (2023)
Li, J., et al.: Autoimplant 2020-first MICCAI challenge on automatic cranial implant design. IEEE Trans. Med. Imaging 40(9), 2329–2342 (2021)
Li, J., et al.: Automatic skull defect restoration and cranial implant generation for cranioplasty. Med. Image Anal. 73, 102171 (2021)
Matzkin, F., et al.: Self-supervised skull reconstruction in brain CT images with decompressive craniectomy. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 390–399. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_38
Meng, D., Keller, M., Boyer, E., Black, M., Pujades, S.: Learning a statistical full spine model from partial observations. In: Reuter, M., Wachinger, C., Lombaert, H., Paniagua, B., Goksel, O., Rekik, I. (eds.) ShapeMI 2020. LNCS, vol. 12474, pp. 122–133. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61056-2_10
Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV), pp. 565–571. IEEE (2016)
Missal, S.: Forensic facial reconstruction of skeletonized and highly decomposed human remains. In: Forensic Genetic Approaches for Identification of Human Skeletal Remains, pp. 549–569. Elsevier (2023)
Morais, A., Egger, J., Alves, V.: Automated computer-aided design of cranial implants using a deep volumetric convolutional denoising autoencoder. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds.) WorldCIST’19 2019. AISC, vol. 932, pp. 151–160. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16187-3_15
Parihar, A., Pandita, V., Kumar, A., Parihar, D.S., Puranik, N., Bajpai, T., Khan, R.: 3D printing: advancement in biogenerative engineering to combat shortage of organs and bioapplicable materials. Regenerative Engineering and Translational Medicine, pp. 1–27 (2021)
Pimentel, P., et al.: Automated virtual reconstruction of large skull defects using statistical shape models and generative adversarial networks. In: Li, J., Egger, J. (eds.) AutoImplant 2020. LNCS, vol. 12439, pp. 16–27. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64327-0_3
Sarmad, M., Lee, H.J., Kim, Y.M.: RL-GAN-Net: a reinforcement learning agent controlled GAN network for real-time point cloud shape completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Seibold, C., et al.: Accurate fine-grained segmentation of human anatomy in radiographs via volumetric pseudo-labeling. arXiv preprint arXiv:2306.03934 (2023)
Seibold, C.M., Reiß, S., Kleesiek, J., Stiefelhagen, R.: Reference-guided pseudo-label generation for medical semantic segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2171–2179 (2022)
Shi, Y., Xu, X., Xi, J., Hu, X., Hu, D., Xu, K.: Learning to detect 3D symmetry from single-view RGB-D images with weak supervision. IEEE Trans. Pattern Analy. Mach. Intell. (2022)
Toscano, J.D., Zuniga-Navarrete, C., Siu, W.D.J., Segura, L.J., Sun, H.: Teeth mold point cloud completion via data augmentation and hybrid RL-GAN. J. Comput. Inf. Sci. Eng. 23(4), 041008 (2023)
Wasserthal, J., Meyer, M., Breit, H.C., Cyriac, J., Yang, S., Segeroth, M.: Totalsegmentator: robust segmentation of 104 anatomical structures in CT images. arXiv preprint arXiv:2208.05868 (2022)
Wilkinson, C.: Facial reconstruction-anatomical art or artistic anatomy? J. Anat. 216(2), 235–250 (2010)
Wodzinski, M., Daniol, M., Socha, M., Hemmerling, D., Stanuch, M., Skalski, A.: Deep learning-based framework for automatic cranial defect reconstruction and implant modeling. Comput. Methods Programs Biomed. 226, 107173 (2022)
Yan, X., Lin, L., Mitra, N.J., Lischinski, D., Cohen-Or, D., Huang, H.: Shapeformer: transformer-based shape completion via sparse representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6239–6249 (2022)
Zhang, Y., Pei, Y., Guo, Y., Chen, S., Xu, T., Zha, H.: Cleft volume estimation and maxilla completion using cascaded deep neural networks. In: Liu, M., Yan, P., Lian, C., Cao, X. (eds.) MLMI 2020. LNCS, vol. 12436, pp. 332–341. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59861-7_34
Acknowledgement
The work is supported by the Plattform für KI-Translation Essen (KITE) from the REACT-EU initiative (EFRE-0801977, https://kite.ikim.nrw/) and “NUM 2.0” (FKZ: 01KX2121) FWF enFaced 2.0 (KLI 1044). The anatomical shape dataset used in this paper can be accessed through MedShapeNet at https://medshapenet.ikim.nrw/.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendix A. Reconstructing Small Anatomies
(see Fig. 6)
Reconstruction results of individual, small anatomies by \(DAE_{agg+res}\) trained with an increased loss aggregation scope (M). From the top: heart (2.4%), spine (4.3%), kidney (1.7%) and spleen (1.2%). The percentages in the brackets are the approximate volume ratio of the anatomy to the corresponding whole-body segmentations. The preliminary results demonstrate that increasing M (in Eq. 3, 4 in the main manuscript) increases also the sensitivity of the reconstructive model, which helps the model identify and reconstruct very small anatomies. Two test instances are presented for each anatomy class.
Appendix B. Anatomy Completion from Skeletons (rib cage + spine)
(see Fig. 7)
The first row shows the input skeleton (ribs and spine), and the second to third row show the reconstruction results in axial and coronal views, respectively. The results are obtained by training \(DAE_{res}\) on 40 such ‘skeleton-full’ pairs for 200 epochs. The preliminary results demonstrate the feasibility of reconstructing the full anatomy based only on the skeleton.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, J., Pepe, A., Luijten, G., Schwarz-Gsaxner, C., Kleesiek, J., Egger, J. (2023). Anatomy Completor: A Multi-class Completion Framework for 3D Anatomy Reconstruction. In: Wachinger, C., Paniagua, B., Elhabian, S., Li, J., Egger, J. (eds) Shape in Medical Imaging. ShapeMI 2023. Lecture Notes in Computer Science, vol 14350. Springer, Cham. https://doi.org/10.1007/978-3-031-46914-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-46914-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46913-8
Online ISBN: 978-3-031-46914-5
eBook Packages: Computer ScienceComputer Science (R0)
