Abstract
Computed tomography (CT) is commonly used for fracture diagnosis because it provides accurate visualization of shape with 3-dimensional(3D) structure. However, CT has some disadvantages such as the high dose of radiation involved in scanning, and relatively high expense compared to X-ray. Also, it is difficult to scan CT in the operation room despite it is necessary to check 3D structure during operation. On the other hand, X-ray is often used in operating rooms because it is relatively simple to scan. However, since X-ray only provides overlapped 2D images, surgeons should rely on 2D images to imagine 3D structure of a target shape. If we can create a 3D structure from a single 2D X-ray image, then it will be clinically valuable. Therefore, we propose Skip-StyleGAN that can efficiently generate rotated images of a given 2D image from 3D rendered shape. Based on the StyleGAN, we arrange training sequence and add skip-connection from the discriminator to the generator. Important discriminative information is transferred through this skip-connection, and it allows the generator to easily produce an appropriately rotated image by making a little variation during the training process. With the effect of skip-connection, Skip-StyleGAN can efficiently generate high-quality 3D rendered images even with small-sized data. Our experiments show that the proposed model successfully generates 3D rendered images of the hand bone complex.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cerrolaza, J.J., et al.: 3D fetal skull reconstruction from 2dus via deep conditional generative networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 383–391. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_44
Cui, J., Li, S., Xia, Q., Hao, A., Qin, H.: Learning multi-view manifold for single image based modeling. Comput. Graph. 82, 275–285 (2019)
Dziugaite, G.K., Roy, D.M., Ghahramani, Z.: Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906 (2015)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems. pp. 2672–2680 (2014)
Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., Smola, A.J.: A kernel method for the two-sample-problem. In: Advances in Neural Information Processing Systems. pp. 513–520 (2007)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: Advances in Neural Information Processing Systems. pp. 5767–5777 (2017)
Huang, R., Zhang, S., Li, T., He, R.: Beyond face rotation: global and local perception gan for photorealistic and identity preserving frontal view synthesis. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2439–2448 (2017)
Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 172–189 (2018)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4401–4410 (2019)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
Moniz, J.R.A., Beckham, C., Rajotte, S., Honari, S., Pal, C.: Unsupervised depth estimation, 3d face rotation and replacement. In: Advances in Neural Information Processing Systems. pp. 9736–9746 (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Tian, Y., Peng, X., Zhao, L., Zhang, S., Metaxas, D.N.: Cr-gan: learning complete representations for multi-view generation. arXiv preprint arXiv:1806.11191 (2018)
Tran, L., Yin, X., Liu, X.: Disentangled representation learning gan for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1415–1424 (2017)
Whitmarsh, T., Humbert, L., De Craene, M., Barquero, L.M.D.R., Frangi, A.F.: Reconstructing the 3d shape and bone mineral density distribution of the proximal femur from dual-energy x-ray absorptiometry. IEEE Trans. Med. Imaging 30(12), 2101–2114 (2011)
Yin, X., Yu, X., Sohn, K., Liu, X., Chandraker, M.: Towards large-pose face frontalization in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3990–3999 (2017)
Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3d solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 146–155 (2016)
Acknowledgement
This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (2016-0-00564, Development of Intelligent Interaction Technology Based on Context Awareness and Human Intention Understanding) and Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (2018-0-00861, Intelligent SW Technology Development for Medical Data Analysis).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ahn, J., Lee, HJ., Choi, I., Lee, M. (2020). Skip-StyleGAN: Skip-Connected Generative Adversarial Networks for Generating 3D Rendered Image of Hand Bone Complex. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12264. Springer, Cham. https://doi.org/10.1007/978-3-030-59719-1_72
Download citation
DOI: https://doi.org/10.1007/978-3-030-59719-1_72
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59718-4
Online ISBN: 978-3-030-59719-1
eBook Packages: Computer ScienceComputer Science (R0)
