We propose a novel methodology for generating synthetic X-rays from 2D RGB images. This method creates accurate simulations for use in non-diagnostic visualization problems where the only input comes from a generic camera. Traditional methods are restricted to using simulation algorithms on 3D computer models. To solve this problem, we propose a method of synthetic X-ray generation using conditional generative adversarial networks (CGANs).
We create a custom synthetic X-ray dataset generator to generate image triplets for X-ray images, pose images, and RGB images of natural hand poses sampled from the NYU hand pose dataset. This dataset is used to train two general-purpose CGAN networks, pix2pix and CycleGAN, as well as our novel architecture called pix2xray which expands upon the pix2pix architecture to include the hand pose into the network.
Our results demonstrate that our pix2xray architecture outperforms both pix2pix and CycleGAN in producing higher-quality X-ray images. We measure higher similarity metrics in our approach, with pix2pix coming in second, and CycleGAN producing the worst results. Our network performs better in the difficult cases which involve high occlusion due to occluded poses or large rotations.
Overall our work establishes a baseline that synthetic X-rays can be simulated using 2D RGB input. We establish the need for additional data such as the hand pose to produce clearer results and show that future research must focus on more specialized architectures to improve overall image clarity and structure.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Bergbäck Knudsen E, Prodi A, Baltser J, Thomsen M, Kjær Willendrup P, Sanchez del Rio M, Ferrero C, Farhi E, Haldrup K, Vickery A, Feidenhans’l R, Mortensen K, Friis Poulsen H, Schmidt S, Lefmann K (2013) McXtrace: a Monte Carlo software package for simulating X-ray optics, beamlines and experiments. J Appl Crystallogr 46(3):679–696. https://doi.org/10.1107/S0021889813007991
Freud N, Duvauchelle P, Létang JM, Babot D (2006) Fast and robust ray casting algorithms for virtual X-ray imaging. Nucl Instrum Methods Phys Res, Sect B 248(1):175–180. https://doi.org/10.1016/j.nimb.2006.03.009
Vidal F P, Garnier M, Freud N, Létang J M, John N W (2009) Simulation of X-ray attenuation on the GPU. In: Proceedings of theory and practice of computer graphics. 15
Villard P-F, Vidal FP, Hunt C, Bello F, John NW, Johnson S, Gould DA (2009) A prototype percutaneous transhepatic cholangiography training simulator with real-time breathing motion. Int J Comput Assist Radiol Surg 4(6):571–578. https://doi.org/10.1007/s11548-009-0367-1
Isola P, Zhu J-Y, Zhou T, Efros A A (2017) Image-to-image translation with conditional adversarial networks (2017) IEEE Conference on computer vision and pattern recognition (CVPR), pp 5967–5976. https://doi.org/10.1109/CVPR.2017.632
Goodfellow, I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. arXiv:1406.2661 [Cs, Stat]
Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv:1411.1784 [Cs, Stat]
Madani A, Moradi M, Karargyris A, Syeda-Mahmood T (2018) Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation. In: 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018), pp 1038–1042. https://doi.org/10.1109/ISBI.2018.8363749
Ying X, Guo H, Ma K, Wu J, Weng Z, Zheng Y (2019) X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks. arXiv:1905.06902 [Cs, Eess]
Sun Y, Liu X, Cong P, Li L, Zhao Z (2018) Digital radiography image denoising using a generative adversarial network. J X-Ray Sci Technol 26(4):523–534. https://doi.org/10.3233/XST-17356
Vidal FP, Villard P-F (2016) Development and validation of real-time simulation of X-ray imaging with respiratory motion. Comput Med Imaging Graph 49:1–15. https://doi.org/10.1016/j.compmedimag.2015.12.002
Tompson J, Stein M, Lecun Y, Perlin K (2014) Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans Gr 33(5):1–10. https://doi.org/10.1145/2629500
Wan T-C, Liu M-Y, Zhu J-Y, Tao A, Kautz J, Catanzaro B (2018) High-resolution image synthesis and semantic manipulation with conditional GANs. In: IEEE/CVF Conference on computer vision and pattern recognition 2018. pp 8798–8807. https://doi.org/10.1109/CVPR.2018.00917
Zhu J-Y, Park T, Isola P, Efros A A (2018) Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593 [Cs]
He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv:1512.03385 [Cs]
Ronneberger O, Fischer P, Brox, T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv:1505.04597 [Cs]
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861
Tmenova O, Martin R, Duong L (2019) CycleGAN for style transfer in X-ray angiography. Int J Comput Assist Radiol Surg 14(10):1785–1794. https://doi.org/10.1007/s11548-019-02022-z
Conflicts of interest
The authors declare that they have no conflict of interest.
This article does not contain any studies with human participants or animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Haiderbhai, M., Ledesma, S., Lee, S.C. et al. pix2xray: converting RGB images into X-rays using generative adversarial networks. Int J CARS 15, 973–980 (2020). https://doi.org/10.1007/s11548-020-02159-2
- Conditional generative adversarial networks
- Image translation
- Machine learning