pix2xray: converting RGB images into X-rays using generative adversarial networks

Abstract

Purpose

We propose a novel methodology for generating synthetic X-rays from 2D RGB images. This method creates accurate simulations for use in non-diagnostic visualization problems where the only input comes from a generic camera. Traditional methods are restricted to using simulation algorithms on 3D computer models. To solve this problem, we propose a method of synthetic X-ray generation using conditional generative adversarial networks (CGANs).

Methods

We create a custom synthetic X-ray dataset generator to generate image triplets for X-ray images, pose images, and RGB images of natural hand poses sampled from the NYU hand pose dataset. This dataset is used to train two general-purpose CGAN networks, pix2pix and CycleGAN, as well as our novel architecture called pix2xray which expands upon the pix2pix architecture to include the hand pose into the network.

Results

Our results demonstrate that our pix2xray architecture outperforms both pix2pix and CycleGAN in producing higher-quality X-ray images. We measure higher similarity metrics in our approach, with pix2pix coming in second, and CycleGAN producing the worst results. Our network performs better in the difficult cases which involve high occlusion due to occluded poses or large rotations.

Conclusion

Overall our work establishes a baseline that synthetic X-rays can be simulated using 2D RGB input. We establish the need for additional data such as the hand pose to produce clearer results and show that future research must focus on more specialized architectures to improve overall image clarity and structure.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. 1.

    Bergbäck Knudsen E, Prodi A, Baltser J, Thomsen M, Kjær Willendrup P, Sanchez del Rio M, Ferrero C, Farhi E, Haldrup K, Vickery A, Feidenhans’l R, Mortensen K, Friis Poulsen H, Schmidt S, Lefmann K (2013) McXtrace: a Monte Carlo software package for simulating X-ray optics, beamlines and experiments. J Appl Crystallogr 46(3):679–696. https://doi.org/10.1107/S0021889813007991

    CAS  Article  Google Scholar 

  2. 2.

    Freud N, Duvauchelle P, Létang JM, Babot D (2006) Fast and robust ray casting algorithms for virtual X-ray imaging. Nucl Instrum Methods Phys Res, Sect B 248(1):175–180. https://doi.org/10.1016/j.nimb.2006.03.009

    CAS  Article  Google Scholar 

  3. 3.

    Vidal F P, Garnier M, Freud N, Létang J M, John N W (2009) Simulation of X-ray attenuation on the GPU. In: Proceedings of theory and practice of computer graphics. 15

  4. 4.

    Villard P-F, Vidal FP, Hunt C, Bello F, John NW, Johnson S, Gould DA (2009) A prototype percutaneous transhepatic cholangiography training simulator with real-time breathing motion. Int J Comput Assist Radiol Surg 4(6):571–578. https://doi.org/10.1007/s11548-009-0367-1

    CAS  Article  PubMed  Google Scholar 

  5. 5.

    Isola P, Zhu J-Y, Zhou T, Efros A A (2017) Image-to-image translation with conditional adversarial networks (2017) IEEE Conference on computer vision and pattern recognition (CVPR), pp 5967–5976. https://doi.org/10.1109/CVPR.2017.632

  6. 6.

    Goodfellow, I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. arXiv:1406.2661 [Cs, Stat]

  7. 7.

    Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv:1411.1784 [Cs, Stat]

  8. 8.

    Madani A, Moradi M, Karargyris A, Syeda-Mahmood T (2018) Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation. In: 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018), pp 1038–1042. https://doi.org/10.1109/ISBI.2018.8363749

  9. 9.

    Ying X, Guo H, Ma K, Wu J, Weng Z, Zheng Y (2019) X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks. arXiv:1905.06902 [Cs, Eess]

  10. 10.

    Sun Y, Liu X, Cong P, Li L, Zhao Z (2018) Digital radiography image denoising using a generative adversarial network. J X-Ray Sci Technol 26(4):523–534. https://doi.org/10.3233/XST-17356

    Article  Google Scholar 

  11. 11.

    Vidal FP, Villard P-F (2016) Development and validation of real-time simulation of X-ray imaging with respiratory motion. Comput Med Imaging Graph 49:1–15. https://doi.org/10.1016/j.compmedimag.2015.12.002

    Article  PubMed  Google Scholar 

  12. 12.

    Tompson J, Stein M, Lecun Y, Perlin K (2014) Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans Gr 33(5):1–10. https://doi.org/10.1145/2629500

    Article  Google Scholar 

  13. 13.

    Wan T-C, Liu M-Y, Zhu J-Y, Tao A, Kautz J, Catanzaro B (2018) High-resolution image synthesis and semantic manipulation with conditional GANs. In: IEEE/CVF Conference on computer vision and pattern recognition 2018. pp 8798–8807. https://doi.org/10.1109/CVPR.2018.00917

  14. 14.

    Zhu J-Y, Park T, Isola P, Efros A A (2018) Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593 [Cs]

  15. 15.

    He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv:1512.03385 [Cs]

  16. 16.

    Ronneberger O, Fischer P, Brox, T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv:1505.04597 [Cs]

  17. 17.

    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  PubMed  Google Scholar 

  18. 18.

    Tmenova O, Martin R, Duong L (2019) CycleGAN for style transfer in X-ray angiography. Int J Comput Assist Radiol Surg 14(10):1785–1794. https://doi.org/10.1007/s11548-019-02022-z

    Article  PubMed  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mustafa Haiderbhai.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Statements

This article does not contain any studies with human participants or animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Haiderbhai, M., Ledesma, S., Lee, S.C. et al. pix2xray: converting RGB images into X-rays using generative adversarial networks. Int J CARS 15, 973–980 (2020). https://doi.org/10.1007/s11548-020-02159-2

Download citation

Keywords

  • Conditional generative adversarial networks
  • Image translation
  • X-rays
  • DRR
  • Machine learning