Skip to main content
SpringerLink
Log in

Deep neural network-based synthetic image digital fluoroscopy using digitally reconstructed tomography

  • Scientific Paper
  • Published:
Physical and Engineering Sciences in Medicine Aims and scope Submit manuscript

Abstract

We developed a deep neural network (DNN) to generate X-ray flat panel detector (FPD) images from digitally reconstructed radiographic (DRR) images. FPD and treatment planning CT images were acquired from patients with prostate and head and neck (H&N) malignancies. The DNN parameters were optimized for FPD image synthesis. The synthetic FPD images’ features were evaluated to compare to the corresponding ground-truth FPD images using mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). The image quality of the synthetic FPD image was also compared with that of the DRR image to understand the performance of our DNN. For the prostate cases, the MAE of the synthetic FPD image was improved (= 0.12 ± 0.02) from that of the input DRR image (= 0.35 ± 0.08). The synthetic FPD image showed higher PSNRs (= 16.81 ± 1.54 dB) than those of the DRR image (= 8.74 ± 1.56 dB), while SSIMs for both images (= 0.69) were almost the same. All metrics for the synthetic FPD images of the H&N cases were improved (MAE 0.08 ± 0.03, PSNR 19.40 ± 2.83 dB, and SSIM 0.80 ± 0.04) compared to those for the DRR image (MAE 0.48 ± 0.11, PSNR 5.74 ± 1.63 dB, and SSIM 0.52 ± 0.09). Our DNN successfully generated FPD images from DRR images. This technique would be useful to increase throughput when images from two different modalities are compared by visual inspection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked Denoising Autoencoders: learning useful representations in a Deep Network with a local Denoising Criterion. J Mach Learn Res 11:3371–3408

    Google Scholar 

  2. Jonathan M, Ueli M, Dan C, Jürgen S (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. International Conference on Artificial Neural Networks. Springer, Heidelberg, pp 52-9

  3. Yang W, Chen Y, Liu Y, Zhong L, Qin G, Lu Z et al (2017) Cascade of multi-scale convolutional neural networks for bone suppression of chest radiographs in gradient domain. Med Image Anal 35:421–433

    Article  PubMed  Google Scholar 

  4. Dong X, Lei Y, Wang TH, Thomas M, Tang L, Curran WJ et al (2019) Automatic multiorgan segmentation in thorax CT images using U-net-GAN. Med Phys 46:2157–2168

    Article  PubMed Central  PubMed  Google Scholar 

  5. Wang J, Lu J, Qin G, Shen L, Sun Y, Ying H et al (2018) Technical note: a deep learning-based autosegmentation of rectal tumors in MR images. Med Phys 45:2560–2564

    Article  PubMed  Google Scholar 

  6. Shen CY, Nguyen D, Chen LY, Gonzalez Y, McBeth R, Qin N et al (2020) Operating a treatment planning system using a deep-reinforcement learning-based virtual treatment planner for prostate cancer intensity-modulated radiation therapy treatment planning. Med Phys 47:2329–2336

    Article  CAS  PubMed  Google Scholar 

  7. Liang X, Chen L, Nguyen D, Zhou Z, Gu X, Yang M et al (2019) Generating synthesized computed tomography (CT) from cone-beam computed tomography (CBCT) using CycleGAN for adaptive radiation therapy. Phys Med Biol 64:125002

    Article  CAS  PubMed  Google Scholar 

  8. de Vos BD, Berendsen FF, Viergever MA, Sokooti H, Staring M, Isgum I (2019) A deep learning framework for unsupervised affine and deformable image registration. Med Image Anal 52:128–143

    Article  PubMed  Google Scholar 

  9. Mori S, Hirai R, Sakata Y (2020) Simulated four-dimensional CT for markerless tumor tracking using a deep learning network with multi-task learning. Phys Med 80:151–158

    Article  PubMed  Google Scholar 

  10. Hirai R, Sakata Y, Tanizawa A, Mori S (2019) Real-time tumor tracking using fluoroscopic imaging with deep neural network analysis. Phys Med 59:22–29

    Article  PubMed  Google Scholar 

  11. Takahashi W, Oshikawa S, Mori S (2020) Real-time markerless tumour tracking with patient-specific deep learning using a personalised data generation strategy: proof of concept by phantom study. Br J Radiol 93:20190420

    Article  PubMed Central  PubMed  Google Scholar 

  12. Mori S, Shirai T, Takei Y, Furukawa T, Inaniwa T, Matsuzaki Y et al (2012) Patient handling system for carbon ion beam scanning therapy. J Appl Clin Med Phys 13:3926

    Article  PubMed  Google Scholar 

  13. Mori S, Kumagai M, Miki K, Fukuhara R, Haneishi H (2015) Development of fast patient position verification software using 2D-3D image registration and its clinical experience. J Radiat Res 56:818–829

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  14. Chang Z, Wang Z, Ma J, O’Daniel JC, Kirkpatrick J, Yin FF (2010) 6D image guidance for spinal non-invasive stereotactic body radiation therapy: comparison between ExacTrac X-ray 6D with kilo-voltage cone-beam CT. Radiother Oncol 95:116–121

    Article  PubMed  Google Scholar 

  15. Penney GP, Weese J, Little JA, Desmedt P, Hill DL, Hawkes DJ (1998) A comparison of similarity measures for use in 2-D-3-D medical image registration. IEEE Trans Med Imaging 17:586–595

    Article  CAS  PubMed  Google Scholar 

  16. Kang E, Min J, Ye JC (2017) A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction. Med Phys 44:e360–e75

    Article  CAS  PubMed  Google Scholar 

  17. Kyong Hwan J, McCann MT, Froustey E, Unser M (2017) Deep convolutional neural network for inverse problems in imaging. IEEE Trans Image Process 26:4509–4522

    Article  Google Scholar 

  18. Gong E, Pauly JM, Wintermark M, Zaharchuk G (2018) Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI. J Magn Reson Imaging 48:330–340

    Article  PubMed  Google Scholar 

  19. Chartsias A, Joyce T, Giuffrida MV, Tsaftaris SA (2018) Multimodal MR Synthesis via modality-invariant latent representation. IEEE Trans Med Imaging 37:803–814

    Article  PubMed  Google Scholar 

  20. Dong X, Lei Y, Wang T, Higgins K, Liu T, Curran WJ et al (2020) Deep learning-based attenuation correction in the absence of structural information for whole-body positron emission tomography imaging. Phys Med Biol 65:055011

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  21. Yang J, Park D, Gullberg GT, Seo Y (2019) Joint correction of attenuation and scatter in image space using deep convolutional neural networks for dedicated brain F-18-FDG PET. Phys Med Biol. ;64

  22. Khoo VS, Joon DL (2006) New developments in MRI for target volume delineation in radiotherapy. Br J Radiol 79(Spec No 1):S2–15

    Article  PubMed  Google Scholar 

  23. Nie D, Trullo R, Lian J, Wang L, Petitjean C, Ruan S et al (2018) Medical Image Synthesis with Deep Convolutional Adversarial Networks. IEEE Trans Biomed Eng 65:2720–2730

    Article  PubMed Central  PubMed  Google Scholar 

  24. Arabi H, Dowling JA, Burgos N, Han X, Greer PB, Koutsouvelis N et al (2018) Comparative study of algorithms for synthetic CT generation from MRI: consequences for MRI-guided radiation planning in the pelvic region. Med Phys 45:5218–5233

    Article  PubMed  Google Scholar 

  25. Shafai-Erfani G, Lei Y, Liu Y, Wang Y, Wang T, Zhong J et al (2019) MRI-Based Proton Treatment Planning for Base of Skull Tumors. Int J Part Ther 6:12–25

    Article  PubMed Central  PubMed  Google Scholar 

  26. McKenzie EM, Santhanam A, Ruan D, O’Connor D, Cao M, Sheng K (2020) Multimodality image registration in the head-and-neck using a deep learning-derived synthetic CT as a bridge. Med Phys 47:1094–1104

    Article  PubMed  Google Scholar 

  27. Lei Y, Dong X, Tian Z, Liu Y, Tian S, Wang T et al (2020) CT prostate segmentation based on synthetic MRI-aided deep attention fully convolution network. Med Phys 47:530–540

    Article  PubMed  Google Scholar 

  28. Dong X, Wang T, Lei Y, Higgins K, Liu T, Curran WJ et al (2019) Synthetic CT generation from non-attenuation corrected PET images for whole-body PET imaging. Phys Med Biol 64:215016

    Article  PubMed Central  PubMed  Google Scholar 

  29. Landry G, Hansen D, Kamp F, Li M, Hoyle B, Weller J et al (2019) Comparing unet training with three different datasets to correct CBCT images for prostate radiotherapy dose calculations. Phys Med Biol 64:035011

    Article  PubMed  Google Scholar 

  30. Yuan N, Dyer B, Rao S, Chen Q, Benedict S, Shang L et al (2020) Convolutional neural network enhancement of fast-scan low-dose cone-beam CT images for head and neck radiotherapy. Phys Med Biol 65:035003

    Article  PubMed Central  PubMed  Google Scholar 

  31. Kim J, Lee JK, Lee KM (2016) Accurateimagesuper-resolutionusing very deep convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition

  32. Papadakis AE, Perisinakis K, Oikonomou I, Damilakis J (2011) Automatic exposure control in pediatric and adult computed tomography examinations: can we estimate organ and effective dose from mean MAS reduction? Invest Radiol 46:654–662

    Article  PubMed  Google Scholar 

  33. Mori S, Inaniwa T, Kumagai M, Kuwae T, Matsuzaki Y, Furukawa T et al (2012) Development of digital reconstructed radiography software at new treatment facility for carbon-ion beam scanning of National Institute of Radiological Sciences. Australas Phys Eng Sci Med 35:221–229

    Article  PubMed  Google Scholar 

  34. Vasiliadis G, Antonatos S, Polychronakis M, Markatos E, Gnort IS (2008) High Performance Network Intrusion Detection Using Graphics Processors. Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (RAID), pp 116–34

  35. Mori S, Shibayama K, Tanimoto K, Kumagai M, Matsuzaki Y, Furukawa T et al (2012) First clinical experience in carbon ion scanning beam therapy: retrospective analysis of patient positional accuracy. J Radiat Res 53:760–768

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  36. Badrinarayanan V, Kendall A, Cipolla R (2017) SegNet: A deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans Pattern Anal Mach Intell 39:2481–2495

    Article  PubMed  Google Scholar 

  37. Ulyanov D, Vedaldi A, Lempitsky VS (2016) Instance normalization: the missing ingredient for fast stylization. ArXiv. ;abs/1607.08022.

  38. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov N (2014) Dropout: A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    Google Scholar 

  39. Sergey I, Christian S (2015) Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. The 32nd International Conference on Machine Learning. :448 – 56

  40. Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted intervention (MICCAI). Springer, Munich, pp 234–241

    Google Scholar 

  41. Radford A, Metz L, Chintala S (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:150806576.

  42. Hwang D, Kim KY, Kang SK, Seo S, Paeng JC, Lee DS et al (2018) Improving the Accuracy of simultaneously reconstructed activity and attenuation maps using deep learning. J Nucl Med 59:1624–1629

    Article  CAS  PubMed  Google Scholar 

  43. Simonya K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. International Conference for Learning Representations

  44. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612

    Article  PubMed  Google Scholar 

  45. Yochai B, Tomer M (2018) The Perception-Distortion Tradeoff. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 228-37

  46. Emami H, Dong M, Nejad-Davarani SP, Glide-Hurst CK (2018) Generating synthetic CTs from magnetic resonance images using generative adversarial networks. Med Phys.

  47. Sakata Y, Hirai R, Kobuna K, Tanizawa A, Mori S (2020) A machine learning-based real-time tumor tracking system for fluoroscopic gating of lung radiotherapy. Phys Med Biol 65:085014

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We wish to thank the staff of the Medical Physics Department of the National Institutes for Quantum Science and Technology, and of our institute hospitals, for their support and discussion. We thank Libby Cone, MD, MA, from DMC Corp. (www.dmed.co.jp) for editing drafts of this manuscript.

Funding

None.

Author information

Authors and Affiliations

  1. National Institutes for Quantum Science and Technology, Quantum Life and Medical Science Directorate, Institute for Quantum Medical Science, Inage-ku, Chiba, 263-8555, Japan

    Shinichiro Mori & Yasuhiko Tachibana

  2. Corporate Research and Development Center, Toshiba Corporation, Kanagawa, 212-8582, Japan

    Ryusuke Hirai & Yukinobu Sakata

  3. QST hospital, National Institutes for Quantum Science and Technology, Inage-ku, Chiba, 263-8555, Japan

    Masashi Koto & Hitoshi Ishikawa

Authors
  1. Shinichiro Mori
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryusuke Hirai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yukinobu Sakata
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yasuhiko Tachibana
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Masashi Koto
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hitoshi Ishikawa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shinichiro Mori.

Ethics declarations

Conflict of interest

Drs. Hirai and Sakata are employed by the Toshiba Corporation, Kawasaki, Japan.

Ethical approval

The study was approved by the Institutional Review Board of our institution (N21-001).

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mori, S., Hirai, R., Sakata, Y. et al. Deep neural network-based synthetic image digital fluoroscopy using digitally reconstructed tomography. Phys Eng Sci Med 46, 1227–1237 (2023). https://doi.org/10.1007/s13246-023-01290-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13246-023-01290-z

Keywords

Search

Navigation