Skip to main content
SpringerLink
Log in

Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging

  • Published:
Radiological Physics and Technology Aims and scope Submit manuscript

Abstract

Since the advent of deep convolutional neural networks (DNNs), computer vision has seen an extremely rapid progress that has led to huge advances in medical imaging. Every year, many new methods are reported at conferences such as the International Conference on Medical Image Computing and Computer-Assisted Intervention and Machine Learning for Medical Image Reconstruction, or published online at the preprint server arXiv. There is a plethora of surveys on applications of neural networks in medical imaging (see [1] for a relatively recent comprehensive survey). This article does not aim to cover all aspects of the field, but focuses on a particular topic, image-to-image translation. Although the topic may not sound familiar, it turns out that many seemingly irrelevant applications can be understood as instances of image-to-image translation. Such applications include (1) noise reduction, (2) super-resolution, (3) image synthesis, and (4) reconstruction. The same underlying principles and algorithms work for various tasks. Our aim is to introduce some of the key ideas on this topic from a uniform viewpoint. We introduce core ideas and jargon that are specific to image processing by use of DNNs. Having an intuitive grasp of the core ideas of applications of neural networks in medical imaging and a knowledge of technical terms would be of great help to the reader for understanding the existing and future applications. Most of the recent applications which build on image-to-image translation are based on one of two fundamental architectures, called pix2pix and CycleGAN, depending on whether the available training data are paired or unpaired (see Sect. 1.3). We provide codes ([2, 3]) which implement these two architectures with various enhancements. Our codes are available online with use of the very permissive MIT license. We provide a hands-on tutorial for training a model for denoising based on our codes (see Sect. 6). We hope that this article, together with the codes, will provide both an overview and the details of the key algorithms and that it will serve as a basis for the development of new applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Various image filters are implemented in the free software Fiji [7], and we can easily try them out to see their characteristics.

  2. Usually, a linear layer involves the constant term as well, so that it has the form \(x \mapsto Ax + b\) for \(b\in {\mathbb {R}}^m\). The term b is often referred to as the bias.

  3. Around the same time, very similar architectures such as UNIT, DiscoGAN, and DualGAN were introduced. Walender et al. [24] evaluated UNIT [25] and CycleGAN for transformation between T1- and T2-weighted MRI images and showed that these two frameworks performed almost equally well.

References

  1. Sahiner B, Pezeshk A, Hadjiiski LM, Wang X, Drukker K, Cha KH, Summers RM, Giger ML. Deep learning in medical imaging and radiation therapy. Med phys. 2018;46:e1–36.

    Article  PubMed  Google Scholar 

  2. Kaji S. Image translation by CNNs trained on unpaired data. 2019. https://github.com/shizuo-kaji/UnpairedImageTranslation. Accessed 18 June 2019.

  3. Kaji S. Image translation for paired image datasets (automap + pix2pix). 2019. https://github.com/shizuo-kaji/PairedImageTranslation. Accessed 18 June 2019.

  4. Lu L, Zheng Y, Carneiro G, Yang L, editors. Deep learning and convolutional neural networks for medical image computing—precision medicine, high performance and large-scale datasets. Advances in computer vision and pattern recognition. Springer; 2017.

  5. Knoll F, Maier AK, Rueckert D. editors. Machine learning for medical image reconstruction—first international workshop, MLMIR 2018, held in conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings, vol. 11074 of Lecture Notes in Computer Science, Springer; 2018.

  6. Litjens GJS, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak J, van Ginneken B, Sánchez CI. A survey on deep learning in medical image analysis. Med Image Anal. 2017;42:60–88.

    Article  PubMed  Google Scholar 

  7. Schindelin J, Arganda-Carreras I, Frise E, Kaynig V, Longair M, Pietzsch T, Preibisch S, Rueden C, Saalfeld S, Schmid B, Tinevez J-Y, White DJ, Hartenstein V, Eliceiri K, Tomancak P, Cardona A. Fiji: an open-source platform for biological-image analysis. Nat Meth. 2012;9:676–82.

    Article  CAS  Google Scholar 

  8. Nielsen MA. Neural networks and deep learning. Determination Press; 2018.

  9. Goodfellow IJ, Bengio Y, Courville A. Deep learning. Cambridge, MA, USA: MIT Press; 2016 http://www.deeplearningbook.org.

  10. Safran I, Shamir O. Depth-width tradeoffs in approximating natural functions with neural networks. In: Proceedings of the 34th international conference on machine learning, Vol 70, ICML’17, p. 2979–87, JMLR.org; 2017.

  11. Scarselli F, Tsoi AC. Universal approximation using feedforward neural networks: a survey of some existing methods, and some new results. Neural Netw. 1998;11:15–37.

    Article  PubMed  Google Scholar 

  12. Lu Z, Pu H, Wang F, Hu Z, Wang L. The expressive power of neural networks: a view from the width. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, editors. Advances in neural information processing systems 30. Red HookRed Hook: Curran Associates, Inc.; 2017. p. 6231–9.

    Google Scholar 

  13. Dumoulin V, Visin F. A guide to convolution arithmetic for deep learning, 2016. arXiv:1603.07285.

  14. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016; 2016. p. 1874–1883.

  15. Odena A, Dumoulin V, Olah C. Deconvolution and checkerboard artifacts. Distill; 2016.

  16. Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach FR, Blei DM (eds) ICML of JMLR workshop and conference proceedings, vol 37. USA: JMLR.org; 2015. p. 448–56.

    Google Scholar 

  17. Ulyanov D, Vedaldi A, Lempitsky VS. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21–26, 2017; 2017. p. 4105–13.

  18. Fredrikson M, Jha S, Ristenpart T. Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22Nd ACM SIGSAC conference on computer and communications security, CCS ’15, (New York, NY, USA); 2015. p. 1322–33, ACM.

  19. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention—MICCAI 2015—18th international conference Munich, Germany, October 5 - 9, 2015, Proceedings, Part III; 2015. p. 234–41.

  20. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ, editors. Advances in neural information processing systems 27. Curran Associates, Inc.; 2014. p. 2672–80.

  21. Yi X. Awesome GAN for medical imaging. 2019. https://github.com/xinario/awesome-gan-for-medical-imaging.

  22. Isola P, Zhu J, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017; 2017. p. 5967–76.

  23. Zhu J, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, 2017; 2017. p. 2242–51.

  24. Welander P, Karlsson S, Eklund A. Generative adversarial networks for image-to-image translation on multi-contrast MR images—a comparison of CycleGAN and UNIT, 2018. arXiv:1806.07777.

  25. Liu M-Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, editors. Advances in neural information processing systems 30. Red Hook: Curran Associates, Inc.; 2017. p. 700–8.

    Google Scholar 

  26. Gatys LA, Ecker AS, Bethge M. Image style transfer using convolutional neural networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR); June 2016. p. 2414–23.

  27. Zhu BO, Liu JZ, Rosen BR, Rosen MS. Image reconstruction by domain-transform manifold learning. Nature. 2018;555:487–92.

    Article  CAS  PubMed  Google Scholar 

  28. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC. Improved training of Wasserstein GANs. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, editors. Advances in neural information processing systems 30. Red Hook: Curran Associates, Inc.; 2017. p. 5767–77.

    Google Scholar 

  29. Karras T, Aila T, Laine S, Lehtinen J. Progressive growing of GANs for improved quality, stability, and variation. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference track proceedings; 2018.

  30. Miyato T, Kataoka T, Koyama M, Yoshida Y. Spectral normalization for generative adversarial networks. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference track proceedings; 2018.

  31. Sinyu J. CT image denoising with deep learning. 2018. https://github.com/SSinyu/CT_DENOISING_REVIEW. Accessed 18 June 2019.

  32. Chen H, Zhang Y, Zhang W, Liao P, Li K, Zhou J, Wang G. Low-dose CT via convolutional neural network. Biomed Opt Express. 2017;8:679–94.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Yang Q, Yan P, Zhang Y, Yu H, Shi Y, Mou X, Kalra MK, Zhang Y, Sun L, Wang G. Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss. IEEE Trans Med Imaging. 2018;37:1348–57.

    Article  PubMed  PubMed Central  Google Scholar 

  34. You C, Yang Q, Shan H, Gjesteby L, Li G, Ju S, Zhang Z, Zhao Z, Zhang Y, Cong W, Wang G. Structurally-sensitive multi-scale deep neural network for low-dose CT denoising. IEEE Access. 2018;6:41839–55.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Yi X, Babyn P. Sharpness-aware low-dose CT denoising using conditional generative adversarial network. J Digit Imaging. 2018;31:655–69.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Kang E, Koo HJ, Yang DH, Seo JB, Ye JC. Cycle-consistent adversarial denoising network for multiphase coronary CT angiography. Med Phys. 2019;46:550–62.

    Article  PubMed  Google Scholar 

  37. Timofte R, Smet VD, Gool LV. Anchored neighborhood regression for fast example-based super-resolution. In: 2013 IEEE International conference on computer vision, 2013; 1920–1927.

  38. Yang J, Wright JN, Huang TS, Ma, Y. Image super-resolution as sparse representation of raw image patches. In: 2008 IEEE conference on computer vision and pattern recognition, 2008; 1–8.

  39. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel M-L. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: BMVC; 2012.

  40. Chang H, Yeung D-Y, Xiong Y. Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004., 2004; 1:I–I.

  41. Kensuke Umehara TI, Ota Junko. Super-resolution imaging of mammograms based on the super-resolution convolutional neural network. Open J Med Imaging. 2017;7:180–95.

    Article  Google Scholar 

  42. Umehara K, Ota J, Ishida T. Application of super-resolution convolutional neural network for enhancing image resolution in chest CT. J Digit Imaging. 2018;31(4):441–50.

    Article  PubMed  Google Scholar 

  43. Plenge E, Poot DHJ, Bernsen M, Kotek G, Houston G, Wielopolski P, van der Weerd L, Niessen WJ, Meijering E. Super-resolution methods in MRI: can they improve the trade-off between resolution, signal-to-noise ratio, and acquisition time? Magn Reson Med. 2012;68:1983–93.

    Article  PubMed  Google Scholar 

  44. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken AP, Tejani A, Totz J, Wang Z, Shi W. Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR. IEEE computer society; 2017. p. 105–14.

  45. Sánchez I, Vilaplana V. Brain MRI super-resolution using generative adversarial networks. In: International conference on medical imaging with deep learning, (Amsterdam, The Netherlands); 2018.

  46. Chuquicusma MJM, Hussein S, Burt JR, Bagci U. How to fool radiologists with generative adversarial networks? a visual turing test for lung cancer diagnosis. 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), 2018; 240–244.

  47. Frid-Adar M, Diamant I, Klang E, Amitai M, Goldberger J, Greenspan H. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing. 2018;321:321–31.

    Article  Google Scholar 

  48. Bermudez C, Plassard AJ, Davis LT, Newton AT, Resnick SM, Landman BA. Learning implicit brain MRI manifolds with deep learning. In: Proceedings of SPIE-the international society for optical engineering, vol 10574; 2018.

  49. Madani A, Moradi M, Karargyris A, Syeda-Mahmood T. Chest x-ray generation and data augmentation for cardiovascular abnormality classification. In: Proceedings of SPIE 10574, Medical Imaging 2018: Image Processing, 105741M; 2018.

  50. Korkinof D, Rijken T, O’Neill M, Yearsley J, Harvey H, Glocker B. High-resolution mammogram synthesis using progressive generative adversarial networks; 2018. arXiv:1807.03401.

  51. Wolterink JM, Dinkla AM, Savenije MHF, Seevinck PR, van den Berg CAT, Išgum I. Deep MR to CT synthesis using unpaired data. In: Tsaftaris SA, Gooya A, Frangi AF, Prince JL, editors. Simulation and synthesis in medical imaging. Cham: Springer International Publishing; 2017. p. 14–23.

    Chapter  Google Scholar 

  52. Hiasa Y, Otake Y, Takao M, Matsuoka T, Takashima K, Carass A, Prince J, Sugano N, Sato Y. Cross-modality image synthesis from unpaired data using cycleGAN: effects of gradient consistency loss and training data size. In: Goksel O, Oguz I, Gooya A, Burgos N, editors. Simulation and synthesis in medical imaging—third international workshop, SASHIMI 2018, held in conjunction with MICCAI 2018, proceedings, lecture notes in computer science, vol. 1. Berlin: Springer Verlag; 2018. p. 31–41 (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics).

    Google Scholar 

  53. Zhang Z, Yang L, Zheng Y. Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, 2018; 9242–9251.

  54. Wu E, Wu K, Cox D, Lotter W. Conditional infilling GANs for data augmentation in mammogram classification. In: Stoyanov D, et al., editors. Image analysis for moving organ, breast, and thoracic images. RAMBO 2018, BIA 2018, TIA 2018. Lecture notes in computer science, vol 11040. Cham: Springer; 2018. p. 98–106.

    Google Scholar 

  55. Mok TCW, Chung ACS. Learning data augmentation for brain tumor segmentation with coarse-to-fine generative adversarial networks. In: Crimi A, Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T, editors. Brainlesion: glioma, multiple sclerosis, stroke and traumatic brain injuries. Cham: Springer International Publishing; 2019. p. 70–80.

    Chapter  Google Scholar 

  56. Frid-Adar M, Klang E, Amitai M, Goldberger J, Greenspan H. Synthetic data augmentation using GAN for improved liver lesion classification. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), 2018; 289–293.

  57. Nie D, Cao X, Gao Y, Wang L, Shen D. Estimating CT image from MRI data using 3D fully convolutional networks. In: Carneiro G, Mateus D, Peter L, Bradley A, Tavares JMRS, Belagiannis V, Papa JP, Nascimento JC, Loog M, Lu Z, Cardoso JS, Cornebise J, editors. Deep learning and data labeling for medical applications. Cham: Springer International Publishing; 2016. p. 170–8.

    Google Scholar 

  58. Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys. 2017;44:1408–19.

    Article  CAS  PubMed  Google Scholar 

  59. Kida S, Nakamoto T, Nakano M, Nawa K, Haga A, Kotoku J, Yamashita H, Nakagawa K. Cone beam computed tomography image quality improvement using a deep convolutional neural network. Cureus. 2018;10:e2548.

    PubMed  PubMed Central  Google Scholar 

  60. Ben-Cohen A, Klang E, Raskin SP, Soffer S, Ben-Haim S, Konen E, Amitai MM, Greenspan H. Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection. Eng Appl Artif Intell. 2019;78:186–94.

    Article  Google Scholar 

  61. Kida S, Kaji S, Nawa K, Imae T, Nakamoto T, Ozaki S, Ohta T, Nozawa Y, Nakagawa K. Cone-beam CT to planning CT synthesis using generative adversarial networks; 2019. arXiv:1901.05773.

  62. Rick Chang JH, Li C-L, Poczos B, Vijaya Kumar BVK, Sankaranarayanan AC. One network to solve them all–solving linear inverse problems using deep projection models. In: The IEEE international conference on computer vision (ICCV); Oct 2017.

  63. Ulyanov D, Vedaldi A, Lempitsky VS. Deep image prior. In: Proceedings of CVPR2018. IEEE Computer Society; 2018. p. 9446–54.

  64. Adler J, Öktem O. Solving ill-posed inverse problems using iterative deep neural networks. Inverse Probl. 2017;33:124007.

    Article  Google Scholar 

  65. Lucas A, Iliadis M, Molina R, Katsaggelos AK. Using deep neural networks for inverse problems in imaging: Beyond analytical methods. IEEE Signal Process Mag. 2018;35:20–36.

    Article  Google Scholar 

  66. Tokui S, Oono K, Hido S, Clayton J. Chainer: a next-generation open source framework for deep learning. In: Proceedings of workshop on machine learning systems (LearningSys) in the twenty-ninth annual conference on neural information processing systems (NIPS); 2015.

  67. National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC), Radiology data from the clinical proteomic tumor analysis consortium sarcomas [CPTAC-SAR] collection [data set]. 2018. https://wiki.cancerimagingarchive.net/display/Public/CPTAC-SAR, Accessed 18 June 2019.

  68. Tanno R, Worrall DE, Ghosh A, Kaden E, Sotiropoulos SN, Criminisi A, Alexander DC. Bayesian image quality transfer with CNNs: exploring uncertainty in dMRI super-resolution. In: Proceedings of Medical image computing and computer assisted intervention—MICCAI 2017, Quebec City, QC, Canada, September 11–13; 2017. p. 611–19.

  69. Adler J, Kohr H, Öktem O. Operator discretization library (ODL). 2017. https://odlgroup.github.io/odl/. Accessed 18 June 2019.

Download references

Acknowledgements

We thank Mrs. Lanzl at the University of Chicago for her English proofreading.

Author information

Author notes

  1. S. Kaji and S. Kida contributed equally to this work.

Authors and Affiliations

  1. Institute of Mathematics for Industry, Kyushu University, Fukuoka, Japan

    Shizuo Kaji

  2. JST PRESTO, Saitama, Japan

    Shizuo Kaji

  3. LPixel Inc., Tokyo, Japan

    Satoshi Kida

  4. The University of Tokyo hospital, Tokyo, Japan

    Satoshi Kida

Authors
  1. Shizuo Kaji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Satoshi Kida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Satoshi Kida.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Funding

Kaji was partially supported by JST PRESTO, Grant Number JPMJPR16E3 Japan.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kaji, S., Kida, S. Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging. Radiol Phys Technol 12, 235–248 (2019). https://doi.org/10.1007/s12194-019-00520-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12194-019-00520-y

Keywords

Search

Navigation