CT Image Enhancement Using Stacked Generative Adversarial Networks and Transfer Learning for Lesion Segmentation Improvement

  • Youbao TangEmail author
  • Jinzheng Cai
  • Le Lu
  • Adam P. Harrison
  • Ke Yan
  • Jing Xiao
  • Lin Yang
  • Ronald M. Summers
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11046)


Automated lesion segmentation from computed tomography (CT) is an important and challenging task in medical image analysis. While many advancements have been made, there is room for continued improvements. One hurdle is that CT images can exhibit high noise and low contrast, particularly in lower dosages. To address this, we focus on a preprocessing method for CT images that uses stacked generative adversarial networks (SGAN) approach. The first GAN reduces the noise in the CT image and the second GAN generates a higher resolution image with enhanced boundaries and high contrast. To make up for the absence of high quality CT images, we detail how to synthesize a large number of low- and high-quality natural images and use transfer learning with progressively larger amounts of CT images. We apply both the classic GrabCut method and the modern holistically nested network (HNN) to lesion segmentation, testing whether SGAN can yield improved lesion segmentation. Experimental results on the DeepLesion dataset demonstrate that the SGAN enhancements alone can push GrabCut performance over HNN trained on original images. We also demonstrate that HNN + SGAN performs best compared against four other enhancement methods, including when using only a single GAN.


CT image enhancement Lesion segmentation Stacked generative adversarial networks Transfer learning 



This research was supported by the Intramural Research Program of the National Institutes of Health Clinical Center and by the Ping An Insurance Company through a Cooperative Research and Development Agreement. We thank Nvidia for GPU card donation.


  1. 1.
    Tang, Y., Harrison, A.P., et al.: Semi-automatic recist labeling on ct scans with cascaded convolutional neural networks. arXiv:1806.09507 (2018)
  2. 2.
    Jin, D., Xu, Z., et al.: Ct-realistic lung nodule simulation from 3d conditional generative adversarial networks for robust lung segmentation. arXiv:1806.04051 (2018)CrossRefGoogle Scholar
  3. 3.
    Tang, Y., Wang, X., et al.: Attention-guided curriculum learning for weakly supervised classification and localization of thoracic diseases on chest radiographs. arXiv:1807.07532 (2018)
  4. 4.
    Cai, J., Tang, Y., et al.: Accurate weakly-supervised deep lesion segmentation using large-scale clinical annotations: Slice-propagated 3d mask generation from 2d recist. arXiv:1807.01172 (2018)
  5. 5.
    Massoptier, L., Casciaro, S.: A new fully automatic and robust algorithm for fast segmentation of liver tissue and tumors from ct scans. Eur. Radiol. 18(8), 1658 (2008)CrossRefGoogle Scholar
  6. 6.
    Christ, P.F., Elshaer, M.E.A., et al.: Automatic liver and lesion segmentation in ct using cascaded fully convolutional neural networks and 3d conditional random fields. MICCA I, 415–423 (2016)Google Scholar
  7. 7.
    Dabov, K., Foi, A.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE TIP 16(8), 2080–2095 (2007)MathSciNetGoogle Scholar
  8. 8.
    Zhang, K., Zuo, W., et al.: Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE TIP 26(7), 3142–3155 (2017)MathSciNetGoogle Scholar
  9. 9.
    Goodfellow, I., Pouget-Abadie, J., et al.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014)Google Scholar
  10. 10.
    Ledig, C., Theis, L., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 4681–4690 (2017)Google Scholar
  11. 11.
    Yan, K., Wang, X., et al.: Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database. In: CVPR, pp. 9261–9270 (2018)Google Scholar
  12. 12.
    Shi, W., Caballero, J., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883 (2016)Google Scholar
  13. 13.
    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)Google Scholar
  14. 14.
    He, K., Zhang, X., et al.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: ICCV, pp. 1026–1034 (2015)Google Scholar
  15. 15.
    Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: CVPRW, pp. 1122–1131 (2017)Google Scholar
  16. 16.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
  17. 17.
    Deng, J., Dong, W., et al.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)Google Scholar
  18. 18.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 (2014)
  19. 19.
    Rother, C., Kolmogorov, V., et al.: Grabcut: interactive foreground extraction using iterated graph cuts. In: ACM TOG, pp. 309–314 (2004)Google Scholar
  20. 20.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV, pp. 1395–1403 (2015)Google Scholar

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2018

Authors and Affiliations

  • Youbao Tang
    • 1
    Email author
  • Jinzheng Cai
    • 1
    • 2
  • Le Lu
    • 1
  • Adam P. Harrison
    • 1
  • Ke Yan
    • 1
  • Jing Xiao
    • 3
  • Lin Yang
    • 2
  • Ronald M. Summers
    • 1
  1. 1.National Institutes of Health Clinical CenterBethesdaUSA
  2. 2.University of FloridaGainesvilleUSA
  3. 3.Ping An Insurance Company of ChinaShenzhenChina

Personalised recommendations