A Mixed-Supervision Multilevel GAN Framework for Image Quality Enhancement

  • Uddeshya Upadhyay
  • Suyash P. AwateEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11768)


Deep neural networks for image quality enhancement typically need large quantities of highly-curated training data comprising pairs of low-quality images and their corresponding high-quality images. While high-quality image acquisition is typically expensive and time-consuming, medium-quality images are faster to acquire, at lower equipment costs, and available in larger quantities. Thus, we propose a novel generative adversarial network (GAN) that can leverage training data at multiple levels of quality (e.g., high and medium quality) to improve performance while limiting costs of data curation. We apply our mixed-supervision GAN to (i) super-resolve histopathology images and (ii) enhance laparoscopy images by combining super-resolution and surgical smoke removal. Results on large clinical and pre-clinical datasets show the benefits of our mixed-supervision GAN over the state of the art.


Image quality enhancement Generative adversarial network (GAN) Mixed-supervision Super-resolution Surgical smoke removal 


  1. 1.
    Aja-Fernandez, S., Estepar, R.J., Alberola-Lopez, C., Westin, C.: Image quality assessment based on local variance. In: IEEE EMBS Annual International Conference, p. 4815 (2006)Google Scholar
  2. 2.
    Baid, A., Kotwal, A., Bhalodia, R., Merchant, S., Awate, S.: Joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and Bayesian inference. In: IEEE International Symposium on Biomedical Imaging, pp. 732–736 (2017)Google Scholar
  3. 3.
    Bertram, C., Klopfleisch, R.: The pathologist 2.0: an update on digital pathology in veterinary medicine. Vet. Pathol. 54(5), 756–766 (2017)CrossRefGoogle Scholar
  4. 4.
    Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. In: International Conference on Learning Representations, pp. 1–17 (2016)Google Scholar
  5. 5.
    Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 275–282 (2004)Google Scholar
  6. 6.
    Chen, L., Tang, W., John, N.: Unsupervised adversarial training of surgical smoke removal. In: Medical Imaging with Deep Learning, pp. 1–3 (2018)Google Scholar
  7. 7.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). Scholar
  8. 8.
    Huisman, A., Looijen, A., van den Brink, S., van Diest, P.: Creation of a fully digital pathology slide archive by high-volume tissue slide scanning. Hum. Pathol. 41(5), 751–757 (2010)CrossRefGoogle Scholar
  9. 9.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)Google Scholar
  10. 10.
    Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep Laplacian pyramid networks for fast and accurate superresolution. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 5835–5843 (2017)Google Scholar
  11. 11.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 105–114 (2017)Google Scholar
  12. 12.
    Luo, X., McLeod, A., Pautler, S., Schlachta, C., Peters, T.: Vision-based surgical field defogging. IEEE Trans. Med. Imag. 36(10), 2021–2030 (2017)CrossRefGoogle Scholar
  13. 13.
    Mousavi, H., Monga, V.: Sparsity-based color image super resolution via exploiting cross channel constraints. IEEE Trans. Image Process. 26(11), 5094–5106 (2017)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  15. 15.
    Shah, M., Bhalgat, Y., Awate, S.P.: Annotation-cost minimization for medical image segmentation using suggestive mixed supervision fully convolutional networks. In: Medical Image Meets Neural Information Processing System, pp. 1–4 (2019)Google Scholar
  16. 16.
    Shah, M., Merchant, S.N., Awate, S.P.: MS-Net: mixed-supervision fully-convolutional networks for full-resolution segmentation. In: Medical Image Computing and Computer-Assisted Intervention, pp. 379–387 (2018)CrossRefGoogle Scholar
  17. 17.
    Tchaka, K., Pawar, V., Stoyanov, D.: Chromaticity based smoke removal in endoscopic images. In: Proceedings of SPIE, p. 101331M (2018)Google Scholar
  18. 18.
    Upadhyay, U., Awate, S.P.: Robust super-resolution GAN, with manifold-based and perception loss. In: IEEE International Symposium on Biomedical Imaging (ISBI) (2019)Google Scholar
  19. 19.
    Wang, Y., et al.: 3D conditional generative adversarial networks for high-quality PET image estimation at low dose. NeuroImage 1(274), 550–562 (2018)CrossRefGoogle Scholar
  20. 20.
    Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  21. 21.
    Yamashita, H., Aoki, H., Tanioka, K., Mori, T., Chiba, T.: Ultra-high definition (8K UHD) endoscope: our first clinical success. SpringerPlus 5(1), 1445 (2016)CrossRefGoogle Scholar
  22. 22.
    Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Ye, M., Johns, E., Handa, A., Zhang, L., Pratt, P., Yang, G.Z.: Self-supervised siamese learning on stereo image pairs for depth estimation in robotic surgery. In: Hamlyn Symposium on Medical Robotics (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Computer Science and EngineeringIndian Institute of TehnologyBombayIndia

Personalised recommendations