Image Super Resolution via Bilinear Pooling: Application to Confocal Endomicroscopy

  • Saeed IzadiEmail author
  • Darren Sutton
  • Ghassan Hamarneh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11905)


Recent developments in image acquisition literature have miniaturized the confocal laser endomicroscopes to improve usability and flexibility of the apparatus in actual clinical settings. However, miniaturized devices collect less light and have fewer optical components, resulting in pixelation artifacts and low resolution images. Owing to the strength of deep networks, many supervised methods known as super resolution have achieved considerable success in restoring low resolution images by generating the missing high frequency details. In this work, we propose a novel attention mechanism that, for the first time, combines 1st- and 2nd-order statistics for pooling operation, in the spatial and channel-wise dimensions. We compare the efficacy of our method to 10 other existing single image super resolution techniques that compensate for the reduction in image quality caused by the necessity of endomicroscope miniaturization. All evaluations are carried out on three publicly available datasets. Experimental results show that our method can produce superior results against state-of-the-art in terms of PSNR, and SSIM metrics. Additionally, our proposed method is lightweight and suitable for real-time inference.



Thanks to the NVIDIA Corporation for the donation of Titan X GPUs used in this research and to the Collaborative Health Research Projects (CHRP) for funding.


  1. 1.
    Ahn, N., et al.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: ECCV (2018)Google Scholar
  2. 2.
    Bray, F., et al.: Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 68(6), 394–424 (2018)CrossRefGoogle Scholar
  3. 3.
    Cheng, X., et al.: Sesr: single image super resolution with recursive squeeze and excitation networks. In: IEEE ICPR, pp. 147–152 (2018)Google Scholar
  4. 4.
    Dong, C., et al.: Image super-resolution using deep convolutional networks. IEEE TPAMI 38(2), 295–307 (2016) CrossRefGoogle Scholar
  5. 5.
    Gao, Z., et al. Global second-order pooling neural networks. arXiv:1811.12006 (2018)
  6. 6.
    Grisan, E., et al.: 239 computer aided diagnosis of barrett’s esophagus using confocal laser endomicroscopy: preliminary data. Gastrointest. Endosc. 75(4), AB126 (2012)CrossRefGoogle Scholar
  7. 7.
    Hu, J., et al.: Squeeze-and-excitation networks. In: IEEE CVPR (2018)Google Scholar
  8. 8.
    Huang, J., et al.: Single image super-resolution from transformed self-exemplars. In: IEEE CVPR, pp. 5197–5206 (2015)Google Scholar
  9. 9.
    Izadi, S., Moriarty, K.P., Hamarneh, G.: Can deep learning relax endomicroscopy hardware miniaturization requirements? In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 57–64. Springer, Cham (2018). Scholar
  10. 10.
    Kiesslich, R., et al.: Confocal laser endoscopy for diagnosing intraepithelial neoplasias and colorectal cancer in vivo. Gastroenterology 127(3), 706–713 (2004)CrossRefGoogle Scholar
  11. 11.
    Kim, J., et al.: Accurate image super-resolution using very deep convolutional networks. In: IEEE CVPR, pp. 1646–1654 (2016)Google Scholar
  12. 12.
    Kim, J., et al.: Deeply-recursive convolutional network for image super-resolution. In: IEEE CVPR, pp. 1637–1645 (2016)Google Scholar
  13. 13.
    Lai, W., et al.: Fast and Accurate Image Super-resolution with Deep Laplacian Pyramid Networks. In: Ieee Tpami, p. 1 (2018) Google Scholar
  14. 14.
    Leong, R.W., et al.: In vivo confocal endomicroscopy in the diagnosis and evaluation of celiac disease. Gastroenterology 135(6), 1870–1876 (2008)CrossRefGoogle Scholar
  15. 15.
    Ravì, D., et al.: Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction. Int. J. Comput. Assist. Radiol. Surg. 13, 917–924 (2018)CrossRefGoogle Scholar
  16. 16.
    Ravì, D., et al.: Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy. Med. Image Anal. 53, 123–131 (2019)CrossRefGoogle Scholar
  17. 17.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883 (2016)Google Scholar
  18. 18.
    Ştefănescu, D., et al.: Computer aided diagnosis for confocal laser endomicroscopy in advanced colorectal adenocarcinoma. PloS ONE 11(5), e0154863 (2016)CrossRefGoogle Scholar
  19. 19.
    Timofte, R., et al.: Anchored neighborhood regression for fast example-based super-resolution. In: IEEE ICCV, pp. 1920–1927 (2013)Google Scholar
  20. 20.
    Timofte, R., et al.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: ACCV, pp. 111–126 (2015)Google Scholar
  21. 21.
    Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.-H., Liao, Q.: Deep learning for single image super-resolution: a brief review. IEEE Trans. Multimedia PP(99), 1 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Computing ScienceSimon Fraser UniversityBurnabyCanada

Personalised recommendations