Learning a Mixture of Deep Networks for Single Image Super-Resolution

  • Ding LiuEmail author
  • Zhaowen Wang
  • Nasser Nasrabadi
  • Thomas Huang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10113)


Single image super-resolution (SR) is an ill-posed problem which aims to recover high-resolution (HR) images from their low-resolution (LR) observations. The crux of this problem lies in learning the complex mapping between low-resolution patches and the corresponding high-resolution patches. Prior arts have used either a mixture of simple regression models or a single non-linear neural network for this propose. This paper proposes the method of learning a mixture of SR inference modules in a unified framework to tackle this problem. Specifically, a number of SR inference modules specialized in different image local patterns are first independently applied on the LR image to obtain various HR estimates, and the resultant HR estimates are adaptively aggregated to form the final HR image. By selecting neural networks as the SR inference module, the whole procedure can be incorporated into a unified network and be optimized jointly. Extensive experiments are conducted to investigate the relation between restoration performance and different network architectures. Compared with other current image SR approaches, our proposed method achieves state-of-the-arts restoration results on a wide range of images consistently while allowing more flexible design choices.


Sparse Code Bicubic Interpolation Convolutional Layer Inference Module Inference Time 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Sig. Process. Mag. 20, 21–36 (2003)CrossRefGoogle Scholar
  2. 2.
    Morse, B.S., Schwartzwald, D.: Image magnification using level-set reconstruction. In: CVPR 2001, vol. 1, 1–333. IEEE (2001)Google Scholar
  3. 3.
    Fattal, R.: Image upsampling via imposed edge statistics. In: ACM Transactions on Graphics (TOG), vol. 26, p. 95. ACM (2007)Google Scholar
  4. 4.
    Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, vol. 1, p. 1. IEEE (2004)Google Scholar
  5. 5.
    Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: ICCV 2009, pp. 349–356. IEEE (2009)Google Scholar
  6. 6.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19, 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1920–1927. IEEE (2013)Google Scholar
  8. 8.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI 38(2), 295–307 (2015)CrossRefGoogle Scholar
  9. 9.
    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206. IEEE (2015)Google Scholar
  10. 10.
    Wang, Z., Yang, Y., Wang, Z., Chang, S., Yang, J., Huang, T.S.: Learning super-resolution jointly from external and internal examples. IEEE Trans. Image Process. 24, 4359–4371 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)Google Scholar
  12. 12.
    Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)Google Scholar
  13. 13.
    Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-16817-3_8 Google Scholar
  14. 14.
    Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. In: Eurographics, vol. 7, p. 8 (2015)Google Scholar
  15. 15.
    Timofte, R., Rasmus, R., Van Gool, L.: Seven ways to improve example-based single image super resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2016)Google Scholar
  16. 16.
    Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X.: Deep network cascade for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 49–64. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10602-1_4 Google Scholar
  17. 17.
    Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 370–378 (2015)Google Scholar
  18. 18.
    Liu, D., Wang, Z., Wen, B., Yang, J., Han, W., Huang, T.S.: Robust single image super-resolution via deep networks with sparse prior. IEEE Trans. Image Process. 25, 3194–3207 (2016)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2016)Google Scholar
  20. 20.
    Yang, J., Wang, Z., Lin, Z., Cohen, S., Huang, T.: Coupled dictionary training for image super-resolution. IEEE Trans. Image Process. 21, 3467–3478 (2012)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., Chenin, P., Cohen, A., Gout, C., Lyche, T., Mazure, M.-L., Schumaker, L. (eds.) Curves and Surfaces 2010. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-27413-8_47 CrossRefGoogle Scholar
  22. 22.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 2, pp. 416–423. IEEE (2001)Google Scholar
  23. 23.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar
  24. 24.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)CrossRefGoogle Scholar
  25. 25.
    Wang, Z., Chang, S., Yang, Y., Liu, D., Huang, T.: Studying very low resolution recognition using deep networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Ding Liu
    • 1
    Email author
  • Zhaowen Wang
    • 2
  • Nasser Nasrabadi
    • 3
  • Thomas Huang
    • 1
  1. 1.Beckman InstituteUniversity of Illinois at Urbana-ChampaignUrbanaUSA
  2. 2.Adobe ResearchSan JoseUSA
  3. 3.Lane Department of Computer Science and Electrical EngineeringWest Virginia UniversityMorgantownUSA

Personalised recommendations