Advertisement

Adaptive Importance Learning for Improving Lightweight Image Super-Resolution Network

  • Lei Zhang
  • Peng Wang
  • Chunhua ShenEmail author
  • Lingqiao Liu
  • Wei Wei
  • Yanning Zhang
  • Anton van den Hengel
Article
  • 22 Downloads

Abstract

Deep neural networks have achieved remarkable success in single image super-resolution (SISR). The computing and memory requirements of these methods have hindered their application to broad classes of real devices with limited computing power, however. One approach to this problem has been lightweight network architectures that balance the super-resolution performance and the computation burden. In this study, we revisit this problem from an orthogonal view, and propose a novel learning strategy to maximize the pixel-wise fitting ability of a given lightweight network architecture. Considering that the initial performance of the lightweight network is very limited, we present an adaptive importance learning scheme for SISR that trains the network with an easy-to-complex paradigm by dynamically updating the importance of image pixels on the basis of the training loss. Specifically, we formulate the network training and the importance learning into a joint optimization problem. With a carefully designed importance penalty function, the importance of individual pixels can be gradually increased through solving a convex optimization problem. The training process thus begins with pixels that are easy to reconstruct, and gradually proceeds to more complex pixels as fitting improves. Furthermore, the proposed learning scheme is able to seamlessly assimilate knowledge from a more powerful teacher network in the form of importance initialization, thus obtaining better initial performance for the network. Through learning the network parameters, and updating pixel importance, the proposed learning scheme enables smaller, lightweight, networks to achieve better performance than has previously been possible. Extensive experiments on four benchmark datasets demonstrate the potential benefits of the proposed learning strategy in lightweight SISR network enhancement. In some cases, our learned network with only \(25\%\) of the parameters and computational complexity can produce comparable or even better results than the corresponding full-parameter network.

Keywords

Important learning Single image super-resolution Lightweight network enhancement 

Notes

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (No. 61671385), and Australian Research Council Grant (FT120100969).

References

  1. Basu, S., & Christensen, J. (2013). Teaching classification boundaries to humans. In AAAI.Google Scholar
  2. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41–48). ACM.Google Scholar
  3. Bevilacqua, M., Roumy, A., Guillemot, C., & Alberi-Morel, ML. (2012). Low-complexity single-image super-resolution based on nonnegative neighbor embedding.Google Scholar
  4. Dong, C., Loy, C. C., He, K., & Tang, X. (2016a). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.CrossRefGoogle Scholar
  5. Dong, C., Loy, C. C., & Tang, X. (2016b). Accelerating the super-resolution convolutional neural network. In European conference on computer vision (pp. 391–407). Springer.Google Scholar
  6. Efrat, N., Glasner, D., Apartsin, A., Nadler, B., & Levin, A. (2013). Accurate blur models vs. image priors in single image super-resolution. In 2013 IEEE international conference on Computer vision (ICCV) (pp. 2832–2839). IEEE.Google Scholar
  7. Github. (2019a). DRRN code. https://github.com/jt827859032/DRRN-pytorch.
  8. Github. (2019b). VDSR code. https://github.com/twtygqyy/pytorch-vdsr.
  9. Glasner, D., Bagon, S., & Irani, M. (2009). Super-resolution from a single image. In 2009 IEEE 12th international conference on computer vision (pp. 349–356). IEEE.Google Scholar
  10. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  11. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017), Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
  12. Huang, J. B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5197–5206).Google Scholar
  13. Jiang, L., Meng, D., Mitamura, T., & Hauptmann, A. G. (2014). Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on multimedia (pp. 547–556). ACM.Google Scholar
  14. Jiang, L., Meng, D., Zhao, Q., Shan, S., & Hauptmann, A. G. (2015). Self-paced curriculum learning. In AAAI (p. 6), Vol. 2.Google Scholar
  15. Khan, F., Mutlu, B., & Zhu, X. (2011). How do humans teach: On curriculum learning and teaching dimension. In Advances in neural information processing systems (pp. 1449–1457).Google Scholar
  16. Kim, J., Kwon Lee, J., & Mu Lee, K. (2016a). Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1646–1654).Google Scholar
  17. Kim, J., Kwon Lee, J., & Mu Lee, K. (2016b). Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1637–1645).Google Scholar
  18. Kim, K. I., & Kwon, Y. (2010). Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(6), 1127–1133.CrossRefGoogle Scholar
  19. Kumar, M. P., Packer, B., & Koller, D. (2010). Self-paced learning for latent variable models. In Advances in neural information processing systems (pp. 1189–1197).Google Scholar
  20. Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 624–632).Google Scholar
  21. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., & Wang, Z., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4681–4690).Google Scholar
  22. Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2980–2988).Google Scholar
  23. Liu, L., Wang, P., Shen, C., Wang, L., Van Den Hengel, A., Wang, C., et al. (2017). Compositional model based fisher vector coding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2335–2348.CrossRefGoogle Scholar
  24. Mao, X. J., Shen, C., Yang, Y. B. (2016). Image restoration using convolutional auto-encoders with symmetric skip connections. arxiv preprint. arXiv preprint arXiv:1606.08921
  25. Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE international conference on computer vision (Vol. 2, pp. 416–423). IEEE.Google Scholar
  26. Paszke, A., Gross, S., Chintala, S., & Chanan, G. (2017). Pytorch.Google Scholar
  27. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014). Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550.
  28. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1874–1883).Google Scholar
  29. Sun, J., Xu, Z., & Shum, H. Y. (2008). Image super-resolution using gradient profile prior. In 2008 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8). IEEE.Google Scholar
  30. Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In The IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 1).Google Scholar
  31. Timofte, R., De Smet, V., & Van Gool, L. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian conference on computer vision (pp. 111–126). Springer.Google Scholar
  32. Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., & Zhang, L. (2017). Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 114–125).Google Scholar
  33. Timofte, R., Gu, S., Wu, J., & Van Gool, L. (2018). Ntire 2018 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 852–863).Google Scholar
  34. Wang, Z., Liu, D., Yang, J., Han, W., & Huang, T. (2015). Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE international conference on computer vision (pp. 370–378).Google Scholar
  35. Wei, W., Zhang, L., Tian, C., Plaza, A., & Zhang, Y. (2017). Structured sparse coding-based hyperspectral imagery denoising with intracluster filtering. IEEE Transactions on Geoscience and Remote Sensing, 55(12), 6860–6876.CrossRefGoogle Scholar
  36. Yang, C. Y., Ma, C., & Yang, M. H. (2014). Single-image super-resolution: A benchmark. In European conference on computer vision (pp. 372–386). Springer.Google Scholar
  37. Yang, J., Wright, J., Huang, T. S., & Ma, Y. (2010). Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11), 2861–2873.MathSciNetCrossRefGoogle Scholar
  38. Zeyde, R., Elad, M., & Protter, M. (2010). On single image scale-up using sparse-representations. In International conference on curves and surfaces (pp. 711–730). Springer.Google Scholar
  39. Zhang, L., Wei, W., Bai, C., Gao, Y., & Zhang, Y. (2018a). Exploiting clustering manifold structure for hyperspectral imagery super-resolution. IEEE Transactions on Image Processing, 27(12), 5969–5982.MathSciNetCrossRefGoogle Scholar
  40. Zhang, L., Wei, W., Shi, Q., Shen, C., Hengel, A. V. D., & Zhang, Y. (2017a). Beyond low rank: A data-adaptive tensor completion method. arXiv preprint arXiv:1708.01008.
  41. Zhang, L., Wei, W., Zhang, Y., Shen, C., van den Hengel, A., & Shi, Q. (2018b). Cluster sparsity field: An internal hyperspectral imagery prior for reconstruction. International Journal of Computer Vision, 126(8), 797–821.CrossRefGoogle Scholar
  42. Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2017b). Deep mutual learning. arXiv preprint arXiv:1706.00384.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Inception Institute of Artificial IntelligenceAbu DhabiUnited Arab Emirates
  2. 2.School of Computing and Information TechnologyUniversity of WollongongWollongongAustralia
  3. 3.School of Computer ScienceThe University of AdelaideAdelaideAustralia
  4. 4.School of Computer ScienceNorthwestern Polytechnical University Xi’anChina

Personalised recommendations