Adaptive Importance Learning for Improving Lightweight Image Super-Resolution Network
- 22 Downloads
Deep neural networks have achieved remarkable success in single image super-resolution (SISR). The computing and memory requirements of these methods have hindered their application to broad classes of real devices with limited computing power, however. One approach to this problem has been lightweight network architectures that balance the super-resolution performance and the computation burden. In this study, we revisit this problem from an orthogonal view, and propose a novel learning strategy to maximize the pixel-wise fitting ability of a given lightweight network architecture. Considering that the initial performance of the lightweight network is very limited, we present an adaptive importance learning scheme for SISR that trains the network with an easy-to-complex paradigm by dynamically updating the importance of image pixels on the basis of the training loss. Specifically, we formulate the network training and the importance learning into a joint optimization problem. With a carefully designed importance penalty function, the importance of individual pixels can be gradually increased through solving a convex optimization problem. The training process thus begins with pixels that are easy to reconstruct, and gradually proceeds to more complex pixels as fitting improves. Furthermore, the proposed learning scheme is able to seamlessly assimilate knowledge from a more powerful teacher network in the form of importance initialization, thus obtaining better initial performance for the network. Through learning the network parameters, and updating pixel importance, the proposed learning scheme enables smaller, lightweight, networks to achieve better performance than has previously been possible. Extensive experiments on four benchmark datasets demonstrate the potential benefits of the proposed learning strategy in lightweight SISR network enhancement. In some cases, our learned network with only \(25\%\) of the parameters and computational complexity can produce comparable or even better results than the corresponding full-parameter network.
KeywordsImportant learning Single image super-resolution Lightweight network enhancement
This work was supported in part by the National Natural Science Foundation of China (No. 61671385), and Australian Research Council Grant (FT120100969).
- Basu, S., & Christensen, J. (2013). Teaching classification boundaries to humans. In AAAI.Google Scholar
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41–48). ACM.Google Scholar
- Bevilacqua, M., Roumy, A., Guillemot, C., & Alberi-Morel, ML. (2012). Low-complexity single-image super-resolution based on nonnegative neighbor embedding.Google Scholar
- Dong, C., Loy, C. C., & Tang, X. (2016b). Accelerating the super-resolution convolutional neural network. In European conference on computer vision (pp. 391–407). Springer.Google Scholar
- Efrat, N., Glasner, D., Apartsin, A., Nadler, B., & Levin, A. (2013). Accurate blur models vs. image priors in single image super-resolution. In 2013 IEEE international conference on Computer vision (ICCV) (pp. 2832–2839). IEEE.Google Scholar
- Github. (2019a). DRRN code. https://github.com/jt827859032/DRRN-pytorch.
- Github. (2019b). VDSR code. https://github.com/twtygqyy/pytorch-vdsr.
- Glasner, D., Bagon, S., & Irani, M. (2009). Super-resolution from a single image. In 2009 IEEE 12th international conference on computer vision (pp. 349–356). IEEE.Google Scholar
- Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
- Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017), Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
- Huang, J. B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5197–5206).Google Scholar
- Jiang, L., Meng, D., Mitamura, T., & Hauptmann, A. G. (2014). Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on multimedia (pp. 547–556). ACM.Google Scholar
- Jiang, L., Meng, D., Zhao, Q., Shan, S., & Hauptmann, A. G. (2015). Self-paced curriculum learning. In AAAI (p. 6), Vol. 2.Google Scholar
- Khan, F., Mutlu, B., & Zhu, X. (2011). How do humans teach: On curriculum learning and teaching dimension. In Advances in neural information processing systems (pp. 1449–1457).Google Scholar
- Kim, J., Kwon Lee, J., & Mu Lee, K. (2016a). Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1646–1654).Google Scholar
- Kim, J., Kwon Lee, J., & Mu Lee, K. (2016b). Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1637–1645).Google Scholar
- Kumar, M. P., Packer, B., & Koller, D. (2010). Self-paced learning for latent variable models. In Advances in neural information processing systems (pp. 1189–1197).Google Scholar
- Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 624–632).Google Scholar
- Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., & Wang, Z., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4681–4690).Google Scholar
- Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2980–2988).Google Scholar
- Mao, X. J., Shen, C., Yang, Y. B. (2016). Image restoration using convolutional auto-encoders with symmetric skip connections. arxiv preprint. arXiv preprint arXiv:1606.08921
- Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE international conference on computer vision (Vol. 2, pp. 416–423). IEEE.Google Scholar
- Paszke, A., Gross, S., Chintala, S., & Chanan, G. (2017). Pytorch.Google Scholar
- Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014). Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550.
- Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1874–1883).Google Scholar
- Sun, J., Xu, Z., & Shum, H. Y. (2008). Image super-resolution using gradient profile prior. In 2008 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8). IEEE.Google Scholar
- Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In The IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 1).Google Scholar
- Timofte, R., De Smet, V., & Van Gool, L. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian conference on computer vision (pp. 111–126). Springer.Google Scholar
- Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., & Zhang, L. (2017). Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 114–125).Google Scholar
- Timofte, R., Gu, S., Wu, J., & Van Gool, L. (2018). Ntire 2018 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 852–863).Google Scholar
- Wang, Z., Liu, D., Yang, J., Han, W., & Huang, T. (2015). Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE international conference on computer vision (pp. 370–378).Google Scholar
- Yang, C. Y., Ma, C., & Yang, M. H. (2014). Single-image super-resolution: A benchmark. In European conference on computer vision (pp. 372–386). Springer.Google Scholar
- Zeyde, R., Elad, M., & Protter, M. (2010). On single image scale-up using sparse-representations. In International conference on curves and surfaces (pp. 711–730). Springer.Google Scholar
- Zhang, L., Wei, W., Shi, Q., Shen, C., Hengel, A. V. D., & Zhang, Y. (2017a). Beyond low rank: A data-adaptive tensor completion method. arXiv preprint arXiv:1708.01008.
- Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2017b). Deep mutual learning. arXiv preprint arXiv:1706.00384.