Skip to main content
SpringerLink
Log in

Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Recently, much attention has been spent on neural architecture search (NAS), aiming to outperform those manually-designed neural architectures on high-level vision recognition tasks. Inspired by the success, here we attempt to leverage NAS techniques to automatically design efficient network architectures for low-level image restoration tasks. In particular, we propose a memory-efficient hierarchical NAS (termed HiNAS) and apply it to two such tasks: image denoising and image super-resolution. HiNAS adopts gradient based search strategies and builds a flexible hierarchical search space, including the inner search space and outer search space. They are in charge of designing cell architectures and deciding cell widths, respectively. For the inner search space, we propose a layer-wise architecture sharing strategy, resulting in more flexible architectures and better performance. For the outer search space, we design a cell-sharing strategy to save memory, and considerably accelerate the search speed. The proposed HiNAS method is both memory and computation efficient. With a single GTX1080Ti GPU, it takes only about 1 h for searching for denoising network on the BSD-500 dataset and 3.5 h for searching for the super-resolution structure on the DIV2K dataset. Experiments show that the architectures found by HiNAS have fewer parameters and enjoy a faster inference speed, while achieving highly competitive performance compared with state-of-the-art methods. Code is available at: https://github.com/hkzhang91/HiNAS

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Ahn, N., Kang, B., and Sohn, K. A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of European Conference Computer Vision, pp. 252–268.

  • Bender, G., Liu, H., Chen, B., Chu, G., Cheng, S., Kindermans, P. J., and Le, Q. V. (2020). Can weight sharing outperform random architecture search? an investigation with tunas. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14323–14332.

  • Cai, H., Chen, T., Zhang, W., Yu, Y., Wang, J. (2018). Efficient architecture search by network transformation. In The AAAI Conference on Artificial Intelligence.

  • Cai, H., Wang, T., Wu, Z., Wang, K., Lin, J., Han, S. (2019). On-device image classification with proxyless neural architecture search and quantization-aware fine-tuning. In Proceedings of IEEE international conference on computer vision, pp. 0–0.

  • Chatterjee, P., & Milanfar, P. (2009). Is denoising dead? IEEE Transactions on Image Processing, 19(4), 895–911.

    Article  MathSciNet  Google Scholar 

  • Chen, S., Chen, Y., Yan, S., and Feng, J. (2019). Efficient differentiable neural architecture search with meta kernels. arXiv preprint arXiv:1912.04749.

  • Chu, X., Zhang, B., Ma, H., Xu, R., Li, J., Li, Q. (2019a). Fast, accurate and lightweight super-resolution with neural architecture search. https://ieeexplore.ieee.org/abstract/document/9413080.

  • Chu, X., Zhang, B., Xu, R., Ma, H. (2019b). Multi-objective reinforced evolution in mobile neural architecture search. https://link.springer.com/chapter/10.1007/978-3-030-66823-5_6.

  • Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.

    Article  MathSciNet  Google Scholar 

  • Dai T, Cai J, Zhang Y, Xia ST, Zhang L (2019) Second-order attention network for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern Recognition pp. 11065–11074.

  • Dong, X., Yang, Y. (2019a). Network pruning via transformable architecture search. arXiv preprint arXiv:1905.09717.

  • Dong, X., Yang, Y. (2019b). One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE conference on computer vision and pattern Recognition pp. 3681–3690.

  • Dong, X., Yang, Y. (2019c). Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 1761–1770.

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2015). Image super-resolution using deep convolutional networks. TPAMI, 38(2), 295–307.

    Article  Google Scholar 

  • Elsken, T., Metzen, J. H., Hutter, F. (2018). Efficient multi-objective neural architecture search via lamarckian evolution. https://arxiv.org/abs/1804.09081.

  • Fang, J., Sun, Y., Zhang, Q., Li, Y., Liu, W., Wang, X. (2020). Densely connected search space for more flexible neural architecture search. In: Proceedings of the IEEE conference on computer vision and pattern Recognition.

  • Ghiasi, G., Lin, T. Y., Le, Q. (2019). NAS-FPN: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 7036–7045.

  • Gu, S., Zhang, L., Zuo, W., Feng, X. (2014). Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 2862–2869.

  • Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L. (2019). Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 1712–1722.

  • Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J. (2020). Single path one-shot neural architecture search with uniform sampling. In European conference on computer vision, Springer, pp. 544–560.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. Proc (pp. 630–645). European conference on computer vision: Springer.

  • He Y, Lin J, Liu Z, Wang H, Li LJ, Han S (2018) Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of European Conference Computer Vision, pp. 784–800.

  • Hu, J., Shen, L., Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 7132–7141.

  • Hui, Z., Gao, X., Yang, Y., Wang, X. (2019). Lightweight image super-resolution with information multi-distillation network. In Proceedings of the ACM international conference on Multimedia, pp. 2024–2032.

  • Hui, Z., Wang, X., Gao, X. (2018). Fast and accurate single image super-resolution via information distillation network. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp 723–731.

  • Jaroensri, R., Biscarrat, C., Aittala, M., Durand, F. (2019). Generating training data for denoising real rgb images via camera pipeline simulation. https://arxiv.org/abs/1904.08825.

  • Kim, J., Lee, J.K. and Lee, K.M.(2016a). Accurate image super-resolution using very deep convolutional networks. In In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654.

  • Kim, J., Lee, J.K. and Lee, K.M. (2016b). Deeply-recursive convolutional network for image super-resolution. In: In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1637–1645.

  • Lai, W. S., Huang, J. B., Ahuja, N., Yang, M. H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 624–632.

  • Lee, W., Lee, J., Kim, D., Ham, B. (2020). Learning with privileged information for efficient image super-resolution. https://link.springer.com/chapter/10.1007/978-3-030-58586-0_28.

  • Li, L., Talwalkar, A. (2020). Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pp. 367–377.

  • Liang, H., Zhang, S., Sun, J., He, X., Huang, W., Zhuang, K., Li, Z. (2019). Darts+: Improved differentiable architecture search with early stopping. https://arxiv.org/abs/1909.06035.

  • Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K. (2017). Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144.

  • Liu, C., Chen, L. C., Schroff, F., Adam, H., Hua, W., Yuille, A. L., Fei-Fei, L. (2019a). Auto-Deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 82–92.

  • Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K. (2018b). Hierarchical representations for efficient architecture search. In Proceedings of the international conference on learning representations.

  • Liu, H., Simonyan, K., Yang, Y. (2019b). Darts: differentiable architecture search. In Proceedings of the international conference on learning representations.

  • Liu, X., Suganuma, M., Sun, Z., Okatani, T. (2019d). Dual residual networks leveraging the potential of paired operations for image restoration. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 7007–7016.

  • Liu, D., Wen, B., Fan, Y., Loy, C. C., Huang, T. S. (2018a). Non-local recurrent network for image restoration. In Proceedings in advances in neural information processing systems, pp. 1673–1682.

  • Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G. (2020). Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2359–2368.

  • Liu, P., El Basha, M., Li, Y., Xiao, Y., Sanelli, P., & Fang, R. (2019). Deep evolutionary networks with expedited genetic algorithms for medical image denoising. Medical Image Analysis, 54, 306–315.

    Article  Google Scholar 

  • Loshchilov, I., Hutter, F. (2017). Sgdr: Stochastic gradient descent with warm restarts. In Proceedings of the international conference on learning representation.

  • Ma, N., Zhang, X., Zheng, H. T., Sun, J. (2018). Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision, pp. 116–131.

  • Mao, X., Shen, C., Yang, Y. B. (2016). Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Proceedings of the advances in neural information processing systems, pp. 2802–2810.

  • Martin, D., Fowlkes, C., Tal, D., Malik, J., et al. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE international conference on computer vision, pp. 416–423.

  • Nekrasov, V., Chen, H., Shen, C., Reid, I. (2019). Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 9126–9135.

  • Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., Dean, J. (2018). Efficient neural architecture search via parameter sharing. In Proceedings of the international conference on machine learning.

  • Plötz, T., and Roth, S. (2018). Neural nearest neighbors networks. In Proceedings of the advances in neural information processing systems, pp. 1087–1098.

  • Plötz, T., Roth, S. (2018). Neural nearest neighbors networks. In Proceedings of the advances in neural information processing systems pp. 1673–1682.

  • Suganuma, M., Ozay, M., Okatani, T. (2018). Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search. In Proceedings of the international conference on machine learning.

  • Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence.

  • Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 2818–2826.

  • Szegedy, C., Wei, L., Jia. Y., Sermanet. P., Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern Recognition.

  • Tai, Y., Yang, J., Liu, X., Xu, C. (2017). Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE international conference on computer vision, pp. 4539–4547.

  • Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q. (2019). Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 2820–2828.

  • Wan, A., Dai, X., Zhang, P., He, Z., Tian, Y., Xie, S., Wu, B., Yu, M., Xu, T., Chen, K., Vajda, P., Gonzalez, J. E. (2020). Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 12965–12974.

  • Wang, N., Gao, Y., Chen, H., Wang, P., Tian, Z., Shen, C. (2020). NAS-FCOS: Fast neural architecture search for object detection. In Proceedings of the IEEE conference on computer vision and pattern Recognition.

  • Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K. (2019). Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 10734–10742.

  • Wu, X., Liu, M., Cao, Y., Ren, D., Zuo, W. (2020). Unpaired learning of deep image denoising. In European conference on computer vision, Springer, pp. 352–368.

  • Xu, Y., Xie, L., Dai, W., Zhang, X., Chen, X., Qi, G. J., Xiong, H., Tian, Q. (2021). Partially-connected neural architecture search for reduced computational redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence.

  • Zhang, H., Li, Y., Chen, H., Shen, C. (2020). Memory-efficient hierarchical neural architecture search for image denoising. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 3657–3666.

  • Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y. (2018d). Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision, pp. 286–301.

  • Zhang, C., Ren, M., Urtasun, R. (2018a). Graph hypernetworks for neural architecture search. arXiv:1810.05749

  • Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y. (2018e). Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 2472–2481.

  • Zhang, K., Zuo, W., Gu, S., Zhang, L. (2017b). Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 3929–3938.

  • Zhang, K., Zuo, W., Zhang, L. (2018c). Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 3262–3271.

  • Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7), 3142–3155.

    Article  MathSciNet  Google Scholar 

  • Zhang, K., Zuo, W., & Zhang, L. (2018). FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, 27(9), 4608–4622.

    Article  MathSciNet  Google Scholar 

  • Zhong, Z., Yan, J., Wu, W., Shao, J., Liu, C. L. (2018). Practical block-wise neural network architecture generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2423–2432.

  • Zoph, B., Le, Q. V. (2017). Neural architecture search with reinforcement learning. In Proceedings of the international conference on learning representations.

  • Zoph, B., Vasudevan, V., Shlens, J., Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 8697–8710.

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, Northwestern Polytechnical University, Xi’an, China

    Haokui Zhang, Ying Li & Chengrong Gong

  2. Intellifusion, Shenzhen, China

    Haokui Zhang

  3. Shaanxi Key Laboratory of Intelligent Processing for Big Energy Data, Yan’an University, Yan’an, China

    Zongwen Bai

  4. Zhejiang University, Hangzhou, China

    Hao Chen & Chunhua Shen

Authors
  1. Haokui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chengrong Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zongwen Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chunhua Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haokui Zhang.

Additional information

Communicated by Bumsub Ham.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Part of this work was done when H. Zhang, H. Chen and C. Shen were with The University of Adelaide. This work was in part supported by National Natural Science Foundation of China (61871460, 61941112, 61761042), Shaanxi Provincial Key RD Program (2020KW-003) and Natural Science Foundation of China of Shaanxi (2020JM-556). Y. Li and C. Shen are the corresponding authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Li, Y., Chen, H. et al. Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration. Int J Comput Vis 130, 157–178 (2022). https://doi.org/10.1007/s11263-021-01537-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-021-01537-w

Keywords

Search

Navigation