Skip to main content
SpringerLink
Log in

Binarizing Super-Resolution Neural Network Without Batch Normalization

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14434))

Included in the following conference series:

  • 513 Accesses

Abstract

In this paper, our objective is to propose a model binarization method aimed at addressing the challenges posed by over-parameterized super-resolution (SR) models. Our analysis reveals that binary SR models experience significant performance degradation, primarily attributed to their sensitivity towards weight/activation distributions, particularly when devoid of Batch Normalization (BN) layers. Consequently, we undertake the following endeavors in this study: First, we conduct a comprehensive analysis to examine the impact of BN layers on SR models based on Binary Neural Networks (BNNs). Second, we propose an asymmetric binarizer that can be reparameterized to adaptively adjust the transition point for activation binarization. Third, we introduce a progressive gradient estimator that modifies weight smoothness and controls weight flipping to stabilize the training procedure in the absence of BN layers. Through extensive experiments, we demonstrate that our proposed method exhibits significant performance improvements. For instance, when binarizing EDSR and scaling up input images by a factor of \(\times 4\), our approach achieves a PSNR decrease of less than 0.4dB on the Urban100 benchmark.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    BN layers are inserted within each channel. For simplicity, we omit the channel-wise index here.

References

  1. Bevilacqua, M., Roumy, A., Guillemot, C., Morel, M.L.A.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: British Machine Vision Conference (BMVC) (2012)

    Google Scholar 

  2. Chen, B., et al.: Arm: any-time super-resolution method. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XIX, pp. 254–270. Springer (2022). https://doi.org/10.1007/978-3-031-19800-7

  3. Chen, T., Zhang, Z., Ouyang, X., Liu, Z., Shen, Z., Wang, Z.: “bnn-bn=?”: training binary neural networks without batch normalization. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pp. 4619–4629 (2021)

    Google Scholar 

  4. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 38(2), 295–307 (2015)

    Article  Google Scholar 

  5. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  6. Gao, G., Li, W., Li, J., Wu, F., Lu, H., Yu, Y.: Feature distillation interaction weighting network for lightweight image super-resolution. In: Proceedings of the AAAI conference on artificial intelligence (AAAI), vol. 36, pp. 661–669 (2022)

    Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  8. Helwegen, K., Widdicombe, J., Geiger, L., Liu, Z., Cheng, K.T., Nusselder, R.: Latent weights do not exist: Rethinking binarized neural network optimization. In: Advances in Neural Information Processing Systems (NeurIPS) 32 (2019)

    Google Scholar 

  9. Hong, C., Baik, S., Kim, H., Nah, S., Lee, K.M.: Cadyq: content-aware dynamic quantization for image super-resolution. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part VII, pp. 367–383. Springer (2022). https://doi.org/10.1007/978-3-031-20071-7_22

  10. Hong, C., Kim, H., Baik, S., Oh, J., Lee, K.M.: Daq: channel-wise distribution-aware quantization for deep image super-resolution networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2675–2684 (2022)

    Google Scholar 

  11. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700–4708 (2017)

    Google Scholar 

  12. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015)

    Google Scholar 

  13. Jiang, X., Wang, N., Xin, J., Li, K., Yang, X., Gao, X.: Training binary neural network without batch normalization for image super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 1700–1707 (2021)

    Google Scholar 

  14. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016)

    Google Scholar 

  15. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690 (2017)

    Google Scholar 

  16. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690 (2017)

    Google Scholar 

  17. Li, H., et al.: PAMS: quantized super-resolution via parameterized max scale. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 564–580. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_34

    Chapter  Google Scholar 

  18. Li, K., et al.: Local means binary networks for image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS) (2022)

    Google Scholar 

  19. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 136–144 (2017)

    Google Scholar 

  20. Lin, M., et al.: Rotated binary neural network. Adv. Neural Inform. Process. Syst. (NeurIPS) 33, 7474–7485 (2020)

    Google Scholar 

  21. Liu, Z., Shen, Z., Li, S., Helwegen, K., Huang, D., Cheng, K.T.: How do adam and training strategies help bnns optimization. In: International Conference on Machine Learning (ICML), pp. 6936–6946. PMLR (2021)

    Google Scholar 

  22. Liu, Z., et al.: Bi-Real Net: enhancing the performance of 1-Bit CNNs with improved representational capability and advanced training algorithm. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 747–763. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0_44

    Chapter  Google Scholar 

  23. Ma, Y., Xiong, H., Hu, Z., Ma, L.: Efficient super resolution using binarized neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (2019)

    Google Scholar 

  24. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 416–423 (2001)

    Google Scholar 

  25. Qin, H., et al.: Forward and backward information retention for accurate binary neural networks. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pp. 2250–2259 (2020)

    Google Scholar 

  26. Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: imagenet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_32

    Chapter  Google Scholar 

  27. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  28. Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1920–1927 (2013)

    Google Scholar 

  29. Tipping, M., Bishop, C.: Bayesian image super-resolution. In: Advances in Neural Information Processing Systems (NeurIPS) 15 (2002)

    Google Scholar 

  30. Tu, Z., Chen, X., Ren, P., Wang, Y.: Adabin: improving binary neural networks with adaptive binary sets. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XI. pp. 379–395. Springer (2022). https://doi.org/10.1007/978-3-031-20083-0_23

  31. Wang, P., He, X., Cheng, J.: Toward accurate binarized neural networks with sparsity for mobile application. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS) (2022)

    Google Scholar 

  32. Wang, Z., Wu, Z., Lu, J., Zhou, J.: Bidet: an efficient binarized object detector. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pp. 2049–2058 (2020)

    Google Scholar 

  33. Xin, J., Wang, N., Jiang, X., Li, J., Huang, H., Gao, X.: Binarized neural network for single image super resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 91–107. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_6

    Chapter  Google Scholar 

  34. Xu, S., et al.: Ida-det: an information discrepancy-aware distillation for 1-bit detectors. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XI, pp. 346–361. Springer (2022). https://doi.org/10.1007/978-3-031-20083-0_21

  35. Xu, S., Zhao, J., Lu, J., Zhang, B., Han, S., Doermann, D.: Layer-wise searching for 1-bit detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5682–5691 (2021)

    Google Scholar 

  36. Xu, Z., et al.: Recu: reviving the dead weights in binary neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5198–5208 (2021)

    Google Scholar 

  37. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2472–2481 (2018)

    Google Scholar 

  38. Zhong, Y., et al.: Dynamic dual trainable bounds for ultra-low precision super-resolution networks. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XVIII, pp. 1–18. Springer (2022). https://doi.org/10.1007/978-3-031-19797-0_1

  39. Zhu, Y., Zhang, Y., Yuille, A.L.: Single image super-resolution using deformable patches. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2917–2924 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Informatics, Xiamen University, Fujian, 361005, People’s Republic of China

    Xunchao Li & Fei Chao

Authors
  1. Xunchao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Chao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fei Chao .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 65 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, X., Chao, F. (2024). Binarizing Super-Resolution Neural Network Without Batch Normalization. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14434. Springer, Singapore. https://doi.org/10.1007/978-981-99-8549-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8549-4_6

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8548-7

  • Online ISBN: 978-981-99-8549-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation