Skip to main content
SpringerLink
Log in

Self-distribution binary neural networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In this work, we study network binarization (i.e., binary neural networks, BNNs), which is one of the most promising techniques in network compression for convolutional neural networks (CNNs). Although prior work has introduced many binarization methods that improve the accuracy of BNNs by minimizing the quantization error, there remains a non-negligible performance gap between the binarized model and the full-precision model. Given that feature representation is critical for deep neural networks and that in BNNs, the features only differ in signs, we argue that the impact on the accuracy of BNNs may be strongly related to the sign distribution of the network parameters in addition to the quantization error. To this end, Self-Distribution Binary Neural Network (SD-BNN) is proposed. First, we utilize Activation Self Distribution (ASD) to adaptively adjust the sign distribution of activations, thereby improving the sign differences of the outputs of the convolution. Second, we adjust the sign distribution of weights through Weight Self Distribution (WSD) and then fine-tune the sign distribution of the outputs of the convolution. Extensive experiments on the CIFAR-10 and ImageNet datasets with various network structures show that the proposed SD-BNN consistently outperforms state-of-the-art (SOTA) BNNs (e.g., 92.5% on CIFAR-10 and 66.5% on ImageNet with ResNet-18) with lower computational cost. Our code is available at https://github.com/pingxue-hfut/SD-BNN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Ding Y, Ma Z, Wen S, Xie J, Chang D, Si Z, Wu M, Ling H (2021) AP-CNN: Weakly supervised attention pyramid convolutional neural network for fine-grained visual classification. IEEE Trans Image Process 30:2826–2836. https://doi.org/10.1109/TIP.2021.3055617

    Article  Google Scholar 

  2. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  3. Wang Z, Lu J, Wu Z, Zhou J (2021) Learning efficient binarized object detectors with information compression. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3050464

  4. Chen C, Wei J, Peng C, Qin H (2021) Depth-quality-aware salient object detection. IEEE Trans Image Process 30:2350–2363. https://doi.org/10.1109/TIP.2021.3052069

    Article  Google Scholar 

  5. Kalayeh MM, Shah M (2021) On symbiosis of attribute prediction and semantic segmentation. IEEE Trans Pattern Anal Mach Intell 43(5):1620–1635. https://doi.org/10.1109/TPAMI.2019.2956039

    Article  Google Scholar 

  6. Tong Z, Xu P, Denoeux T (2021) Evidential fully convolutional network for semantic segmentation. Appl Intell 51(9):6376–6399. https://doi.org/10.1007/s10489-021-02327-0

    Article  Google Scholar 

  7. Ding G, Zhang S, Jia Z, Zhong J, Han J (2021) Where to prune: Using LSTM to guide data-dependent soft pruning. IEEE Trans Image Process 30:293–304. https://doi.org/10.1109/TIP.2020.3035028

    Article  Google Scholar 

  8. Singh P, Verma VK, Rai P, Namboodiri VP (2020) Acceleration of deep convolutional neural networks using adaptive filter pruning. IEEE J Sel Top Signal Process 14 (4):838–847. https://doi.org/10.1109/JSTSP.2020.2992390

    Article  Google Scholar 

  9. Gao H, Wang Z, Cai L, Ji S (2021) Channelnets: Compact and efficient convolutional neural networks via channel-wise convolutions. IEEE Trans Pattern Anal Mach Intell 43(8):2570–2581. https://doi.org/10.1109/TPAMI.2020.2975796

    Article  Google Scholar 

  10. Han K, Wang Y, Xu C, Xu C, Wu E, Tao D (2021) Learning versatile convolution filters for efficient visual recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3114368

  11. Li X, Li S, Omar B, Wu F, Li X (2021) Reskd: Residual-guided knowledge distillation. IEEE Trans Image Process 30:4735–4746. https://doi.org/10.1109/TIP.2021.3066051

    Article  Google Scholar 

  12. Gou J, Yu B, Maybank SJ, Tao D (2021) Knowledge distillation: A survey. Int J Comput Vis 129(6):1789–1819. https://doi.org/10.1007/s11263-021-01453-z

    Article  Google Scholar 

  13. Hubara I, Courbariaux M, Soudry D, El-yaniv R, Bengio Y (2017) Quantized neural networks: training neural networks with low precision weights and activations. J Mach Learn Res 18:187–118730

    MathSciNet  MATH  Google Scholar 

  14. Gong R, Liu X, Jiang S, Li T, Hu P, Lin J, Yu F, Yan J (2019) Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In: 2019 IEEE/CVF international conference on computer vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp 4851–4860. https://doi.org/10.1109/ICCV.2019.00495

  15. Tung F, Mori G (2020) Deep neural network compression by in-parallel pruning-quantization. IEEE Trans Pattern Anal Mach Intell 42(3):568–579. https://doi.org/10.1109/TPAMI.2018.2886192

    Article  Google Scholar 

  16. Huang C, Liu P, Fang L (2021) MXQN: Mixed quantization for reducing bit-width of weights and activations in deep convolutional neural networks. Appl Intell 51 (7):4561–4574. https://doi.org/10.1007/s10489-020-02109-0

    Article  Google Scholar 

  17. Fan Y, Pang W, Lu S (2021) HFPQ: Deep neural network compression by hardware-friendly pruning-quantization. Appl Intell 51(10):7016–7028. https://doi.org/10.1007/s10489-020-01968-x

    Article  Google Scholar 

  18. Hubara I, Courbariaux M, Soudry D, El-yaniv R, Bengio Y (2016) Binarized neural networks. In: Annual conference on neural information processing systems 2016, December 5-10, 2016, Barcelona, Spain, pp 4107–4115

  19. Qiao GC, Hu S, Chen TP, Rong LM, Ning N, Yu Q, Liu Y (2020) STBNN: Hardware-friendly spatio-temporal binary neural network with high pattern recognition accuracy. Neurocomputing 409:351–360. https://doi.org/10.1016/j.neucom.2020.06.084

    Article  Google Scholar 

  20. Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. In: Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pp 525–542. https://doi.org/10.1007/978-3-319-46493-0_32

  21. Bulat A, Tzimiropoulos G (2019) Xnor-net++: Improved binary neural networks. In: 30Th british machine vision conference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019, pp 62

  22. Martínez B, Yang J, Bulat A, Tzimiropoulos G (2020) Training binary neural networks with real-to-binary convolutions. In: 8Th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

  23. Qin H, Gong R, Liu X, Shen M, Wei Z, Yu F, Song J (2020) Forward and backward information retention for accurate binary neural networks. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp 2247–2256. https://doi.org/10.1109/CVPR42600.2020.00232

  24. Shen M, Liu X, Gong R, Han K (2020) Balanced binary neural networks with gated residual. In: 2020 IEEE International conference on acoustics, speech and signal processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pp 4197–4201. https://doi.org/10.1109/ICASSP40776.2020.9054599

  25. Ding R, Chin T, Liu Z, Marculescu D (2019) Regularizing activation distribution for training binarized deep networks. In: IEEE Conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp 11408–11417. https://doi.org/10.1109/CVPR.2019.01167

  26. Wang Z, Lu J, Zhou J (2021) Learning channel-wise interactions for binary convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 43(10):3432–3445. https://doi.org/10.1109/TPAMI.2020.2988262

    Article  Google Scholar 

  27. Torralba A, Fergus R, Freeman WT (2008) 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Trans Pattern Anal Mach Intell 30(11):1958–1970. https://doi.org/10.1109/TPAMI.2008.128

    Article  Google Scholar 

  28. Deng J, Dong W, Socher R, Li L, Li K, Fei-fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Computer society conference on computer vision and pattern recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp 248–255

  29. Liu Z, Luo W, Wu B, Yang X, Liu W, Cheng K (2020) Bi-real net: Binarizing deep network towards real-network performance. Int J Comput Vis 128 (1):202–219. https://doi.org/10.1007/s11263-019-01227-8

    Article  Google Scholar 

  30. Wu L, Lin X, Chen Z, Huang J, Liu H, Yang Y (2021) An efficient binary convolutional neural network with numerous skip connections for fog computing. IEEE Internet Things J 8(14):11357–11367. https://doi.org/10.1109/JIOT.2021.3052105

    Article  Google Scholar 

  31. Kim H, Kim K, Kim J, Kim J (2020) Binaryduo: Reducing gradient mismatch in binary activation network by coupling binary activations. In: 8Th international conference on learning representations, ICLR 2020, Addis Ababa, ethiopia, April 26-30, 2020

  32. Lin M, Ji R, Xu Z, Zhang B, Wang Y, Wu Y, Huang F, Lin C (2020) Rotated binary neural network. In: Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, neurIPS 2020, December 6-12, 2020, virtual

  33. Lan W, Lan L (2021) Compressing deep convolutional neural networks by stacking low-dimensional binary convolution filters. In: Thirty-fifth AAAI conference on artificial intelligence, AAAI 2021, virtual event, February 2-9, 2021, pp 8235–8242

  34. Lin X, Zhao C, Pan W (2017) Towards accurate binary convolutional neural network. In: Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp 345–353

  35. Pouransari H, Tu Z, Tuzel O (2020) Least squares binary quantization of neural networks. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition, CVPR workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp 2986–2996

  36. Liu C, Ding W, Hu Y, Xia X, Zhang B, Liu J, Doermann D (2020) Circulant binary convolutional networks for object recognition. IEEE J Sel Top Signal Process 14(4):884–893. https://doi.org/10.1109/JSTSP.2020.2969516

    Article  Google Scholar 

  37. Bethge J, Bartz C, Yang H, Chen Y, Meinel C (2021) Meliusnet: An improved network architecture for binary neural networks. In: IEEE Winter conference on applications of computer vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pp 1438–1447. https://doi.org/10.1109/WACV48630.2021.00148

  38. Zhuang B, Shen C, Tan M, Liu L, Reid ID (2019) Structured binary neural networks for accurate image classification and semantic segmentation. In: IEEE Conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp 413–422. https://doi.org/10.1109/CVPR.2019.00050

  39. Zhu S, Dong X, Su H (2019) Binary ensemble neural network: More bits per network or more networks per bit?. In: IEEE Conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp 4923–4932

  40. Zagoruyko S, Komodakis N (2017) Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In: 5Th international conference on learning representations, ICLR 2017, Toulon, France, April 24-26, 2017, conference track proceedings

  41. Bengio Y, Lėonard N, Courville AC (2013) Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR arXiv:abs/1308.3432

  42. Huang K, Ni B, Yang X (2019) Efficient quantization for neural networks with binary weights and low bitwidth activations. In: The thirty-third AAAI conference on artificial intelligence, AAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pp 3854–3861. https://doi.org/10.1609/aaai.v33i01.33013854

  43. Zhang D, Yang J, Ye D, Hua G (2018) Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In: Computer Vision - ECCV 2018 - 15th European conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII, pp 373–390. https://doi.org/10.1007/978-3-030-01237-3_23

  44. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  45. Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems 28: annual conference on neural information processing systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp 1135–1143

  46. Bulat A, Tzimiropoulos G, Kossaifi J, Pantic M (2019) Improved training of binary networks for human pose estimation and image recognition. CoRR arXiv:abs/1904.05868

  47. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: 2015 IEEE International conference on computer vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp 1026–1034. https://doi.org/10.1109/ICCV.2015.123

Download references

Acknowledgments

This work was supported in part by the Anhui Provincial Key Research and Development Program under Grant 202004a05020040, in part by the National Key Research and Development Program under Grant 2018YFC0604404, in part by Intelligent Network and New Energy Vehicle Special Project of Intelligent Manufacturing Institute of HFUT under Grant IMIWL2019003, and in part by Fundamental Research Funds for the Central Universities under Grant PA2021GDGP0061.

Author information

Authors and Affiliations

  1. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230009, China

    Ping Xue, Yang Lu, Jingfei Chang, Xing Wei & Zhen Wei

  2. Anhui Mine IOT and Security Monitoring Technology Key Laboratory, Hefei, 230009, China

    Yang Lu, Xing Wei & Zhen Wei

  3. Engineering Research Center of Safety Critical Industrial Measurement and Control Technology, Ministry of Education, Hefei University of Technology, Hefei, 230009, China

    Yang Lu & Zhen Wei

  4. Intelligent Manufacturing Institute of HeFei University of Technology, Hefei, 230009, China

    Xing Wei

Authors
  1. Ping Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingfei Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xing Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yang Lu or Zhen Wei.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, P., Lu, Y., Chang, J. et al. Self-distribution binary neural networks. Appl Intell 52, 13870–13882 (2022). https://doi.org/10.1007/s10489-022-03348-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03348-z

Keywords

Search

Navigation