Skip to main content
SpringerLink
Log in

How to Train Accurate BNNs for Embedded Systems?

  • Chapter
  • First Online:
Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing

Abstract

A key enabler of deploying convolutional neural networks on resource-constrained embedded systems is the binary neural network (BNN). BNNs save on memory and simplify computation by binarizing both features and weights. Unfortunately, binarization is accompanied by a severe decrease in accuracy. To reduce the accuracy gap between binary and full-precision networks, many repair methods have been proposed in the recent past, which we have classified and put into a single overview. The repair methods are divided into two main branches, training techniques and network topology changes, which are further split into smaller categories. We observe progress in reducing the accuracy gap, but BNN papers are not aligned on what repair methods should be used. Therefore, we conduct an empirical review that evaluates the benefits of repair methods in isolation on two benchmarks: CIFAR10 on ResNet-20 and CIFAR100 on ResNet-18. We found three repair categories most beneficial: feature binarizer, feature normalization, and double residual. Based on this review, we discuss future directions and research opportunities. We sketch the benefit and costs associated with BNNs on embedded systems because it remains to be seen whether BNNs will be able to close the accuracy gap while staying highly energy-efficient on resource-constrained embedded systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We think that the authors have made a typing mistake in their paper and meant median instead of mean.

  2. 2.

    We have spent over 4000 GPU-hours to get the results of our review.

References

  1. Alizadeh, M., Fernández-Marqués, J., Lane, N., Gal, Y.: An Empirical study of Binary Neural Networks’ Optimisation. In: International Conference on Learning Representations (2019)

    Google Scholar 

  2. Andri, R., Karunaratne, G., Cavigelli, L., Benini, L.: ChewBaccaNN: A flexible 223 TOPS/W BNN accelerator. Proceedings of the IEEE International Symposium on Circuits and Systems 2021-May (2021). https://doi.org/10.1109/ISCAS51556.2021.9401214

  3. Bethge, J., Bartz, C., Yang, H., Chen, Y., Meinel, C.: MeliusNet: An Improved Network Architecture for Binary Neural Networks. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1438–1447. IEEE, New York (2021). https://doi.org/10.1109/WACV48630.2021.00148

  4. Bethge, J., Yang, H., Bornstein, M., Meinel, C.: BinaryDenseNet: Developing an Architecture for Binary Neural Networks. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 1951–1960. IEEE, New York (2019). https://doi.org/10.1109/ICCVW.2019.00244

  5. Bulat, A., Martínez, B., Tzimiropoulos, G.: High-Capacity Expert Binary Networks. In: International Conference on Learning Representations (2021)

    Google Scholar 

  6. Bulat, A., Tzimiropoulos, G.: XNOR-Net++: Improved Binary Neural Networks. In: 30th British Machine Vision Conference 2019, BMVC 2019. BMVA Press, New York (2019)

    Google Scholar 

  7. Bulat, A., Tzimiropoulos, G., Kossaifi, J., Pantic, M.: Improved Training of Binary Networks for Human Pose Estimation and Image Recognition (2019)

    Google Scholar 

  8. Chen, T., Zhang, Z., Ouyang, X., Liu, Z., Shen, Z., Wang, Z.: “BNN - BN = ?”: Training Binary Neural Networks without Batch Normalization. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4614–4624. IEEE, New York (2021). https://doi.org/10.1109/CVPRW53098.2021.00520

  9. Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. In: Advances in Neural Information Processing Systems (2016)

    Google Scholar 

  10. Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: AutoAugment: Learning Augmentation Policies from Data. In: Cvpr 2019 (Section 3), pp. 113–123 (2018). https://doi.org/10.48550/arxiv.1805.09501

  11. Darabi, S., Belbahri, M., Courbariaux, M., Nia, V.P.: Regularized binary network training. In: EMCˆ2: Workshop on Energy Efficient Machine Learning and Cognitive Computing (2018)

    Google Scholar 

  12. Diffenderfer, J., Kailkhura, B.: Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network. In: International Conference on Learning Representations (2021)

    Google Scholar 

  13. Guo, N., Bethge, J., Yang, H., Zhong, K., Ning, X., Meinel, C., Wang, Y.: BoolNet: Minimizing The Energy Consumption of Binary Neural Networks (2021)

    Google Scholar 

  14. Han, K., Wang, Y., Xu, Y., Xu, C., Wu, E., Xu, C.: Training binary neural networks through learning with noisy supervision. In: Proceedings of the 37th International Conference on Machine Learning, pp. 4017–4026 (2020)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2016-December, 770–778 (2015). https://doi.org/10.48550/arxiv.1512.03385

  16. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015). https://doi.org/10.1109/ICCV.2015.123

  17. He, X., Mo, Z., Cheng, K., Xu, W., Hu, Q., Wang, P., Liu, Q., Cheng, J.: ProxyBNN: Learning Binarized Neural Networks via Proxy Matrices. In: Proceedings of the European Conference on Computer Vision (ECCV), vol. 12348 LNCS, pp. 223–241. Springer Science and Business Media Deutschland GmbH, New York (2020). https://doi.org/10.1007/978-3-030-58580-8_14

  18. Helwegen, K., Widdicombe, J., Geiger, L., Liu, Z., Cheng, K.T., Nusselder, R.: Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization. In: Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  19. Ignatov, D., Ignatov, A.: Controlling information capacity of binary neural network. Pattern Recogn. Lett. 138, 276–281 (2020). https://doi.org/10.1016/j.patrec.2020.07.033

    Article  Google Scholar 

  20. Jiang, X., Wang, N., Xin, J., Li, K., Yang, X., Gao, X.: Training Binary Neural Network without Batch Normalization for Image Super-Resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1700–1707 (2021)

    Google Scholar 

  21. Jouppi, N.P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P.L., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T.V., Gottipati, R., Gulland, W., Hagmann, R., Richard Ho, C., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Killebrew, D., Koch, A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Ross, M., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang, W., Wilcox, E., Yoon, D.H.: In-datacenter performance analysis of a tensor processing unit. In: Proceedings of the International Symposium on Computer Architecture Part F128643, pp. 1–12 (2017). https://doi.org/10.1145/3079856.3080246

  22. Lee, J., Kim, D., Ham, B.: Network Quantization with Element-wise Gradient Scaling. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6444–6453. IEEE, New York (2021). https://doi.org/10.1109/CVPR46437.2021.00638

  23. Li, Y., Geng, T., Li, A., Yu, H.: BCNN: Binary complex neural network. Microprocess. Microsyst. 87, 104359 (2021). https://doi.org/10.1016/J.MICPRO.2021.104359

    Article  Google Scholar 

  24. Li, Y., Pintea, S.L., van Gemert, J.C.: Equal Bits: Enforcing Equally Distributed Binary Network Weights. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)

    Google Scholar 

  25. Lin, M., Ji, R., Xu, Z., Zhang, B., Wang, Y., Wu, Y., Huang, F., Lin, C.W., Lab, P.C.: Rotated Binary Neural Network. In: Advances in Neural Information Processing Systems, pp. 7474–7485 (2020)

    Google Scholar 

  26. Lin, X., Zhao, C., Pan, W.: Towards Accurate Binary Convolutional Neural Network. In: Advances in Neural Information Processing Systems, vol. 2017-December, pp. 345–353. Neural Information Processing Systems Foundation, California (2017)

    Google Scholar 

  27. Liu, C., Chen, P., Zhuang, B., Shen, C., Zhang, B., Ding, W.: SA-BNN: State-Aware Binary Neural Network. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2091–2099 (2021)

    Google Scholar 

  28. Liu, C., Ding, W., Xia, X., Zhang, B., Gu, J., Liu, J., Ji, R., Doermann, D.: Circulant Binary Convolutional Networks: Enhancing the Performance of 1-Bit DCNNs With Circulant Back Propagation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2019-June, pp. 2686–2694. IEEE, New York (2019). https://doi.org/10.1109/CVPR.2019.00280

  29. Liu, Z., Shen, Z., Li, S., Helwegen, K., Huang, D., Cheng, K.T.: How Do Adam and Training Strategies Help BNNs Optimization? In: Proceedings of the 38th International Conference on Machine Learning, pp. 6936–6946 (2021)

    Google Scholar 

  30. Liu, Z., Shen, Z., Savvides, M., Cheng, K.T.: ReActNet: towards precise binary neural network with generalized activation functions. In: Proceedings of the European Conference on Computer Vision (ECCV), vol. 12359 LNCS, pp. 143–159. Springer Science and Business Media Deutschland GmbH, New York (2020). https://doi.org/10.1007/978-3-030-58568-6_9

  31. Liu, Z., Wu, B., Luo, W., Yang, X., Liu, W., Cheng, K.T.: Bi-Real Net: Enhancing the Performance of 1-Bit CNNs with Improved Representational Capability and Advanced Training Algorithm. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 722–737 (2018). https://doi.org/10.1007/978-3-030-01267-0_44

  32. Martínez, B., Yang, J., Bulat, A., Tzimiropoulos, G.: Training binary neural networks with real-to-binary convolutions. In: International Conference on Learning Representations (2020)

    Google Scholar 

  33. Qin, H., Gong, R., Liu, X., Bai, X., Song, J., Sebe, N.: Binary neural networks: A survey. Pattern Recogn. 105, 107,281 (2020). https://doi.org/10.1016/j.patcog.2020.107281

    Article  Google Scholar 

  34. Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., Song, J.: Forward and backward information retention for accurate binary neural networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2247–2256. IEEE, New York (2020). https://doi.org/10.1109/CVPR42600.2020.00232

  35. Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. In: Proceedings of the European Conference on Computer Vision (ECCV), vol. 9908 LNCS, pp. 525–542. Springer, Berlin (2016). https://doi.org/10.1007/978-3-319-46493-0_32

  36. Redfern, A.J., Zhu, L., Newquist, M.K.: BCNN: A Binary CNN With All Matrix Ops Quantized To 1 Bit Precision. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4599–4607. IEEE, New York (2021). https://doi.org/10.1109/CVPRW53098.2021.00518

  37. Sakr, C., Choi, J., Wang, Z., Gopalakrishnan, K., Shanbhag, N.: True gradient-based training of deep binary activated neural networks via continuous binarization. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 2018-April, pp. 2346–2350. Institute of Electrical and Electronics Engineers Inc., New York (2018). https://doi.org/10.1109/ICASSP.2018.8461456

  38. Scherer, M., Rutishauser, G., Cavigelli, L., Benini, L.: CUTIE: Beyond PetaOp/s/W Ternary DNN Inference Acceleration with Better-than-Binary Energy Efficiency. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 41(4), 1020–1033 (2021). https://doi.org/10.1109/TCAD.2021.3075420

    Article  Google Scholar 

  39. Shen, M., Liu, X., Gong, R., Han, K.: Balanced binary neural networks with gated residual. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 2020-May, pp. 4197–4201. IEEE, New York (2020). https://doi.org/10.1109/ICASSP40776.2020.9054599

  40. Simons, T., Lee, D.J.: A review of binarized neural networks. Electronics 8(6), 661 (2019). https://doi.org/10.3390/electronics8060661

    Article  Google Scholar 

  41. Suarez-Ramirez, C.D., Gonzalez-Mendoza, M., Chang, L., Ochoa-Ruiz, G., Duran-Vega, M.A.: A bop and beyond: a second order optimizer for binarized neural networks. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1273–1281. IEEE, New York (2021). https://doi.org/10.1109/CVPRW53098.2021.00140

  42. Tang, W., Hua, G., Wang, L.: How to train a compact binary neural network with high accuracy? In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  43. Tieleman, T., Hinton, G.: COURSERA: Neural networks for machine learning. In: Lecture 9c: Using Noise as a Regularizer (2012)

    Google Scholar 

  44. Wang, P., He, X., Li, G., Zhao, T., Cheng, J.: Sparsity-inducing binarized neural networks. Proceedings of the AAAI Conference on Artificial Intelligence 34(07), 12192–12199 (2020). https://doi.org/10.1609/AAAI.V34I07.6900

    Article  Google Scholar 

  45. Wang, Y., Yang, Y., Sun, F., Yao, A.: Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5360–5369 (2021)

    Google Scholar 

  46. Wang, Z., Lu, J., Tao, C., Zhou, J., Tian, Q.: Learning channel-wise interactions for binary convolutional neural networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 568–577. IEEE (2019). https://doi.org/10.1109/CVPR.2019.00066

  47. Xin, J., Wang, N., Jiang, X., Li, J., Huang, H., Gao, X.: Binarized neural network for single image super resolution. In: Proceedings of the European Conference on Computer Vision (ECCV), vol. 12349 LNCS, pp. 91–107. Springer Science and Business Media Deutschland GmbH, New York (2020). https://doi.org/10.1007/978-3-030-58548-8_6

  48. Xu, W., Chen, Q., He, X., Wang, P., Cheng, J.: Improving Binary Neural Networks through Fully Utilizing Latent Weights (2021)

    Google Scholar 

  49. Xu, Z., Lin, M., Liu, J., Chen, J., Shao, L., Gao, Y., Tian, Y., Ji, R.: ReCU: Reviving the Dead Weights in Binary Neural Networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5198–5208 (2021)

    Google Scholar 

  50. Xue, P., Lu, Y., Chang, J., Wei, X., Wei, Z.: Self-Distribution Binary Neural Networks (2021). https://doi.org/10.1007/s10489-022-03348-z

  51. Yuan, C., Agaian, S.S.: A comprehensive review of Binary Neural Network (2021)

    Google Scholar 

  52. Zhang, J., Su, Z., Feng, Y., Lu, X., Pietikäinen, M., Liu, L.: Dynamic Binary Neural Network by learning channel-wise thresholds (2021)

    Google Scholar 

  53. Zhang, Y., Zhang, Z., Lew, L.: PokeBNN: A Binary Pursuit of Lightweight Accuracy (2021)

    Google Scholar 

  54. Zhao, J., Yang, L., Zhang, B., Guo, G., Doermann, D.: Uncertainty-aware Binary Neural Networks. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 3441–3447. International Joint Conferences on Artificial Intelligence Organization, California (2021). https://doi.org/10.24963/ijcai.2021/474

  55. Zhu, S., Dong, X., Su, H.: Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit? In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2019-June, pp. 4918–4927. IEEE, New York (2019). https://doi.org/10.1109/CVPR.2019.00506

  56. Zhuang, B., Shen, C., Tan, M., Liu, L., Reid, I.: Structured binary neural networks for accurate image classification and semantic segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2019-June, pp. 413–422. IEEE, New York (2019). https://doi.org/10.1109/CVPR.2019.00050

  57. Zou, W., Cheng, S., Wang, L., Fu, G., Shang, D., Zhou, Y., Zhan, Y.: Increasing information entropy of both weights and activations for the binary neural networks. Electronics 10(16), 1943 (2021). https://doi.org/10.3390/electronics10161943

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Eindhoven Artificial Intelligence Systems Institute and PARsE lab, Eindhoven University of Technology, Eindhoven, The Netherlands

    F. A. M. de Putter & Henk Corporaal

Authors
  1. F. A. M. de Putter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henk Corporaal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. A. M. de Putter .

Editor information

Editors and Affiliations

  1. Colorado State University, Fort Collins, CO, USA

    Sudeep Pasricha

  2. New York University Abu Dhabi, Abu Dhabi, Abu Dhabi, United Arab Emirates

    Muhammad Shafique

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Putter, F.A.M.d., Corporaal, H. (2024). How to Train Accurate BNNs for Embedded Systems?. In: Pasricha, S., Shafique, M. (eds) Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-39932-9_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-39932-9_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-39931-2

  • Online ISBN: 978-3-031-39932-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation