Skip to main content

Pruning and quantization algorithm with applications in memristor-based convolutional neural network

Cognitive Neurodynamics (2023)Cite this article

Abstract

The human brain’s ultra-low power consumption and highly parallel computational capabilities can be accomplished by memristor-based convolutional neural networks. However, with the rapid development of memristor-based convolutional neural networks in various fields, more complex applications and heavier computations lead to the need for a large number of memristors, which makes power consumption increase significantly and the network model larger. To mitigate this problem, this paper proposes an SBT-memristor-based convolutional neural network architecture and a hybrid optimization method combining pruning and quantization. Firstly, SBT-memristor-based convolutional neural network is constructed by using the good thresholding property of the SBT memristor. The memristive in-memory computing unit, activation unit and max-pooling unit are designed. Then, the hybrid optimization method combining pruning and quantization is used to improve the SBT-memristor-based convolutional neural network architecture. This hybrid method can simplify the memristor-based neural network and represent the weights at the memristive synapses better. Finally, the results show that the SBT-memristor-based convolutional neural network reduces a large number of memristors, decreases the power consumption and compresses the network model at the expense of a little precision loss. The SBT-memristor-based convolutional neural network obtains faster recognition speed and lower power consumption in MNIST recognition. It provides new insights for the complex application of convolutional neural networks.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

References

  • Abdel-Hamid O, Ar Mohamed, Jiang H et al (2014) Convolutional neural networks for speech recognition. IEEE/ACM Trans Audio, Speech, Lang Process 22(10):1533–1545

    Article  Google Scholar 

  • Ajit A, Acharya K, Samanta A (2020) A review of convolutional neural networks. In: 2020 international conference on emerging trends in information technology and engineering (ic-ETITE), IEEE, pp 1–5

  • Albawi S, Mohammed TA, Al-Zawi S (2017) Understanding of a convolutional neural network. In: 2017 international conference on engineering and technology (ICET), IEEE, pp 1–6

  • Alzubaidi L, Zhang J, Humaidi AJ et al (2021) Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J big Data 8(1):1–74

    Article  Google Scholar 

  • Bisen D (2021) Deep convolutional neural network based plant species recognition through features of leaf. Multimed Tools Appl 80(4):6443–6456

    Article  Google Scholar 

  • Burr GW, Shelby RM, Sebastian A et al (2017) Neuromorphic computing using non-volatile memory. Adv Phys: X 2(1):89–124

    Google Scholar 

  • Chen B, Cai F, Zhou J, et al (2015) Efficient in-memory computing architecture based on crossbar arrays. In: 2015 IEEE International Electron Devices Meeting (IEDM), IEEE, pp 17–5

  • Chen J, Wu Y, Yang Y et al (2021) An efficient memristor-based circuit implementation of squeeze-and-excitation fully convolutional neural networks. IEEE Trans Neural Netw Learn Syst 33(4):1779–1790

    Article  Google Scholar 

  • Dou G, Yu Y, Guo M et al (2017) Memristive behavior based on Ba-doped SrTIO3 films. Chin Phys Lett 34(3):038502

    Article  Google Scholar 

  • Drakopoulos F, Baby D, Verhulst S (2021) A convolutional neural-network framework for modelling auditory sensory cells and synapses. Commun Biol 4(1):1–17

    Article  Google Scholar 

  • Gandhi UD, Malarvizhi Kumar P, Chandra Babu G, et al (2021) Sentiment analysis on twitter data by using convolutional neural network (CNN) and long short term memory (LSTM). Wirel Pers Commun. https://doi.org/10.1007/s11277-021-08580-3

  • Gao F, Huang T, Sun J et al (2019) A new algorithm for SAR image target recognition based on an improved deep convolutional neural network. Cogn Comput 11(6):809–824

    Article  Google Scholar 

  • Ghimire D, Kil D, Kim Sh (2022) A survey on efficient convolutional neural networks and hardware acceleration. Electronics 11(6):945

    Article  Google Scholar 

  • Goldani MH, Safabakhsh R, Momtazi S (2021) Convolutional neural network with margin loss for fake news detection. Inf Process Manag 58(1):102418

    Article  Google Scholar 

  • Gu J, Wang Z, Kuen J et al (2018) Recent advances in convolutional neural networks. Pattern Recogn 77:354–377

    Article  Google Scholar 

  • Guo M, Zhu Y, Liu R et al (2022) An associative memory circuit based on physical memristors. Neurocomputing 472:12–23

    Article  Google Scholar 

  • Han S, Mao H, Dally WJ (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

  • Hu X, Shi W, Zhou Y, et al (2022) Quantized and adaptive memristor based CNN (QA-mCNN) for image processing. Sci chin-inf Sci 65(1):119104. https://doi.org/10.1007/s11432-020-3031-9

  • Huang HM, Wang Z, Wang T et al (2020) Artificial neural networks based on memristive devices: from device to system. Adv Intell Syst 2(12):2000149

    Article  Google Scholar 

  • Jeong H, Shi L (2018) Memristor devices for neural networks. J Phys D Appl Phys 52(2):023003

    Article  Google Scholar 

  • Ji X, Hu X, Zhou Y et al (2019) Adaptive sparse coding based on memristive neural network with applications. Cogn Neurodyn 13(5):475–488

    Article  Google Scholar 

  • LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  CAS  Google Scholar 

  • Lehtonen E, Laiho M (2010) CNN using memristors for neighborhood connections. In: 2010 12th International Workshop on Cellular Nanoscale Networks and their Applications (CNNA 2010), IEEE, pp 1–4

  • Li C, Belkin D, Li Y et al (2018) Efficient and self-adaptive in-situ learning in multilayer memristor neural networks. Nat Commun 9(1):1–8

    Google Scholar 

  • Li H, Wang S, Zhang X et al (2021a) Memristive crossbar arrays for storage and computing applications. Adv Intell Syst 3(9):2100017

    Article  Google Scholar 

  • Li Z, Liu F, Yang W et al (2021) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3084827

    Article  Google Scholar 

  • Liu J, Zhou Y, Duan S, et al (2022) Memristive neural network circuit implementation of associative learning with overshadowing and blocking. Cogn Neurodyn. https://doi.org/10.1007/s11571-022-09882-3

  • Mao WL, Chen WC, Wang CT et al (2021) Recycling waste classification using optimized convolutional neural network. Resour Conserv Recycl 164(105):132

    Google Scholar 

  • Shafiee A, Nag A, Muralimanohar N et al (2016) ISAAC: a convolutional neural network accelerator with in-situ analog arithmetic in crossbars. ACM SIGARCH Comput Archit News 44(3):14–26

    Article  Google Scholar 

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  • Wang J, Zhuge F (2019) Memristive synapses for brain-inspired computing. Adv Mater Technol 4(3):1800544

    Article  Google Scholar 

  • Wang Z, Li C, Lin P et al (2019) In situ training of feed-forward and recurrent convolutional memristor networks. Nat Mach Intell 1(9):434–442

    Article  Google Scholar 

  • Wen S, Wei H, Yan Z et al (2019) Memristor-based design of sparse compact convolutional neural network. IEEE Trans Netw Sci Eng 7(3):1431–1440

    Article  Google Scholar 

  • Xia Z, Chen J, Huang Q et al (2020) Neural synaptic plasticity-inspired computing: a high computing efficient deep convolutional neural network accelerator. IEEE Trans Circuits Syst I Regul Pap 68(2):728–740

    Article  Google Scholar 

  • Xu X, Zhu Z, Wang Y et al (2022) Odor pattern recognition of a novel bio-inspired olfactory neural network based on kernel clustering. Commun Nonlinear Sci Numer Simul 109(106):274

    Google Scholar 

  • Yakopcic C, Alom MZ, Taha TM (2016) Memristor crossbar deep network implementation based on a convolutional neural network. In: 2016 International joint conference on neural networks (IJCNN), IEEE, pp 963–970

  • Yakopcic C, Alom MZ, Taha TM (2017) Extremely parallel memristor crossbar architecture for convolutional neural network implementation. In: 2017 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1696–1703

  • Yang CHH, Qi J, Chen SYC et al (2021) Decentralizing feature extraction with quantum convolutional neural network for automatic speech recognition. In: ICASSP 2021–2021 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), IEEE, pp 6523–6527

  • Yao P, Wu H, Gao B et al (2020) Fully hardware-implemented memristor convolutional neural network. Nature 577(7792):641–646

    Article  CAS  Google Scholar 

  • Zhang Q, Zhang M, Chen T et al (2019) Recent advances in convolutional neural network acceleration. Neurocomputing 323:37–51

    Article  Google Scholar 

  • Zhang Y, Wang X, Friedman EG (2017) Memristor-based circuit design for multilayer neural networks. IEEE Trans Circuits Syst I Regul Pap 65(2):677–686

    Article  Google Scholar 

  • Zhang Y, Wang Z, Zhu J et al (2020) Brain-inspired computing with memristors: challenges in devices, circuits, and systems. Appl Phys Rev 7(1):011308

    Article  CAS  Google Scholar 

  • Zhang Y, Wu Z, Liu S et al (2021) A quantized convolutional neural network implemented with memristor for image denoising and recognition. Front Neurosci. https://doi.org/10.3389/fnins.2021.717222

    Article  Google Scholar 

  • Zhou Y, Hu X, Wang L et al (2021) Quantbayes: weight optimization for memristive neural networks via quantization-aware bayesian inference. IEEE Trans Circuits Syst I Regul Pap 68(12):4851–4861

    Article  Google Scholar 

  • Zhu M, Gupta S (2017) To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 62176143, 61703246), the Natural Science Foundation of Shandong Province (ZR2022MF225, ZR2021MF001), the Talented Young Teachers Training Program of Shandong University of Science and Technology, and the Elite Project of Shandong University of Science and Technology.

Author information

Authors and Affiliations

  1. College of Electrical Engineering and Automation, Shandong University of Science and Technology, Qingdao, 266590, China

    Mei Guo, Yurui Sun, Yongliang Zhu & Gang Dou

  2. The University of Nottingham Ningbo China, Ningbo, 315100, China

    Mingqiao Han

  3. The Australian AI Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, 2007, Australia

    Shiping Wen

Authors
  1. Mei Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yurui Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongliang Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mingqiao Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gang Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shiping Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gang Dou.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Guo, M., Sun, Y., Zhu, Y. et al. Pruning and quantization algorithm with applications in memristor-based convolutional neural network. Cogn Neurodyn (2023). https://doi.org/10.1007/s11571-022-09927-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11571-022-09927-7

Keywords

  • Memristor
  • Convolutional neural network
  • Network pruning
  • Quantization weight