Skip to main content
SpringerLink
Log in

Lightweight deep neural network from scratch

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In general, deep neural networks (DNNs) are seriously overparameterized with enormous hardware resources demanded, which creates a heavy burden for inference applications especially for resource-constrained edge devices. To overcome this difficulty, there are two principal solutions: optimizing the overparameterized DNNs and designing high-efficiency DNN algorithms with lightweight architectures. In terms of the optimization methods, pruning is the most effective technique because the solution fundamentally optimizes the bloated DNNs by removing redundant structures from the network and can be seamlessly incorporated into all other optimization solutions as well as all kinds of DNN architectures. Nevertheless, the study reported in this paper reveals that the various excellent but also complicated pruning algorithms may not be as effective as proposals demonstrate and do not yield optimal solutions for all cases. In addition, the current lightweight DNN architectures are also overparameterized to a large extent. In this research, we propose a mechanism for determining lightweight DNN networks From Scratch (FS-DNN). First, we conduct a thorough study on the theoretical basis of evaluating the hardware resources demanded by DNNs, and establish the objective function for determining a lightweight DNN network. Based on the study, the theoretical FS-DNN for determining lightweight DNNs from scratch with high efficiency is proposed. Then, we perform a series of experiments with FS-DNN based lightweight DNNs on the public dataset CIFAR10/100 and a private dataset Kuzushiji, which prove the feasibility and efficiency of FS-DNN. According to the research, instead of adopting bloated DNN networks that demand complicated pruning algorithms to optimize the networks after the fact or the current so-called lightweight DNNs, the experimental results demonstrate that lightweight networks based on FS-DNN achieve superior performance in computing consumption with competitive or even better accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data Availability

All data generated or analysed during this study is included in this published article (and its supplementary information files).

References

  1. Yue X, Li H, Shimizu M, Kawamura S, Meng L (2022) Yolo-gd: a deep learning-based object detection algorithm for empty-dish recycling robots. Machines

  2. Kim J, Kim Han-Joon, Kim H (2019) Fraud detection for job placement using hierarchical clusters-based deep neural networks. Appl Intell 49(8):2842–2861

    Article  Google Scholar 

  3. Yue X, Li H, Fujikawa Y, Meng L (2022) Dynamic dataset augmentation for deep learning-based oracle bone inscriptions recognition. ACM Journal on Computing and Cultural Heritage (JOCCH)

  4. Tiwari S, Goel S, Bhardwaj A (2022) Midnn- a classification approach for the eeg based motor imagery tasks using deep neural network. Appl Intell 52(5):4824–4843

    Article  Google Scholar 

  5. Ji D, Shin D, Park J (2021) An error compensation technique for low-voltage dnn accelerators. IEEE Trans Very Large Scale Integr (VLSI) Syst 29(2):397–408

    Article  Google Scholar 

  6. Trappey AJC, Trappey CV, Govindarajan UH, Sun JJH (2021) Patent value analysis using deep learning models—the case of iot technology mining for the manufacturing industry. IEEE Trans Eng Manag 68 (5):1334–1346

    Article  Google Scholar 

  7. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, vol 25. Curran Associates, Inc, Lake Tahoe, Nevada, USA

  8. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: The 3rd international conference on learning representations(ICLR), San diego, CA, USA

  9. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: 2015 IEEE Conference on computer vision and pattern recognition (CVPR), Boston, MA, USA, pp 1–9

  10. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: The 32nd international conference on machine learning (ICML), vol 37, Lille, France, pp 448–456

  11. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA, pp 2818–2826

  12. He K, Zhang X, Ren S, Sun J (June 2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA, pp 770–778

  13. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA, pp 2261–2269

  14. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE Conference on computer vision and pattern recognition (CVPR), Salt Lake City, pp 6848–6856

  15. Ma N et al (2018) Shufflenet v2: practical guidelines for efficient CNN architecture design. In: The 15th european conference on computer vision (ECCV), vol 11218, Munich, Germany, pp 122–138

  16. Tan M, Bo C, Pang R, Vasudevan V, Le QV (2019) Mnasnet: platform-aware neural architecture search for mobile. In: 2019 IEEE/CVF Conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, pp 2815–2823

  17. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen Liang-Chieh (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition, pp 4510–4520

  18. Li H, Wang Z, Yue X, Wang W, Tomiyama H, Meng L (2022) An architecture-level analysis on deep learning models for low-impact computations. Artif Intell Rev, 55

  19. Liu W, Wu G, Ren F, Kang X (2020) Dff-resnet: an insect pest recognition model based on residual networks. Big Data Mining and Analytics 3(4):300–310

    Article  Google Scholar 

  20. Yu X, Kang C, Guttery DS, Kadry S, Chen Y, Zhang Y-D (2021) Resnet-scda-50 for breast abnormality classification. IEEE/ACM Trans Comput Biol Bioinform 18(1):94–102

    Article  Google Scholar 

  21. Lin C, Zhao G, Yang Z, Yin A, Wang X, Li G, Chen H, Ma Z, Zhao L, Luo H, Wang T, Ding B, Pang X, Chen Q (2022) Cir-net: automatic classification of human chromosome based on inception-resnet architecture. IEEE/ACM Trans Comput Biol Bioinform 19(3):1285–1293

    Google Scholar 

  22. Huang L, Pun C-M (2020) Audio replay spoof attack detection by joint segment-based linear filter bank feature extraction and attention-enhanced densenet-bilstm network. IEEE/ACM Trans Audio Speech Lang Process 28:1813–1825

    Article  Google Scholar 

  23. Bakshi S, Rajan S (2021) Fall event detection system using inception-densenet inspired sparse siamese network. IEEE Sensors Lett 5(8):1–4

    Article  Google Scholar 

  24. Yang X, Ye Q, Cai G, Wang Y, Cai G (2022) Pd-resnet for classification of parkinson’s disease from gait. IEEE J Transl Eng Health Med 10:1–11

    Article  Google Scholar 

  25. Li H, Yue X, Wang Z, Chai Z, Wang W, Tomiyama H, Meng L (2022) Optimizing the deep neural networks by layer-wise refined pruning and the acceleration on fpga. Computational Intelligence and Neuroscience, 2022

  26. Li T, Wu B, Yang Y, Fan Y, Zhang Y, Liu W (2019) Compressing convolutional neural networks via factorized convolutional filters. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, pp 3972–3981

  27. Zhang Q, Han R, Xin G, Liu CH, Wang G, Chen LY (2022) Lightweight and accurate dnn-based anomaly detection at edge. IEEE Trans Parallel Distrib Syst 33(11):2927–2942

    Article  Google Scholar 

  28. Li B, Qu S, Wang Y (2022) An automated quantization framework for high-utilization rram-based pim. IEEE Trans Comput-Aided Des Integr Circuits Syst 41(3):583–596

    Article  Google Scholar 

  29. Choi Y, El-Khamy M, Lee J (2020) Learning sparse low-precision neural networks with learnable regularization. IEEE Access 8:96963–96974

    Article  Google Scholar 

  30. Gong C, Chen Y, Ye L u, Li T, Hao C, Chen D (2021) Vecq: minimal loss dnn model compression with vectorized weight quantization. IEEE Trans Comput 70(5):696–710

    Article  MathSciNet  MATH  Google Scholar 

  31. Kim N, Shin D, Choi W, Kim G, Park J (2021) Exploiting retraining-based mixed-precision quantization for low-cost dnn accelerator design. IEEE Trans Neural Netw Learn Syst 32(7):2925–2938

    Article  Google Scholar 

  32. Su Y, Hong D, Li Y, Jing P (2020) Low-rank regularized deep collaborative matrix factorization for micro-video multi-label classification. IEEE Signal Process Lett 27:740–744

    Article  Google Scholar 

  33. Xue Z, Du J, Du D, Li G, Huang Q, Lyu S (2019) Deep constrained low-rank subspace learning for multi-view semi-supervised classification. IEEE Signal Process Lett 26(8):1177–1181

    Article  Google Scholar 

  34. Liao Z, Huang J, Cheng Y, Li C, Liu PX (2022) A novel decomposition-based ensemble model for short-term load forecasting using hybrid artificial neural networks. Appl Intell 52(10):11043–11057

    Article  Google Scholar 

  35. Yazdani R, Riera M, Arnau J-M, González A (2018) The dark side of dnn pruning. In: 2018 ACM/IEEE 45Th annual international symposium on computer architecture (ISCA), pp 790–801

  36. Fang Y u, Li C, Wang P, Han C, Huang R, Xi H (2021) Easiedge: a novel global deep neural networks pruning method for efficient edge computing. IEEE Internet Things J 8(3):1259–1271

    Article  Google Scholar 

  37. Zhang T, Ye S, Feng X, Ma X, Zhang K, Li Z, Tang J, Liu S, Lin X, Liu Y, Fardad M, Wang Y (2022) Structadmm: achieving ultrahigh efficiency in structured pruning for dnns. IEEE Trans Neural Netw Learn Syst 33(5):2259–2273

    Article  Google Scholar 

  38. Zhuang T, Zhang Z, Huang Y, Zeng X, Shuang K, Li X (2020) Neuron-level structured pruning using polarization regularizer. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H (eds) Advances in Neural Information Processing Systems, vol 33. Curran Associates, Inc., pp 9865–9877

  39. Chen T, Bo J i, Tianyu DING, Fang B, Wang G, Zhu Z, Liang L, Shi Y, Yi S, Xiao T u (2021) Only train once: a one-shot neural network training and pruning framework. In: Thirty-fifth conference on neural information processing systems

  40. Loshchilov I, Hutter F (2017) SGDR: Stochastic gradient descent with warm restarts. In: The 5th international conference on learning representations (ICLR), Toulon, France

  41. He Y, Liu P, Wang Z, Hu Z, Yi Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: 2019 IEEE/CVF Conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 06, pp 4335–4344

  42. Huang Z, Li L i, Hailin Sun. (2020) Global biased pruning considering layer contribution. IEEE Access 8:173521–173529

    Article  Google Scholar 

  43. Center for Open Data in the Humanities. Kuzushiji dataset (in japanese). 2018. http://codh.rois.ac.jp/char-shape/book/100249416/2019,. (Apr. 22, 2021 accessed)

  44. Meng L Character recognition, 2022-08 accessed

  45. Huang Y, Qiao X, Ren P, Liu L, Pu C, Dustdar S, Chen J (2022) A lightweight collaborative deep neural network for the mobile web in edge cloud. IEEE Trans Mob Comput 21(7):2289– 2305

    Article  Google Scholar 

  46. Geng T, Li A, Wang T, Wu C, Li Y, Shi R, Wu W, Herbordt M (2021) O3bnn-r: an out-of-order architecture for high-performance and regularized bnn inference. IEEE Transactions on Parallel and Distributed Systems 32(1):199–213

    Article  Google Scholar 

  47. Pham T-N, Trinh Q-K, Chang I-J, Alioto M (2022) Stt-bnn: a novel stt-mram in-memory computing macro for binary neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 12 (2):569–579

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Science and Engineering, Ritsumeikan University, 1-1-1 Noji-higashi, Kusatsu, 525-8577, Shiga, Japan

    Hengyi Li, Xuebin Yue & Chengyan Zhao

  2. College of Science and Engineering, Ritsumeikan University, 1-1-1 Noji-higashi, Kusatsu, 525-8577, Shiga, Japan

    Lin Meng

Authors
  1. Hengyi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuebin Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengyan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lin Meng.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Block structures of Lightweight-DNNs

Appendix A: Block structures of Lightweight-DNNs

MobileNetV2: :

Figure 2 shows two kinds of MobileNet-bottlenecks. The bottleneck adopts depthwise convolution (DWConv) for the [3 × 3] convolution to reduce the model size and complexity. The [1 × 1] convolution is used to increase/decrease the dimensions.

ShuffleNet Block: :

Figure 3 shows the ShuffleNet-block structures. In addition to taking depthwise convolution (DWConv) to adjust the dimensions, the ShuffleNet block has two branches, which are concatenated and then the channels are shuffled.

Fig. 3
figure 3

ShuffleNet Block [15]

Full size image

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Yue, X., Zhao, C. et al. Lightweight deep neural network from scratch. Appl Intell 53, 18868–18886 (2023). https://doi.org/10.1007/s10489-022-04394-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04394-3

Keywords

Search

Navigation