Skip to main content
SpringerLink
Log in

Lightweight residual densely connected convolutional neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Extremely efficient convolutional neural network architectures are one of the most important requirements for limited-resource devices (such as embedded and mobile devices). The computing power and memory size are two important constraints of these devices. Recently, some architectures have been proposed to overcome these limitations by considering specific hardware-software equipment. In this paper, the lightweight residual densely connected blocks are proposed to guaranty the deep supervision, efficient gradient flow, and feature reuse abilities of convolutional neural network. The proposed method decreases the cost of training and inference processes without using any special hardware-software equipment by just reducing the number of parameters and computational operations while achieving a feasible accuracy. Extensive experimental results demonstrate that the proposed architecture is more efficient than the AlexNet and VGGNet in terms of model size, required parameters, and even accuracy. The proposed model has been evaluated on the ImageNet, MNIST, Fashion MNIST, SVHN, CIFAR-10, and CIFAR-100. It achieves state-of-the-art results on Fashion MNIST dataset and reasonable results on the others. The obtained results show the superiority of the proposed method to efficient models such as the SqueezNet. It is also comparable with state-of-the-art efficient models such as CondenseNet and ShuffleNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alam F, Mehmood R, Katib I, Altowaijri SM, Albeshri A (2019) TAAWUN: A decision fusion and feature specific road detection approach for connected autonomous vehicles. Mobile Networks and Applications, pp 1–17 Springer

  2. Assunċao F, Lourenċo N, Machado P, Ribeiro B (2019) Denser: deep evolutionary network structured representation. Genet Program Evolvable Mach 20(1):5–35

    Article  Google Scholar 

  3. Baker B, Gupta O, Naik N, Raskar R (2017) Designing neural network architectures using reinforcement learning International Conference on Learning Representations (ICLR)

  4. Cai H, Zhu L, Han S (2019) Proxylessnas: Direct neural architecture search on target task and hardware International Conference on Learning Representations (ICLR)

  5. Chen W, Wilson J, Tyree S, Weinberger K, Chen Y (2015) Compressing neural networks with the hashing trick. In: International conference on machine learning, pp. 2285–2294

  6. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258

  7. Courbariaux M, Hubara I, Soudry D, El-Yaniv R, Bengio Y (2016) Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv:1602.02830

  8. Deng L (2012) The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Proc Mag 29(6):141–142

    Article  Google Scholar 

  9. Denil M, Shakibi B, Dinh L, De Freitas N, et al. (2013) Predicting parameters in deep learning. In: Advances in neural information processing systems, pp. 2148–2156

  10. Denton EL, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in neural information processing systems, pp. 1269–1277

  11. Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision, pp. 2650–2658

  12. Fooladgar F, Kasaei S (2019) 3m2rnet: Multi-modal multi-resolution refinement network for semantic segmentation. In: Science and information conference, pp. 544–557. Springer

  13. Fooladgar F, Kasaei S (2020) A survey on indoor rgb-d semantic segmentation: From hand-crafted features to deep convolutional neural networks. Multimedia Tools and Applications 79(7):4499–4524

    Article  Google Scholar 

  14. Grigorescu S, Trasnea B, Cocias T, Macesanu G (2019) A survey of deep learning techniques for autonomous driving Journal of Field Robotics

  15. Han S, Liu X, Mao H, Pu J, Pedram A, Horowitz MA, Dally WJ (2016) Eie: efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News 44(3):243–254

    Article  Google Scholar 

  16. Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems, pp. 1135–1143

  17. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

  18. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision, pp. 630–645. sSpringer

  19. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  20. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141

  21. Hu Y, Wen G, Luo M, Dai D (2018) Competitive inner-imaging squeeze and excitation for residual network. arXiv:1807.08920

  22. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol. 1, p. 3

  23. Huang G, Liu S, Van der Maaten L, Weinberger KQ (2018) Condensenet: An efficient densenet using learned group convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2752–2761

  24. Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. In: European conference on computer vision, pp. 646–661. Springer

  25. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv:1602.07360

  26. Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, Adam H, Kalenichenko D (2018) Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704–2713

  27. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Tech. rep. Citeseer

  28. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105

  29. Kuutti S, Bowden R, Jin Y, Barber P, Fallah S (2020) A survey of deep learning applications to autonomous vehicle control IEEE Transactions on Intelligent Transportation Systems

  30. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv:1608.08710

  31. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744

  32. Liu C, Zoph B, Neumann M, Shlens J, Hua W, Li LJ, Fei-Fei L, Yuille A, Huang J, Murphy K (2018) Progressive neural architecture search. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 19–34

  33. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440

  34. Naseer M, Khan SH, Porikli F (2018) Indoor scene understanding in 2.5/3d: A survey. arXiv:1803.03352

  35. Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY (2011) Reading digits in natural images with unsupervised feature learning

  36. Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. In: European conference on computer vision, pp. 525–542. Springer

  37. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems, pp. 3856–3866

  38. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  39. Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:1505.00387

  40. Srivastava RK, Greff K, Schmidhuber J (2015) Training very deep networks. In: Advances in neural information processing systems, pp. 2377–2385

  41. Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820–2828

  42. Tan M, Le QV (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, vol. 97, pp. 6105–6114. PMLR. http://proceedings.mlr.press/v97/tan19a.html

  43. Wang X, Yu F, Dou ZY, Darrell T, Gonzalez JE (2018) Skipnet: Learning dynamic routing in convolutional networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 409–424

  44. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. In: Advances in neural information processing systems, pp. 2074–2082

  45. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms

  46. Yang Y, Zhong Z, Shen T, Lin Z (2018) Convolutional neural networks with alternately updated clique. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2413–2422

  47. Zagoruyko S, Komodakis N (2016) Wide residual networks. arXiv:1605.07146

  48. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image restoration. arXiv:1812.10477

  49. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481

  50. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856

  51. Zhong Z, Yang Z, Deng B, Yan J, Wu W, Shao J, Liu CL (2020) Blockqnn: Efficient block-wise neural network architecture generation IEEE Transactions on Pattern Analysis and Machine Intelligence

  52. Zhong Z, Zheng L, Kang G, Li S, Yang Y (2017) Random erasing data augmentation. arXiv:1708.04896

  53. Zhou H, Alvarez JM, Porikli F (2016) Less is more: Towards compact cnns. In: European conference on computer vision, pp. 662–677. Springer

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

    Fahimeh Fooladgar & Shohreh Kasaei

Authors
  1. Fahimeh Fooladgar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shohreh Kasaei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shohreh Kasaei.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fooladgar, F., Kasaei, S. Lightweight residual densely connected convolutional neural network. Multimed Tools Appl 79, 25571–25588 (2020). https://doi.org/10.1007/s11042-020-09223-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09223-8

Keywords

Search

Navigation