An Image Enhancing Pattern-Based Sparsity for Real-Time Inference on Mobile Devices

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)


Weight pruning has been widely acknowledged as a straightforward and effective method to eliminate redundancy in Deep Neural Networks (DNN), thereby achieving acceleration on various platforms. However, most of the pruning techniques are essentially trade-offs between model accuracy and regularity which lead to impaired inference accuracy and limited on-device acceleration performance. To solve the problem, we introduce a new sparsity dimension, namely pattern-based sparsity that comprises pattern and connectivity sparsity, and becoming both highly accurate and hardware friendly. With carefully designed patterns, the proposed pruning unprecedentedly and consistently achieves accuracy enhancement and better feature extraction ability on different DNN structures and datasets, and our pattern-aware pruning framework also achieves pattern library extraction, pattern selection, pattern and connectivity pruning and weight training simultaneously. Our approach on the new pattern-based sparsity naturally fits into compiler optimization for highly efficient DNN execution on mobile platforms. To the best of our knowledge, it is the first time that mobile devices achieve real-time inference for the large-scale DNN models thanks to the unique spatial property of pattern-based sparsity and the help of the code generation capability of compilers.



This work is supported by the National Science Foundation CCF-1919117, CCF-1937500, CNS-1909172, CNS-2011260, and is sponsored by DiDi GAIA Research Collaboration Initiative. We thank all anonymous reviewers for their feedback.


  1. 1.
  2. 2.
  3. 3.
    Aravindh, M., Andrea, V.: Understanding deep image representations by inverting them. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 (2015)Google Scholar
  4. 4.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn. 3(1), 1–122 (2011)Google Scholar
  5. 5.
    Chen, C.F., Oh, J., Fan, Q., Pistoia, M.: SC-CONV: sparse-complementary convolution for efficient model utilization on CNNs. In: 2018 IEEE International Symposium on Multimedia (ISM), pp. 97–100. IEEE (2018)Google Scholar
  6. 6.
    Chen, T., et al.: TVM: an automated end-to-end optimizing compiler for deep learning. In: OSDI (2018)Google Scholar
  7. 7.
    Dai, X., Yin, H., Jha, N.K.: Nest: a neural network synthesis tool based on a grow-and-prune paradigm. IEEE Trans. Comput. 68(10), 1487–1497 (2019)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Dong, X., Yang, Y.: Network pruning via transformable architecture search. In: Advances in Neural Information Processing Systems, pp. 759–770 (2019)Google Scholar
  9. 9.
    Freeman, W., Adelson, E.: The design and use of steerable filters. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, pp. 891–906. IEEE (1991)Google Scholar
  10. 10.
    Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: International Conference on Learning Representations (ICLR) (2016)Google Scholar
  11. 11.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  12. 12.
    He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)Google Scholar
  13. 13.
    He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: AutoML for model compression and acceleration on mobile devices. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 815–832. Springer, Cham (2018).
  14. 14.
    He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1398–1406. IEEE (2017)Google Scholar
  15. 15.
    Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  16. 16.
    Lin, S., et al.: Towards optimal structured CNN pruning via generative adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2799 (2019)Google Scholar
  17. 17.
    Liu, B., Wang, M., Foroosh, H., Tappen, M., Pensky, M.: Sparse convolutional neural networks. In: CVPR, pp. 806–814 (2015)Google Scholar
  18. 18.
    Liu, N., Ma, X., Xu, Z., Wang, Y., Tang, J., Ye, J.: Autocompress: an automatic DNN structured pruning framework for ultra-high compression rates. In: AAAI, pp. 4876–4883 (2020)Google Scholar
  19. 19.
    Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: International Conference on Learning Representations (2019)Google Scholar
  20. 20.
    Ma, X., et al.: PCONV: the missing but desirable sparsity in DNN weight pruning for real-time execution on mobile devices. In: AAAI, pp. 5117–5124 (2020)Google Scholar
  21. 21.
    Ma, X., et al.: Tiny but accurate: a pruned, quantized and optimized memristor crossbar framework for ultra efficient DNN implementation. In: ASP-DAC (2020)Google Scholar
  22. 22.
    Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016)
  23. 23.
    Mukund, S., Ankur, T., Qiqi, Y.: Axiomatic attribution for deep networks. In: 2017 International Conference on Machine Learning (ICML). ACM/IEEE (2017)Google Scholar
  24. 24.
    Niu, W., et al.: PatDNN: achieving real-time DNN execution on mobile devices with pattern-based weight pruning. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 907–922 (2020)Google Scholar
  25. 25.
    Parashar, A., et al.: SCNN: an accelerator for compressed-sparse convolutional neural networks. In: ISCA (2017)Google Scholar
  26. 26.
    Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends® Optim. 1(3), 127–239 (2014)Google Scholar
  27. 27.
    Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: NeurIPS (2019)Google Scholar
  28. 28.
    Ren, A., et al.: ADMM-NN: an algorithm-hardware co-design framework of DNNs using alternating direction methods of multipliers. In: ASPLOS, pp. 925–938 (2019)Google Scholar
  29. 29.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  30. 30.
    Siyuan, M., Raef, B., Mikhail, B.: The power of interpolation: understanding the effectiveness of SGD in modern over-parametrized learning. In: 2018 International Conference on Machine Learning (ICML). ACM/IEEE (2018)Google Scholar
  31. 31.
    Springenberg, J.T., Alexey Dosovitskiy, T.B.a.R.: Striving for simplicity: the all convolutional net. In: ICLR-2015 Workshop Track (2015)Google Scholar
  32. 32.
    Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems, pp. 2074–2082 (2016)Google Scholar
  33. 33.
    Xu, M., Zhu, M., Liu, Y., Lin, F.X., Liu, X.: Deepcache: principled cache for mobile deep vision. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pp. 129–144. ACM (2018)Google Scholar
  34. 34.
    Yao, S., Hu, S., Zhao, Y., Zhang, A., Abdelzaher, T.: Deepsense: a unified deep learning framework for time-series mobile sensing data processing. In: Proceedings of the 26th International Conference on World Wide Web (2017)Google Scholar
  35. 35.
    You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 2130–2141 (2019)Google Scholar
  36. 36.
    Zhu, X., Zhou, W., Li, H.: Improving deep neural network sparsity through decorrelation regularization. In: IJCAI (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Northeastern UniversityBostonUSA
  2. 2.College of William and MaryWilliamsburgUSA
  3. 3.Syracuse UniversitySyracuseUSA
  4. 4.IBM ResearchYorktown HeightsUSA
  5. 5.Lehigh UniversityBethlehemUSA
  6. 6.George Mason UniversityFairfaxUSA
  7. 7.DiDi AI LabsMountain ViewUSA
  8. 8.Tsinghua UniversityBeijingChina

Personalised recommendations