Skip to main content
SpringerLink
Log in

A Comparative Study of Neural Network Compilers on ARMv8 Architecture

  • Conference paper
  • First Online:
Architecture of Computing Systems (ARCS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13949))

Included in the following conference series:

  • 383 Accesses

Abstract

The deployment of Deep Neural Network (DNN) models in far edge devices is a challenging task, because these devices are characterized by scarce resources. To address these challenges various deep learning toolkits and model compression techniques have been developed both from industry and academia. The available DNN toolchains can perform optimizations at different levels e.g., graph level, Intermediate Representation (IR) or machine-dependent optimizations, while they operate in an Ahead-of-Time (AOT) or Just-in-Time (JIT) manner. Although the area of DNN toolchains is an active research area, there is no available study that analyses the performance benefits achieved by the different optimization levels e.g., the performance boost reported by the graph-level vs. the machine-dependent optimizations. This work performs a comprehensive study of three popular neural network (NN) compiler frameworks that target (mainly) far edge devices: TensorFlow Lite for MCUs, GLOW, and IREE. For a fair comparison, our performance analysis targets to reveal the performance benefits offered by the different optimization levels for the three studied frameworks as well as the strength of specific graph-level optimizations e.g., in quantizing the input NN models. Our evaluation is based on various NN models with different computational/memory resources and the experiments are performed in a state-of-the-art high-performance embedded platform by Nvidia.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aakanksha, C., Warden, P., Shlens, J., Howard, A., Rhodes, R.: Visual wake words dataset. arXiv preprint arXiv:1906.05721 (2019)

  2. AlexNet. url: https://cvml.ista.ac.at/courses/DLWT_W17/material/AlexNet.pdf

  3. Cafee2 framework. url: https://caffe2.ai/

  4. Chen, T., et al.: An automated end-to-end optimizing compiler for deep learning. arXiv preprint arXiv:1802.04799 (2018)

  5. Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017)

  6. CuDNN. url: https://developer.nvidia.com/cudnn

  7. David, R., et al.: TensorFlow Lite Micro: Embedded machine learning on TinyML systems. arXiv preprint arXiv:2010.08678 (2020)

  8. Diniz, P.C., Cardoso, J.M.P., Coutinho, J.G.F.: Embedded computing for high performance (2017)

    Google Scholar 

  9. GLOW graph-level optimization. url: https://github.com/pytorch/glow/blob/master/docs/Optimizations.md

  10. GLOW. url: https://github.com/pytorch/glow

  11. GLOW with CMSIS support. url: https://github.com/Theoo1997/glow

  12. Graph Optimizations url: https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html#graph-optimization-levels

  13. Halide library. url: https://halide-lang.org/

  14. HLO IR. url: https://github.com/tensorflow/mlir-hlo

  15. Howard, A.G., et al.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

  16. Intel ngraph: An intermediate representation, compiler, and executor for deep learning. url: https://github.com/NervanaSystems/ngraph

  17. Jiang, X., et al.: MNN: a universal and efficient inference engine. arXiv preprint arXiv: 2002.12418v1 (2020)

  18. Keras FLOPs. url: https://pypi.org/project/keras-flops/

  19. Koizumi, Y., et al.: Unsupervised anomalous sound detection for machine condition monitoring. arXiv:2006.05822 (2020)

  20. Koizumi, Y., Saito, S., Harada, N Uematsu, H., Imoto, K.: ToyADMOS: A dataset of miniature-machine operating sounds for anomalous sound detection. Workshop on Applications of Signal Processing to Audio and Acoustics (2019)

    Google Scholar 

  21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2017)

    Google Scholar 

  22. Lai, L., Suda, N., Chandra, V.: CMSIS-NN: efficient neural network kernels for Arm Cortex-M CPUs. arXiv preprint arXiv:1801.06601, 2018

  23. Lattner, C., et al.: MLIR: A compiler infrastructure for the end of moore's law. arXiv preprint arxiv:2002.11054 (2020)

  24. Leary, C., Wang, T.: XLA: TensorFlow (2017)

    Google Scholar 

  25. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Google Scholar 

  26. LeCun, Y., Cortes, C., Burges, J.C.: The MNIST database of handwritten digits. Microsoft Res. 1998

    Google Scholar 

  27. M. Li, et al.: The deep learning compiler: a comprehensive survey. arXiv preprint arXiv: 2002.03794 (2020)

  28. Lin, W.F., et al.: ONNC: a compilation framework connecting ONNX to proprietary deep learning accelerators. In: 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) (2019)

    Google Scholar 

  29. Luo, C., He, X., Zhan, J., Wang, L., Gao, W., Dai, J.: Comparison and benchmarking of AI models and frameworks on mobile devices. arXiv preprint arXiv: 2005.05085 (2020)

  30. nGraph IR. url: https://docs.openvino.ai/2020.2/_docs_IE_DG_nGraph_Flow.html

  31. NNAPI. url: https://developer.android.com/ndk/guides/neuralnetworks

  32. oneAPI. url: https://github.com/oneapi-src/oneDNN

  33. ONNX. url: https://onnx.ai/

  34. OpenVino toolkit: Neural network compression framework (NNCF). url: https://docs.openvino.ai/2022.1/docs_nncf_introduction.html

  35. Paszke, A., et al.: PyTorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 1912.01703 (2019)

  36. Purohit, H., et al.: MIMII dataset: sound dataset for malfunctioning industrial machine investigation and inspection. In: Workshop on Detection & Classification of Acoustic Scenes and Events (2019)

    Google Scholar 

  37. PyTorch Mobile. url: https://pytorch.org/mobile/home/

  38. Roesch, J., et al.: Relay: a high-level compiler for deep learning. arXiv preprint arXiv:1904.08368 (2019)

  39. M. Sponner, B. Waschneck, and A. Kumar. Compiler toolchains for deep learning workloads on embedded platforms. arXiv preprint arXiv:2104.04576, 2021

  40. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  41. Tan, M., Le, Q.V.: EfficientNet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946arXiv: 1905.11946, 2019

  42. Tensor comprehensions. url: https://github.com/facebookresearch/TensorComprehensions

  43. TensorFlow. url: https://www.tensorflow.org

  44. TensorRT constant folding example. url: https://www.ccoderun.ca/programming/doxygen/tensorrt/md_TensorRT_tools_onnx-graphsurgeon_examples_05_folding_constants_README.html

  45. TinyIREE. url: https://openxla.github.io/iree/

  46. TinyML Benchmarks. url: https://github.com/mlcommons/tiny

  47. Grosser, T., Größlinger, A., Lengauer, C.: Polly - performing polyhedral optimizations on a low-level intermediate representation. Parallel Processing Letters (2012)

    Google Scholar 

  48. Tollenaere, N., et al.: Autotuning convolutions is easier than you think. ACM Trans. Archit. Code Optimizations 20, 1–24 (2023)

    Google Scholar 

  49. The CIFAR-10 dataset. url: http://www.cs.toronto.edu/~kriz/cifar.html

  50. Wade, A.W., Kulkarni, P.A., Jantz, M.R.: AOT vs. JIT: Impact of profile data on code quality, In: Proceedings of the 18th ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (2017)

    Google Scholar 

  51. Warden, P.: Speech commands: a dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018

  52. Xia, X., et al.: TRT-ViT: TensorRT-oriented vision transformer. arXiv preprint arXiv: 2205.09579 (2022)

  53. Yi, X., et al.: Optimizing DNN compilation for distributed training with joint OP and tensor fusion. arXiv preprint arXiv:2209.12769 (2022)

  54. L. Zheng, et al.: Ansor: generating high-performance tensor programs for deep learning. In: 14th USENIX symposium on operating systems design and implementation (OSDI 20) (2020)

    Google Scholar 

Download references

Acknowledgements

This research has been supported by a sponsored research agreement between Applied Materials, Inc. And Aristotle University of Thessaloniki, Greece.

Author information

Authors and Affiliations

  1. School of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece

    Theologos Anthimopulos & Georgios Keramidas

  2. Think Silicon, S.A. An Applied Materials Company, Patras, Greece

    Georgios Keramidas & Iakovos Stamoulis

  3. School of Engineering, Computing and Mathematics, University of Plymouth, Plymouth, UK

    Vasilios Kelefouras

Authors
  1. Theologos Anthimopulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Georgios Keramidas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vasilios Kelefouras
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Iakovos Stamoulis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Theologos Anthimopulos .

Editor information

Editors and Affiliations

  1. National Technical University of Athens, Athens, Greece

    Georgios Goumas

  2. Kiel University, Kiel, Germany

    Sven Tomforde

  3. Gottfried Wilhelm Leibniz Universität Hannover, Hannover, Germany

    Jürgen Brehm

  4. Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany

    Stefan Wildermann

  5. Otto-von-Guericke University Magdeburg, Magdeburg, Germany

    Thilo Pionteck

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Anthimopulos, T., Keramidas, G., Kelefouras, V., Stamoulis, I. (2023). A Comparative Study of Neural Network Compilers on ARMv8 Architecture. In: Goumas, G., Tomforde, S., Brehm, J., Wildermann, S., Pionteck, T. (eds) Architecture of Computing Systems. ARCS 2023. Lecture Notes in Computer Science, vol 13949. Springer, Cham. https://doi.org/10.1007/978-3-031-42785-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42785-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42784-8

  • Online ISBN: 978-3-031-42785-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation