Skip to main content
SpringerLink
Log in

Overviewing AI-Dedicated Hardware for On-Device AI in Smartphones

  • Chapter
  • First Online:
Artificial Intelligence and Hardware Accelerators

Abstract

Machine learning applications for smartphones, such as camera applications, and automatic speech recognition, require a real-time response of deep learning processing. Although one inference processing of deep learning requires more than a billion operations, the development of hardware and deep learning has allowed on-device real-time deep learning processing in smartphones. The performance of smartphone neural processing unit (NPU) has been doubling every year since NPU was integrated into smartphone application processors in 2017. As of 2021, dozens of on-device deep learning applications have been employed in a smartphone. This chapter aims to provide an overview of the history of progress in achieving on-device inference in the smartphone industry. This chapter describes the development and overview of system-on-chip technology and single instruction multiple data architecture such as central processing unit, graphics processing unit, digital signal processor, and NPU, to achieve on-device artificial intelligence used in smartphones. We will also briefly introduce the architectures and features of NPUs widely used in the mobile phone industry. In addition, it introduces strategies such as quantization, pruning, and compression that can increase computational efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE. 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  2. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Proces. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  3. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

    Google Scholar 

  4. Galaxy Unpacked January 2021: Official Replay | Samsung, YouTube. https://youtube.be/TD_BZN0bn_U. Accessed 5 Jan 2022

  5. Exynos 990 Mobile Processor: Official Introduction, YouTube. https://www.youtube.com/watch?v=13RgDxD83vl. Accessed 5 Jan 2022

  6. Samsung Exynos 2100 official introduction webpage, Samsung. https://www.samsung.com/semiconductor/minisite/exynos/products/mobileprocessor/exynos-2100/. Accessed 5 Jan 2022

  7. Qualcomm’s Snapdragon 820 processor product brief document, Qualcomm. https://www.qualcomm.com/media/documents/files/snapdragon-820-processor-product-brief.pdf. Accessed 5 Jan 2022

  8. Huawei Kirin 970 AI processor presentation 10/16/17, YouTube. https://www.youtube.com/watch?v=5v2D_QddnFc. Accessed 5 Jan 2022

  9. Qualcomm’s introduction material of Snapdragon 865 5G, Qualcomm. https://www.qualcomm.com/media/documents/files/2019-snapdragon-865-5g-ai-deep-dive-ziad-asghar-jeff-gehlhaar.pdf. Accessed 5 Jan 2022

  10. Qualcomm Redefines Premium with the Flagship Snapdragon 888 5G Mobile Platform, Qualcomm. https://www.qualcomm.com/news/releases/2020/12/02/qualcomm-redefines-premium-flagship-snapdragon-888-5g-mobile-platform. Accessed 5 Jan 2022

  11. Antony, A., Sarika, S.: A review on IoT operating systems. Int. J. Comput. Appl. 975, 8887

    Google Scholar 

  12. Jang, M., Kim, K., Kim, K.: The performance analysis of ARM NEON technology for mobile platforms. In: Proceedings of the 2011 ACM Symposium on Research in Applied Computation, pp. 104–106 (2011)

    Google Scholar 

  13. The A14 Bionic | Apple Special Event 2020, YouTube. https://www.youtube.com/watch?v=5toYNtqbiwg

  14. Han, S., Pool, J., Tran, J., Dally, W.J.: Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.02626 (2015)

    Google Scholar 

  15. Chen, Y.-H., Krishna, T., Emer, J.S., Sze, V.: Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE J. Solid State Circuits. 52(1), 127–138 (2016)

    Article  Google Scholar 

  16. Horowitz, M.: 1.1 computing’s energy problem (and what we can do about it). In: 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10–14. IEEE (2014)

    Google Scholar 

  17. Overton, M.L.: Numerical computing with IEEE floating point arithmetic. Society for Industrial and Applied Mathematics (2001)

    Book  MATH  Google Scholar 

  18. Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M.W., Keutzer, K.: A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630 (2021)

    Google Scholar 

  19. Tools in Snapdragon Neural Processing Engine SDK Reference Guide, Qualcomm. https://developer.qualcomm.com/sites/default/files/docs/snpe/tools.html. Accessed 5 Jan 2022

  20. September Event 2018 – Apple, YouTube. https://www.youtube.com/watch?v=wFTmQ27S7OQ. Accessed 5 Jan 2022

  21. MediaTek Dimensity 800 introduction, Mediatek. https://www.mediatek.com/products/smartphones/dimensity-800. Accessed 5 Jan 2022

  22. Huawei Kirin 990 5G launch, YouTube. https://www.youtube.com/watch?v=9zkxuxpObKI. Accessed 5 Jan 2022

  23. Song, J., Cho, Y., Park, J.-S., Jang, J.-W., Lee, S., Song, J.-H., Lee, J.-G., Kang, I.: 7.1 an 11.5 tops/w 1024-mac butterfly structure dual-core sparsity-aware neural processing unit in 8nm flagship mobile soc. In: 2019 IEEE International Solid-State Circuits Conference-(ISSCC), pp. 130–132. IEEE (2019)

    Google Scholar 

  24. Liao, H., Tu, J., Xia, J., Zhou, X.: Davinci: A scalable architecture for neural network computing. In: 2019 IEEE Hot Chips 31 Symposium (HCS), pp. 1–44. IEEE Computer Society (2019)

    Google Scholar 

  25. Lin, C.-H., Cheng, C.-C., Tsai, Y.-M., Hung, S.-J., Kuo, Y.-T., Wang, P.H., Tsung, P.-K., et al.: 7.1 a 3.4-to-13.3 tops/w 3.6 tops dual-core deep-learning accelerator for versatile AI applications in 7nm 5g smartphone SOC. In: IEEE International Solid-State Circuits Conference-(ISSCC) 2020, pp. 134–136. IEEE (2020)

    Google Scholar 

  26. Introducing Neon for Armv8-A, Arm. https://developer.arm.com/architectures/instruction-sets/simd-isas/neon/neon-programmers-guide-for-armv8-a/introducing-neon-for-armv8-a/single-page. Accessed 5 Jan 2022 [27]

  27. Codrescu, L.: Architecture of the Hexagon™ 680 DSP for mobile imaging and computer vision. In: 2015 IEEE Hot Chips 27 Symposium (HCS), pp. 1–26. IEEE (2015)

    Google Scholar 

  28. Arm Mali-T760 GPU, Arm. https://developer.arm.com/ip-products/graphics-and-multimedia/mali-gpus/mali-t760-gpu. Accessed 5 Jan 2022

  29. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) [31]

    Google Scholar 

  30. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510–4520 (2018)

    Google Scholar 

  31. Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  32. Jolivet, R., Lewis, T.J., Gerstner, W.: Generalized integrate-and-fire models of neuronal activity approximate spike trains of a detailed model to a high degree of accuracy. J. Neurophysiol. 92(2), 959–976 (2004)

    Article  Google Scholar 

  33. Peng, P., Mingyu, Y., Weisheng, X.: Running 8-bit dynamic fixed-point convolutional neural network on low-cost arm platforms. In: 2017 Chinese Automation Congress (CAC), pp. 4564–4568. IEEE (2017)

    Google Scholar 

  34. Courbariaux, M., Bengio, Y., David, J.-P.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: Advances in neural information processing systems, pp. 3123–3131 (2015)

    Google Scholar 

  35. Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnor-net: Imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision, pp. 525–542. Springer, Cham (2016) [36]

    Google Scholar 

  36. Li, F., Zhang, B., Liu, B.: Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016)

    Google Scholar 

  37. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Quantized neural networks: Training neural networks with low precision weights and activations. J. Mach. Learn. Res. 18(1), 6869–6898 (2017)

    MathSciNet  MATH  Google Scholar 

  38. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., Zou, Y.: Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016) [39]

    Google Scholar 

  39. Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: Towards lossless CNNs with low-precision weights. arXiv preprint arXiv:1702.03044 (2017)

    Google Scholar 

  40. Szymon Migacz: 8-bit inference with TensorRT. NVIDIA. https://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf

  41. Park, H., Kim, D., Kim, S.: TMA: Tera-MACs/W neural hardware inference accelerator with a multiplier-less massive parallel processor. Int. J. Circuit Theory Appl. 49(5), 1399–1409 (2021)

    Article  Google Scholar 

  42. Model optimization in TensorFlow Lite guide: Google. https://www.tensorflow.org/lite/performance/model_optimization. Accessed 5 Jan 2022.

  43. Quantizing a Model in Snapdragon Neural Processing Engine SDK Reference Guide, Qualcomm. https://developer.qualcomm.com/sites/default/files/docs/snpe/model_conversion.html. Accessed 5 Jan 2022.

  44. He, Y., Dong, X., Kang, G., Yanwei, F., Yan, C., Yang, Y.: Asymptotic soft filter pruning for deep convolutional neural networks. IEEE Trans. Cybernet. 50(8), 3594–3604 (2019)

    Article  Google Scholar 

  45. Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., Vajda, P.: Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677 (2020)

    Google Scholar 

  46. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    Google Scholar 

  47. Li, Y., Yuan, G., Wen, Y., Hu, E., Evangelidis, G., Tulyakov, S., Wang, Y., Ren, J.: Efficient former: vision transformers at MobileNet speed. arXiv preprint arXiv:2206.01191 (2022)

    Google Scholar 

Download references

Acknowledgments

This work was supported by two Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (“No.2020-0-00056, To create AI systems that act appropriately and effectively in novel situations that occur in open worlds”).

Author information

Authors and Affiliations

  1. School of Integrated Technology, Yonsei University, Incheon, South Korea

    Hyunbin Park & Shiho Kim

Authors
  1. Hyunbin Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiho Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiho Kim .

Editor information

Editors and Affiliations

  1. School of Integrated Technology, Yonsei University, Incheon, Korea (Republic of)

    Ashutosh Mishra

  2. Yonsei University, Incheon, Korea (Republic of)

    Jaekwang Cha

  3. School of Integrated Technology, Yonsei University, Incheon, Korea (Republic of)

    Hyunbin Park

  4. School of Integrated Technology, Yonsei University, Incheon, Korea (Republic of)

    Shiho Kim

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Park, H., Kim, S. (2023). Overviewing AI-Dedicated Hardware for On-Device AI in Smartphones. In: Mishra, A., Cha, J., Park, H., Kim, S. (eds) Artificial Intelligence and Hardware Accelerators. Springer, Cham. https://doi.org/10.1007/978-3-031-22170-5_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22170-5_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22169-9

  • Online ISBN: 978-3-031-22170-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation