Skip to main content
SpringerLink
Log in

Intelligence Inference on IoT Devices

  • Chapter
  • First Online:
Learning Techniques for the Internet of Things

  • 212 Accesses

Abstract

With the rapid advancement of artificial intelligence (AI), the proliferation of deep neural networks (DNNs) has ushered in a transformative era, revolutionizing modern lifestyles and enhancing production efficiency. However, the substantial computational and data requirements generated by Internet of Things (IoT) devices present a significant bottleneck, rendering traditional cloud-based computing models inadequate for real-time processing tasks. In response to these challenges, developers have increasingly turned to cloud offloading as a solution, despite the high infrastructure costs and heavy reliance on network conditions associated with this approach. Meanwhile, the emergence of SoCs has enabled on-device execution, particularly on high-tier platforms capable of effectively handling SOTA DNNs. This chapter offers a comprehensive review of intelligent inference approaches, with a specific emphasis on reducing inference time and minimizing transmitted bandwidth between IoT devices and the cloud. The review encompasses various aspects, including the background of inference, hardware architectures supporting inference, a diverse range of intelligent applications, inference libraries tailored for IoT devices, and different types of inference techniques for applications. Additionally, this work addresses the current challenges in intelligent inference, discusses future development trends, and provides future research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Adadi, Amina, and Mohammed Berrada. 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE access 6: 52138–52160.

    Article  Google Scholar 

  • Ain, Qurat-ul et al. 2018. IoT operating system based fuzzy inference system for home energy management system in smart buildings. Sensors 18 (9): 2802.

    Article  Google Scholar 

  • Alkhabbas, Fahed, et al. 2020. A goal-driven approach for deploying self-adaptive IoT systems. In 2020 IEEE International Conference on Software Architecture (ICSA), 146–156. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Almeida, Mario, et al. 2022. Dyno: Dynamic onloading of deep neural networks from cloud to device. ACM Transactions on Embedded Computing Systems 21 (6): 1–24.

    Article  Google Scholar 

  • Azizi, Shekoofeh, et al. 2023. Synthetic data from diffusion models improves imagenet classification. arXiv preprint. arXiv:2304.08466.

    Google Scholar 

  • Bajrami, Xhevahir, et al. 2018. Face recognition performance using linear discriminant analysis and deep neural networks. International Journal of Applied Pattern Recognition 5 (3): 240–250.

    Article  Google Scholar 

  • Bradski, Gary, Adrian Kaehler, et al. 2000. OpenCV. Dr. Dobb’s Journal of Software Tools 3 (2): 1–81.

    Google Scholar 

  • Cheng, Yu, et al. 2015. An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEEE International Conference on Computer Vision, 2857–2865.

    Google Scholar 

  • Choudhary, Tejalal, et al. 2020. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review 53: 5113–5155.

    Article  Google Scholar 

  • Courville, Vanessa, and Vahid Partovi Nia. 2019. Deep learning inference frameworks for ARM CPU. Journal of Computational Vision and Imaging Systems 5 (1): 3–3.

    Google Scholar 

  • Deng, Yunbin. 2019. Deep learning on mobile devices: A review. In Mobile Multimedia/Image Processing, Security, and Applications 2019. Vol. 10993, 52–66. Bellingham: SPIE.

    Google Scholar 

  • Developers, TensorFlow. 2022. TensorFlow. In Zenodo.

    Google Scholar 

  • Donta, Praveen Kumar, and Schahram Dustdar. 2022. The promising role of representation learning for distributed computing continuum systems. In 2022 IEEE International Conference on Service-Oriented System Engineering (SOSE), 126–132. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Donta, Praveen Kumar, Boris Sedlak, et al. 2023. Governance and sustainability of distributed continuum systems: A big data approach. Journal of Big Data 10 (1): 1–31.

    Article  Google Scholar 

  • Dustdar, Schahram, and Ilir Murturi. 2020. Towards distributed edge-based systems. In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI), 1–9. Piscataway: IEEE.

    Google Scholar 

  • Dustdar, Schahram, and Ilir Murturi. 2021. Towards IoT processes on the edge. In Next-Gen Digital Services. A Retrospective and Roadmap for Service Computing of the Future: Essays Dedicated to Michael Papazoglou on the Occasion of His 65th Birthday and His Retirement, 167–178.

    Google Scholar 

  • Flamis, Georgios, et al. 2021. Best practices for the deployment of edge inference: The conclusions to start designing. Electronics 10 (16): 1912.

    Article  Google Scholar 

  • Girshick, Ross, et al. 2015. Region-based convolutional networks for accurate object detection and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (1): 142–158.

    Article  Google Scholar 

  • Guo, Anqi, et al. 2023. Software-hardware co-design of heterogeneous SmartNIC system for recommendation models inference and training. In Proceedings of the 37th International Conference on Supercomputing, 336–347.

    Google Scholar 

  • Guo, Peizhen, and Wenjun Hu. 2018. Potluck: Cross-application approximate deduplication for computation-intensive mobile applications. In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, 271–284.

    Google Scholar 

  • Han, Seungyeop, et al. 2016. MCDNN: An approximation-based execution framework for deep stream processing under resource constraints. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services, 123–136.

    Google Scholar 

  • Haris, Jude, Gibson, Perry, Cano, José, Agostini, Nicolas Bohm and Kaeli, David. 2022. Hardware/Software Co-Design of Edge DNN Accelerators with TFLite. 107 (8): 1–4.

    Google Scholar 

  • Hu, Chuang, et al. 2019. Dynamic adaptive DNN surgery for inference acceleration on the edge. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications, 1423–1431. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Huang, Jin, et al. 2020. Clio: Enabling automatic compilation of deep learning pipelines across iot and cloud. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 1–12.

    Google Scholar 

  • Huynh, Loc N., et al. 2017. DeepMon: Mobile GPU-based deep learning framework for continuous vision applications. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, 82–95.

    Google Scholar 

  • Iandola, Forrest N., et al. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and\(<\) 0.5 MB model size. arXiv preprint. arXiv:1602.07360.

    Google Scholar 

  • Jayakodi, Nitthilan Kanappan, Janardhan Rao Doppa, et al. 2020. SETGAN: Scale and energy trade-off gans for image applications on mobile platforms. In Proceedings of the 39th International Conference on Computer-Aided Design, 1–9.

    Google Scholar 

  • Jayakodi, Nitthilan Kanappan, Syrine Belakaria, et al. 2020. Design and optimization of energy-accuracy tradeoff networks for mobile platforms via pretrained deep models. ACM Transactions on Embedded Computing Systems (TECS) 19 (1): 1–24.

    Article  Google Scholar 

  • Jeong, Hyuk-Jin, et al. 2018. IONN: Incremental offloading of neural network computations from mobile devices to edge servers. In Proceedings of the ACM Symposium on Cloud Computing, 401–411.

    Google Scholar 

  • Jiang, Xiaotang, et al. 2020. MNN: A universal and efficient inference engine. In Proceedings of Machine Learning and Systems. Vol. 2, 1–13.

    Google Scholar 

  • Jiao, Meng, et al. 2020. A GRU-RNN based momentum optimized algorithm for SOC estimation. Journal of Power Sources 459: 228051.

    Article  Google Scholar 

  • Kang, Yiping, et al. 2017. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News 45 (1): 615–629.

    Article  Google Scholar 

  • Kounoudes, Alexia Dini et al. 2021. User-centred privacy inference detection for smart home devices. 2021 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/IOP/SCI), 210–218. Piscataway: IEEE.

    Google Scholar 

  • Kouris, Alexandros, et al. 2022. Multi-exit semantic segmentation networks. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXI, 330–349. Berlin: Springer.

    Chapter  Google Scholar 

  • Laskaridis, Stefanos, Stylianos I. Venieris, Hyeji Kim, et al. 2020. HAPI: Hardware-aware progressive inference. In Proceedings of the 39th International Conference on Computer-Aided Design, 1–9.

    Google Scholar 

  • Laskaridis, Stefanos, Stylianos I. Venieris, Mario Almeida, et al. 2020. SPINN: Synergistic progressive inference of neural networks over device and cloud. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 1–15.

    Google Scholar 

  • Lebedev, Mikhail, and Pavel Belecky. 2021. A survey of open-source tools for FPGA-based inference of artificial neural networks. In 2021 Ivannikov Memorial Workshop (IVMEM), 50–56. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Lebedev, Vadim, and Victor Lempitsky. 2016. Fast convnets using group-wise brain damage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2554–2564.

    Google Scholar 

  • Leiserson, Charles E., et al. 2020. There’s plenty of room at the Top: What will drive computer performance after Moore’s law? Science 368 (6495): eaam9744.

    Google Scholar 

  • Leon, Vasileios, et al. 2022. Systematic embedded development and implementation techniques on intel myriad VPUs. In 2022 IFIP/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC), 1–2. Piscataway: IEEE.

    Google Scholar 

  • Li, En, et al. 2018. Edge intelligence: On-demand deep learning model co-inference with device-edge synergy. In Proceedings of the 2018 Workshop on Mobile Edge Communications, 31–36.

    Google Scholar 

  • Li, Hongshan, et al. 2018. JALAD: Joint accuracy-and latency-aware deep structure decoupling for edge-cloud execution. In 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), 671–678. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Li, Liangzhi, et al. 2018. Deep learning for smart industry: Efficient manufacture inspection system with fog computing. IEEE Transactions on Industrial Informatics 14 (10): 4665–4673.

    Article  Google Scholar 

  • Li, Ying, et al. 2023. Federated domain generalization: A survey. arXiv preprint. arXiv:2306.01334.

    Google Scholar 

  • LiKamWa, Robert, and Lin Zhong. 2015. Starfish: Efficient concurrency support for computer vision applications. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, 213–226.

    Google Scholar 

  • Liu, Hongye, et al. 2016. Deep relative distance learning: Tell the difference between similar vehicles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2167–2175.

    Google Scholar 

  • Liu, Shaoshan, et al. 2019. Edge computing for autonomous driving: Opportunities and challenges. Proceedings of the IEEE 107 (8): 1697–1716.

    Article  Google Scholar 

  • Lucas, Bruce D., and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. In IJCAI’81: 7th International Joint Conference on Artificial Intelligence. Vol. 2, 674–679.

    Google Scholar 

  • Mao, Jiachen, et al. 2017. MoDNN: Local distributed mobile computing system for deep neural network. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017, 1396–1401. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Mohammadi, Mehdi, and Ala Al-Fuqaha. 2018. Enabling cognitive smart cities using big data and machine learning: Approaches and challenges. IEEE Communications Magazine 56 (2): 94–101.

    Article  Google Scholar 

  • Owens, John D., et al. 2008. GPU computing. In Proceedings of the IEEE 96 (5): 879–899.

    Article  Google Scholar 

  • Panda, Priyadarshini, et al. 2016. Conditional deep learning for energy-efficient and enhanced pattern recognition. In 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), 475–480. Piscataway: IEEE.

    Google Scholar 

  • Polino, Antonio, et al. 2018. Model compression via distillation and quantization. arXiv preprint. arXiv:1802.05668.

    Google Scholar 

  • Rastegari, Mohammad, et al. 2016. XNOR-Net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, 525–542. Berlin: Springer.

    Google Scholar 

  • Ren, Wei-Qing, et al. 2023. A survey on collaborative DNN inference for edge intelligence. In Machine Intelligence Research, 1–25.

    Google Scholar 

  • Romero, Adriana, et al. 2014. Fitnets: Hints for thin deep nets. arXiv preprint. arXiv:1412.6550.

    Google Scholar 

  • Sedlak, Boris, et al. 2022. Specification and operation of privacy models for data streams on the edge. In 2022 IEEE 6th International Conference on Fog and Edge Computing (ICFEC), 78–82. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Sengupta, Abhronil, et al. 2019. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience 13: 95.

    Article  Google Scholar 

  • Soto, José Angel Carvajal, et al. 2016. CEML: Mixing and moving complex event processing and machine learning to the edge of the network for IoT applications. In Proceedings of the 6th International Conference on the Internet of Things, 103–110.

    Google Scholar 

  • Sun, Yi, Chen, Yuheng, Wang, Xiaogang, Tang, Xiaoou. 2014. Deep learning face representation by joint identification-verification. Advances in Neural Information Processing Systems 27 (8): 1–8.

    Google Scholar 

  • Targ, Sasha, et al. 2016. Resnet in resnet: Generalizing residual architectures. arXiv preprint. arXiv:1603.08029.

    Google Scholar 

  • Teerapittayanon, Surat, et al. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), 2464–2469. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Tsigkanos, Christos, et al. 2019. Dependable resource coordination on the edge at runtime. Proceedings of the IEEE 107 (8): 1520–1536.

    Article  Google Scholar 

  • Viola, Paul, and Michael Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. Vol. 1, I–I. Piscataway: IEEE.

    Google Scholar 

  • Wang, Qipeng, et al. 2022. Melon: Breaking the memory wall for resource-efficient on-device machine learning. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 450–463.

    Google Scholar 

  • Wang, Yang, et al. 2017. Effective multi-query expansions: Collaborative deep networks for robust landmark retrieval. IEEE Transactions on Image Processing 26 (3): 1393–1404.

    Article  MathSciNet  Google Scholar 

  • Wu, Jiaxiang, et al. 2016. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4820–4828.

    Google Scholar 

  • Xiang, Yecheng, and Hyoseung Kim. 2019. Pipelined data-parallel CPU/GPU scheduling for multi-DNN real-time inference. In 2019 IEEE Real-Time Systems Symposium (RTSS), 392–405. Piscataway: IEEE.

    Chapter  Google Scholar 

  • Xu, Daliang, et al. 2022. Mandheling: Mixed-precision on-device DNN training with DSP offloading. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking, 214–227.

    Google Scholar 

  • Xu, Mengwei, Jiawei Liu, et al. 2019. A first look at deep learning apps on smartphones. In The World Wide Web Conference, 2125–2136.

    Google Scholar 

  • Xu, Mengwei, Tiantu Xu, et al. 2021. Video analytics with zero-streaming cameras. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), 459–472.

    Google Scholar 

  • Xu, Mengwei, Xiwen Zhang, et al. 2020. Approximate query service on autonomous iot cameras. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, 191–205.

    Google Scholar 

  • Yim, Junho, et al. 2017. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4133–4141.

    Google Scholar 

  • Yu, Yong, et al. 2019. A review of recurrent neural networks: LSTM cells and network architectures. Neural Computation 31 (7): 1235–1270.

    Article  MathSciNet  Google Scholar 

  • Zhang, Qiyang, Xiang Li, et al. 2022. A comprehensive benchmark of deep learning libraries on mobile devices. In Proceedings of the ACM Web Conference 2022, 3298–3307.

    Google Scholar 

  • Zhang, Qiyang, Zuo Zhu, et al. 2023. Energy-efficient federated training on mobile device. IEEE Network 35 (5): 1–14.

    MathSciNet  Google Scholar 

  • Zhang, Xiangyu, et al. 2018. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6848–6856.

    Google Scholar 

  • Zhao, Zhuoran, et al. 2018. Deepthings: Distributed adaptive deep learning inference on resource-constrained IoT edge clusters. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 37 (11): 2348–2359.

    Article  Google Scholar 

  • Zhou, Kanglei, et al. 2022. TSVMPath: Fast regularization parameter tuning algorithm for twin support vector machine. Neural Processing Letters 54 (6): 5457–5482.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Network and Switching, Beijing University of Posts and Telecommunications, Beijing, China

    Qiyang Zhang

  2. Distributed Systems Group, TU Wien, Vienna, Austria

    Qiyang Zhang

  3. College of Computer Science and Engineering, Northeastern University, Shenyang, China

    Ying Li

  4. Distributed Systems Group, TU Wien, Vienna, Austria

    Ying Li

  5. State Key Laboratory of Network and Switching, Beijing University of Posts and Telecommunications, Beijing, China

    Dingge Zhang & Shangguang Wang

  6. Distributed Systems Group, TU Wien, Vienna, Austria

    Ilir Murturi, Victor Casamayor Pujol & Schahram Dustdar

Authors
  1. Qiyang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dingge Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ilir Murturi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Victor Casamayor Pujol
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Schahram Dustdar
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shangguang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiyang Zhang .

Editor information

Editors and Affiliations

  1. Distributed Systems Group, TU Wein, Vienna, Austria

    Praveen Kumar Donta

  2. Indian Institute of Information Technology, Sri City, India

    Abhishek Hazra

  3. Center for Ubiquitous Computing, University of Oulu, Oulu, Finland

    Lauri Lovén

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Zhang, Q. et al. (2024). Intelligence Inference on IoT Devices. In: Donta, P.K., Hazra, A., Lovén, L. (eds) Learning Techniques for the Internet of Things. Springer, Cham. https://doi.org/10.1007/978-3-031-50514-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50514-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50513-3

  • Online ISBN: 978-3-031-50514-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation