Skip to main content
SpringerLink
Log in

Real-time task scheduling with image resizing for criticality-based machine perception

  • Published:
Real-Time Systems Aims and scope Submit manuscript

Abstract

This paper extends a previous conference publication that proposed a real-time task scheduling framework for criticality-based machine perception, leveraging image resizing as the tool to control the accuracy and execution time trade-off. Criticality-based machine perception reduces the computing demand of on-board AI-based machine inference pipelines (that run on embedded hardware) in applications such as autonomous drones and cars. By segmenting inputs, such as individual video frames, into smaller parts and allowing the downstream AI-based perception module to process some segments ahead of (or at a higher quality than) others, limited machine resources are spent more judiciously on more important parts of the input (e.g., on foreground objects in lieu of backgrounds). In recent work, we explored the use of image resizing as a way to offer a middle ground between full-resolution processing and dropping, thus allowing more flexibility in handling less important parts of the input. In this journal extension, we make the following contributions: (i) We relax a limiting assumption of our prior work; namely, the need for a “perfect sensor" to identify which parts of the image are more critical. Instead, we investigate the use of real LiDAR measurements for quick-and-dirty image segmentation ahead of AI-based processing. (ii) We explore another dimension of freedom in the scheduler: namely, merging several nearby objects into a consolidated segment for downstream processing. We formulate the scheduling problem as an optimal resize-merge problem and design a solution for it. Experiments on an AI-powered embedded platform with a real-world driving dataset demonstrate the practicality and effectiveness of our proposed framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Note that in the presence of multiple objects in the segment, this empirically computed number can be the average of per-object confidence values.

  2. https://github.com/ultralytics/yolov5

References

  • Abdelzaher T, Ayanian N, Basar T, Diggavi S, Diesner J, Ganesan D, Govindan R, Jha S, Lepoint T, Marlin B, Nahrstedt K, Nicol D, Rajkumar R, Russell S, Seshia S, Sha F, Shenoy P, Srivastava M, Sukhatme G, Swami A, Tabuada P, Towsley D, Vaidya N, Veeravalli V (2018) Toward an internet of battlefield things: a resilience perspective. Computer 51(11):24–36

    Article  Google Scholar 

  • Alcon M, Tabani H, Kosmidis L, Mezzetti E, Abella J, Cazorla FJ (2020) Timing of autonomous driving software: problem analysis and prospects for future solutions. In: 2020 IEEE real-time and embedded technology and applications symposium (RTAS), pp 267–280

  • Bamburry D (2015) Drones: designed for product delivery. Des Manag Rev 26(1):40–48

    Google Scholar 

  • Bar-Shalom Y, Tse E (1975) Tracking in a cluttered environment with probabilistic data association. Automatica 11(5):451–460

    Article  MATH  Google Scholar 

  • Bateni S, Liu C (2018) Apnet: approximation-aware real-time neural network. In: 2018 IEEE real-time systems symposium (RTSS), pp 67–79

  • Bhattacharya S, Lane ND (2016) Sparsification and separation of deep learning layers for constrained resource inference on wearables. In: Proceedings of the 14th ACM conference on embedded network sensor systems CD-ROM. SenSys ’16. Association for Computing Machinery, New York, NY, USA, pp 176–189

  • Bogoslavskyi I, Stachniss C (2017) Efficient online segmentation for sparse 3d laser scans. PFG J Photogramm Remote Sens Geoinf Sci 85:41–52

    Google Scholar 

  • Chen J, Yu S, Tabish R, Bansal A, Liu S, Abdelzaher T, Sha L (2021) Lidar cluster first and camera inference later: a new perspective towards autonomous driving. arXiv preprint arXiv:2111.09799

  • CUDA Concurrent Kernel Execution. https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#concurrent-kernel-execution

  • Dai X, Yin H, Jha NK (2020) Incremental learning using a grow-and-prune paradigm with efficient neural networks. IEEE Trans Emerg Top Comput. https://doi.org/10.1109/TETC.2020.3037052

    Article  Google Scholar 

  • Denton E, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation

  • Floreano D, Wood RJ (2015) Science, technology and the future of small autonomous drones. Nature 521:460–466

    Article  Google Scholar 

  • Han S, Mao H, Dally W.J (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding

  • Heo S, Cho S, Kim Y, Kim H (2020) Real-time object detection system with multi-path neural networks. In: 2020 IEEE real-time and embedded technology and applications symposium (RTAS), pp 174–187

  • Hu Y, Liu S, Abdelzaher T, Wigness M, David P (2021) On exploring image resizing for optimizing criticality-based machine perception. In: 2021 IEEE 27th international conference on embedded and real-time computing systems and applications (RTCSA). IEEE, pp 169–178

  • Jin Q, Yang L, Liao Z (2020) Adabits: neural network quantization with adaptive bit-widths. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Jung S, Son C, Lee S, Son J, Han J-J, Kwak Y, Hwang S.J, Choi C (2019) Learning to quantize deep networks by optimizing quantization intervals with task loss. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Kellerer H, Pferschy U, Pisinger D (2004) The multiple-choice knapsack problem. Springer, Berlin, pp 317–347

    MATH  Google Scholar 

  • Kim J-E, Bradford R, Shao Z (2020a) Anytimenet: controlling time-quality tradeoffs in deep neural network architectures. In: 2020 design, automation test in Europe conference exhibition (DATE), pp 945–950

  • Kim J-E, Bradford R, Yoon M-K, Shao Z (2020b) Abc: abstract prediction before concreteness. In: 2020 design, automation test in Europe conference exhibition (DATE), pp 1103–1108

  • Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollar P, Zitnick L (2014) Microsoft coco: common objects in context. In: European conference on computer vision (ECCV)

  • Lin S-C, Zhang Y, Hsu C-H, Skach M, Haque ME, Tang L, Mars J (2018) The architectural implications of autonomous driving: constraints and acceleration. SIGPLAN Not 53(2):751–766

    Article  Google Scholar 

  • Liu S, Yao S, Fu X, Tabish R, Yu S, Bansal A, Yun H, Sha L, Abdelzaher T (2020) On removing algorithmic priority inversion from mission-critical machine inference pipelines. In: 2020 IEEE real-time systems symposium (RTSS), pp 319–332

  • Liu S, Yao S, Fu X, Shao H, Tabish R, Yu S, Bansal A, Yun H, Sha L, Abdelzaher T (2021) Real-time task scheduling for machine perception in intelligent cyber-physical systems. IEEE Trans Comput. https://doi.org/10.1109/TC.2021.3106496

    Article  MATH  Google Scholar 

  • Milioto A, Vizzo I, Behley J, Stachniss C (2019) Rangenet++: fast and accurate lidar semantic segmentation. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 4213–4220

  • Minnehan B, Savakis A (2019) Cascaded projection: end-to-end network compression and acceleration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Olmedo IS, Capodieci N, Martinez JL, Marongiu A, Bertogna M (2020) Dissecting the cuda scheduling hierarchy: a performance and predictability perspective. In: 2020 IEEE real-time and embedded technology and applications symposium (RTAS), pp 213–225

  • Otterness N, Anderson JH (2020) Amd gpus as an alternative to nvidia for supporting real-time workloads. 32nd Euromicro conference on real-time systems (ECRTS 2020), vol 165. Leibniz International Proceedings in Informatics (LIPIcs). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, pp 10–11023

  • Otterness N, Yang M, Rust S, Park E, Anderson JH, Smith FD, Berg A, Wang S (2017) An evaluation of the nvidia tx1 for supporting real-time computer-vision workloads. In: 2017 IEEE real-time and embedded technology and applications symposium (RTAS), pp 353–364

  • Sinha P, Zoltners AA (1979) The multiple-choice knapsack problem. Oper Res 27(3):503–515

    Article  MathSciNet  MATH  Google Scholar 

  • Sun P, Kretzschmar H, Dotiwalla X, Chouard A, Patnaik V, Tsui P, Guo J, Zhou Y, Chai Y, Caine B, Vasudevan V, Han W, Ngiam J, Zhao H, Timofeev A, Ettinger S, Krivokon M, Gao A, Joshi A, Zhang Y, Shlens J, Chen Z, Anguelov D (2020) Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Wang Y, Xu C, You S, Tao D, Xu C (2016) Cnnpack: packing convolutional neural networks in the frequency domain. In: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R (eds) Advances in neural information processing systems, vol 29. Curran Associates, Inc

  • Yang M, Otterness N, Amert T, Bakita J, Anderson JH, Smith FD (2018) Avoiding pitfalls when using nvidia gpus for real-time tasks in autonomous systems. In: Altmeyer S (ed) 30th Euromicro conference on real-time systems (ECRTS 2018), vol 106. Leibniz International Proceedings in Informatics (LIPIcs). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, pp 20–12021

  • Yang M, Wang S, Bakita J, Vu T, Smith FD, Anderson JH, Frahm J-M (2019) Re-thinking cnn frameworks for time-sensitive autonomous-driving applications: addressing an industrial challenge. In: 2019 IEEE real-time and embedded technology and applications symposium (RTAS), pp 305–317

  • Yao S, Zhao Y, Zhang A, Su L, Abdelzaher T (2017) Deepiot: compressing deep neural network structures for sensing systems with a compressor-critic framework. In: Proceedings of the 15th ACM conference on embedded network sensor systems. SenSys ’17. Association for Computing Machinery, New York, NY, USA

  • Yao S, Zhao Y, Zhang A, Hu S, Shao H, Zhang C, Su L, Abdelzaher T (2018a) Deep learning for the internet of things. Computer 51(5):32–41

    Article  Google Scholar 

  • Yao S, Zhao Y, Shao H, Liu S, Liu D, Su L, Abdelzaher T (2018b) Fastdeepiot: towards understanding and optimizing neural network execution time on mobile and embedded devices. In: Proceedings of the 16th ACM conference on embedded networked sensor systems. SenSys ’18. Association for Computing Machinery, New York, NY, USA, pp 278–291

  • Yao S, Hao Y, Zhao Y, Shao H, Liu D, Liu S, Wang T, Li J, Abdelzaher T (2020) Scheduling real-time deep learning services as imprecise computations. In: 2020 IEEE 26th international conference on embedded and real-time computing systems and applications (RTCSA), pp 1–10

Download references

Acknowledgements

This research was sponsored in part by DARPA award W911NF-17-C-0099 and the Army Research Laboratory under Cooperative Agreement W911NF-17-2-0196.

Author information

Authors and Affiliations

  1. University of Illinois at Urbana-Champaign, Champaign, USA

    Yigong Hu, Shengzhong Liu & Tarek Abdelzaher

  2. US DEVCOM Army Research Laboratory, Adelphi, USA

    Maggie Wigness & Philip David

Authors
  1. Yigong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shengzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tarek Abdelzaher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maggie Wigness
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Philip David
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yigong Hu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, Y., Liu, S., Abdelzaher, T. et al. Real-time task scheduling with image resizing for criticality-based machine perception. Real-Time Syst 58, 430–455 (2022). https://doi.org/10.1007/s11241-022-09387-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11241-022-09387-6

Keywords

Search

Navigation