Abstract
Nano-sized unmanned aerial vehicles (UAVs) have been attracting growing attention from both academia and industry with their compact and versatile nature. With the size of merely several centimeters, they are particularly well-suited for executing targeted missions in confined spaces, such as tracking a single pedestrian indoors. However, due to strict constraints on payload and power consumption, the onboard computing unit of nano-UAV is limited to the microcontroller Unit (MCU), which makes the novel single object tracking algorithms not applicable on nano-UAV platform. In this paper, we present a lightweight single pedestrian tracking algorithm to meet the strict resource constraints of nano-UAV system. The algorithm consists of an object detection frontend and a motion-based tracking backend. Based on our deployment methodology, covering from dataset production, model training to layer fusion and deployment, the proposed algorithm achieves extreme complexity reduction (13\(\times \) fewer operations and 106\(\times \) less memory), and keeps good tracking performance compared with state-of-the-art (SOTA) research work. The full range of onboard experiments illustrate the efficiency our algorithm for real-time tracking (up to 43 fps). Meanwhile, both high computing efficiency (5.2 MACs/cycle) and energy efficiency (5.3 mJ/frame) are achieved and exceed similar works on nano-UAV.
Similar content being viewed by others
Code Availability
Not applicable
References
Liu, Y., Meng, Z., Zou, Y., Cao, M.: Visual object tracking and servoing control of a nano-scale quadrotor: system, algorithms, and experiments. IEEE CAA J. Autom. Sinica 8(2), 344–360 (2021)
Niculescu, V., Lamberti, L., Conti, F., Benini, L., Palossi, D.: Improving autonomous nano-drones performance via automated end-to-end optimization and deployment of dnns. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 11(4), 548–562 (2021)
Palossi, D., Zimmerman, N., Burrello, A., Conti, F., Müller, H., Gambardella, L.M., Benini, L., Giusti, A., Guzzi, J.: Fully onboard ai-powered human-drone pose estimation on ultra-low power autonomous flying nano-uavs. IEEE Internet of Things Journal (2021)
Palossi, D., Conti, F., Benini, L.: An open source and open hardware deep learning-powered visual navigation engine for autonomous nano-uavs. In: 2019 15th International conference on Distributed Computing in Sensor Systems (DCOSS), pp. 604–611 (2019)
Palossi, D., Loquercio, A., Conti, F., Flamand, E., Scaramuzza, D., Benini, L.: A 64-mw dnn-based visual navigation engine for autonomous nano-drones. IEEE Internet Things J. 6(5), 8357–8371 (2019)
Palossi, D., Gomez, A., Draskovic, S., Marongiu, A., Thiele, L., Benini, L.: Extending the lifetime of nano-blimps via dynamic motor control. Journal of Signal Processing Systems 91(3), 339–361 (2019)
Kang, K., Belkhale, S., Kahn, G., Abbeel, P., Levine, S.: Generalization through simulation: integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 6008–6014 (2019)
Srisamosorn, V., Kuwahara, N., Yamashita, A., Ogata, T., Ota, J.: Human-tracking system using quadrotors and multiple environmental cameras for face-tracking application. Int. J. Adv. Rob. Syst. 14(5), 1729881417727357 (2017)
Palossi, D., Singh, J., Magno, M., Benini, L.: Target following on nano-scale unmanned aerial vehicles. In: 2017 7th IEEE International Workshop on Advances in Sensors and Interfaces (IWASI), pp. 170–175 (2017)
Zhang, X., Xian, B., Zhao, B., Zhang, Y.: Autonomous flight control of a nano quadrotor helicopter in a gps-denied environment using on-board vision. IEEE Trans. Industr. Electron. 62(10), 6392–6403 (2015)
Cai, G., Dias, J., Seneviratne, L.: A survey of small-scale unmanned aerial vehicles: recent advances and future development trends. Unmanned Systems 2(02), 175–199 (2014)
Briod, A., Zufferey, J.-C., Floreano, D.: Optic-flow based control of a 46g quadrotor. In: Workshop on vision-based closed-loop control and navigation of micro helicopters in GPS-denied environments, IROS 2013. Workshop on vision-based closed-loop control and navigation of micro helicopters in GPS-denied environments, IROS 2013 (2013). CONF
Shakhatreh, H., Sawalmeh, A.H., Al-Fuqaha, A., Dou, Z., Almaita, E., Khalil, I., Othman, N.S., Khreishah, A., Guizani, M.: Unmanned aerial vehicles (uavs): a survey on civil applications and key research challenges. Ieee Access 7, 48572–48634 (2019)
Motlagh, N.H., Taleb, T., Arouk, O.: Low-altitude unmanned aerial vehicles-based internet of things services: comprehensive survey and future perspectives. IEEE Internet Things J. 3(6), 899–922 (2016)
Lo, L.-Y., Yiu, C.H., Tang, Y., Yang, A.-S., Li, B., Wen, C.-Y.: Dynamic object tracking on autonomous uav system for surveillance applications. Sensors 21(23), 7888 (2021)
Rodriguez-Ramos, A., Alvarez-Fernandez, A., Bavle, H., Campoy, P., How, J.P.: Vision-based multirotor following using synthetic learning techniques. Sensors 19(21), 4794 (2019)
Cheng, H., Lin, L., Zheng, Z., Guan, Y., Liu, Z.: An autonomous vision-based target tracking system for rotorcraft unmanned aerial vehicles. In: 2017 IEEE/RSJ International conference on Intelligent Robots and Systems (IROS), pp. 1732–1738 (2017)
Floreano, D., Wood, R.J.: Science, technology and the future of small autonomous drones. Nature 521(7553), 460–466 (2015)
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: multi-object tracking by associating every detection box. arXiv preprint arXiv:2110.06864 (2021)
Li, R., Pang, M., Zhao, C., Zhou, G., Fang, L.: Monocular long-term target following on uavs. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 29–37 (2016)
Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: Yolov4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)
Conti, F., Palossi, D., Marongiu, A., Rossi, D., Benini, L.: Enabling the heterogeneous accelerator model on ultra-low power microcontroller platforms. In: 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1201–1206 (2016)
Gautschi, M., Schiavone, P.D., Traber, A., Loi, I., Pullini, A., Rossi, D., Flamand, E., Gürkaynak, F.K., Benini, L.: Near-threshold risc-v core with dsp extensions for scalable iot endpoint devices. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 25(10), 2700–2713 (2017)
Pullini, A., Rossi, D., Haugou, G., Benini, L.: \(\mu \)dma: an autonomous i/o subsystem for iot end-nodes. In: 2017 27th International symposium on Power and Timing Modeling, Optimization and Simulation (PATMOS), pp. 1–8 (2017)
Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., Yan, J.: Siamrpn++: Evolution of siamese visual tracking with very deep networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4282–4291 (2019)
Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.: Fast online object tracking and segmentation: a unifying approach. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1328–1338 (2019)
Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: 2010 IEEE Computer society conference on computer vision and pattern recognition, pp. 2544–2550 (2010)
Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2014)
Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: Exploiting the circulant structure of tracking-by-detection with kernels. In: European conference on computer vision, pp. 702–715 (2012)
Danelljan, M., Häger, G., Khan, F.S., Felsberg, M.: Discriminative scale space tracking. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1561–1575 (2016)
Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8971–8980 (2018)
Zhou, X., Wang, D., Krähenbühl P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)
Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848–6856 (2018)
Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly 2(1–2), 83–97 (1955)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European conference on computer vision, pp. 740–755 (2014)
Wu, Y., Lim, J., Yang, M.-H.: Online object tracking: a benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2411–2418 (2013)
Acknowledgements
This work is supported by the Qiyuan Laboratory and the National Natural Science Foundation of China (62171156).
Funding
This work is supported by the National Natural Science Foundation of China (62171156).
Author information
Authors and Affiliations
Contributions
Haolin Chen, Ruidong Wu, Wenshuai Lu, Xinglong Ji, Tao Wang, Haolun Ding, Yuxiang Dai and Bing Liu designed the study. Haolin Chen, Ruidong Wu, Tao Wang and Haolun Ding designed the tracking algorithm. Haolin Chen, Wenshuai Lu and Xinglong Ji designed the development methodology. Haolin Chen, Xionglong Ji, Tao Wang and Bing Liu designed the dataset and the experiment. Haolin Chen and Ruidong Wu coordinated the experiment and analyzed the data. All authors wrote the final manuscript.
Corresponding author
Ethics declarations
Ethics approval
Not applicable as this study does not contain biological applications.
Consent for participate
All authors of this research paper have consented to participate in the research study.
Consent for publication
All authors and involved researchers give their consent for the publication of the present paper, including photographs and video, to be published in the Journal of Intelligent and Robotic Systems.
Conflicts of interest
Not applicable
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Chen, H., Wu, R., Lu, W. et al. Fully Onboard Single Pedestrian Tracking on Nano-UAV Platform. J Intell Robot Syst 109, 50 (2023). https://doi.org/10.1007/s10846-023-01979-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-01979-z