Abstract
The capability of the unmanned aerial vehicle (UAV) to capture highly informative data has expanded its utility in multiple sectors. Surveillance-based UAV applications have highly relied on accurate object tracking. During UAV monitoring, issues such as changes in object appearance and occlusion are common. Tracking the objects under such a scenario is a challenging task. On the other hand, accurate object tracking is quintessential in critical scenarios like security surveillance. In this work, a novel deep learning-based framework for accurate multiple objects tracking with UAV videos is proposed. Tiny-Deeply Supervised Object Detector (Tiny-DSOD) is adopted for accurate object detection. A novel stacked bidirectional-forward LSTM (SBF-LSTM) tracker with spatial and visual features is proposed for object tracking. The spatial and visual features obtained from Tiny-DSOD are trained with the tracker, which predicts object location during tracking. The choice of SBF-LSTM as the tracker enables accurate prediction of object location. Object association is dealt with based on bounding box distance, appearance, and size metrics. With the proposed model, switching object identities is lessened to a greater extent, thereby increasing the tracking accuracy. The proposed methodology outperforms the state-of-the-art methods on UAV videos.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B., (2016). Simple online and realtime tracking, In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464–3468. https://doi.org/10.1109/ICIP.2016.7533003\
Bochinski, E., Eiselein, V., Sikora, T., (2017). High-Speed tracking-by-detection without using image information, In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)., pp. 1–6. https://doi.org/10.1109/AVSS.2017.8078516
Chen, X., & Meng, Q. (2013). Vehicle detection from UAVs by using SIFT with implicit shape model. Proceedings - 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013, 3139–3144. https://doi.org/10.1109/SMC.2013.535
Cui, Z., Ke, R., & Wang, Y. (2018). Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction. arXiv preprint arXiv:1801.02143
Dai, K., Wang, Y., & Song, Q. (2019). Real-time object tracking with template tracking and foreground detection network. Sensors, 19(18), 3945. https://doi.org/10.3390/s19183945
Dirir, A., & Elsayed, H. (2019). Object tracking framework for unmanned aerial vehicles. In: Dubai: 2019 IEEE Global Conference on Internet of Things (GCIoT). https://doi.org/10.1109/GCIoT47977.2019.9058406
Drone war. https://www.thehindu.com/news/national/telangana/drone-war-on-drinking-in-public-spaces/article19914315.ece. Accessed 10 December 2019
Drones Monitor. https://www.ndtv.com/mumbai-news/drones-used-to-monitor-traffic-on-mumbai-pune-expressway-1451540. Accessed 10 December 2019
Du, D., Qi, Y., Yu, H., Yang, Y., Duan, K., Li, G., et al. (2018). The unmanned aerial vehicle benchmark: Object detection and tracking. http://arxiv.org/abs/1804.00518
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017). LSTM: A search space Odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924
Iter, D., Kuck, J., & Zhuang, P. (2016). Target Tracking with Kalman Filtering, KNN and LSTMs, 2
Lee, B. Y., Liew, L. H., Cheah, W. S., & Wang, Y. C. (2014). Occlusion handling in videos object tracking: A survey. In: IOP Conference Series: Earth and Environmental Science, 18(1). https://doi.org/10.1088/1755-1315/18/1/012020
Li, Y., Li, J., Lin, W., & Li, J. (2019). Tiny-DSOD: Lightweight object detection for resource-restricted usages. British Machine Vision Conference 2018, BMVC 2018
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single shot multibox detector. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 9905 LNCS, pp. 21–37). Springer Verlag. https://doi.org/10.1007/978-3-319-46448-0_2
Liu, S., Li, X., Lu, H., He, Y., 2022. Multi-Object Tracking Meets Moving UAV. In: Presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8876–8885
Máttyus, G., Benedek, C., & Szirányi, T. (2015). Multi target tracking on Aerial videos. International Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences ISPRS Archives, 38(1W17):0-6
Micheal, A. A., Vani, K., Sanjeevi, S., & Lin, C. H. (2020). A tool to enhance the capacity for deep learning based object detection and tracking with UAV data. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives (Vol. 43, pp. 221–226). International Society for Photogrammetry and Remote Sensing. https://doi.org/10.5194/isprs-archives-XLIII-B5-2020-221-2020
Mueller, M., Smith, N., & Ghanem, B. (2016). A benchmark and simulator for UAV tracking. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 9905 LNCS, pp. 445–461). Springer Verlag. https://doi.org/10.1007/978-3-319-46448-0_27
Pirsiavash, H., Ramanan, D., Fowlkes, C.C., 2011. Globally-optimal greedy algorithms for tracking a variable number of objects, In: CVPR 2011. Presented at the CVPR 2011, pp. 1201–1208. https://doi.org/10.1109/CVPR.2011.5995604
Police drone. https://www.thehindu.com/news/national/tamil-nadu/police-deploy-drone-to-nab-dacoits/article21285524.ece. Accessed 10 December 2019
Real-World Drones. https://cacm.acm.org/magazines/2019/11/240355-real-world-applications-for-drones/fulltext. Accessed 10 December 2019
Sadeghi-Tehran, P., Clarke, C., Angelov, P., (2014). A Real-time approach for autonomous detection and tracking of moving objects from UAV In: IEEE Symposium on Evolving and Autonomous Learning Systems (EALS) https://doi.org/10.1109/EALS.2014.7009502
Sǎrǎcin, CG., Dragoş, I., Chirilǎ, AI. (2017) Powering aerial surveillance drones. In: 2017 10th International Symposium on Advanced Topics in Electrical Engineering, ATEE 2017, 237–240. https://doi.org/10.1109/ATEE.2017.7905185
Shen, Z., Liu, Z., Li, J., Jiang, Y. G., Chen, Y., & Xue, X. (2017). DSOD: Learning Deeply Supervised Object Detectors from Scratch. In: Proceedings of the IEEE International Conference on Computer Vision (Vol. 2017-October, pp. 1937–1945). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ICCV.2017.212
Wojke, N., Bewley, A., & Paulus, D. (2018). Simple online and realtime tracking with a deep association metric. In: Proceedings - International Conference on Image Processing, ICIP, 2017-Septe, 3645–3649. https://doi.org/10.1109/ICIP.2017.8296962
Xu, Y., Yu, G., Wang, Y., Wu, X., & Ma, Y. (2017). Car detection from low-altitude UAV imagery with the faster R-CNN. Journal of Advanced Transportation. https://doi.org/10.1155/2017/2823617
Xue, X., Li, Y., & Shen, Q. (2018). Unmanned aerial vehicle object tracking by correlation filter with adaptive appearance model. Sensors (Switzerland), 18(9). https://doi.org/10.3390/s18092751
Zhang, W., Song, K., Rong, X., & Li, Y. (2019). Coarse-to-Fine UAV Target Tracking with Deep Reinforcement Learning. IEEE Transactions on Automation Science and Engineering, 16(4), 1522–1530. https://doi.org/10.1109/TASE.2018.2877499
Zhu, P., Wen, L., Du, D., Bian, X., Fan, H., Hu, Q., & Ling, H. (2020). Detection and Tracking Meet Drones Challenge. http://arxiv.org/abs/2001.06303
Acknowledgements
This work was funded by International Society for Photogrammetry and Remote Sensing(ISPRS) under ISPRS Scientific Initiatives 2019
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Micheal, A.A., Vani, K. Deep Learning-Based Multi-class Multiple Object Tracking in UAV Video. J Indian Soc Remote Sens 50, 2543–2552 (2022). https://doi.org/10.1007/s12524-022-01615-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12524-022-01615-7