Abstract
Accurate yield estimation of passion fruits is essential for planning acreage and harvest timing. However, due to the complexity of the natural environment and tracking instability, the existing yield estimation methods suffer from excessively large models that are difficult to deploy or repetitive counting of fruit. Therefore, an improved approach for efficient passion fruit yield estimation was proposed using the lightweight YOLOv5s and improved DeepSORT. First, the video is fed into the proposed lightweight YOLOv5s called YOLOv5s-little to obtain coordinates and confidence information about the fruits within each frame. Then, the information obtained from the detection model is input into improved DeepSORT for continuous frame tracking of passion fruit. Considering the frequent error IDs (ID switching), two improvements based on DeepSORT are proposed: delaying the creation of tracks and adding a second round of IoU matching. Finally, to overcome the problem of repetitive counting, a specific tracking counting method based on the track information and state is used for accurate passion fruit counting. Our method achieved a competitive result in tests. YOLOv5s-little detector achieved precision of 98.9%, 98.3% recall, 99.5% mAP, and only 0.9MB model size. The improved DeepSORT algorithm achieved higher order tracking accuracy (HOTA) of 79.6%, multi-object tracking accuracy (MOTA) of 92.58%, identification F1 (IDF1) of 95.02%, and ID switch (IDSW) of 11 respectively. Compared with DeepSORT, it improved by 4.66%, 1.8%, and 9.16% in HOTA, MOTA and IDF1, respectively, and IDSW improved the most with 85%. Compared with FairMOT and TransTrack, the HOTA of YOLOv5s-little + improved DeepSORT achieved improvements of 11.56% and 25.24%, respectively. The statistical average counting accuracy of our proposed counting method reaches 95.1%, which is a 7.09% improvement over the maximum ID value counting method. The counting results from test videos are highly correlated with the manual counting results (\({\text{R}}^{2}\) = 0.96), indicating that the counting method has high accuracy and effectiveness. These results show that YOLOv5s-little + improved DeepSORT can meet the practical needs of passion fruit yield estimation in real scenarios.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ariza-Sentís, M., Baja, H., Vélez, S., & Valente, J. (2023). Object detection and tracking on UAV RGB videos for early extraction of grape phenotypic traits. Computers and Electronics in Agriculture, 211, 108051. https://doi.org/10.1016/j.compag.2023.108051
Assunção, E. T., Gaspar, P. D., Mesquita, R. J. M., Simões, M. P., Ramos, A., Proença, H., & Inacio, P. R. M. (2022). Peaches detection using a deep learning technique—a contribution to yield estimation, resources management, and circular economy. Climate, 10(2), 11. https://doi.org/10.3390/cli10020011
Bewley, A., Ge, Z., Ott, L., Ramos, F., & Upcroft, B. (2016). Simple online and realtime tracking. 2016 IEEE international conference on image processing (ICIP) (pp. 3464–3468). IEEE.
Bezerra, A. D. M., Pacheco Filho, A. J. S., Bomfim, I. G. A., Smagghe, G., & Freitas, B. M. (2019). Agricultural area losses and pollinator mismatch due to climate changes endanger passion fruit production in the Neotropics. Agricultural Systems, 169, 49–57. https://doi.org/10.1016/j.agsy.2018.12.002
Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M. (2020). Yolov4: Optimal speed and accuracy of object detection. Preprint retrieved from https://arxiv.org/abs/2004.10934
De Jong, S., Baja, H., Tamminga, K., & Valente, J. (2022). APPLE MOTS: Detection, segmentation and tracking of homogeneous objects using MOTS. IEEE Robotics and Automation Letter, 7(4), 11418–11425. https://doi.org/10.1109/LRA.2022.3199026
Egi, Y., Hajyzadeh, M., & Eyceyurt, E. (2022). Drone-computer communication based tomato generative organ counting model using YOLO V5 and deep-sort. Agriculture, 12(9), 1290. https://doi.org/10.3390/agriculture12091290
Gao, F., Fang, W., Sun, X., Wu, Z., Zhao, G., Li, G., Li, R., Fu, L., & Zhang, Q. (2022). A novel apple fruit detection and counting methodology based on deep learning and trunk tracking in modern orchard. Computers and Electronics in Agriculture, 197, 107000. https://doi.org/10.1016/j.compag.2022.107000
Ge, Y., Lin, S., Zhang, Y., Li, Z., Cheng, H., Dong, J., Shao, S., Zhang, J., Qi, X., & Wu, Z. (2022). Tracking and counting of tomato at different growth period using an improving YOLO-deepsort network for inspection robot. Machines, 10(6), 489. https://doi.org/10.3390/machines10060489
Guo, S., Wang, S., Yang, Z., Wang, L., Zhang, H., Guo, P., Gao, Y., & Guo, J. (2022). A review of deep learning-based visual multi-object tracking algorithms for autonomous driving. Applied Sciences, 12(21), 10741. https://doi.org/10.3390/app122110741
He, L., Fang, W., Zhao, G., Wu, Z., Fu, L., Li, R., Majeed, Y., & Dhupia, J. (2022a). Fruit yield prediction and estimation in orchards: A state-of-the-art comprehensive review for both direct and indirect methods. Computers and Electronics in Agriculture, 195, 106812. https://doi.org/10.1016/j.compag.2022.106812
He, L., Wu, F., Du, X., & Zhang, G. (2022b). Cascade-SORT: A robust fruit counting approach using multiple features cascade matching. Computers and Electronics in Agriculture, 200, 107223. https://doi.org/10.1016/j.compag.2022.107223
Holzinger, A., Saranti, A., Angerschmid, A., Retzlaff, C. O., Gronauer, A., Pejakovic, V., Medel-Jimenez, F., Krexner, T., Gollob, C., & Stampfer, K. (2022). Digital transformation in smart farm and forest operations needs human-centered AI: Challenges and future directions. Sensors, 22(8), 3043. https://doi.org/10.3390/s22083043
Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO.’ Precision Agriculture, 20(6), 1107–1135. https://doi.org/10.1007/s11119-019-09642-0
Liu, S., Cossell, S., Tang, J., Dunn, G., & Whitty, M. (2017). A computer vision system for early stage grape yield estimation based on shoot detection. Computers and Electronics in Agriculture, 137, 88–101. https://doi.org/10.1016/j.compag.2017.03.013
Liu, T. H., Nie, X. N., Wu, J. M., Zhang, D., Liu, W., Cheng, Y. F., Zheng, Y., Qiu, J., & Qi, L. (2023). Pineapple (Ananas comosus) fruit detection and localization in natural environment based on binocular stereo vision and improved YOLOv3 model. Precision Agriculture, 24(1), 139–160. https://doi.org/10.1007/s11119-022-09935-x
Liu, X., Chen, S. W., Liu, C., Shivakumar, S. S., Das, J., Taylor, C. J., Underwood, J., & Kumar, V. (2019). Monocular camera based fruit counting and mapping with semantic data association. IEEE Robotics and Automation Letter, 4(3), 2296–2303. https://doi.org/10.1109/LRA.2019.2901987
Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., & Leibe, B. (2021). HOTA: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, 129(2), 548–578. https://doi.org/10.1007/s11263-020-01375-2
Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., & Kim, T. K. (2021). Multiple object tracking: A literature review. Artificial Intelligence, 293, 103448. https://doi.org/10.1016/j.artint.2020.103448
Padilla, R., Netto, S. L., & da Silva, E. A. B. (2020). A survey on performance metrics for object-detection algorithms. 2020 international conference on systems, signals and image processing (IWSSIP) (pp. 237–242). Niterói: IEEE.
Qi, X., Dong, J., Lan, Y., & Zhu, H. (2022). Method for identifying litchi picking position based on YOLOv5 and PSPNet. Remote Sensing, 14(9), 2004. https://doi.org/10.3390/rs14092004
Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. Preprint retrieved from https://arxiv.org/abs/1804.02767
Rong, J., Zhou, H., Zhang, F., Yuan, T., & Wang, P. (2023). Tomato cluster detection and counting using improved YOLOv5 based on RGB-D fusion. Computers and Electronics in Agriculture, 207, 107741. https://doi.org/10.1016/j.compag.2023.107741
Shen, L., Su, J., He, R., Song, L., Huang, R., Fang, Y., Song, Y., & Su, B. (2023). Real-time tracking and counting of grape clusters in the field based on channel pruning with YOLOv5s. Computers and Electronics in Agriculture, 206, 107662. https://doi.org/10.1016/j.compag.2023.107662
Sun, P., Cao, J., Jiang, Y., Zhang, R., Xie, E., Yuan, Z., Wang, C., & Luo, P. (2020). Transtrack: Multiple object tracking with transformer. Preprint retrieved form https://arxiv.org/abs/2012.15460
Tan, C., Li, C., He, D., & Song, H. (2022). Towards real-time tracking and counting of seedlings with a one-stage detector and optical flow. Computers and Electronics in Agriculture, 193, 106683. https://doi.org/10.1016/j.compag.2021.106683
Thokchom, R., & Mandal, G. (2017). Production preference and importance of passion fruit (Passiflora edulis): A review. Journal of Agricultural Engineering and Food Technology, 4(1), 27–30.
Tu, S., Pang, J., Liu, H., Zhuang, N., Chen, Y., Zheng, C., Wan, H., & Xue, Y. (2020). Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images. Precision Agriculture, 21(5), 1072–1091. https://doi.org/10.1007/s11119-020-09709-3
Vasconez, J. P., Delpiano, J., Vougioukas, S., & Auat Cheein, F. (2020). Comparison of convolutional neural networks in fruit detection and counting: A comprehensive evaluation. Computers and Electronics in Agriculture, 173, 105348. https://doi.org/10.1016/j.compag.2020.105348
Wang, D., & He, D. (2021). Channel pruned YOLO V5s-based deep learning approach for rapid and accurate apple fruitlet detection before fruit thinning. Biosystems Engineering, 210, 271–281. https://doi.org/10.1016/j.biosystemseng.2021.08.015
Wang, L., Zhao, Y., Xiong, Z., Wang, S., Li, Y., & Lan, Y. (2022a). Fast and precise detection of litchi fruits for yield estimation based on the improved YOLOv5 model. Frontiers in Plant Science, 13, 965425. https://doi.org/10.3389/fpls.2022.965425
Wang, Z., Jin, L., Wang, S., & Xu, H. (2022b). Apple stem/calyx real-time recognition using YOLO-v5 algorithm for fruit automatic loading system. Postharvest Biology and Technology, 185, 111808. https://doi.org/10.1016/j.postharvbio.2021.111808
Wang, X., Wu, Z., Jia, M., Xu, T., Pan, C., Qi, X., & Zhao, M. (2023). Lightweight SM-YOLOv5 tomato fruit detection algorithm for plant factory. Sensors, 23(6), 3336. https://doi.org/10.3390/s23063336
Wojke, N., Bewley, A., & Paulus, D. (2017). Simple online and realtime tracking with a deep association metric. 2017 IEEE international conference on image processing (ICIP) (pp. 3645–3649). IEEE.
Yang, H., Chang, F., Huang, Y., Xu, M., Zhao, Y., Ma, L., & Su, H. (2022). Multi-object tracking using Deep SORT and modified CenterNet in cotton seedling counting. Computers and Electronics in Agriculture, 202, 107339. https://doi.org/10.1016/j.compag.2022.107339
Zhang, W., Wang, J., Liu, Y., Chen, K., Li, H., Duan, Y., Wu, W., Shi, Y., & Guo, W. (2022). Deep-learning-based in-field citrus fruit detection and tracking. Horticulture Research, 9, 1–10.
Zhang, Y., Wang, C., Wang, X., Zeng, W., & Liu, W. (2021). FairMOT: On the fairness of detection and re-identification in multiple object tracking. International Journal of Computer Vision, 129(11), 3069–3087. https://doi.org/10.1007/s11263-021-01513-4
Zhou, Z., Song, Z., Fu, L., Gao, F., Li, R., & Cui, Y. (2020). Real-time kiwifruit detection in orchard using deep learning on Android™ smartphones for yield estimation. Computers and Electronics in Agriculture, 179, 105856. https://doi.org/10.1016/j.compag.2020.105856
Funding
Funding was provided by National Natural Science Foundation of China (Grant Nos. 61772209, 31600591), Science and Technology Planning Project of Guangdong Province (Grant No. 2019A050510034), National College Students Innovation and Entrepreneurship Training Program (Grant No. 202110564025), Key Research and Development Program of Guangzhou (No. 2024B03J1358).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Tu, S., Huang, Y., Liang, Y. et al. A passion fruit counting method based on the lightweight YOLOv5s and improved DeepSORT. Precision Agric (2024). https://doi.org/10.1007/s11119-024-10132-1
Accepted:
Published:
DOI: https://doi.org/10.1007/s11119-024-10132-1