Skip to main content
SpringerLink
Log in

U-DPnet: an ultralight convolutional neural network for the detection of apples in orchards

  • Research
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Efficient and accurate detection of apples is critical for the successful implementation of harvesting robots in orchards. However, due to limited memory resources on robotic platforms, it is imperative to develop lightweight detection algorithms that can operate in real-time. To address this challenge, we propose an ultralight convolutional neural network, U-DPnet, based on depth-separable convolution. Our approach incorporates the cross-stage deep separable module (CDM) and the multi-cascade deep separable module (MDM) in the backbone for nonlinear unit addition and attention mechanisms, which reduce the volume of the network while improving the feature representation capability. A simplified bi-directional feature pyramid network (BiFPN) is constructed in the neck for multi-scale feature fusion, and Adaptive feature propagation (AFP) is designed between the neck and the backbone for smooth feature transitions across different scales. To further reduce the network volume, we develop a uniform channel downsampling and network weight-sharing strategy. Multiple loss functions and label assignment strategies are used to optimize the training process. The performance of U-DPnet is verified on a homemade Apple dataset. Experimental results demonstrate that U-DPnet achieves detection accuracy and speed comparable to that of the 7 SOTA models. Moreover, U-DPnet exhibits an absolute advantage in model volume and computations (only 1.067M Params and 0.563G FLOPs, 39.79% and 36.36% less than yolov5-n).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability statement

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Wu, L., Ma, J., Zhao, Y., Liu, H.: Apple detection in complex scene using the improved yolov4 model. Agronomy 11(3), 476 (2021). https://doi.org/10.3390/agronomy11030476

    Article  Google Scholar 

  2. Yan, B., Fan, P., Lei, X., Liu, Z., Yang, F.: A real-time apple targets detection method for picking robot based on improved yolov5. Remote Sens. 13(9), 1619 (2021). https://doi.org/10.3390/rs13091619

    Article  Google Scholar 

  3. Xuan, G., Gao, C., Shao, Y., Zhang, M., Wang, Y., Zhong, J., Li, Q., Peng, H.: Apple detection in natural environment using deep learning algorithms. IEEE Access 8, 216772–216780 (2020). https://doi.org/10.1109/ACCESS.2020.3040423

    Article  Google Scholar 

  4. Zhang, T., Dai, F.: A review of machine vision based fruit recognition applications (2022)

  5. Gené-Mola, J., Gregorio, E., Guevara, J., Auat, F., Sanz-Cortiella, R., Escolà, A., Llorens, J., Morros, J.-R., Ruiz-Hidalgo, J., Vilaplana, V., et al.: Fruit detection in an apple orchard using a mobile terrestrial laser scanner. Biosyst. Eng. 187, 171–184 (2019). https://doi.org/10.1016/j.biosystemseng.2019.08.017

    Article  Google Scholar 

  6. Naranjo-Torres, J., Mora, M., Hernández-García, R., Barrientos, R.J., Fredes, C., Valenzuela, A.: A review of convolutional neural network applied to fruit image processing. Appl. Sci. 10(10), 3443 (2020). https://doi.org/10.3390/app10103443

    Article  Google Scholar 

  7. Wan, S., Goudos, S.: Faster r-cnn for multi-class fruit detection using a robotic vision system. Comput. Netw. 168, 107036 (2020). https://doi.org/10.1016/j.comnet.2019.107036

    Article  Google Scholar 

  8. Chen, J., Yuan, Z., Peng, J., Chen, L., Huang, H., Zhu, J., Liu, Y., Li, H.: Dasnet: dual attentive fully convolutional siamese networks for change detection in high-resolution satellite images. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 14, 1194–1206 (2020). https://doi.org/10.1109/JSTARS.2020.3037893

    Article  Google Scholar 

  9. Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

  10. Liang, Q., Zhu, W., Long, J., Wang, Y., Sun, W., Wu, W.: A real-time detection framework for on-tree mango based on ssd network. In: Intelligent Robotics and Applications: 11th International Conference, ICIRA 2018, Newcastle, NSW, Australia, August 9–11, 2018, Proceedings, Part II 11, pp. 423–436 (2018). https://doi.org/10.1007/978-3-319-97589-4_36. Springer

  11. Kang, H., Zhou, H., Chen, C.: Visual perception and modeling for autonomous apple harvesting. IEEE Access 8, 62151–62163 (2020). https://doi.org/10.1109/ACCESS.2020.2984556

    Article  Google Scholar 

  12. Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pvt v2: improved baselines with pyramid vision transformer. Comput. Vis. Media 8(3), 415–424 (2022). https://doi.org/10.1007/s41095-022-0274-8

    Article  Google Scholar 

  13. Liu, J., Jia, R., Li, W., Ma, F., Abdullah, H.M., Ma, H., Mohamed, M.A.: High precision detection algorithm based on improved retinanet for defect recognition of transmission lines. Energy Rep. 6, 2430–2440 (2020). https://doi.org/10.1016/j.egyr.2020.09.002

    Article  Google Scholar 

  14. Ji, W., Gao, X., Xu, B., Pan, Y., Zhang, Z., Zhao, D.: Apple target recognition method in complex environment based on improved yolov4. J. Food Process Eng. 44(11), 13866 (2021). https://doi.org/10.1111/jfpe.13866

    Article  Google Scholar 

  15. Zhou, J., Hu, W., Zou, A., Zhai, S., Liu, T., Yang, W., Jiang, P.: Lightweight detection algorithm of kiwifruit based on improved yolox-s. Agriculture 12(7), 993 (2022). https://doi.org/10.3390/agriculture12070993

    Article  Google Scholar 

  16. Zheng, Y.-Y., Kong, J.-L., Jin, X.-B., Wang, X.-Y., Su, T.-L., Zuo, M.: Cropdeep: the crop vision dataset for deep-learning-based classification and detection in precision agriculture. Sensors 19(5), 1058 (2019). https://doi.org/10.3390/s19051058

    Article  Google Scholar 

  17. Fragapane, G., De Koster, R., Sgarbossa, F., Strandhagen, J.O.: Planning and control of autonomous mobile robots for intralogistics: Literature review and research agenda. Eur. J. Oper. Res. 294(2), 405–426 (2021). https://doi.org/10.1016/j.ejor.2021.01.019

    Article  MathSciNet  MATH  Google Scholar 

  18. Kang, H., Chen, C.: Fruit detection, segmentation and 3d visualisation of environments in apple orchards. Comput. Electron. Agric. 171, 105302 (2020). https://doi.org/10.1016/j.compag.2020.105302

    Article  Google Scholar 

  19. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018). https://doi.org/10.48550/arXiv.1801.04381

  20. Wang, D., He, D.: Channel pruned yolo v5s-based deep learning approach for rapid and accurate apple fruitlet detection before fruit thinning. Biosyst. Eng. 210, 271–281 (2021)

    Article  Google Scholar 

  21. Tan, M., Pang, R., Le, Q.V.: Efficientdet: scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10781–10790 (2020). https://doi.org/10.1109/CVPR42600.2020.01079

  22. Wang, C.-Y., Liao, H.-Y.M., Wu, Y.-H., Chen, P.-Y., Hsieh, J.-W., Yeh, I.-H.: Cspnet: a new backbone that can enhance learning capability of cnn. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 390–391 (2020). https://doi.org/10.48550/arXiv.1911.11929

  23. Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8759–8768 (2018). https://doi.org/10.48550/arXiv.1803.01534

  24. Yu, G., Chang, Q., Lv, W., Xu, C., Cui, C., Ji, W., Dang, Q., Deng, K., Wang, G., Du, Y., Lai, B., Liu, Q., Hu, X., Yu, D., Ma, Y.: PP-PicoDet: a better real-time object detector on mobile devices (2021). https://doi.org/10.48550/arXiv.2111.00902

  25. Ge, Z., Liu, S., Wang, F., Li, Z., Sun, J.: Yolox: exceeding yolo series in 2021. arXiv:2107.08430 (2021)

  26. Zhang, H., Wang, Y., Dayoub, F., Sunderhauf, N.: Varifocalnet: an iou-aware dense object detector. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8514–8523 (2021)

  27. Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., Ren, D.: Distance-iou loss: faster and better learning for bounding box regression. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12993–13000 (2020)

  28. Li, X., Wang, W., Wu, L., Chen, S., Hu, X., Li, J., Tang, J., Yang, J.: Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection. Adv. Neural Inf. Process. Syst. 33, 21002–21012 (2020)

    Google Scholar 

  29. Authors, P.: PaddleDetection, object detection and instance segmentation toolkit based on PaddlePaddle. https://github.com/PaddlePaddle/PaddleDetection (2019)

  30. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part V 13, pp. 740–755 (2014). Springer

  31. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. (2017)

  32. Long, X., Deng, K., Wang, G., Zhang, Y., Dang, Q., Gao, Y., Shen, H., Ren, J., Han, S., Ding, E., Wen, S.: PP-YOLO: an effective and efficient implementation of object detector (2020). https://doi.org/10.48550/arXiv.2007.12099

  33. Ge, Z., Liu, S., Wang, F., Li, Z., Sun, J.: Yolox: exceeding yolo series in 2021. arXiv:2107.08430 (2021). https://doi.org/10.48550/arXiv.2107.08430

  34. Zili Liu, T.Z., Xu, G., Yang, Z., Liu, H., Cai, D.: Training-time-friendly network for real-time object detection. arXiv:1909.00700 (2019). https://doi.org/10.1609/aaai.v34i07.6838

  35. Wang, C.-Y., Bochkovskiy, A., Liao, H.-Y.M.: YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv:2207.02696 (2022). https://doi.org/10.48550/arXiv.2207.02696

Download references

Author information

Authors and Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, 710100, Shaanxi, China

    Hao Wan, Xilei Zeng, Zeming Fan, Shanshan Zhang & Ke Zhang

Authors
  1. Hao Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xilei Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zeming Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shanshan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ke Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zeming Fan.

Ethics declarations

Conflict of interest/Conflict of interest:

The authors declare that the publication of this paper has no conflicts of interest/Conflict of interest.

Ethics approval:

None.

Consent to participate:

Confirm.

Consent for publication:

Confirm.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wan, H., Zeng, X., Fan, Z. et al. U-DPnet: an ultralight convolutional neural network for the detection of apples in orchards. J Real-Time Image Proc 20, 76 (2023). https://doi.org/10.1007/s11554-023-01330-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-023-01330-7

Keywords

Search

Navigation