Skip to main content
SpringerLink
Log in

Color-, depth-, and shape-based 3D fruit detection

  • Published:
Precision Agriculture Aims and scope Submit manuscript

Abstract

A novel detection algorithm based on color, depth, and shape information is proposed for detecting spherical or cylindrical fruits on plants in natural environments and thus guiding harvesting robots to pick them automatically. A probabilistic image segmentation method is first presented to segment a red–green–blue image as a binary mask. Multiplied by this mask, a filtered depth image is obtained. Region growing, a region-based image segmentation method, is then applied to group the depth image into multiple clusters. Each cluster represents a fruit, leaf, or branch that is later transformed into a point cloud. Next, a 3D shape detection method based on M-estimator sample consensus, a model parameter estimator, is employed to detect potential fruits from each point cloud. Finally, an angle/color/shape-based global point cloud descriptor (GPCD) is developed to extract a feature vector for an entire point cloud, and a support vector machine classifier trained on the GPCD features is used to exclude false positives. Pepper, eggplant, and guava datasets were captured in the field. For the pepper, eggplant, and guava datasets, the detection precision was 0.864, 0.886, and 0.888, and the recall was 0.889, 0.762, and 0.812, respectively. Experiments revealed that the proposed algorithm was universal and robust and hence applicable to an agricultural harvesting robot.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Ahonen, T., Matas, J., He, C., & Pietikäinen, M. (2009). Rotation invariant image description with local binary pattern histogram fourier features. In Proceedings of the 16th Scandinavian Conference on Image Analysis (pp. 61–70).

  • Bac, C. W., Henten, E. J., Hemming, J., & Edan, Y. (2015). Harvesting robots for high-value crops: State-of-the-art review and challenges ahead. Journal of Field Robotics,31(6), 888–911.

    Article  Google Scholar 

  • Barnea, E., Mairon, R., & Ben-Shahar, O. (2016). Colour-agnostic shape-based 3D fruit detection for crop harvesting robots. Biosystems Engineering,146, 57–70.

    Article  Google Scholar 

  • Bulanon, D. M., Kataoka, T., Ota, Y., & Hiroma, T. (2003). A segmentation algorithm for the automatic recognition of fuji apples at harvest. Biosystems Engineering,83(4), 405–412.

    Article  Google Scholar 

  • Cupec, R., Filko, D., Vidović, I., Nyarko, E. K., & Željko Hocenski. (2014). Point cloud segmentation to approximately convex surfaces for fruit recognition. In Proceedings of the Croatian Computer Vision Workshop (pp. 56–61).

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition (pp. 886–893).

  • Duda, R., Hart, P., & Stork, D. (2001). Pattern classification. New York: Wiley.

    Google Scholar 

  • Font, D., Pallejà, T., Tresanchez, M., Runcan, D., Moreno, J., Martínez, D., et al. (2014). A proposal for automatic fruit harvesting by combining a low cost stereovision camera and a robotic arm. Sensors,14(7), 11557.

    Article  PubMed  Google Scholar 

  • Harrell, R. C., Slaughter, D. C., & Adsit, P. D. (1989). A fruit-tracking system for robotic harvesting. Machine Vision and Applications,2(2), 69–80.

    Article  Google Scholar 

  • Hoppe, H., Derose, T., Duchamp, T., Mcdonald, J., & Stuetzle, W. (1992). Surface reconstruction from unorganized points. ACM SIGGRAPH Computer Graphics,26(26), 71–78.

    Article  Google Scholar 

  • Kusumam, K., Krajník, T., Pearson, S., Duckett, T., & Cielniak, G. (2017). 3D-vision based detection, localization, and sizing of broccoli heads in the field. Journal of Field Robotics,34(8), 1505–1518.

    Article  Google Scholar 

  • Li, H., Lee, W. S., & Wang, K. (2016). Immature green citrus fruit detection and counting based on fast normalized cross correlation (fncc) using natural outdoor colour images. Precision Agriculture,17(6), 678–697.

    Article  Google Scholar 

  • Lu, J., & Sang, N. (2015). Detecting citrus fruits and occlusion recovery under natural illumination conditions. Computers and Electronics in Agriculture,110(C), 121–130.

    Article  Google Scholar 

  • Luo, L., Tang, Y., Zou, X., Wang, C., Zhang, P., & Feng, W. (2016). Robust grape cluster detection in a vineyard by combining the adaboost framework and multiple color components. Sensors,16(12), 2098.

    Article  Google Scholar 

  • Monta, M., & Namba, K. (2003). Three-dimensional sensing system for agricultural robots. In Proceedings of 2003 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (pp. 1216–1221).

  • Murillo-Bracamontes, E. A., Martinez-Rosas, M. E., Miranda-Velasco, M. M., Martinez-Reyes, H. L., Martinez-Sandoval, J. R., & Cervantes-De-Avila, H. (2012). Implementation of hough transform for fruit image segmentation. Procedia Engineering,35(12), 230–239.

    Article  Google Scholar 

  • Nguyen, T. T., Vandevoorde, K., Wouters, N., Kayacan, E., Baerdemaeker, J. G. D., & Saeys, W. (2016). Detection of red and bicoloured apples on tree with an RGB-D camera. Biosystems Engineering,146, 33–44.

    Article  Google Scholar 

  • Osada, R., Funkhouser, T., Chazelle, B., & Dobkin, D. (2001). Matching 3D models with shape distributions. In Proceedings International Conference on Shape Modeling and Applications, (pp. 154–166).

  • Qureshi, W. S., Payne, A., Walsh, K. B., Linker, R., Cohen, O., & Dailey, M. N. (2017). Machine vision for counting fruit on mango tree canopies. Precision Agriculture,18(2), 224–244.

    Article  Google Scholar 

  • Rachmawati, E., Khodra, M. L., & Supriana, I. (2016). Fruit image segmentation by combining color and depth data. International Conference on Information System & Applied Mathematics,1746(1), 651–666.

    Google Scholar 

  • Rakun, J., Stajnko, D., & Zazula, D. (2011). Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry. Computers & Electronics in Agriculture,76(1), 80–88.

    Article  Google Scholar 

  • Ren, C. Y., Prisacariu, V. A., Reid, I. D., & Murray, D. W. (2017). Real-time tracking of single and multiple objects from depth-colour imagery using 3d signed distance functions. International Journal of Computer Vision,124(1), 80–95.

    Article  Google Scholar 

  • Roscher, R., Herzog, K., Kunkel, A., & Kicherer, A. (2014). Automated image analysis framework for high-throughput determination of grapevine berry sizes using conditional random fields. Computers & Electronics in Agriculture,100(1), 148–158.

    Article  Google Scholar 

  • Rusu, R. B. (2009). Semantic 3D object maps for everyday manipulation in human living environment. PhD thesis. Germany: Computer Science Department, Technische Universit€at Mu¨ nchen.

  • Rusu, R. B., Blodow, N., & Beetz, M. (2009). Fast point feature histograms (FPFH) for 3D registration. In Proceedings of the IEEE International Conference on Robotics and Automation (pp. 3212–3217).

  • Schnabel, R., Wahl, R., & Klein, R. (2010). Efficient RANSAC for point-cloud shape detection. Computer Graphics Forum,26(2), 214–226.

    Article  Google Scholar 

  • Song, Y., Glasbey, C. A., Horgan, G. W., Polder, G., Dieleman, J. A., & van der Heijden, G. W. A. M. (2014). Automatic fruit recognition and counting from multiple images. Biosystems Engineering,118(1), 203–215.

    Article  Google Scholar 

  • Stein, M., Bargoti, S., & Underwood, J. (2016). Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors,16(11), 1915.

    Article  Google Scholar 

  • Tao, Y., & Zhou, J. (2017). Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking. Computers & Electronics in Agriculture,142, 388–396.

    Article  Google Scholar 

  • Torr, P. H. S., & Murray, D. W. (1997). The development and comparison of robust methods for estimating the fundamental matrix. International Journal of Computer Vision,24(3), 271–300.

    Article  Google Scholar 

  • Tremeau, A., & Borel, N. (1997). A region growing and merging algorithm to color segmentation. Pattern Recognition,30(7), 1191–1203.

    Article  Google Scholar 

  • Wachs, J. P., Stern, H. I., Burks, T., & Alchanatis, V. (2010). Low and high-level visual feature-based apple detection from multi-modal images. Precision Agriculture,11(6), 717–735.

    Article  Google Scholar 

  • Wahabzada, M., Paulus, S., Kersting, K., & Mahlein, A. K. (2015). Automated interpretation of 3D laserscanned point clouds for plant organ segmentation. BMC Bioinformatics,16(1), 1–11.

    Article  Google Scholar 

  • Wang, Z., Walsh, K. B., & Verma, B. (2017). On-tree mango fruit size estimation using RGB-D images. Sensors,17(12), 20170154.

    Google Scholar 

  • Xiang, R., Jiang, H., & Ying, Y. (2014). Recognition of clustered tomatoes based on binocular stereo vision. Computers & Electronics in Agriculture,106, 75–90.

    Article  Google Scholar 

  • Zou, X., Ye, M., Luo, C., Xiong, J., Luo, L., Wang, H., & Chen, Y. (2016). Fault-tolerant design of a limited universal fruit-picking end-effector based on vision-positioning error. Applied Engineering in Agriculture, 32(1), 5–18.

    Article  Google Scholar 

  • Zou, X., Zou, H., & Lu, J. (2012). Virtual manipulator-based binocular stereo vision positioning system and errors modelling. Machine Vision and Applications,23(1), 43–63.

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded by a grant from the National Natural Science Foundation of China (No. 31571568) and a grant from the National Key Research and Development Program of China (No. 2017YFD0700103).

Author information

Authors and Affiliations

  1. Key Laboratory of Key Technology on Agricultural Machine and Equipment, Ministry of Education, South China Agricultural University, 483 Wushan Road, Guangzhou, 510642, China

    Guichao Lin, Xiangjun Zou, Juntao Xiong & Yamei Fang

  2. College of Mechanical and Automotive Engineering, Chuzhou University, 1 Huifeng West Road, Chuzhou, 239000, China

    Guichao Lin

  3. College of Urban and Rural Construction, Zhongkai University of Agriculture and Engineering, Guangxin Road, Guangzhou, 510225, China

    Yunchao Tang

Authors
  1. Guichao Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunchao Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiangjun Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juntao Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yamei Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yunchao Tang or Xiangjun Zou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, G., Tang, Y., Zou, X. et al. Color-, depth-, and shape-based 3D fruit detection. Precision Agric 21, 1–17 (2020). https://doi.org/10.1007/s11119-019-09654-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11119-019-09654-w

Keywords

Search

Navigation