Skip to main content
SpringerLink
Log in

A new deep-learning strawberry instance segmentation methodology based on a fully convolutional neural network

  • Review Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Instance segmentation is one of the image processing problems where deep learning techniques are beginning to show potential. In agriculture, one of its main application is automatic fruit harvesting. This study focuses on its application on strawberry crops, where the development of automatic harvesting machines is of particular interest. At present, the reference methodology to deal with instance segmentation is Mask R-CNN. However, Mask R-CNN requires a large processing power which limits its implementation in real-time systems. This work proposes a new methodology to carry out instance segmentation of strawberries based on the use of a fully convolutional neural network. Instance segmentation is achieved by adding two new channels to the network output so that each strawberry pixel predicts the centroid of its strawberry. The final segmentation of each strawberry is obtained by applying a grouping and filtering algorithm. The methodology was tested using the publicly available StrawDI_Db1 database. The evaluation results show values of mean average precision (mAP) and mean instance intersection over union (I\(^{2}\)oU) of 52.61 and 93.38, respectively, with a processing speed of 30 fps. These figures mean an increase in precision higher than 15% and a fps rate six times higher than those obtained in the reference methodologies based on Mask R-CNN. Therefore, the methodology presented in this paper can be considered as the latest reference methodology for strawberry segmentation, meeting the precision and speed requirements needed for it to be used in the automatic strawberry harvesting systems that work in real time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Agrobot. http://agrobot.com (2020)

  2. Champ J, Mora-Fallas A, Goëau H, Mata-Montero E, Bonnet P, Joly A (2020) Instance segmentation for the fine detection of crop and weed plants by precision agricultural robots. Appl Plant Sci p. e11373

  3. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vision 88(2):303–338

    Article  Google Scholar 

  4. Ganesh P, Volle K, Burks T, Mehta S (2019) Deep orange: Mask R-CNN based orange detection and segmentation. IFAC-PapersOnLine 52(30):70–75

    Article  MathSciNet  Google Scholar 

  5. Ge Y, Xiong Y, Tenorio GL, From PJ (2019) Fruit localization and environment perception for strawberry harvesting robots. IEEE Access 7:147,642-147,652

    Article  Google Scholar 

  6. Gené-Mola J, Sanz-Cortiella R, Rosell-Polo JR, Morros JR, Ruiz-Hidalgo J, Vilaplana V, Gregorio E (2020) Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Comput Electron Agric 169:105,165

    Article  Google Scholar 

  7. Gonzalez S, Arellano C, Tapia JE (2019) Deepblueberry: quantification of blueberries in the wild using instance segmentation. IEEE Access 7:105,776-105,788

    Article  Google Scholar 

  8. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969

  9. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE international conference on computer vision, pp. 1026–1034

  10. Jia W, Tian Y, Luo R, Zhang Z, Lian J, Zheng Y (2020) Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Comput Electron Agric 172:105,380

    Article  Google Scholar 

  11. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105

  12. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  13. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: European conference on computer vision, pp. 740–755. Springer

  14. Liu X, Zhao D, Jia W, Ji W, Ruan C, Sun Y (2019) Cucumber fruits detection in greenhouses based on instance segmentation. IEEE Access 7:139,635-139,642

    Article  Google Scholar 

  15. Naranjo-Torres J, Mora M, Hernández-García R, Barrientos RJ, Fredes C, Valenzuela A (2020) A review of convolutional neural network applied to fruit image processing. Appl Sci 10(10):3443

    Article  Google Scholar 

  16. Ni X, Li C, Jiang H, Takeda F (2020) Deep learning image segmentation and extraction of blueberry fruit traits associated with harvestability and yield. Hortic Res 7(1):1–14

    Article  Google Scholar 

  17. Octinion. http://octinion.com (2020)

  18. Pérez-Borrero I, Marín-Santos D, Gegúndez-Arias ME, Cortés-Ancos E (2020) A fast and accurate deep learning method for strawberry instance segmentation. Comput Electron Agric 178:105,736

    Article  Google Scholar 

  19. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp. 91–99

  20. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer

  21. Santos TT, de Souza LL, dos Santos AA, Avila S (2020) Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association. Comput Electron Agric 170:105,247

    Article  Google Scholar 

  22. Tian Y, Yang G, Wang Z, Li E, Liang Z (2020) Instance segmentation of apple flowers using the improved mask R-CNN model. Biosyst Eng 193:264–278

    Article  Google Scholar 

  23. Uhrig J, Rehder E, Fröhlich B, Franke U, Brox T (2018) Box2pix: single-shot instance segmentation by assigning pixels to object boxes. In: IEEE Intelligent Vehicles Symposium (IV), pp. 292–299. IEEE

  24. Velesaca HO, Mira R, Suarez PL, Larrea CX, Sappa AD (2020) Deep learning based corn kernel classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 66–67

  25. Watanabe T, Wolf D (2018) Distance to center of mass encoding for instance segmentation. In: 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 3825–3831. IEEE

  26. Yu Y, Zhang K, Yang L, Zhang D (2019) Fruit detection for strawberry harvesting robot in non-structural environment based on mask R-CNN. Comput Electron Agric 163:104,846

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Science and Technology Research Centre, University of Huelva, Huelva, Spain

    Isaac Perez-Borrero, Diego Marin-Santos, Manuel J. Vasallo-Vazquez & Manuel E. Gegundez-Arias

Authors
  1. Isaac Perez-Borrero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diego Marin-Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel J. Vasallo-Vazquez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Manuel E. Gegundez-Arias
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Isaac Perez-Borrero.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Perez-Borrero, I., Marin-Santos, D., Vasallo-Vazquez, M.J. et al. A new deep-learning strawberry instance segmentation methodology based on a fully convolutional neural network. Neural Comput & Applic 33, 15059–15071 (2021). https://doi.org/10.1007/s00521-021-06131-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06131-2

Keywords

Search

Navigation