Skip to main content
SpringerLink
Log in

AI-based fruit identification and quality detection system

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The technological development in today’s era has unlocked the measures to propose new applications for the fruit industry. Automation boosts the economic growth and productivity of the country. Fruit quality detection in complex backgrounds using an automated system is significant for this sector. Fruit sorting has an impact on the export market and quality evaluation. One of the crucial qualities of grading fruits is their appearance, which affects their market value and the choice of the consumers. The manual sorting and inspection method takes a long time and is more tedious and exhaustive. Hence, an automated system is required to evaluate fruit, detect defects, and sort them based on their quality. Deep learning algorithms have highly influenced the area of object detection. Mask R-CNN and YOLOv5 are two object detection algorithms that have been experimented. YOLOv5 outperforms the Mask R-CNN approach when real-time object detection is required. The fruit identification and quality detection model is developed based on the YOLOv5 object detection system in the proposed work. The dataset includes 10,545 images of four different fruits, i.e., apple, banana, orange, and tomato, based on their quality. The model works in two phases. In phase 1, fruit is identified, and in phase 2, fruit quality detection is performed. The mosaic augmentation on the dataset has been applied for phase 1 training resulting in high detection performance and a robust system. The model classifies the fruit, and then the predicted image is passed to phase 2 for corresponding fruit quality detection. The mAP value of phase 1 is 92.80%. For phase 2, the mAP values for apple and banana quality detection models are 99.60% and 93.1%, respectively. The mAP values are 96.70% and 95% for orange and tomato quality detection models. The results show that the proposed method could realize fruit identification and quality detection on the validation dataset. The samples have been passed to show the real-time performance of the system. The efficiency of the trained model has been validated in different scenarios, including simple, complex, low-quality camera inputs. The fruit identification and quality detection model has been compared with several state-of-the-art detection methods, and the results are very encouraging.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Data availability

The datasets generated during the current study are available from the corresponding author on reasonable request.

References

  1. 40% vegetables, fruits get wasted in India: IARI Director. https://www.nationalheraldindia.com/national/40-vegetables-fruits-get-wasted-in-india-iari-director. Accessed 16 May 2022

  2. ACFR FRUIT DATASET. http://data.acfr.usyd.edu.au/ag/treecrops/2016-multifruit/. Accessed 27 Jan 2021

  3. Azizah LM, Umayah SF, Riyadi S, Damarjati C, Utama NA (2017) Deep learning implementation using convolutional neural network in mangosteen surface defect detection. In: 2017 7th IEEE international conference on control system, computing and engineering (ICCSCE), pp 242–246

  4. Bargoti S, Underwood J (2017) Deep fruit detection in orchards. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp 3626–3633

  5. Bhargava A, Bansal A (2021) Fruits and vegetables quality evaluation using computer vision: a review. J King Saud Univ-Comput Inf Sci 33(3):243–257

    Google Scholar 

  6. Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) no-reference image quality score - MATLAB brisque. https://www.mathworks.com/help/images/ref/brisque.html. Accessed 01 June 2022

  7. Chen W, Lu S, Liu B, Li G, Qian T (2020) Detecting citrus in orchard environment by using improved YOLOv4. Sci Programm 2020:1–13

    Google Scholar 

  8. Ciocca G, Napoletano P, Schettini R (2016) Food recognition: a new dataset, experiments, and results. IEEE J Biomed Health Inform 21(3):588–598

    Article  Google Scholar 

  9. Ciocca G, Napoletano P, Schettini R (2017) Learning CNN-based features for retrieval of food images. In: International conference on image analysis and processing, pp 426–434

  10. Ciocca G, Napoletano P, Schettini R (2018) IVLFood-WS: recognizing food in the wild using deep learning. In: 2018 IEEE 8th International Conference on Consumer Electronics-Berlin (ICCE-Berlin), pp 1–6

  11. Fu Y Fruit freshness grading using deep learning. Auckland University of Technology.

  12. Fu Z, Chen D, Li H (2017) Chinfood1000: a large benchmark dataset for chinese food recognition. In: International conference on intelligent computing, pp 273–281

  13. Fu L, Duan J, Zou X, Lin J, Zhao L, Li J, Yang Z (2020) Fast and accurate detection of Banana fruits in complex background orchards. IEEE Access 8:196835–196846

    Article  Google Scholar 

  14. Fu L et al (2022) YOLO-Banana: a lightweight neural network for rapid detection of banana bunches and stalks in the natural environment. Agronomy 12(2):391. https://doi.org/10.3390/AGRONOMY12020391

    Article  Google Scholar 

  15. Gan H, Lee WS, Alchanatis V, Ehsani R, Schueller JK (2018) Immature green citrus fruit detection using color and thermal images. Comput Electron Agric 152:117–125

    Article  Google Scholar 

  16. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448

  17. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  18. GitHub - facebookresearch/detectron2: Detectron2 is a platform for object detection, segmentation and other visual recognition tasks. https://github.com/facebookresearch/detectron2. Accessed 01 June 2022

  19. GitHub - tzutalin/labelImg: LabelImg is a graphical image annotation tool and label object bounding boxes in images. https://github.com/tzutalin/labelImg. Accessed 27 Jan 2022

  20. Goyal K, Kumar P, Verma K (2021) Food adulteration detection using artificial intelligence: a systematic review. Arch Comput Methods Eng:1–30

  21. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969

  22. He K, Gkioxari G, Dollar P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, vol 2017-October, pp 2980–2988. https://doi.org/10.1109/ICCV.2017.322

  23. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  24. Hussain D, Hussain I, Ismail M, Alabrah A, Ullah SS, Alaghbari HM (2022) A simple and efficient deep learning-based framework for automatic fruit recognition. Comput Intell Neurosci 2022. https://doi.org/10.1155/2022/6538117

  25. Ismail N, Malik OA (2022) Real-time visual inspection system for grading fruits using computer vision and deep learning techniques. Inf Process Agric 9(1):24–37. https://doi.org/10.1016/J.INPA.2021.01.005

    Article  Google Scholar 

  26. Jia W, Tian Y, Luo R, Zhang Z, Lian J, Zheng Y (2020) Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Comput Electron Agric 172:105380

    Article  Google Scholar 

  27. Kang H, Chen C (2020) Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Comput Electron Agric 171:105302

    Article  Google Scholar 

  28. Kuznetsova A, Maleva T, Soloviev V (2020) Using YOLOv3 algorithm with pre-and post-processing for apple detection in fruit-harvesting robot. Agronomy 10(7):1016

    Article  Google Scholar 

  29. Lawal MO (2021) Tomato detection based on modified YOLOv3 framework. Sci Rep 11(1):1–11. https://doi.org/10.1038/s41598-021-81216-5

    Article  Google Scholar 

  30. Lawal OM (2021) Development of tomato detection model for robotic platform using deep learning. Multimed Tools Appl 80(17):26751–26772. https://doi.org/10.1007/S11042-021-10933-W/FIGURES/14

    Article  Google Scholar 

  31. Li Y, Feng X, Liu Y, Han X (2021) Apple quality identification and classification by image processing based on convolutional neural networks. Sci Rep 11(1):1–15. https://doi.org/10.1038/s41598-021-96103-2

    Article  Google Scholar 

  32. Li G, Huang X, Ai J, Yi Z, Xie W (2021) Lemon-YOLO: an efficient object detection method for lemons in the natural environment. IET Image Process 15(9):1998–2009. https://doi.org/10.1049/IPR2.12171

    Article  Google Scholar 

  33. Liu G, Nouaze JC, Touko Mbouembe PL, Kim JH (2020) YOLO-tomato: a robust algorithm for tomato detection based on YOLOv3. Sensors 20(7):2145

    Article  Google Scholar 

  34. Martinel N, Foresti GL, Micheloni C (2018) Wide-slice residual networks for food recognition. In: Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, vol 2018-January, pp 567–576. https://doi.org/10.1109/WACV.2018.00068

  35. Mezgec S, Koroušić Seljak B (2017) NutriNet: a deep learning food and drink image recognition system for dietary assessment. Nutrients 9(7):657

    Article  Google Scholar 

  36. Minh Trieu N, Truong Thinh N, Editors A, Biancolillo A, Graña M (2021) Quality classification of dragon fruits based on external performance using a convolutional neural network. Appl Sci 11(22):10558. https://doi.org/10.3390/APP112210558

    Article  Google Scholar 

  37. Mirhaji H, Soleymani M, Asakereh A, Abdanan Mehdizadeh S (2021) Fruit detection and load estimation of an orange orchard using the YOLO models through simple approaches in different imaging and illumination conditions. Comput Electron Agric 191:106533. https://doi.org/10.1016/J.COMPAG.2021.106533

    Article  Google Scholar 

  38. Mithun BS et al (2018) Non-destructive method to detect artificially ripened banana using hyperspectral sensing and RGB imaging. In: Sensing for agriculture and food quality and safety X, vol 10665, p 106650T

  39. Momeny M, Jahanbakhshi A, Jafarnezhad K, Zhang YD (2020) Accurate classification of cherry fruit using deep CNN based on hybrid pooling approach. Postharvest Biol Technol 166:111204. https://doi.org/10.1016/J.POSTHARVBIO.2020.111204

    Article  Google Scholar 

  40. Redmon J, Farhadi A YOLOv3: an incremental improvement. Accessed 31 May 2022. [Online]. Available: https://pjreddie.com/yolo/

  41. Redmon J, Divvala S, Girshick R, Farhadi A (2015) You only look once: unified, real-time object detection. Accessed 04 June 2022. [Online]. Available: http://arxiv.org/abs/1506.02640

  42. Ren S, He K, Girshick R, Sun J (2016) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  43. Rodríguez FJ, García A, Pardo PJ, Chávez F, Luque-Baena RM (2018) Study and classification of plum varieties using image analysis and deep learning techniques. Prog Artif Intell 7(2):119–127

    Article  Google Scholar 

  44. Sa I, Ge Z, Dayoub F, Upcroft B, Perez T, McCool C (2016) Deepfruits: a fruit detection system using deep neural networks. Sensors 16(8):1222

    Article  Google Scholar 

  45. Tan W, Zhao C, Wu H (2016) Intelligent alerting for fruit-melon lesion image based on momentum deep learning. Multimed Tools Appl 75(24):16741–16761

    Article  Google Scholar 

  46. Tian Y, Yang G, Wang Z, Li E, Liang Z (2019) Detection of apple lesions in orchards based on deep learning methods of cyclegan and yolov3-dense. J Sensors 2019:1–3

    Google Scholar 

  47. Tian Y, Yang G, Wang Z, Wang H, Li E, Liang Z (2019) Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Comput Electron Agric 157:417–426

    Article  Google Scholar 

  48. Wan S, Goudos S (2020) Faster R-CNN for multi-class fruit detection using a robotic vision system. Comput Netw 168:107036

    Article  Google Scholar 

  49. Wang K, Liew JH, Zou Y, Zhou D, Feng J (2019) Panet: few-shot image semantic segmentation with prototype alignment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 9197–9206

  50. Wang HJ, Mou Q, Yue Y, Zhao H (2020) Research on detection technology of various fruit disease spots based on mask R-CNN. 2020 IEEE international conference on mechatronics and automation, ICMA 2020, pp 1083–1087. https://doi.org/10.1109/ICMA49215.2020.9233575

  51. Wang C-Y, Liao H-YM, Wu Y-H, Chen P-Y, Hsieh J-W, Yeh I-H (2020) CSPNet: A new backbone that can enhance learning capability of CNN. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 390–391

  52. Wang P, Niu T, He D (2021) Tomato young fruits detection method under near color background based on improved Faster R-CNN with attention mechanism. Agriculture 11(11):1059. https://doi.org/10.3390/AGRICULTURE11111059

    Article  Google Scholar 

  53. Wang L, Zhao Y, Liu S, Li Y, Chen S, Lan Y (2022) Precision detection of dense plums in orchards using the improved YOLOv4 model. Front Plant Sci 13:839269–839269. https://doi.org/10.3389/FPLS.2022.839269

    Article  Google Scholar 

  54. Wu A, Zhu J, Ren T (2020) Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network. Comput Electr Eng 81:106454. https://doi.org/10.1016/J.COMPELECENG.2019.106454

    Article  Google Scholar 

  55. Xu R, Lin H, Lu K, Cao L, Liu Y (2021) A forest fire detection system based on ensemble learning. Forests 12(2):217. https://doi.org/10.3390/F12020217

    Article  Google Scholar 

  56. Yan B, Fan P, Lei X, Liu Z, Yang F (2021) A real-time apple targets detection method for picking robot based on improved YOLOv5. Remote Sens 13(9):1619

    Article  Google Scholar 

  57. Yanai K, Kawano Y (2015) Food image recognition using deep convolutional network with pre-training and fine-tuning. In: 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp 1–6

  58. Yao J, Qi J, Zhang J, Shao H, Yang J, Li X (2021) A real-time detection algorithm for Kiwifruit defects based on YOLOv5. Electronics (Basel) 10(14):1711

    Google Scholar 

  59. Yu Y, Zhang K, Yang L, Zhang D (2019) Fruit detection for strawberry harvesting robot in non-structural environment based on mask-RCNN. Comput Electron Agric 163:104846. https://doi.org/10.1016/J.COMPAG.2019.06.001

    Article  Google Scholar 

  60. Zhang J, He L, Karkee M, Zhang Q, Zhang X, Gao Z (2018) Branch detection for apple trees trained in fruiting wall architecture using depth features and regions-convolutional neural network (R-CNN). Comput Electron Agric 155:386–393

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by Thapar-TAU Center for Excellence in Food Security (T2CEFS) under the research project “A Data-Driven Approach to Precision Agriculture in Small Farms Project.”

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Thapar Institute of Engineering and Technology, Patiala, Punjab, 147004, India

    Kashish Goyal, Parteek Kumar & Karun Verma

Authors
  1. Kashish Goyal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parteek Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karun Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kashish Goyal.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have influenced the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goyal, K., Kumar, P. & Verma, K. AI-based fruit identification and quality detection system. Multimed Tools Appl 82, 24573–24604 (2023). https://doi.org/10.1007/s11042-022-14188-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14188-x

Keywords

Search

Navigation