Automatic fruit picking technology: a comprehensive review of research advances

Abstract

In recent years, the fruit industry has become an important part of agricultural development, and fruit harvesting is a key stage in the production process. However, picking fruits during the harvest season is always a major challenge. In order to solve the challenges of time-consuming, costly, and inefficient fruit picking, researchers have conducted a lot of studies on automatic fruit picking equipment. Existing picking technologies still require further research and development to improve efficiency and reduce fruit damage. Aiming at the efficient and non-destructive picking of fruits, this paper reviews machine vision and mechanical fruit picking technology and the current research status, including the current application status, equipment structure, working principle, picking process, and experimental results. As a promising tool, machine vision technology has been widely researched and applied due to its low hardware cost and rich visual information. With the development of science and technology, automated fruit picking technology integrates information technology, integrates automatic perception, transmission, control, and operation, etc., saves manpower costs, and continuously promotes the development of modern agriculture in the direction of refinement of equipment technology, automation, and intelligence. Finally, the challenges faced by automated fruit picking are discussed, and future development is looked forward to with a view to contributing to its sustainable development.

1 Introduction

With the dramatic increase in the world's population, the fruit industry is under pressure to increase acreage and production (Eigenbrod and Gruda 2015; Horrigan et al. 2002; Gongal et al. 2015). Due to increased social diversity and an aging population, the number of people involved in the processing of agricultural products is decreasing, and the labor required for fruit picking accounts for 60–70% of the entire growing process, making inefficient picking and high labor costs a major challenge for fruit farmers (Fess et al. 2011; Sibhatu et al. 2015; He and Schupp 2018; Vougioukas 2019). In recent years, there has been a significant increase in interest in automated fruit picking technology, which reduces the cost of picking while increasing the efficiency of picking and has great potential for future development (Hua et al. 2023).

The twenty-first century is a critical period for the transition from agricultural mechanization to intelligent automated machinery, and industrial intelligent automation is of indelible importance to the development of modern agriculture in terms of scale, diversification, and precision (An et al. 2022; Yang and Zhang 2014; Lad et al. 2022). According to the United Nations, the world population will reach 9.7 billion in 2050. This means that the world’s annual agricultural production will have to increase by about 60% to meet people’s needs (Walker 2016; Mahmud et al. 2023). China, for example, has been the world’s largest producer and consumer of fruit since 1993 (Chen et al. 2017). The most recent data in Fig. 1A shows that the area of orchards and production have increased every year. Fruit harvesting has become one of the most time-consuming and labor-intensive aspects of fruit production operations (Vrochidou et al. 2022; Jatoi et al. 2017). In recent years, researchers around the world have been working on mechanized and intelligent fruit picking techniques (Ayaz et al. 2019).

Fig. 1

(Source National Bureau of Statistics of China). B Statistics from the Web of Science (https://www.webofscience.com/) The topics are “Fruit picking techniques” and “Fruit picking”

A Fruit production and acreage in China, 2012 to 2021

As early as the 1980s, American scholars Schertz and Brown proposed the use of robots for fruit picking, and more research results have emerged successively (Jia et al. 2020). However, previous studies have focused on the current various intelligent fruit and vegetable picking robots and their performance characteristics (Wang et al. 2022), vision control technology (Zhao et al. 2016), identification and localization (Li et al. 2022a, b), image processing technology (Hua et al. 2023), and the study of key structures (end-effector and robotic arm, etc.) (Li et al. 2022a, b; Cai et al. 2010; Lu et al. 2022; Oliveira et al. 2022). It can be concluded from this literature that these techniques have been successfully used for picking the corresponding fruits or vegetables, but all of them suffer from low accuracy and efficiency in judging the ripeness of the fruits, as well as a high level of damage. Although extensive and in-depth research has been carried out in the field of fruit picking at home and abroad and a large number of agricultural fruit picking equipment has been developed, a detailed and systematic summary of the latest research results in this field has not yet been made, and the existing review articles only collect and compare the picking equipment from one or a few specific perspectives, which is not conducive to the comprehension of the latest technology of agricultural picking equipment.

In recent years, extensive development and detailed studies based on various fruit picking techniques have been conducted Fig. 1B. Statistics on the keywords “fruit picking technology” and “fruit picking” from the Web of Science (https://www.webofscience.com/) show a rapid increase in the number of publications and citation frequency of the literature in the last 10 years. Among them, the citation frequency has increased significantly from 2019 to 2021. In this paper, we will review the picking equipment from two aspects of mechanization and machine vision-based fruit picking technology, respectively, and analyze the structure, process, picking efficiency, fruit damage level, and picking environment of the equipment. Summarize the future development trend of automated picking equipment in order to promote the modernization of the fruit industry and the promotion of high technology as a reference.

This review is organized as follows: Sect. 2 introduces mechanized fruit picking technologies, describing the characteristics and picking efficiency of seven picking technologies. Section 3 focuses on the principles and application status of machine vision-based picking technology and presents the current research status of automatic fruit picking equipment for apples, strawberries, tomatoes, citrus, and strawberries, respectively. In addition, agricultural drone fruit picking technology is introduced. Section 4 focuses on the challenges and future trends of automated fruit picking technology. Section 5 concludes the paper. The framework of this work is shown in Fig. 2.

Fig. 2

Overview of automated fruit picking technologies frame

2 Mechanical automatic fruit picking technology and equipment

In 1968, research on mechanical fruit and vegetable harvesting equipment commenced, and the United States took the lead in studying mechanical vibratory shaking and pneumatic vibratory shaking types of picking machinery (Navas et al. 2021; Bao et al. 2022). However, while these machines can complete the basic task of picking, they cause significant damage to the fruit and suffer from low picking efficiency, severely limiting their usefulness (Atanda et al. 2011). With the continuous development of science and technology worldwide, mechanical fruit harvesting equipment has undergone rapid development (Yuan and Chen 2014). Current research on mechanical fruit harvesting equipment can be broadly categorized into two types: human–machine cooperative harvesting and machine vision-based fruit picking equipment (Silwal 2016). In the former, the picking machine is operated by a worker to complete the harvesting task, while in the latter, fruit identification, positioning, and picking actions are all performed by the equipment itself (Bechar and Vigneault 2017; Yin 2020).

This section provides a comprehensive overview of the technologies used for mechanical harvesting of fruits, including vibratory, pneumatic, impact, comb and brush, push and shear, and shear technologies (Chen 2021). An extensive search was conducted to gather relevant information, and a comprehensive assessment of the current state of research and issues in the field was provided to assist scholars in their research projects.

2.1 Vibratory shaking type

Vibratory harvesting machinery utilizes a vibration device to transmit vibrations. By setting the optimal vibration frequency and amplitude, the vibration device transmits vibrations to the fruit tree, causing it to vibrate and the fruit to move with variable acceleration, leading to the fruit falling off and completing the harvest (Wu et al. 2022; Gupta et al. 2016). However, high vibration frequencies can cause damage to the fruit (Kou et al. 2022). The vibratory device of the vibratory shaking picker mainly generates vibrations and comes in two forms: the eccentric wheel type and the crank-slider mechanism type Fig. 3. Experiments on citrus harvesting using these two forms demonstrated that the harvesting rate using the crank-slider type ranged from 50 to 55%, while the eccentric type yielded 70 to 75% (Zhu et al. 2021a, b, c, d). The shaking method can be classified based on the position of the vibration: branch-trunk vibration and crown vibration. In branch trunk vibration, the clamping mechanism grips the trunk or branch, causing the eccentric wheel or crank-slider device to vibrate the tree to separate the fruit (Whitney et al. 1988).

Fig. 3

Two forms of vibration device. A Crank-slider type. B Eccentric type

Vibratory fruit harvesting equipment is the most widely used form of mechanical harvesting for forest fruits due to its high efficiency and low cost (Afsah-Hejri et al. 2022). However, the vibrations produced by the equipment can cause damage to the fruit (Hinsch et al. 1993). As a result, both domestic and foreign research on vibratory shaking harvesting equipment has focused mainly on blueberries, wine grapes, apples, and other fruits.

Simple berry harvesting equipment was first developed by the French in the 1940s, and by the 1960s, picking highbush blueberries with vibration was gaining importance (Calnitsky 2017). Initially, the main reliance was on manual picking and simple tools, which led to the loss of fruit due to slippage and consequent economic losses (Elik et al. 2019). Typical research shows that the United States OXBO Manufacturing Company has 25 years of experience with blueberry and raspberry harvesters, and the use of the rotary vibratory picking method can minimize plant damage (Huffman 2014; Arak 2021). BEI Company in the United States is in an international leading position in the field of blueberry picker research; its LBT Harvester, BEI Tracks Blueberry Harvester, Rotary Harvester, and other series of vibration blueberry pickers in the United States are widely used (Zhao et al. 2018; Yarborough and Hergeri 2010; Yu et al. 2014; DeVetter et al. 2019). Bao et al. (2014) conducted extensive research on automatic blueberry picking technology and developed a vibration strategy based on the blueberry harvesting equipment, which is 10 times more efficient than manual picking. A hand-pushed dwarf shrub blueberry picker and a vibratory harvester with an efficiency of 12 kg/h were developed for dwarf shrub blueberries, both of which are hand-held, easy to operate, have low manufacturing and maintenance costs, and are significantly more efficient than manual picking (Guo et al. 2012). Li et al. (2020a, b, c) designed a blueberry picker using a vibrator consisting of a crank linkage and a double rocker mechanism. It was 10.6 times more efficient than manual picking, with high efficiency, low fruit breakage, and labor savings. Another type of longitudinal vibration picker for berry bush fruits, adding a spring system as a buffer, established and analyzed the vibration model of the picking location and the lateral branches of the berry bushes and obtained the optimal vibration relationship during the picking process, with a harvest rate of 95.5% and an average error of 0.0095 (Hou et al. 2023; Chen et al. 2021a, b).

In recent years, the growing demand for high-quality wine grapes has posed a significant challenge to the development of the industry (Keller 2010). Wine grape harvesting has been mechanized in places such as Europe and the US, for example, in France, 70% of wine grapes are harvested by machine (Sarig 2012; Jones et al. 2014). The New Holland Braud series of self-propelled grape harvesters, which was launched in 1975 under the umbrella of the Case New Holland Industrial Group, has been an industry leader for more than 40 years, and experiments have shown that a single Braud series harvester can replace up to 60 harvesters at a harvest cost of around a quarter of the cost of manual harvesting (Douthie 2019; Pezzi and Martelli 2015; Fornari et al. 2021). Therefore, the development of mechanized harvesting is considered an inevitable trend in the industry (Fu et al. 2022). Yuan et al. (2020) developed a crankshaft vibratory threshing and harvesting unit with a flexible clamping vibratory mechanism, drive train, and frame, which had a harvesting success rate of 93.06% and a breakage rate of 4.57%, which is significantly more efficient than manual harvesting. Another method is to use a 4R mechanism combined with a planar hinge mechanism to achieve variable speed and directional movement during wine grape harvesting, with an average harvest rate of 85.55% (Zhu et al. 2023). It is worth noting that damage during harvest is still a major disadvantage of vibratory grape harvesting, and future research should focus on how to reduce fruit damage.

For apple picking, most of them are still manually based with low picking efficiency (Zhu et al. 2021a, b, c, d; Zhang et al. 2021a, b). Due to factors such as the large size of apples and the thin skin and thick flesh, automated robotic picking has been realized in Europe and the United States (Ghahremani 2020). However, there is a lack of research on mechanized apple picking. Figure 4A shows a hydraulically controlled vibratory apple picker. During the picking operation, the vibratory picking device clamps the trunk of a high-acid apple tree and vibrates, causing the fruit to be dislodged under inertia for picking, with a net picking rate of 95.9% and a damage rate of 1.3%, which suggests that the machine meets the requirements for picking high-acid apples (Shang et al. 2023).

Fig. 4

(Source Shang et al. 2023). B Overall view of the air-suction harvesting equipment (Source Jutras et al. 1963). C Comb picking machinery (Source Guirado et al. 2016). D Comb picking machinery (Source Savary et al. 2011). E Structure of the push-pruner picke (Source Yu et al. 2022). F Working principle diagram of a shear fruit picker (Source Li 2021)

Current status of mechanical fruit picking equipment. A Structure of the vibratory apple picker

Mechanized harvesting techniques using vibroseis in the fruit industry have been extensively researched in a large body of literature and have developed considerably over the last few decades. However, because of the tender and fragile nature of fruits, damage to both fruit and fruit trees often occurs. Therefore, more efforts in the future should be focused on efficient and non-destructive fruit harvesting, and the selection of the optimal vibration frequency can effectively reduce the damage to the fruit. Actuator selection is still an important aspect we are very concerned about. According to the characteristics of the fruit, the choice of flexible materials for the damage situation has also improved. With the continuous development of automation and mechanization technology, improve the applicability of orchard machinery and the reliability of mechanical equipment to achieve low fruit damage or even lossless harvesting, so as to meet the needs of the fruit market.

2.2 Pneumatic type

Pneumatic fruit harvesting machines have been a popular method for fruit harvesting since their inception (Elfferich et al. 2022). These machines utilize powerful air flow to induce movement in the fruit, which allows them to be dislodged from the tree once the force applied is greater than the resistance of the fruit stalk to the branch (Brown and Sukkarieh 2021). The air flow is generated by powerful fans attached to large tractors, and the direction of the air flow is controlled by a guiding device.

Pneumatic fruit picking machinery was first researched in the USA, particularly for citrus harvesting. Pneumatic mechanical harvesting methods were proposed in Fig. 4B. The early experiments were not very efficient due to outdated technology, low-powered equipment, and low efficiency (Jutras et al. 1963). By the 1970s, air-powered harvesting experiments were conducted with the strongest airflow speed of 43.8 m/s, where continuous impact on citrus trees for 10 s achieved a net harvesting rate of 85–90% for ripe citrus. However, continuous airflow speed during harvesting could cause mild damage to the citrus trees and, in severe cases, affect citrus yields the following year (Whitney and Patterson 1972). To improve the operational performance of the air-suction floor date picker, a prototype was designed, built, and tested to optimize its key parameters (Zhang et al. 2021a, b).

Compared with vibratory harvesting, pneumatic harvesting has the advantages of high picking efficiency, less impurity, and convenient operation. It is worth noting that pneumatic harvesting machinery is not in direct contact with the fruit plant and will not cause physical damage to the fruit skin and trunk, but because of its high-speed airflow on the fruit skin, branches, and leaves and other unavoidable damage, the application of large-scale harvesting is less. The future development of this technology should focus on reducing energy consumption, reducing damage, analyzing the correlation between the spatial structure of fruit trees and harvesting efficiency, and adapting the growth characteristics of fruit trees to mechanized operations by improving their growth characteristics.

2.3 Percussive type

Impact fruit harvesting technology uses mechanical springs or electromagnetic excitation to generate high-intensity input excitation that acts directly on the fruit tree. Make the tree branches produce instantaneous acceleration, produce vibration, and make the fruit fall off. Harvesting methods directly acting on the impact of the fruit tree effectively avoid damage to the fruit in the harvesting process, but damage to the fruit tree and branches affects the yield of the second year (Burks et al. 2005).

Studies have shown that 80–90% of sweet cherries can be harvested by both shock and vibratory shaking, so the performance of the two methods is close to each other (Erdoǧan et al. 2003; Liu et al. 2018). However, increasing the frequency of vibration during harvest may cause damage to the fruit. Therefore, it is important to control the shock or vibration frequency to avoid fruit damage (Norton et al. 1962). In order to minimize damage to the fruit tree caused by shock and to ensure that the fruit is smoothly removed from the canopy, both tree shapes were tested. The results showed that both shapes had a fruit picking rate of 90–95% with minimal damage to fruit and branches (Pellerin et al. 1978). In order to address the challenges associated with mechanical harvesting of sweet cherries and to reduce the damage caused to the fruit surface during dislodgement, an impact cherry picker was designed, characterized by the installation of an impact device on both sides of the branch. While the machine was able to achieve high efficiency (1480 kg/h) and high net picking rates, it was found that the use of two impact devices increased damage to the branches and affected fruiting rates the following year (Peterson et al. 2003). A spring-loaded impact trunk shaker was designed for apple trees, and its performance was evaluated. Test results showed that the shaker was capable of delivering up to 1151 J of energy at a speed of 5.16 m/s (Pacheco and Rehkugler 1980). Impact harvesting equipment seems to be used efficiently, but it can also lead to damage to the fruit surface and branches, which can seriously affect fruit production in the coming year and cause losses to the fruit grower. This method is suitable for fruit trees with thick stems and hard fruits with hard skin, such as walnuts, olives, and cherries (Yuan et al. 2022).

Thus, it seems that the impact fruit harvesting technology causes irreversible damage to fruit trees during harvesting, and future research should be directed towards continuously analyzing the causes of damage to fruits and plants during the harvesting process of impact machinery, optimizing the mechanism of the harvesting machinery, improving the harvesting efficiency, and reducing the impact damage. Strengthen the exploration of new technologies to replace the use of impact fruit harvesting methods to ensure fruit quality.

2.4 Brush type

The comb fruit harvester uses a combing device to act on the canopy of the fruit tree, increasing the rotation and oscillation of the combing device to dislodge the fruit (Castro-García et al. 2012). This equipment can be used for fruits such as goji berries and blueberries. In terms of blueberry harvesting, the current state-of-the-art technology is best typified by vibratory mechanical harvesting technology. According to the growth characteristics of blueberry fruits, a combing device is used to act directly on the canopy of the blueberry plant, and the fruits are dislodged through the combing device. Guo et al. (2012) proposed a hand-pushed dwarf bush blueberry harvester for dwarf bush blueberry harvesting, with a single harvesting capacity of 12 kg/h, a fruit damage rate of 10%, and a harvesting net rate of 86%. Peterson et al. (1997) developed an experimental mechanical blueberry harvester (V45) with an inclined double spike drum vibrator to ensure the reliability of harvesting ripe blueberries. Tests showed that the efficiency of the blueberry harvester was significantly higher than that of a rotary harvester and that the quality of the fruit was comparable to that of the hand-harvested blueberries, with a guaranteed quality. Figure 4C shows a comb vibratory harvester developed for field harvesting of olive trees, where parameters such as the position of picked fruit on the capture frame, the position of unseparated fruit on the tree, the degree of tree damage, and ground speed were considered during the analysis (Guirado et al. 2016). In order to analyze the force distribution within the citrus canopy, two different field experiments were carried out using a comb vibratory picker with a complete machine structure, as shown in Fig. 4D. The first experiment was designed to investigate the effect of fruit position on the forces applied, while the second experiment investigated the distribution of forces and accelerations along the length of the branch. The results showed that the forces applied to the fruit within the canopy were higher than those applied to the fruit at the edges (Savary et al. 2011).

In order to harvest apples, Hu (2020) proposed a flexible comb harvesting method using a flexible comb harvesting platform. The principle of fruit detachment during the harvesting process was investigated using a fruit-branch system dynamics model, and the motions and forces involved were analyzed. It was found that the rigid-flexible coupling model was the most effective for fruit detachment. Comb-type apple harvesting equipment has been shown to be effective in reducing fruit damage, thereby significantly increasing harvest and damage rates (Zhao 2022). The spiral brush apple harvester was developed for machine harvesting of apple trees, and this machine can efficiently pick, transport, and collect apples in batches. The results showed that it is suitable for apple harvesting as it achieves the highest efficiency. It can be seen that comb-brush harvesting of fruit is widely used in the small berry sector, and harvesting efficiency is significantly higher.

Compared with vibration, impact, and other harvesting methods, brush-type harvesting machinery greatly reduces the damage to the fruit and plant, and the harvesting efficiency of the fruit has improved. For the development of this technology, it should be focused on the continuous optimization of the brush structure, selecting suitable brush materials for the fruit so that the brush-type harvester achieves the best harvesting effect, studying the characteristics of the bonding force between the branch and the fruit stalk, analyzing the factors affecting fruit shedding, and conducting field harvesting experiments to determine the best harvesting machinery parameters.

2.5 Push-shear type

The push-shear picking technique is less automated and is suitable for large orchards with flat terrain. However, a push-shear cherry picker based on the pruning principle can quickly pick cherries with a wide range of stem profiles. Field trials have shown that the picker can pick more than 9.7 kg of cherries in half an hour, with a stalk picking success rate of more than 98% and a damage rate of less than 1%. The researchers invented a push-shear cherry picking device that can be recommended for cherry trees, improving operational efficiency and saving manpower. The device drops cherries onto a picking net for easy sorting and storage (Fig. 4E) (Yu and Ampazidis 2022; Xu et al. 2021).

Push-and-shear harvesting technology is also currently facing gradual elimination because it requires manual assistance to complete harvesting, has a low degree of automation, is inefficient, and is not suitable for large-scale production. Currently, with the development of computer technology, many people will turn their attention to agriculturally intelligent robots to reduce the dependence on manual labor and improve overall productivity, which is also the future trend of automated fruit harvesting.

2.6 Shear type

In order to solve the difficulties and safety hazards of manual fruit picking for a wide range of fruits with high growth, such as apples, pears, peaches, etc., Gong (2020) invented a high-level fruit shear picker that can pick fruits quickly and non-destructively, improves the efficiency of picking, and reduces the cost. Another invention is a shear-type woody fruit picker (Li 2021), which harvests high branches by adjusting a telescopic rod and using a clamping and shearing device to shear off the fruit, with the picking principle shown in Fig. 4F. Although these inventions are effective in practice, their overall operation and structure are complicated, and each mechanism works independently. The operation is cumbersome, which limits their use. However, it solves the dilemma of the high cost, low efficiency, fruit damage, and worker injury of traditional manual ladder climbing.

Although shear-type harvesting machinery can complete the basic harvesting tasks, there are still drawbacks, such as the low degree of automation and cumbersome structure. Future research should focus on simplifying the structure and reducing the cost, and we believe that the main thing should be to strengthen the degree of automation, improve flexibility, and choose intelligent actuators to ensure that the fruit will be harvested smoothly, avoiding the impact of the harvesting process on the surrounding flowers and foliage.

3 Fruit picking technology based on machine vision

In recent years, although mechanized fruit harvesting technology has matured, it is more damaging to branches and fruits, less efficient, and still requires manual involvement in the process to achieve fruit harvesting. Therefore, there is an urgent need to develop an efficient and damage-free fruit picking technology. This section describes the principles of machine vision technology, the current status of research in the agricultural field and the research progress of fruit picking equipment applying machine vision technology.

3.1 Principles of machine vision picking technology

Machine vision technology is used to mimic human visual functions, using a control system to analyse and process image information from objective objects for practical applications in harvesting, measurement and control (Kamkar et al. 2020; Pathare et al. 2013; Erol et al. 2007). This technology involves mechanics, computer technology, image processing, image recognition and localisation, artificial intelligence, signal processing and many other fields (Chen and Gong 2015). As fresh fruit products require good eating and appearance quality, selective harvesting methods are needed to ensure that ripe fruit can be harvested quickly and without damage (Paturi and Cheruku 2021; De Corato 2020). The basic structure of a fruit harvesting device based on machine vision technology consists of an autonomous mobile platform, a multi-degree-of-freedom robotic arm, a force feedback system with flexible end-effectors, a multi-sensor machine vision system, a drive control system, an intelligent decision-making system and auxiliary hardware and software (Duan et al. 2021; Jia et al. 2020; Tang et al. 2020a, b).

The initial phase of picking focuses on visual sensing perception techniques and learning crop information (Zou et al. 2012; Zhao et al. 2016), including camera components to recognize target fruit information (Wang et al. 2019), fruit localization, target background recognition (features such as colour, shape, texture and pose), 3D reconstruction, robot behavior planning based on visual localization mechanisms and vision (Fang 2019). For smooth fruit picking, vision systems have multiple sensing capabilities such as vision-mechanical cooperative control, vision recognition, coordinated vision-mechanical positioning and fault tolerance (Davidson et al. 2020). The control system should be collaborative and use a vision servo-controlled picking mechanism to perform the operation of pinching and cutting ripe fruit stalks (Ronzhin et al. 2022). The end-effector is required to accurately receive commands from the control system and collaborate with the robot arm to complete the fruit picking (Arad et al. 2020).

With the rapid development of computer and automation technology and the application and popularity of agricultural high technology, robotics is gradually entering the field of agricultural production (Wakchaure et al. 2023; Mao et al. 2021). It is worth noting that at present, there are still some challenges to be solved in the process of fruit harvesting robot picking machine vision technology. Due to the complexity of the crop-growing environment, such as branches and leaves, neighboring fruits, and some uncertain factors, this may cause inaccurate recognition and positioning, affecting the picking efficiency (Xiong et al. 2018; Wang et al. 2017). To address this challenge, Li et al. proposed a multi-armed apple harvesting robotic system, which utilizes a fruit accurate recognition and localization algorithm based on multi-task deep convolutional neural network (DCNN) technology to improve the recognition rate and localization accuracy of potentially occluded fruits (Li et al. 2023a, b), and at the same time, determining the shape and size of the fruits, the growing environment, the planting method, the biological characteristics, etc. are also beneficial for accurate fruit harvesting (Li et al. 2020a, b, c). In addition, the path planning and obstacle avoidance speed of harvesting robots in the field will also have a great impact on the operation speed of harvesting robots, and the use of machine learning combined with artificial neuron network technology to improve the recognition and processing ability of harvesting robots in complex environments will be the key to the development of navigation and positioning technology in the future (Li et al. 2020a, b, c; Wang and Liu 2020; Kondratenko et al. 2022; Zhou et al. 2022). In harvesting, for fresh fruits, picking robot arm picking is easy to break the fruit skin, affecting the commercialization of fresh fruits, and there is a phenomenon of missed picking; therefore, for different fruits, the research and development of a specialized, lightweight, flexible picking robot arm is imminent (Vrochidou et al. 2022; Zhou et al. 2021).

3.2 Current status of machine vision technology research

With the development of automation technology, investment in artificial intelligence research is increasing. Machine vision, a branch of AI, is now widely used for recognition and identification purposes in various work environments, such as the fruit picking equipment in Fig. 5A. This technology simulates human visual functions by capturing and processing images through a camera and then uploading the images to a personal computer for practical applications and control. Machine vision technology combines expertise from several fields such as image processing, machine automation, optics, vision sensors, virtual control, and computer applications (Gao et al. 2020). In recent years, with the rapid development of industrial intelligence, machine vision technology, which has the advantages of convenience, accuracy, speed, and intelligence, has been widely used in various fields such as industrial inspection, vision robotics, intelligent agriculture, and unmanned driving and has received more and more attention (Dong and Han 2021; Sun et al. 2018; Ren et al. 2022; Pérez et al. 2016; Mavridou et al. 2019). The development of computer vision technology has mainly included target detection and recognition, image segmentation, pose estimation, behavior analysis, etc., and the commonly used algorithms include convolutional neural networks (CNN), support vector machines (SVM), and deep learning (ResNet, YOLO) (Yang et al. 2021; Leo et al. 2017; Sharma et al. 2020; Srivastava et al. 2021). The development of machine vision technology, on the one hand, is thanks to the improvement of computer and camera performance; on the other hand, it is also inseparable from the optimization and innovation of the core algorithms, and currently, in the context of the AI big data era, the deep learning algorithms of 5G deep fusion will increase the accuracy of machine vision exponentially, so improving the accuracy of AI algorithms is the focus of future research and the difficulty(Ren and Wang 2022; Tang et al. 2023; Mahmood et al. 2022). Although machine vision technology has been better developed in various fields, there is still a gap between the diversity and complexity of research objects and meeting the needs of practical applications (Arrieta et al. 2020). Vision technology used in the field of automated fruit picking faces a huge amount of data, redundant information, feature space dimensions, and other characteristics (Cubero et al. 2016). A single simple feature extraction algorithm is difficult to meet the algorithm’s requirements for universality because of the field environment of light and shade conditions, the crop’s environment changes, and the accuracy of the recognition and positioning will be affected (Peng et al. 2023; Fu et al. 2021; Zhu et al. 2020; Shekhar et al. 2020). Future research and development should further improve autonomous navigation, fruit localization, and identification, among other issues.

Fig. 5

Schematic diagram of machine vision picking technology. A Flow chart of machine vision based fruit picking equipment. B Physical diagram of the harvesting process of the intelligent harvesting equipment

The introduction of the technology of machine vision, instead of the traditional manual inspection methods, greatly improves the quality of the products put on the market and increases the production efficiency (Syam and Sharma 2018; Tao et al. 2018). It is widely used in food and beverage, cosmetics, pharmaceuticals, building materials and chemicals, metal processing, electronics manufacturing, packaging, automotive manufacturing and other industries (Malik et al. 2023; Kabbour and Luque 2020). It can be broadly divided into two directions, one is non-destructive testing and the other is vision robotics. For example, Zhu et al. (2021a, b, c, d) reviewed the research progress in the application of machine vision technology in food processing, presented the challenges and future trends. Dowlati et al. (2012) applied machine vision technology to fish quality assessment and gave an outlook of future development. Penumuru et al. (2020) introduced a machine vision robot using machine vision and introduces a generic approach to automatic material recognition using machine vision and machine learning techniques to improve the cognitive capabilities of machine tools deployed in Industry 4.0 as well as material handling equipment such as robots. Li et al. (2023a, b) provides an overview of the current state of machine vision technology in furniture manufacturing and summarises the challenges faced by machine vision. In recent years, the scale of research on machine vision technology in the field of smart agriculture has been increasing, especially in the field of smart fruit and vegetable picking (Sharma et al. 2020; Shaikh et al. 2022; Saleem et al. 2021). The picking equipment typically consists of mobile platforms, PCs, machine vision components, control system cameras, end-effectors, and robotic arms, as shown in Fig. 5B. The operation process mainly relies on the detection and recognition of external features such as fruit color, texture, and shape, and operates the end-effector via image information processing on the PC system (Chen et al. 2022).

The key to the future development of vision technology in the field of fruit picking is still based on fruit recognition, positioning, and fruit separation (Yang et al. 2023a, b; Tian et al. 2020). Strengthen the research on the biomechanical characteristics of the picking object, optimize the recognition algorithm, develop adaptive scene analysis algorithms and adaptive control systems to adapt to new types of scene analysis algorithms and adaptive control systems, integrate 5G and IoT technologies, and realize the development of the picking robot’s operational capabilities for multi-scene and multi-crop types (Rong et al. 2021; Riaz et al. 2022; Liu et al. 2023a, b). Research and development of advanced materials suitable for picking, microsensors, actuators, and soft machinery applied to the picking process, and the gradual realization of strong adaptability and efficient autonomous harvesting (Zhang et al. 2023; Wang and Chortos 2022).

3.3 Application status of machine vision picking equipment

In recent years, machine vision technology, as an important branch of artificial intelligence, has been at the core of achieving intelligent perception, which is one of the inevitable key technologies for the development of smart agriculture, effectively liberating the labor force and also improving the quality and yield of crop products. Some representative research results are presented in Sect. 3.3, reviewing the current status of harvesting fruits such as tomato, apple, citrus, strawberry, and kiwi. In addition, in order to explore the challenges of harvesting fruits growing at high altitudes, several machine vision-based UAV harvesting techniques are presented.

3.3.1 Apple picking

In 1985, France developed the first apple-picking robot, which can basically pick an apple in 10 s in a test environment. Then later, Japan, the United States, Israel, and other agriculturally developed countries also joined the research on apple picking robots, which was also quite successful (Karkee et al. 2021; Tillett 1993). For example, Israel’s apple picking robot on the Gala apple picking test has 12 picking robotic arms, the use of a “3 claw” design, similar to catching dolls, and the apple one by one to pick down is 8–10 times more than the manual apple picking (Hohimer et al. 2019; Bergerman et al. 2016). Apple picking, as one of the advantageous industries in American agriculture, is even more automated. As shown in Fig. 6A, six hands are installed on the fruit picking robot, picking about 30 apples in a minute, from identification to picking in only 1.5 s. With satellite navigation combined with visual analysis, it can accurately identify ripe apples on the tree, and it adopts a vacuum suction type for all-weather picking (Zhang et al. 2018).

Fig. 6

(Source Hohimer et al. 2019; Bergerman et al. 2016). B Diagram of the complete apple picking equipment (Source Bu et al. 2022). C Diagram of the complete apple picking equipment (Source De-An et al. 2011). D Structure of strawberry picking equipment (Source Xiong et al. 2019). E Construction and specification diagram of the end-effector (Source Han et al. 2012). F Harvesting diagram of tomato picking equipment (Source Feng et al. 2015). G Structural diagram of citrus harvesting equipment (Source Wang et al. 2019). H Kiwifruit picking robot (Source Fu 2023). I Structure of kiwifruit picking equipment (Source Mu et al. 2020). J Structure of kiwifruit picking equipment (Source Mu et al. 2017). K Kiwifruit picking equipment based on two-armed collaboration (Source He et al. 2022). L UAV fruit picker developed by FAV in 2019 (Source Maor 2022). M Drone picker developed by TEVEL in 2020 (Source Maor 2023). (Color figure online)

Current status of fruit picking robot applications based on machine vision technology. A Apple picking robots in Israel. (Color figure onlined)

To evaluate the efficiency of different harvesting maneuvers, Bu et al. (2022) developed an apple harvesting device with an integrated vision system, flexible end-effector, and manipulator (Fig. 6B) and evaluated it using two harvesting maneuvers with a harvesting success rate of more than 80% and a damage rate of 0, but with a longer harvesting cycle. In order to improve the efficiency of harvesting equipment in various environments, Zhao et al. (2022) proposed a fast, high-precision harvesting device with advanced vision capabilities. Tests were conducted under different light conditions to evaluate the performance of the mobile platform, the reliability of the vision system, and the harvesting efficiency of the robotic arm. The results show that it is an efficient harvesting device that can identify ripe fruits at an average speed of about 1 s, with a picking speed of about 25 s/pc and a picking success rate of more than 95%.

To improve harvesting efficiency and reduce fruit damage, De-An et al. (2011) developed an apple picking robot (Fig. 6C). During harvesting, the pressure value on the apple surface was continuously monitored to prevent damage caused by excessive clamping force, and experimental results showed that the average recognition time was 15.4 s and the picking success rate was 77%. In order to solve the problem of damage caused to branches and fruits during harvesting, Davidson et al. (2016) developed an automatic apple harvester using a low-cost “non-sensor” system, and in order to facilitate the detection of clustering and occlusion of fruits, the machine vision system incorporates the Circular Hough Transform (Circular Hough Transform) and the Circular Block Analysis method. Using a six-degree-of-freedom (6DOF) serial link design for the manipulator and end-effector, the picking results showed that 95 out of 100 fruits were picked, with an average positioning and picking time of 1.2 and 6.8 s per fruit, respectively, and an accuracy of up to 90% could be achieved, with a success rate of more than 95% of the picking, proving the efficiency and accuracy of the picking of the fruits. Therefore, the study of apple picking robots is one of the research hotspots and difficulties in the field of agricultural robotics, which involves a number of disciplines such as mechanics, electronics, control, artificial intelligence, etc., and has a wide range of development prospects.

3.3.2 Strawberry picking

Strawberry is a labor-intensive crop whose harvesting mechanization is hindered by its own physiological characteristics, such as inconsistent ripening, thin and soft skin, and the irregular arrangement of each cluster of fruits after ripening (Baur and Iles 2023; Yarborough and Hergeri 2010). Therefore, the research and development of a non-destructive and efficient strawberry picking robot are imminent (Zhu et al. 2023; Abasi et al. 2018). In recent years, the German company Organifarms designed the “BERRY” strawberry picking robot, which is the most typical and can automatically detect the location of strawberries, ripeness, and quality and complete the picking machine collection work (Parsa et al. 2023). The strawberry picking robot from the Belgian company Octinion can identify a strawberry in 5 s, is equipped with multiple cameras with machine vision capabilities, can automatically generate 3D images and locate them, and can determine on its own whether a strawberry is ripe or not (Bogue 2020; Woo et al. 2020; Van Henten 2019; Verbiest et al. 2021).

Currently, strawberry picking still faces problems such as low efficiency, high damage rate, inaccurate recognition and localization, and high cost, and in recent years, relevant researchers have been committed to improving these problems with a view to designing efficient and damage-free strawberry picking robots (Yu et al. 2020; Rehman et al. 2022). Ling et al. (2021) designed a strawberry intelligent picking robot that can accurately identify and locate the position of ripe strawberries and harvest the fruits by gripping and twisting the fruit stalks. The experimental results showed that the average speed of discrimination of strawberries by this machine was 1 s, the rate of misjudgment of the fruits was 7%, and the success rate of the picking was 90%. To address the severe labor shortage in Western agriculture, Octinion has developed an autonomous strawberry picking device. Three RGB cameras were used for 3D vision and strawberry detection based on color differences, and picking was done by a robotic arm, which showed a picking speed of 4 s/pc with significantly higher efficiency and lower damage rates (De Preter et al. 2018).

To overcome the effects of the picking environment, Xiong et al. (2020) designed a porous strawberry continuous picking robot with a Hokuyo radar for navigation sensing at the front position and proposed a new obstacle separation algorithm to enable the picking system to pick bunches of strawberries. The results showed that the success rate of the first picking ranged from 50.0 to 97.1%, and the success rate of the second picking increased to 75.0 to 100.0%, with the failure being attributed to the limitations of the vision system as well as the lack of flexibility of the gripper. Meanwhile, to improve factors such as low picking efficiency and slow picking cycle, Xiong et al. (2019) designed a cable-driven strawberry picking robot as shown in Fig. 6D. Field experiments showed that the picking success rate was 96.8%, the average cycle time for picking a single strawberry consecutively was 7.5 s, or 10.6 s if all procedures were included, and the average picking success rate in a farm environment was 53.6%, or 59.0% if “success with damage” was included. To enable picking robots to work in complex environments, Huang et al. (2020) developed an automated strawberry picking device that utilized a human–robot cooperation approach for target recognition and a robotic arm controller for picking. Feng et al. (2015) developed a strawberry harvester that utilized an image segmentation algorithm based on the OHTA color space, and harvesting experiments were evaluated for performance. The device was found to be highly accurate in detecting strawberry stems, with an accuracy of 93% and more than 90% accuracy in determining the maturity and shape quality of strawberries against a black and white background. As shown in Fig. 6E, Han et al. (2012) developed an autonomous robot for harvesting strawberries cultivated in a tabletop system, where strawberry detection was based on a stereoscopic CCD color camera and laser device, respectively. The strawberry detection was carried out based on the 3D image and distance information obtained from the stereo CCD color camera and laser device, respectively. A DC servo motor-driven end-effector was designed to pick strawberries without any damage in less than 7 s, which greatly improves the problem of long harvesting cycles and damages. Hayashi et al. (2010) developed a strawberry harvesting robot that was designed based on the concepts of nighttime operation, pedal handling, and task sharing with the workers. The experiments showed that the machine vision device recognized the stalks with an accuracy of 60% and successfully picked a fruit in 11.5 s.

The above study shows that although the existing strawberry picking technology has gradually matured, it is still some distance from being practical and commercialized. The existing strawberries are planted in a regular and disorderly manner, which leads to an increase in the difficulty of identification and positioning of the harvesting robot and an increase in the picking cycle. Therefore, in order to make strawberry picking more efficient, the planting of strawberries should be regular and orderly, and the intelligent recognition and positioning of the harvesting robot, the design of the mechanical structure, path planning, and control system need further research and improvement.

3.3.3 Tomato picking

As one of the three major world-trade fruits and vegetables, tomatoes contain rich nutritional value (Iloh et al. 2020; Diop and Jaffee 2005). The global tomato planting area and production have increased significantly, but they are facing problems such as concentrated ripening and low picking efficiency (Li et al. 2018). In response to the above problems, the development of tomato picking robots basically solves the problem of time-consuming and labor-intensive tomato picking (Tian et al. 2022a, b; Bachche 2015).

Israeli agribusiness research and development of the Metomotion tomato automatic picking machine is the most typical and applicable to greenhouse tomato picking (Hughes et al. 2022; Saraiva et al. 2023). The development of a multi-purpose robotic system, GRoW, equipped with the most advanced robotics and automation technology, can achieve efficient and non-destructive tomato picking (Xie et al. 2022a, b; Cheng et al. 2023). Japanese agri-tech company Inaho has developed a tomato picking robot that can realize fully automated picking and can be used for up to 12 h on a single charge, working around the clock (de Bourgogne 2021). U.S. greenhouse company Appharvest, through the collection of a large number of tomato image data, can develop a tomato picking robot that can identify 50 kinds of tomato varieties and determine the maturity of tomatoes (Childers 2020). In recent years, the development of large language models such as ChatGPT has set off a revolution in the field of robotics, and the first tomato picking robot co-developed by ChatGPT and humans has appeared, advancing the development of fruit and vegetable picking robots (Gill and Kaur 2023; AlZu’bi et al. 2022).

In order to improve the efficiency of tomato picking, Feng et al. (2015) designed a tomato intelligent picking robot, as shown in Fig. 6F. The robot used an image recognition algorithm to detect ripe tomatoes in the field of view and determined the spatial position based on the linear laser positioning principle. The results showed that the picking success rate was as high as 83.9%, but the picking time was long, about 24 s/pc. In order to realize the robot’s ability to pick ripe tomatoes in a greenhouse, a picking robot for solar greenhouses with a vertical trellis was designed to automatically detect and identify the fruits during the picking and collection process, resulting in an efficient and damage-free picking of tomatoes, which may be inaccurately identified and located due to external factors (Yu et al. 2022). In order to achieve automated tomato picking, a tomato picking robotic arm was modeled, and a recognition method was developed to create a stereo vision model using the circular Hough transform and RGB color space. Fifty tests were conducted in a simulated environment with a success rate of 78% (Zhou et al. 2018). Yaguchi et al. (2016) developed a stereoscopic camera tomato picking device using infinite rotating joints for gripping, which allows depth measurements to be taken in direct sunlight. The device was evaluated through tomato picking competitions and on-farm experiments, with a picking speed of 80 s/fruit and a success rate of 60%, which is inefficient and much lower than manual picking speeds. Yasukawa et al. (2017) designed an infrared image- and specular reflection-based tomato picking robot for indoor fruit detection and recognition, with an accuracy in real-world image evaluation of 88.1%.

A dual-arm tomato harvesting robot for greenhouses was designed using binocular vision sensors with a 95% success rate in detecting ripe tomatoes, with only a few missed detections due to leaf shading. Tests showed that the positioning error was less than 10 mm and the harvesting success rate was 87.5%, which was significantly higher than the manual harvesting efficiency when applied to actual harvesting (Ling et al. 2019). Rong et al. (2022) designed an integrated adsorption and gripping manipulator in order to accurately identify the position of the tomato fruits and estimate the grasping posture to improve the success and efficiency of the robotic harvesting and developed an optimal sorting algorithm and an optimal sorting algorithm for the fruits. Optimal sorting algorithm and fruit nearest-neighbor localization algorithm, and designed directional grasping and sequential picking control strategies. Evaluation results showed that the string and fruit recognition accuracy based on the YOLOv5m was 90.2% and 97.3%, respectively, and the success rate of fruit harvesting was increased to 72.1%, with an average harvesting time for individual fruits reaching 14.6 s. Gao et al. (2022) developed a pneumatic finger end-effector for cherry tomato picking equipment that uses a combination of gripping and rotation to pick ripe fruits consistently and stably, and field tests showed that the average cycle time for picking a single cherry tomato was 6.4 s. The picking success rates for pickable cherry tomatoes in different directions were 84% (right), 83.3% (rear), 79.8% (left), and 69.4% (front).

Currently, strawberry picking has been automated by robots, but from the existing literature, it can be seen that in the current shortage of labor, the research and development of picking robots basically improves the problem of low efficiency in strawberry picking, but because of its complex growth environment, there is still a long period of picking and inaccurate identification and positioning problems.

3.3.4 Citrus picking

Citrus is rich in resources and excellent varieties, and in recent years, there has been an increasing demand for citrus (Liu et al. 2022). Currently, citrus harvesting is still a labor-intensive and time-intensive task, with the risk of injury to the fruit growers (Khatri et al. 2021). Automated citrus harvesting can reduce the labor risk and improve the picking efficiency of the fruit growers (Ferreira et al. 2018; Blasco et al. 2019).

Spain’s Agri-Tech invented a citrus picking robot that can pick 60 citrus a minute, compared to 8 a minute by hand. The robot can also sort the picked citrus by size via a video recorder (Zhao et al. 2020; Harman and Sklar 2022). To address the challenges of commercializing citrus harvesting robots, Yin et al. (2023) provide a fully integrated, autonomous, and innovative solution for citrus harvesting robots to overcome the difficulties of harvesting citrus due to its natural growth characteristics, propose a new visual estimation of the fruit pose, and design a new end-effector, which allows the robot to harvest citrus continuously with an overall success rate of 87.2% and an average picking time of 10.9 s/pc.

For the automated picking of citrus, Yang et al. (2019) proposed a citrus robot, which had a success rate of 83.6% and 91.9% in identifying obstacles and fruit ripeness, respectively, with an error of 5.9 mm, a processing time of 0.4 s for a single image frame, and a success rate of 80.51% and 75.79% in picking ripe citrus and avoiding obstacles, respectively. In order to improve the low automation of picking, a citrus fruit picking robot was developed, which was divided into a control part and a mechanical part. The picking results showed that the average picking time was 5.4 s and the maximum picking height was 1.85 m. The robot has the ability to identify, localize, pick, sort, and box citrus (Liu et al. 2019). As shown in Fig. 6G, Wang et al. (2019) developed a citrus harvesting device that uses a tongue-shaped end-effector to randomly harvest citrus fruits along the direction of the stalk. The results of laboratory citrus stalk cutting and harvesting experiments showed that the average cutting success rate was as high as ~ 98% over the range of deflection angles [− 50°, 50°], and the harvesting rate was ~ 89% in the optimal position. The harvesting results in the natural environment showed that the harvesting rate was as high as 74% in the optimal position. Yang et al. (2023a, b) developed a lightweight, Raspberry Pi-based platform. A citrus intelligent recognition picking robot was developed based on the Raspberry Pi platform, and a citrus deep recognition picking system was designed. A deep convolutional neural network (YOLOv4-tiny) was used to verify the validity of the citrus dataset, with a recognition rate of 98%, which can achieve the recognition of citrus three-dimensional positional coordinates and accurate picking, and 20 picking tests were conducted, with an average success rate of picking of more than 90%.

With the rapid development of modern agricultural technology, the automation of citrus production and harvesting is an inevitable trend. It should be noted that although the citrus automated harvesting robot has successfully realized the identification and harvesting of citrus, the automated harvesting of citrus still faces some challenges, such as adaptability to different varieties and growing environments, stability of the robot, etc., but these challenges will be gradually overcome with the continuous progress of technology.

3.3.5 Kiwifruit picking

Kiwifruit is a nutritious fruit with a unique flavor and high economic returns (Barman et al. 2021). At present, kiwifruit harvesting is mainly done manually, which requires a lot of manpower for picking, sorting, and packing due to the relatively concentrated harvesting period and short harvesting cycle (Sarkar 2021; Barbole et al. 2022). Therefore, it is necessary to promote the automation of kiwifruit harvesting to make up for the problem of labor shortage and to achieve the mechanization, informationization, and standardization of the kiwifruit industry (Ren et al. 2023).

The automatic kiwifruit picker developed by Robotics Plus of New Zealand picks ripe kiwifruit from the fruit rack through four machine picking arms and then transmits them to the collection box through the tube bag under the machine arm to realize the work of picking, sorting, packaging, etc. The robot can pick kiwifruit from the fruit rack through four machine-picking arms (Zhang et al. 2019; Karkee et al. 2019). As shown in Fig. 6H, northwest A&F University in China has developed a kiwifruit-picking robot. After the robot starts working, the camera will accurately identify and locate the fruit, after which the bionic manipulator is able to quickly clamp and pick without damage, combining with the hardening characteristics of kiwifruit harvesting to achieve efficient kiwifruit collection (Fu 2023).

Mu et al. (2020) developed a kiwifruit picking robot (Fig. 6I), which proposed an automated picking method based on kiwifruit characteristics, where the end-effector approached the fruit from below, wrapped and grasped the fruit with two bionic fingers, and then bending the fingers would allow separation of the fruit from the trunk. Tests showed an average picking time of 4–5 s for 240 fruits with a picking success rate of 94.2%, demonstrating the potential of the grasp-pick-gather end-effector. A compact and lightweight kiwifruit picker was developed based on kiwifruit characteristics. It recognizes and locates ripe fruits and picks them by clamping and twisting. The design of the machine lays the foundation for further research in automatic sorting technology (Gao et al. 2013). Williams et al. (2019) developed a multi-armed kiwifruit harvesting device with a novel harvesting mechanism to efficiently harvest kiwifruit from the canopy. Field trials demonstrated a harvest rate of 51% with an average harvest cycle time of 5.5 s/unit, but with long wastage times and fruit losses of up to 23.4%. With the new improved vision system and two manipulators, the success rate for harvesting fruit was 86.0% and 55.8% for kiwifruit, with a cycle time of 2.78 s per unit (Williams et al. 2020).

As shown in Fig. 6J, Mu et al. (2017) proposed a picking method with “grab-pick-slide” and designed an end-effector with bionic fingers, information sensing, and machine vision for non-destructive kiwifruit picking. Tests showed a picking success rate of 90%, an average loss rate of 10%, and an average picking cycle time of 4 s. The device has great utility in achieving non-destructive fruit picking. As shown in Fig. 6K, He et al. (2022) proposed a two-arm cooperative method for mechanical picking of kiwifruit in orchards. The method consists of three steps: determining the picking position, collision detection, and a continuous picking cycle. The picking test of the dual-arm collaborative platform showed an average picking success rate of 86.67% and a collision detection time of 3.95 ± 0.83 s for each fruit.

To improve the efficiency of kiwifruit harvesting, a multi-mechanical-arm kiwifruit harvesting device was proposed by Barnett et al. (2020). Ten harvesting tests were conducted, all of which yielded good results, and the use of a multi-mechanical arm was the most effective method for completing the harvesting operation. In order to automate kiwifruit harvesting, Chen et al. (2012) designed an end-effector for a kiwifruit harvesting device that was mounted on the front end of the robotic arm. In addition, the end-effector mechanism, sensing system, and control system were designed. Tests showed that the end-effector had a gripping success rate of 100%, a picking success rate of 90%, and an average picking cycle time of 9 s/pc.

Kiwifruit harvesting is a labour-intensive task, and it is clear from existing research that automated kiwifruit harvesting has become a need and a trend for the future as kiwifruit production becomes more and more significant.

3.3.6 Drone picking techniques

Existing picking robots are mostly designed with ground mobile platforms, which cannot complete the picking operations targeting higher-growing fruit trees and inter-mountain orchards (Xie et al. 2022a, b; Hussain et al. 2022). Currently, using UAVs as mobile carriers to carry end-picking end-effectors is one of the solutions to high-altitude fruit picking, which has the advantages of high harvesting efficiency and flexibility (Varadaramanujan et al. 2017; Suryawanshi et al. 2022).

There are fewer studies on fruit picking by drones, with the apple picking drone developed by Israeli agricultural drone manufacturer Tevel Aerobotics being the most typical (Neupane et al. 2023; https://www.tevel-tech.com/, 2022). As shown in Fig. 6L, this UAV combines flight technology, robotics, and AI technology. It is also equipped with a gripper, a camera, a protective frame, etc., and has achieved more than 90% accuracy in picking ripe fruits in trials (Maor 2022). Meanwhile, another achievement of Tevel’s fruit picking UAV system is fruit grading. When the fruit picking UAV is selectively picking fruits, the harvesting result is significantly better than manual picking and 10% higher than manual grading harvesting (Vrochidou et al. 2022; Eminoglu and Yegul 2022). In 2020, Tevel proposed a harvesting solution based on UAV picking technology, as shown in Fig. 6M, where a robotic arm mounted on the UAV mainly recognizes the location and ripeness of fruits through visual sensors. When ripe fruits are detected within a certain range, the UAV flies near the fruit trees, grabs the fruits with the robotic arm, and picks them by rotating and twisting them. The collected fruits end up in a collection device (Maor 2023).

Jagadeeswaran has proposed a smart coconut and palm-cutting drone for efficient fruit harvesting. The quadcopter UAV is equipped with a vision system, a slider crank mechanism to lift a payload of 1 kg, and a cutter on the top to separate the fruits from the tree (Jagadeeswaran et al. 2021). Chu et al. (2022) proposed a novel device for harvesting red pine fruits divided into a drone and a harvesting mechanism, and the tests have shown that this harvesting device is highly automated, simple to operate, and cost-effective. Tang et al. (2020a, b) developed a bionic snake mouth harvesting mechanism and an unmanned harvesting device based on visual positioning that have high harvesting efficiency, low labor costs, and can effectively prevent fruit damage and personnel injuries during the harvesting process. Yan et al. (2021) developed a fruit tree picking UAV that uses a micro-hydraulic rod to move a mobile blade, which, together with a fixed blade, accurately separates the fruit stalks from the fruit tree through probe detection. This UAV enables non-destructive fruit picking, reduces labor costs, and improves picking efficiency. Zhu et al. (2021a, b, c, d) proposed an autonomous lychee picking device based on a quadcopter UAV. The system uses an identification control system to detect ripe lychee fruits and an actuator-end scissor assembly to cut the stalks, which allows for non-destructive fruit picking that is cost-effective, safe, and reliable.

From the existing literature, it can be seen that the existing research on picking UAVs is mostly in the design stage, many designs are idealized and not tested on the ground, and the related control problems have to be further solved. Compared with ground-picking robots, picking UAVs have higher requirements for range, identification and positioning, and obstacle avoidance.

4 Challenges and future trends

As mentioned earlier, most fruit picking experimental studies have shown that fruit picking equipment based on mechanization and machine vision demonstrates unique advantages and development prospects in the agricultural field. In recent years, many research teams have conducted extensive scientific studies on automated fruit picking technologies and have made significant breakthroughs in key technologies. However, experiences gained from picking experiments have concluded that low efficiency, a high damage rate, low automation, inaccurate identification and positioning, and the high cost of complex structures are the main challenges of current fruit picking technologies. In order to address these challenges, it is crucial to reduce costs, increase identification and positioning accuracy, and improve efficiency.

4.1 Mechanised fruit picking technology

Mechanical fruit picking equipment can cause greater damage to fruit and branches during harvesting. Vibratory picking techniques may cause the fruit to fall directly to the ground, damaging the fruit skin and branches. Chen et al. (2021a, b) designed a vibratory blueberry picker, and experimental results showed that the picker caused 6.7% damage to branches. It has been shown in the literature that pneumatic picking may cause damage to the fruit surface, branches, and leaves due to larger air currents (Afsah-Hejri et al. 2022). Junming et al. (2021) conducted harvesting trials on apples using an impact vibratory picking machine, which achieved an efficiency of more than 90% but caused damage to the plant. Zhang et al. (2015) designed a comb-and-brush berry picking machine to harvest mature fruits with a picking rate of 86.70% and a fruit damage rate of 8.62%, pointing out that comb-and-brush harvesting can cause different degrees of damage to branches and leaves. The push-shear harvesting technique caused less damage to fruits, but the net harvesting rate was lower (Yu and Ampazidis 2022). Shear harvesting may cause damage to the fruits as the stalks are not cut at a good distance, which tends to destroy the fruits (Xu et al. 2023). Although the use of orchard harvesting machinery has become widespread, damage to fruits and stems during harvesting still occurs, which constrains the development of this equipment. To solve this problem, future research and development should focus on reducing the damage rate and improving the picking efficiency, such as by adding some flexible materials as actuators, giving priority to materials that cause less damage to fruits when selecting materials, and determining the optimal picking parameters through a large number of experiments. In the future, fruit harvesting machinery may develop in the direction of low damage, high efficiency, and low cost. Equipment’s ability to operate in complex environments, overcoming the effects of external factors, or effectively responding to or avoiding these situations through means such as sensors and software analysis. In addition, researchers and developers should have the expertise to solve problems in a timely manner and to effectively improve and maintain the performance of vision systems.

Complex orchard environments pose a challenge to achieving a high level of automation in fruit picking, and picking machinery requires a large amount of information to ensure that the task is completed (Jia et al. 2022; Yoshida et al. 2022). However, due to the high vibration frequency, air velocity, impact frequency, and combing speed in complex environments, unripe fruits may be picked prematurely, leading to economic losses. Researchers have designed human–machine-assisted devices, but most of them are used for semi-automatic picking and still require human intervention to identify and locate ripe fruits. As a result, a large amount of manual labor is still required during fruit picking. Picking machinery is complex and expensive to maintain, which may lead to increased equipment damage. The internal structure of the machinery is cumbersome and requires several mechanical parts to work together, which is relatively costly. Safe management and maintenance of equipment are critical, but neglecting both can lead to problems, deterioration, and reduced harvesting efficiency (Duckett et al. 2018). In addition, the complexity of the equipment can lead to high maintenance costs and operator difficulties (Ampatzidis et al. 2014). Therefore, it is urgent to accelerate the development towards automation, intelligence, and user-friendliness so that harvesting equipment can be developed in the direction of simple operation, low failure rate, safety, and reliability.

In view of the above problems, automatic fruit harvesting machinery is one of the difficulties and hot spots in the field of agricultural high technology. Its harvesting efficiency has been a difficult problem for researchers. Future research and development work should be devoted to improving the harvesting efficiency of harvesting machinery, such as through a large number of experiments to determine the optimal operating parameters of harvesting machinery, to maximize the possibility of overcoming some of the difficulties in harvesting, and to improve the timeliness of the equipment for harvesting fruits. It is also helpful. In addition, most of the currently available harvesting equipment is designed for specific fruits or shapes with poor versatility, which increases the production cost. To address this issue, future R&D efforts should focus on upgrading existing harvesting equipment to be multifunctional and adaptable to different fruits and environments, and such multifunctional equipment would only require replacing the corresponding modules, which reduces costs and improves efficiency. For the design of actuators, such as shear mechanisms, flexible materials should be used, which can greatly avoid the damage caused to the fruits.

Complex orchard environments can reduce the efficiency of harvesting fruit. Existing mechanized harvesting technology requires manual assistance to complete the harvesting task, which is not fully automated and therefore imposes strict requirements on the fruit-growing environment. In order to achieve a high degree of automation of harvesting equipment, it is necessary to develop standardized planting specifications for different types of fruits. To achieve fruit planting standardization, scaling, specialization, and factory, it will help to reduce the complexity of the harvesting operation and improve efficiency, which will lay the foundation for the development of harvesting machinery and is also the future development trend in this field.

Harvesting machinery has a complex structure and high maintenance costs, increasing the risk of equipment damage (El-Termezy et al. 2022). Its structure involves several mechanical components, leading to high costs. Neglecting safety management and maintenance can lead to equipment problems, deterioration, and loss of efficiency. The complexity of the equipment can also lead to high maintenance costs and difficulties for operators (Bechar and Vigneaul 2016). As orchard harvesting machinery technology continues to mature, harvesting efficiency and fruit quality have improved. However, the structure of harvesting equipment often consists of numerous auxiliary components, which increases manufacturing and maintenance costs. In order to better serve the processing of agricultural products, future harvesting equipment should focus on simplicity, high precision, high automation, and low cost. For example, it is possible to combine the harvesting process with the grading and transport of fruits to reduce the costs required for segmented harvesting.

4.2 Machine vision picking technology

The lower recognition efficiency of vision systems used for fruit picking can affect the accuracy of fruit picking (Wan and Goudos 2020). In addition, the recognition efficiency of picking equipment can be reduced by a variety of external factors, including the complexity of the fruit-growing environment, such as whether the fruit is obscured by debris, branches, or overlapping growth (Xiao et al. 2023; Karkee et al. 2018). Natural conditions, such as the light, color, shape, texture of the fruit, inclement weather, noise, and geography, can also negatively affect the recognition efficiency, making it difficult for the visual recognition system to accurately identify the location of the fruit, and these problems seriously affect the picking efficiency (Tang et al. 2020a, b; Liu et al. 2020). Some studies have shown that Zhuang et al. (2023) introduced an artificial potential field (APF) for path planning of a robot based on a six-degree-of-freedom robot for situations such as obstacle obstruction and combined it with the A* algorithm, which is highly adaptive to obstacle avoidance path planning and is able to complete the obstacle avoidance path planning in a faster and more reasonable way. In addition, the use of deep learning techniques can improve the target recognition performance of harvesting robots in complex environments, but there are still many uncontrollable influencing factors, and the stability of visual recognition is also problematic (Ghasemi et al. 2022; Wu et al. 2021). Therefore, in order to maximize the potential of harvesting equipment, future R&D efforts should focus on improving the equipment’s ability to operate in complex environments, overcoming the effects of external factors, or effectively responding to or avoiding these situations through means such as sensors and software analysis. In addition, researchers and developers should have the expertise to solve problems in a timely manner and to effectively improve and maintain the performance of vision systems.

Picking cycle time and efficiency are key factors in measuring the performance of fruit picking equipment (Li et al. 2011). Although the equipment must ensure that fruit damage is minimized and does not affect the next year’s yield, it is less efficient and not as effective as manual labor (Connor et al. 2014). Wang et al. (2023) pointed out that in the case of cherry tomato harvesting, factors such as inefficient recognition, long processing time, and susceptibility to subjective factors are often encountered, which limit the accuracy of the harvesting robots in complex scenarios and robustness, most of which require manual participation. A self-built target detection algorithm for hollies is proposed, and the results show that the average accuracy rate reaches 95.2%, and the improved model can perform real-time target recognition and maturity detection for hollies. Therefore, to improve the efficiency of harvesting equipment, future research should focus on enhancing infrastructure maintenance to ensure optimal component interaction, improving the response time of the recognition and localization system, increasing the harvesting speed of the end-effector, and constructing finer target recognition and detection methods.

Future research and development of fruit harvesting equipment should focus on simplifying structures, improving harvesting efficiency, and reducing manufacturing and maintenance costs (Bac et al. 2014). Existing equipment involves multiple fields such as physical, mechanical, electronic information, and intelligent system control, requiring expensive hardware and software facilities to achieve high efficiency, and maintenance of complex equipment requires specialized personnel, leading to increased operating costs (Zhang et al. 2020). Data suggests that the world’s first AI-picking robot, Robocrop, cost a staggering £700,000 to develop. Meanwhile, new research from Interact Analysis suggests that the robotic harvesting market is in the early stages of growth but has huge potential—valued at $236 million in 2022 but set to rise to $6.8 billion by 2030 (https://interactanalysis.com/). Reducing R&D costs has become the trend for future harvesting robots, and improving the relevant supporting facilities of harvesting equipment can reduce costs. The various supporting facilities of harvesting equipment should be fully utilized to improve their maintenance. In order to achieve intelligent, efficient, and low-cost harvesting, operators should monitor the working status of each facility in real time and find the best parameters through experiments. In addition, future development should focus on miniaturization, intelligence, and user-friendliness to fully replace manual harvesting and contribute to rural revitalization.

Existing fruit picking equipment is designed for specific types or shapes of fruits and is therefore less versatile (Li et al. 2022a, b). Most fruit picking is done in a short period of time, which poses a challenge for storage and extended shelf life. In order to reduce manufacturing costs and better serve agriculture, fruit picking equipment must be highly versatile to pick a wide range of fruit types (Tian et al. 2020). Mechanical damage during fruit harvesting often leads to a significant reduction in fruit quality and economic losses. It has been shown that during harvesting, fruit damage due to factors such as abrasion or friction accelerates water loss, and bacteria can penetrate into the fruit, leading to rapid decay and spoilage (Komarnicki et al. 2017; Tensaw 2020; Sudheer and Indira 2007). Therefore, measures need to be taken to minimize damage to the fruits, and the materials used for the pusher and collection device should be improved; the use of soft materials can minimize damage caused by direct contact between the fruits and the end of the pusher, avoiding direct dropping of the fruits and reducing the impact of environmental factors. At present, the development and promotion of various intelligent fruit picking equipment is accelerating this process, greatly reducing costs and improving efficiency. This is also the future development trend of automated fruit picking.

5 Conclusions

This paper reviews the progress of the application of mechanized technology and machine vision-based technology in the field of fruit harvesting. It clearly points out the categories, harvesting principles, and harvesting efficiencies of existing mechanized harvesting technologies at home and abroad, and the use of this technology for fruit harvesting needs to take into account the damage to the fruit and the degree of automation. In order to achieve efficient and damage-free fruit harvesting, machine vision-based fruit harvesting equipment has become a hot research topic in recent years. In order to obtain detailed information about the environment and fruits, the harvesting equipment is equipped with various types of vision sensors and image analysis algorithms. Fruit picking such as apples, strawberries, tomatoes, citrus, and kiwifruit is reviewed in detail, and the use of drone picking technology can achieve fast and efficient picking for high-growth fruit trees and inter-mountain orchards.

In addition, on the basis of the overview of the current situation of the application of the above picking equipment, this paper summarizes some of the difficulties that still exist in the practical application of mechanized technology and machine vision technology in the field of fruit picking, such as low efficiency, high cost, high damage, and difficulties in identification and positioning. In view of the above problems in practical application, the future development of the two picking technologies is worth looking forward to. In summary, with the rapid development of computer and automation technology, the application of agricultural robots, and the development of a new generation of information technology such as the Internet of Things and the accelerated deployment of 5G networks, there is an urgent need to design a new machine vision algorithm to improve the feature extraction capability, the feature selection capability, and the feature classification capability, and the researchers have found that the 5G deep fusion deep learning algorithm combines the efficiency and robustness of machine vision with the flexibility of human vision, and the combined machine vision harvesting system not only has the ability to detect in complex environments, but also improves greatly in real-time. In the face of manual picking labor, damage and other problems are endless. The fruit of the automated harvesting of accurate and non-destructive problems needs to be solved. In recent years, non-contact image sensors accordingly came out, not only to detect the size and shape of the fruit but also to the fruit of the damage to the appearance of the analysis of the fruit, suitable for a variety of types of fruit harvesting and sorting, so that efficient and accurate harvesting of fruits has become a reality. It is believed that with the continuous progress of technology and the support of government policy, the research of automatic fruit harvesting equipment will be a direction with broad prospects.