Evaluation of approach strategies for harvesting robots: Case study of sweet pepper harvesting

Robotic harvesters that use visual servoing must choose the best direction from which to approach the fruit to minimize occlusion and avoid obstacles that might interfere with the detection along the approach. This work proposes different approach strategies, compares them in terms of cycle times, and presents a failure analysis methodology of the different approach strategies. The different approach strategies are: in-field assessment by human observers, evaluation based on an overview image using advanced algorithms or remote human observers, or attempting multiple approach directions until the fruit is successfully reached. In the latter approach, each attempt costs time, which is a major bottleneck in bringing harvesting robots into the market. Alternatively, a single approach strategy that only attempts one direction can be applied if the best approach direction is known a-priori. The different approach strategies were evaluated for a case study of sweet pepper harvesting in laboratorial and greenhouse conditions. The first experiment, conducted in a commercial greenhouse, revealed that the fruit approach cycle time increased 8% and 116% for reachable and unreachable fruits respectively when the multiple approach strategy was applied, compared to the single approach strategy. The second experiment measured human observers’ ability to provide insights to approach directions based on overview images taken in both greenhouse and laboratorial conditions. Results revealed that human observers are accurate in detecting unapproachable directions while they tend to miss approachable directions. By detecting fruits that are unreachable (via automatic algorithms or human operators), harvesting cycle times can be significantly shortened leading to improved commercial feasibility of harvesting robots.

Detection is considered to be one of the major limitations preventing commercialization of autonomous harvesting robots today with state of the art detection rate limited at 85% [4]. A major problem is the unstructured and dynamic nature of the agricultural environments [24]: fruits have a high inherent variability in size, shape, texture, and location; in addition, variable illumination conditions and occlusion significantly influence the detection performance. Significant research have been focused on developing detection algorithms [4,18,22]. Variable illumination conditions have been overcome using different techniques such as adaptive thresholding (e.g. [41]), adding controlled illumination (e.g. [17]), applying high dynamic range cameras [37], and color modification [39]. Attempts to deal with occlusion have previously been made using hyperspectral imaging [28]. However, due to the high cost as well as the weight of hyperspectral cameras, RGB cameras have become the most commonly used sensor since fruits, particularly ripe, tend to be different in color than the background [4]. Additional solutions include mechanical removal of occlusion in a temporal manner using air blowing [42], or permanently by pruning leaves. Both methods, from internal conversations with growers and experts in the field, cause disagreement in the growers community due to the possible impact on yield and damage to the plant. Pruning of leaves also involves a lot of manual work which will impact the economical feasibility of a harvesting robot.
Detecting the fruit is typically done from an overview waypoint W 0 where several plants are visible [9] (Figs. 1 & 5). This can be done either by using external static sensors or eye-in-hand cameras mounted on a robotic manipulator [9] with different algorithms [4,24,31,35,41,43]. Viewpoints analyses in harvesting robotics indicate that only 60% of the fruit can be detected from a single detection direction [22]. Improved performance can be achieved by including a human in the interpretation of the visual information received from the robot sensors [7,8].
Approaching the fruit when eye-in-hand cameras are used is often divided into two steps [6,19]. The first step involves moving from the overview waypoint W 0 to an approach waypoint W 1 where a single fruit is centerd in the image. This waypoint is identified using the approach direction θ 1 , defined as the angle the robot should use to approach the fruit from, and distance d to the fruit, as seen in Fig. 1. The second step uses visual servoing [6,10] to move towards the fruit until it is reached. The first step is not always used [25], but can be introduced to shorten the time it takes to Fig. 1 Overview waypoint W 0 (camera facing front direction) and approach waypoint W 1 , identified using the distance to fruit d and approach direction θ 1 reach the fruit since the distance needed for visual servoing will be shorter, while the second step is used to continuously refine the fruit position detected from the overview image. The first step is important so as to reduce the long overall harvesting cycle time which is another major limitation preventing commercializing harvesting robots [4]. Planning a path towards the fruit needs to take into account plant stems and other obstacles to prevent harming the vegetation [1,3,5].
When the fruit has been reached, or sometimes on the approach towards the fruit, the maturity of the fruit must be evaluated to determine if it should be harvested or not [20]. If it is determined to be ripe enough, the fruit must then be grasped. The accurate grasping of a fruit is a difficult problem due to the limitations of available robotic grippers and the inherent difficulties of grasp planning [14,34]. Eizicovits and Berman [14] developed geometrybased grasp quality measures based on 3D point cloud to determine the best grasping pose of different objects, including sweet peppers. This kind of solution depends on detailed 3D sensor information of the object [15,29] which is very difficult to achieve in dense greenhouse environments. Obtaining this information in enough detail prolongs the harvesting cycle. Some gripper solutions that do not need an accurate grasping pose have been reported, but currently the harvesting success is limited [2]. Once the fruit has been grasped, it must be detached without damaging the plant or fruit. This operation is fruit dependent (e.g., for sweet peppers the detaching is performed by cutting the peduncle of the pepper; for apples a twist and snap operation is needed).
Given the dense environment and the continuous detection required when using visual servoing, there is a high risk of losing the fruit along the approach due to occlusion, regardless of the detection algorithm used.
To reduce the risk of losing a fruit due to occlusion it is important to approach it from a waypoint from where the fruit is not occluded by leaves and other obstacles along the approach. An approach strategy is the method of finding an approach waypoint from which visual servoing will not lose the fruit due to occlusion. The aim of this paper is to propose different approach strategies and compare them in terms of cycle times and success rates and identify failure causes. To focus on the approach task, standard color segmentation algorithms for detecting fruits and ripeness are used while limitations of grasping and detaching the fruits are not regarded. The work is demonstrated for a case study of sweet pepper (Capsicum annuum) harvesting in a research greenhouse as part of the Horizon 2020 EU SWEEPER project (G.A. 644313) 1 using a robotic manipulator equipped with an eye-in-hand camera. Previous limited work in lab conditions revealed that the choice of approach strategy influenced cycle time up to 40-45% but did not influence the success rate, which was 100% regardless of strategy [33]. However, in greenhouse conditions one would expect lower success rate and cycle times. This paper aims to analyze the effect of the approach direction on performance. Additionally, a failure analysis methodology of the different approach strategies provides several conclusions for implementation in robotic harvesters.
Previous research on human-robot collaboration for target recognition has indicated that improved performance can be achieved by including a human in the interpretation of the visual information received from the robot sensors [7,8]. Therefore, another aim with this work is to measure the ability of human observers to provide insights on approach directions based on overview images taken in the field.

Approach Strategies
An approach strategy is the method of finding an approach waypoint from which visual servoing will not lose the fruit due to occlusion. It is computed using the approach direction θ 1 and distance to fruit d (Fig. 1). Two approach strategies are proposed and evaluated: -Single approach strategy: the robot attempts only one approach direction which is considered to be the best one with least occlusion. This direction can be obtained either by advanced algorithms that map the environment or by a human operator doing an assessment in-field or remotely by looking at images taken by the robot at the overview waypoint. -Multiple approach strategy: the robot autonomously attempts each approach direction from a sequence of apriori defined approach directions until it finds one that leads to a successfully reached fruit.
This section describes how approach waypoints are calculated, and provides a detailed description of the two strategies.

Approach Waypoint Calculation
To calculate the pose of the approach waypoint, the following information must be known: the location of the pepper, the approach direction, and the visual servoing distance, i.e. the distance to the fruit from the approach waypoint. Given position P i (x i , y i , z i ) of (the surface of) fruit i at approach direction θ i = 90 • the position for approach waypoint W i (x i , y i , z i ), located at a predefined distance d from fruit i at an approach direction θ i , is calculated according to: where r is the fruit radius.

Proposed Approach Strategies Description
In the single approach strategy, a single approach direction θ i should be determined for each fruit i. If possible, approaching from front (θ i = 90 • ) is preferred [5], otherwise the least occluded direction should be chosen. The approach cycle starts with moving the end-effector to a pre-defined overview waypoint W 0 (x, y, z) where the location of all visible fruits are recorded. From there, using the position and selected approach direction of each target fruit i, the approach waypoint W i (x i , y i , z i ) is calculated according to Eqs. 1-3. The control unit plans a path for the end-effector to the first waypoint. After reaching it, a visual servo procedure guides the manipulator towards the target until the end-effector reaches the fruit. If the manipulator is able to reach the target fruit, the fruit is marked as reachable.
If the manipulator cannot reach the fruit for some reason, e.g. the view of the fruit is lost during visual servoing or the controller is not able to plan a path there, the fruit is marked as unreachable. After the fruit has been marked as either reachable or unreachable, the approach cycle for fruit i ends. The next approach cycle starts with the robot moving to the approach waypoint of the next fruit, W i+1 . The cycle ends when all fruits detected from the overview image have been attempted to be approached. The left part of Fig. 2 shows a flowchart of this approach strategy.
In the multiple approach strategy, the best approach direction θ i is unknown, and therefore must be searched from a list of predefined potential approach directions θ i,1 ..θ i,k as seen in Fig. 3. For each target fruit i and potential approach direction θ ij the control unit calculates the path of the robotic manipulator to a waypoint W ij (x ij , y ij , z ij ) according to Eqs. 1-3. The manipulator moves to each waypoint in turn for the first fruit until the fruit is marked as reachable or all waypoints have been tried. In the case that all approach directions θ i,1 ..θ i,k have been attempted without being able to reach the fruit, the target fruit is marked as unreachable. After success or fail, the path to the waypoint W i+1,1 for the next fruit and its first approach direction θ i+1,1 is calculated. The right part of Fig. 2 shows a flowchart of this approach strategy.

Experimental Methods
Two experiments were conducted to compare the performance of the approach strategies. In the first experiment, the multiple approach strategy is compared to the single approach strategy assessed by a human operator placed in the field. In-field human assessment was selected as opposed to fully autonomous algorithms due to lack of sensor technology and algorithms to date. Current algorithms are often partially based on manual data [6] and not sufficiently accurate or fast to map greenhouse environments for determining possible approach directions. Stems are not detected in the system used in this work and are thereby not avoided unless they block the view of the fruit.
The second experiment measures the ability of human observers to provide insights to approach directions based on overview images collected in laboratorial and greenhouse conditions.

Equipment
A 6DOF robotic manipulator Fanuc LR Mate 200iD equipped with an eye-in hand iDS UI-5250RE RGB camera and a Sick DT20HI displacement measurement laser sensor was placed in-front of each scene. The end-effector used in  the greenhouse experiments was slightly different than the one used in the previous lab experiment [33]. Primarily the lab version had a suction cup mounted in front that could touch the peppers while the greenhouse version lacked this. Therefore, the greenhouse version never touched the peppers but stopped just before it reached the pepper. In laboratorial conditions, an artificial plastic pepper crop with yellow plastic fruits and green plastic leaves were used for the experiments. An example from the laboratorial and greenhouse setups can be seen in Fig. 4.
The workflow of the robot was implemented using a generic ROS software framework for development of agricultural and forestry robots previously developed [21]. The framework is constructed with a hybrid robot architecture, using a state machine implementing a flowchart as described by Ringdahl et al. [32].

In-field Experimental Protocol
The experimental protocol used for the robotic experiments in the greenhouse resembles the previously described laboratory experiment protocol [33], while addressing some of the conclusions and their implementation in greenhouse conditions. Table 1 outlines the protocol used for laboratorial and greenhouse experiments respectively. The difference between greenhouse and laboratorial conditions can be  seen in Fig. 5, which shows overview images taken from the respective environment.
The greenhouse experiment was conducted in a research greenhouse at Sint-Katelijne-Waver, Belgium, early in the season. The robotic manipulator and its sensory system were placed in an aisle and manually centerd in front of a pepper or a cluster of peppers, defined as a scene. First, the locations of all the fruits in the scene were registered. In this experiment the system was not equipped with a RGB-D camera, which would typically be used for automatic registration of fruits locations. Instead, the manipulator was manually positioned in front of each fruit before the experiment began. The end-effector was brought close to the fruit and the position was determined by adding the distance to the fruit, given by the laser mounted on the end-effector (Fig. 6), to the current end-effector position. The system used in the experiment lacked a sensor to accurately measure the fruit radius, but even with such sensor it can be difficult to estimate the correct radius due to occlusion of parts of the fruit. Peppers in a greenhouse have a natural variation in size but normally have an average radius r = 0.04m [4,38], so this value was used for the calculations in Eqs. 1-3. All fruit locations were added into the system in form of spheres so that the Fig. 6 The position of each pepper was measured by manually moving the robotic arm close to the fruit and using the laser to estimate the distance to the fruit (the end-effector used in the greenhouse lacked the suction cup seen in the picture) robot could avoid them when planning a path. Next, an approach cycle was performed five times for each fruit; once for the single approach strategy and four times for the multiple approach strategy, one for each potential approach direction (Table 1). Since ripe peppers are in high color contrast with the foliage and branches around them, color is the most useful visual cue. Therefore the visual servoing used for each approach attempt employs a color based blob detection algorithm for the continuous detection of the target. Minimum and maximum thresholds of the blobs' bounding and inner circles radii were introduced to limit false positive artefacts. Different visual servoing distances were evaluated in this experiment as opposed to a fixed distance in the lab, as well as a different number of approach directions and their order. The list is ordered in two possible ways: center first (CF) or left first (LF) and can have either 3 or 5 directions. Previous results [33] showed no significant difference for different robotic maximum speeds, therefore each configuration is performed at 25% of maximum speed to assure safe operation in the field.

Measurements
At the end of each approach attempt, the following measures were registered: the result of the attempt (success/failure) and the reason of any attempt failure (collision, planning failure, or lost fruit from sight during visual servoing). For each fruit, additional measures were registered: the fruit approach cycle time and the number of attempted approach directions. Fruit approach cycle time is the time it takes from when the robot starts moving to the first waypoint of a fruit until it has been marked as reachable or unreachable. Additionally, the following measures were calculated for each approach strategy and are presented in form of descriptive statistics in the results section: -Cycle time relative increase. Defined as the ratio between the average fruit approach cycle time for the multiple approach strategy and the single approach strategy. It measures the impact on the fruit approach cycle time given an approach direction known a-priori.
-Ratio of reachable fruits in each approach strategy. Allows a comparison between the success ratio of each approach strategy. -Ratio of successfully approach attempts. Allows insights into which approach directions are most successful. -Approach attempt failure ratio. Provides insights into the reasons an approach attempt fails and the frequency at which failures occur.
Additionally, the statistical significance of the differences in the value of the measures was calculated. The fruit approach cycle time is analysed in a box-cox transformed linear regression [36]. The reachable fruit rate as function of approach strategy and visual servoing distance is analysed in a logistic regression [23].

Failure Analysis
To be able to investigate the reasons leading to unsuccessful approaches, the following failure analysis methodology is presented. The failure analysis looks into the reasons why an approach strategy fails, resulting in a fruit being marked as unreachable, as well as the reasons for failure of individual approach attempts in the multiple approach strategy. The main failure reason this paper addresses is related to occlusion, but one should consider the tight space between plant lanes in which the robot has to operate in a safe manner, minimizing the harm caused to the plants around. Therefore, other failures, caused by the planner, are considered and measured. As mentioned in Section 3.2 the visual servoing failures are registered at the end of each unsuccessful approach attempt. The failure reasons are then separated into three categories: -Occlusion related failure. Cases where the fruit was lost from sight during the visual servoing stage. -Visual servoing planning failure. Cases where the next coordinate generated in the visual servoing path planner cannot be reached due to physical constraints of the manipulator, collision between the robot and that the environment, or the planner fails to find a solution. -Approach waypoint planning failure. Cases where approach waypoint W ij is unreachable due to the physical constraints, collision between the robot and the environment, or that the planner fails to find a solution.
Looking into the different reasons causing an approach to fail is important from two aspects. First, it provides insights into the reasons for failure due to non-occlusion related issues. It also provides an estimation of the expected success rate of an approach attempt that has been made with no prior knowledge about the best approach direction as in the multiple-approach strategy. The failure analysis presents the approach attempt failure ratio divided into the three categories listed above. It also provides analyses of the difference in success rates for different approach directions with and without planning related failures.

Overview Images Evaluation Protocol
To evaluate the human ability to provide insights into possible approach directions θ i or limiting the number of directions that should be attempted by determining directions that are not approachable, two datasets containing images taken at the overview waypoint W 0 were collected: 1. DB-LAB -Laboratorial images acquired at the overview waypoint W 0 in the laboratorial experiment described in our previous publication [33]. 2. DB-Greenhouse -Greenhouse images acquired midseason in a commercial greenhouse in Ijsselmuiden, Netherlands using the same camera as in DB-LAB mounted on a Fanuc LR Mate 200iD/7L manipulator. The images were automatically acquired from 14 waypoints [26].
Images from the greenhouse experiment described in Section 3.2 were not used for the evaluation due to the excess number of non-visual servoing related failures. Fig. 7 Example lab image of a fruit from three different viewpoints given to the questionnaire participants. They were told that the left and right viewpoints are reachable, while the center one is not A questionnaire was handed out to 13 human observer participants and included images from the two datasets, divided into two parts: 1. Laboratory part: contained 6 images: one example image (Fig. 7) and five from DB-Lab (one of them is presented in Fig. 5). 2. Greenhouse part: contained 6 images from DB-Greenhouse. The images were randomly selected with the following limitations: a) image was taken in daytime; b) at least 2 ripe peppers were visible from front view. One of the images is presented in Fig. 8.
The fruits in the images displayed to the participants were bounded by red boxes. The participants were asked to mark all directions (if any) from which, in their opinion, each pepper could be successfully approached (left/front/right). According to the instructions a pepper must be visible in the robot camera for a successful approach. They were given an example image (Fig. 7) with views of one fruit from the laboratorial setting from the three potential approach directions. Even if a fruit is partly occluded by a stem it should still be regarded as approachable, as seen in Fig. 7 (right). By purpose the participants were not given a definition of how visible a fruit must be to be approached. The participants were also asked to indicate if they had any previous experience working in greenhouse environment or doing research within agricultural robotics. By processing the answers an observed approachability list AP ij k which defines for each fruit i marked in the overview image if it is reachable from the approach direction θ j , by the vote of observer k is generated. Given the results of the approach attempt performed in Fig. 8 An overview image from a greenhouse scene. The questionnaire participants were asked to determine all possible approach directions for the peppers marked with red boxes the laboratory a ground truth approachability list AP * ij is defined as the rounded ratio of successful attempts for fruit i from approach direction θ j , to the overall number of attempts performed for that fruit. Given the ground truth approachability and the observed approachability list AP ij k , the contribution of a human participant's ability, from an overview image only, to predict if an approach direction will lead to a successful approach is evaluated. The evaluation is performed using precision and recall measures, defined as follows:

Approach Strategies Cycle Time and Success Rate Comparison
The in-field experimental protocol yielded 150 fruits that were attempted to be approached in eight scenes with 2-6 peppers in each scene. The platform was placed in an aisle orientated approximately southwest -northeast. A total of 18 peppers facing the aisle approximately northwest, divided in five scenes, were approached from 9 cm and a total of 12 peppers facing the aisle approximately southeast, divided in 3 scenes, were approached from 15 cm. This creates a balanced dataset aimed to cancel the influence of the foliage growth direction (due to sunlight direction) on approachability. Each pepper was attempted to be approached five times, one for each approach strategy described in Section 3.2. In total, 244 approach attempts were made. The total ratio of reachable fruits was 86%. 84% and 87% of the fruit are reachable in the single approach strategy and in the multiple approach strategy respectively ( Table 2). As seen in Fig. 9, the most approachable direction θ i was 90 • , with 74% successful approaches made from that direction. Very few approaches were made at 135 and 155 degrees, implying that the fruit was successfully approached at one of the other approach directions (right was always tested last in all approach strategies). Removing all cases where the robot failed to plan a safe path or collided with another fruit in a cluster during approach shows that 125 • is the most approachable direction (although only approached 7 times) with 90 • the second best (approached 93 times).
Comparing performance of the single approach strategy to the multiple approach strategy revealed that the fruit approach cycle time increased when a multiple approach strategy was applied by 8% for reachable and by 116% for unreachable fruits respectively ( Table 2).
In 16% of the cases for the single approach strategy, the fruit was unreachable even though it was assessed manually in-field as the best approach direction (see Section 4.1.1 for an analysis of failure cases). For the single approach strategy, the fruit approach cycle time of unreachable fruits was 33% shorter than for the reachable fruits. In the multiple approach strategy, the fruits were unreachable in 13% of the cases and the fruit approach cycle time was 33% longer for unreachable fruits than for reachable fruits. Table 3 shows the average fruit approach cycle time in seconds for the different approach strategies described in Table 1 for the two visual servoing distances (9 and 15 cm). The single approach strategy yielded the shortest overall time for 9 cm visual servoing distance with 13% shorter time than the next best strategy, CF3. At 15 cm approach distance, the CF3 and single approach strategy took approximately the same amount of time on average. As can be seen in Fig. 10, CF3 and CF5 needed on average only one approach to succeed, meaning that most of the time it only needed to go to the center position without needing to try another direction. The LF5 method needed most trials, with an average of two approaches before success. Figure 11 shows the success rate for the different approach strategies (Table 1) for the two visual servoing distances in the greenhouse. In general, the success rate was between 75-95% with slightly lower success when approaching from 15 cm than when approaching from 9 cm, with the exception of the LF5 strategy. Using five approach directions yielded slightly higher success rate than when using three approach directions. Fig. 9 Rate of successfully approach attempts by approach direction θ i , and the number of times each approach direction was attempted. In the right bars, cases where the robot failed to plan or collided with other fruits in the scene were removed Table 3 Average fruit approach cycle time in seconds for the different approach strategies described in Table 1

Failure Analysis of Greenhouse Data
Of the total 244 approach attempts, 30 attempts were part of a single approach strategy and 214 were part of the multiple approach strategy. Forty-seven percent (115 approaches) of the approach attempts failed, indicating the importance of determining a correct approach. The most common reason (41%) for failure was due to the robot planner not being able to find a safe path to the fruit during visual servoing or while moving to the next approach waypoint. 37% of the failed approach attempts were because the fruit was lost from view along the visual servoing. The remaining 22% of the failures were due to a collision between the robot and another fruit in the scene. Figure 12 presents the ratio of all approach attempts outcomes for the two approach strategies separately. Approach attempts made from an approach direction obtained by manual in-field assessment as part of the single approach strategy was successful in 84% of the cases. In 10% of the unsuccessful approaches the fruit was lost during visual servoing (i.e. the fruit was no longer detected) while the remaining cases (6%) were a result of a collision between some part of the robot (usually the end-effector) and another fruit in the cluster or that the planner could not find a safe path while moving to the next approach waypoint. Looking at each individual approach attempt as part of the multiple approach strategy, only 49% of the approaches were successful, while 18% were lost in visual servoing and 10% were lost as a result of planning failure during visual servoing. While moving to the next approach waypoint, 23% of all approach attempts failed due to planning failure or collision. Figure 9 presents the approach success rate by approach direction θ i . Removing all the cases where the robot failed to plan a safe path or collided with another fruit in the cluster during approach shows that almost no planning related failures happened at 90 • (81% compared to 74%) while the side views caused a significant amount of planning failures, increasing as function of |θ i − 90| (e.g., 65% compared to 31% for 25 • ). This supports the assumption made in Section 2.2 that approaching from front 90 • should be preferred as long as this approach is not occluded. The low rate of planning related failures for 90 • can be explained by the fact that this scene contains much fewer fruits and other obstacles from the front approach direction, making the planning task less complex and the risk of colliding with obstacles lower. When side approach directions are applied (|θ i −90| > 0) the robotic manipulator needs to reach deeper into the plant making it more likely to cause collisions between the robot and other fruits. Additionally, it seems the approach directions from the right (θ i > 90 • ) fail more often due to planning than from the left. Important to note that these differences are not due to any botanical changes Fig. 10 Average number of approaches till successful fruit approach for the different potential approach directions of the multiple approach strategy described in Table 1 for two visual servoing distances Fig. 11 Success rate for the different approach strategies described in Table 1 for two different approach distances Fig. 12 Ratio of all approach attempts outcomes for the two approach strategies separately. Lost in VS means the fruit was no longer detected while doing visual servoing. Failure occurred due to not finding a safe path or due to collision between the end-effector and a fruit since these are planning failures, which are not connected to the growth of foliage in greenhouse conditions. Since much fewer attempts were made from the right this needs to be verified in future work.

Approach Cycle Times and Fruit Reachability Analyses
The results of a Box-Cox transformed (λ = 0.44) fruit approach cycle time regression shows statistical significant (sig < 0.0001) differences between the multiple approach strategies (CF and LF) and the single approach strategy. The multiple approach strategy adds on average 3.5 seconds to the fruit approach cycle time as compared to the single approach strategy. No statistical significant differences were found between the CF and LF approach direction sorting (sig = 0.72). The number of approach directions were not statistically significant either (sig = 0.18). The visual servoing distance was found to be a significant factor (sig = 0.001), increasing the fruit approach cycle time by 0.28 seconds for each extra cm of visual servoing. The approach cycle time of a reachable fruit was on average 2.4 seconds longer than an unreachable fruit. Significant differences in approach cycle time between scenes were found, implying that additional variations between the scenes also affects the cycle time. This requires further investigation on larger datasets.
Looking into the interaction between the approach strategy and if the pepper was reachable or unreachable (Fig. 13), reachable fruits yield shorter approach cycle times than unreachable fruits for the multiple approach strategy. In the single approach strategy, unreachable fruits yield shorter approach cycle time than reachable fruits. These  Table 2 and found statistically significant (sig = 0.001).
The result of a logistic regression on reachable fruit rate as function of approach strategy and visual servoing distance revealed no statistically significant differences. This might be due to the relatively small dataset gathered in the greenhouse experiment.

Human Contribution to Approach Direction Assessment
In both the greenhouse and the previous laboratory experiments, the average fruit approach cycle time increased for the multiple approach strategy as compared to the single approach strategy using in-field assessment by human observer. In this section the human ability to determine the best approach direction from overview images is evaluated.
All 13 participants answered all questions. Four participants had prior experience of work in greenhouse conditions and 9 never worked in greenhouses before.
In laboratorial images collected, the average precision per person (Eq. 4) was 92% (min = 85%, max = 97%) and the average recall per person (Eq. 5) was 71% (min = 40%, max = 83%). No significant differences were found between the four participants who had prior work experience in greenhouse conditions, to the nine who were not. As aforementioned, the precision and recall on the greenhouse data cannot be calculated due to the lack of ground truth information. Instead, the evaluation includes investigation of the degree of agreeability between the participants, e.g. if all answered front as an approachable direction for fruit i. Fleiss' Kappa for the laboratorial images is 0.514, corresponding to moderate agreement level, and for the greenhouse collected data was 0.217 which corresponds to a slight-to-fair agreement. This is a significant reduction in the inter-rater agreement between participants which indicates that as expected the greenhouse task is by far more difficult to analyse. A closer look into the measured Kappa as a function of approach direction and the participants work experience in greenhouse conditions can be found in Table 4.

Approach Strategies Experiments
The results from the greenhouse experiments support previously reported results [33] indicating an increased fruit approach cycle time when using the multiple approach strategy as compared to a single approach direction yielded from manual-infield assessment of human observers. Therefore, additional information about approachable directions is necessary to shorten robotic harvesting times. The data from the greenhouse experiment showed a less prominent time increase (ca 8%) compared to the previously published laboratorial experiment (ca 40-45%). Even so, it was found to be statistically significant. Longer visual servoing distances was also found to increase fruit approach cycle time. On the other hand, the number of approach directions (3 or 5) or their order in the multiple approach strategy did not significantly influence the fruit approach cycle time. This validates the conclusions reported from the laboratorial experiments. While the robot eventually managed to approach all peppers in the laboratorial experiment, in the greenhouse each approach strategy resulted in only between 75-95% approachable peppers. The most successful approach direction was from front, supporting the decision to prioritise it in the single approach strategy. Failure analysis showed multiple reasons for failure during an approach attempt. The most common failure (41%) was the inability to plan a safe path in the tight space between plant lanes and stems in the greenhouse. Another reason for failure was that the end-effector collided with another fruit in a cluster while moving towards the next approach viewpoint. This failure is expected to happen much less often in a real application where the peppers are harvested and thereby would not be in the way for the robot when moving towards the next fruit. Only 37% of all failures were due to fruit loss from view due to occlusion during visual servoing towards the fruit. In ideal conditions, one would expect this to be the only reason for not being able to approach the pepper. This is also the only failure one could expect a human observer to predict in field or by observing overview images. If we remove all other reasons for failure (73 cases of the total 244 approach attempts), the robot not reaching the fruit due to occlusion only occurs in 25% of all approach attempts.
Results showed slightly lower reachable fruit rate for the single approach strategy than for the multiple approach strategy. However, no statistically significant difference could be seen in the greenhouse data, probably due to the limited size of the dataset. Since only one approach attempt is made per fruit in single approach strategy, slightly lower fruit reachability is to be expected since it cannot attempt any other approach directions if the first was unsuccessful. Furthermore, the non-occlusion related failures mentioned above were not considered when best approach direction was chosen by an in-field human observer for the single approach strategy.

Human Contribution
The evaluation of the human ability to assist in determining the best approach direction in laboratorial conditions revealed high precision (92%), implying that the participants rarely predicted non approachable directions to be approachable. Consequently, the participants showed a high ability to detect unapproachable directions. On the other hand, the 70% recall suggests that they tend to miss possible approach directions. The implications of these conclusions are that one should consider humans for reducing the list of possible approach directions used in the search pattern, but should not rely on a human operator to define the best approach direction. The greenhouse experiment showed that an increased number of approach directions does not significantly increase fruit approach cycle time. The implication of this is that the system would be best served by letting humans determine which peppers are unreachable and thereby saving a considerable amount of time, especially since the multiple approach strategy takes significantly longer time to fail. A deeper view into the values of the kappa for front/side approach directions reveals significant agreement on the front direction in laboratorial conditions, and lower agreement on the side views. This implies that a human finds the side view to be more complex for approachability prediction. This difference was not found in greenhouse conditions, where the agreement level remained low for both front and side approach directions. In greenhouse conditions though, participants with prior experience in agricultural robotics or greenhouse environments showed greater agreement among themselves compared to the nonexperienced participants. No such difference is seen in the laboratorial setting. The results imply that though no significant difference was found in precision and recall in laboratorial conditions for experienced and non-experienced participants, these differences might become significant when the experiment is performed in the more complicated greenhouse conditions where experience can become advantageous.

Limitations
This work aimed to evaluate the human's ability to determine that a pepper is not approachable at all by a robot. As concluded in Section 5.2 the laboratorial tests showed that the humans are quite good at this. The data from the greenhouse experiment described in Section 3.2 was unfortunately not sufficient to generalize these results to greenhouse conditions due to multiple non-occlusion failures as noted in Section 5.1. Future work with a more optimal harvesting system should reduce these failures and allow for a fair comparison.
In this work, stems were not avoided unless they covered significant parts of the fruit and thereby causing the visual servoing to lose the fruit from sight. Future work on selecting approach directions should incorporate stem detection in the robotic harvesting cycle to avoid stems that are between the fruit and the approach waypoint in order not to damage them. Some initial work on detecting and avoiding stems has been done [3] and could be extended to be used in this application in the future.
Sunlight is a significant factor in the botanical growth models of the leaves. As mentioned in Section 4.1, the protocol aimed at creating a balanced dataset to cancel the influences of growth direction of the foliage due to different sunlight direction. The results showed that the vast majority of the successful approaches were for θ i <= 90, regardless of side of aisle (and thereby sunlight direction). One would expect that if the sunlight direction was a factor in approachability, there would be a difference in which angles are more successful for different sides of the aisle. The results showed no significant differences in which directions were successful and not between the different sides of the aisle, indicating that the sunlight direction did not matter. However, since the visual servoing distance were different for the different sides of the aisle it is not possible to prove explicitly that sunlight did not play a part.
Current detection rates (up to 85% [4,9,24]) allow evaluation of the approach direction strategies independent of detection performance. Any effect of sunlight direction (or other environmental conditions) on the visual servoing performance is not addressed in this paper (e.g. if the fruit was lost from sight due to the sunlight).

Conclusions
The main contribution of this work is to suggest and compare approach strategies, while measuring cycle times and success rate as well as analyzing causes of approach attempt failures. For the proposed greenhouse environment and hardware configuration it was shown that a multiple approach strategy results in 8% longer cycle time than a single approach strategy and that the most common failure was the inability to plan a safe path in the tight space between plant lanes and stems in the greenhouse. The exact measures might vary between different harvesting systems, but the evaluation method of different approach strategies can be applied to any robotic system using visual servoing in highly cluttered environments.
The issue of human in the loop is a question that is often raised in the literature as a possible alternative to fully autonomous systems which have limited performance. The slight increase in fruit approach cycle time in the multiple approach direction strategy is highly unlikely to justify the inclusion of a human observer in the field to assess the best approach direction. Therefore, it is important to develop algorithms that will be able to provide this information in shorter times and lower costs. For other hardware configurations and greenhouse environments, the increase might be significant enough to justify a human in the loop.
To identify these cases one should consider comparing the different approach strategies using the methods proposed in this work.
Since the ability to detect fruits that are unreachable from overview images can significantly shorten the overall harvesting cycle time methods should be explored to determine this. More research is needed to investigate the times it takes for a human to detect unreachable fruits in order to analyse the economics of human robot collaboration for the proposed tasks.
Given the high failure rate of the single approach direction due to non-occlusion related failures, one should consider a hybrid approach strategy. This strategy would start with using the single approach strategy, since it yields the shortest cycle times. If the direction assessed to be the best one fails, the multiple approach strategy can be applied to search for a possible approach direction. The hybrid approach aims to increase the ratio of reachable fruits and should be investigated further. If no information about the best approach direction is available, the multiple approach strategy should be considered to lower the rate of unreachable fruits. If a fixed approach direction is the only applicable strategy then the proposed approach direction for sweet pepper harvesting is from the front (90 • ). appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.