Fleets of robots for environmentally-safe pest control in agriculture

Abstract

Feeding the growing global population requires an annual increase in food production. This requirement suggests an increase in the use of pesticides, which represents an unsustainable chemical load for the environment. To reduce pesticide input and preserve the environment while maintaining the necessary level of food production, the efficiency of relevant processes must be drastically improved. Within this context, this research strived to design, develop, test and assess a new generation of automatic and robotic systems for effective weed and pest control aimed at diminishing the use of agricultural chemical inputs, increasing crop quality and improving the health and safety of production operators. To achieve this overall objective, a fleet of heterogeneous ground and aerial robots was developed and equipped with innovative sensors, enhanced end-effectors and improved decision control algorithms to cover a large variety of agricultural situations. This article describes the scientific and technical objectives, challenges and outcomes achieved in three common crops.

Introduction

The European Union uses approximately 360 million kg of pesticides per year for agricultural and horticultural tasks (PSS 2015). Assuming that the return in crops saved is approximately four times the investment (Pimentel 2005), these products clearly produce a major economic benefit. However, such assessments do not consider the indirect but important environmental and economic costs associated with pesticide use, which are estimated to total approximately 10 000 million dollars per year in the US (Pimentel 2005). This heavy cost for local and national administrations could be reduced by diminishing pesticide use, which would, in turn, reduce the cost to farmers. The use of precise management and control technologies in agriculture can provide substantial savings of herbicides and pesticides by making use of newly available technologies such as global navigation satellite systems (GNSS), geographic information systems (GIS), automated agricultural machinery, high-resolution image systems, sophisticated sensors, automatic control and robotics. A modern approach is to use existing information and communication technologies (ICT) to design and build improved pest and crop sensors, enhanced actuators and mobile robots to perform proper pest control. Specifically, in recent decades, several attempts to build agricultural autonomous systems for implementing precision agriculture techniques have been carried out and tested under real-world conditions. These efforts have followed different approaches. The first approach has consisted of integrating modified commercial agricultural vehicles and intervention mechanisms (Pilarski et al. 2002; Blackmore et al. 2004; Nørremark et al. 2008; Johnson et al. 2009; Bergerman et al. 2012; Kohanbash et al. 2012; Moorehead et al. 2012). An illustrative example is the work described in Nørremark et al. (2008), which developed an unmanned self-propelled hoeing system for intra-row weed control consisting of an autonomous vehicle based on a commercial agricultural tractor and a cycloid hoe.

A second approach has developed new robot structures (individual steering/traction in the wheels, electrically powered vehicles, adjustable ground clearance and track widths, etc.) and integrated intervention mechanisms for specific agricultural applications. Some of these developments will be operative in the coming years (Bakker et al. 2010; Ruckelshausen et al. 2009).

One more approach has aimed to build groups of small vehicles under unified control because they can provide clear advantages over existing equipment (Blackmore et al. 2005; Peleg 2005; Cheung et al. 2008; Sørensen and Bochtis 2010). For example, small vehicles ensure higher positioning accuracy during operation and are intrinsically lighter than big machines. This last feature reduces the soil compaction and makes the vehicles safer in terms of safety to others, own safety and crop safety (Blackmore et al. 2001). However, small robots manage smaller payloads, and thus smaller agricultural tools, than do big machines. Therefore, several small robots have to work jointly to accomplish the work of a traditional machine in the same time. This raises the concept of fleets of robots with additional advantages regarding price (it allows farmers to get high-technology equipment in an increasing manner), fault tolerance (failure in a small robot means one less robot at work, while failure in a big vehicle means the whole process on the field is stopped), mission co-ordination and reconfiguration (at any time the fleet behavior can be changed to optimize the mission, taking into account sudden changes in field conditions), etc. (Emmi 2014). The theoretical foundations of fleets of robots have already been investigated (Bautin et al. 2011; Bouraqadi et al. 2012), but the first real tests for applications of precision agriculture techniques have been conducted recently (RHEA 2014). Thus, this article describes one of the works carried out to check fleets of robots.

Within this context, a consortium of 19 experienced multi-disciplinary groups from 15 institutions/companies (3 research centers, 4 universities, 7 small and medium-sized enterprises and one large company) was formed to build, conduct experiments with and evaluate a new generation of automatic systems and fleets of robots for chemical and physical (mechanical and thermal) weed management and tree crop spraying. The project Robot Fleet for Highly Effective Agricultural and Forestry Management (RHEA) was envisaged to change traditional agricultural and forestry production by providing smart, autonomous commercial robots to farms, thus promoting the evolution toward fully automated operations and approaching the concept of the “farm of the future.” The system involves a number of dissimilar ground and aerial robots (Fig. 1) endowed with advanced sensors, innovative end-effectors and improved decision control algorithms that were developed to reduce the use of chemical inputs in agriculture and forestry and to decrease production costs while improving crop quality, health and safety to both humans and animals. Such a large, interdisciplinary project was funded by the European Union under the Seventh Framework Programme.

Fig. 1

The RHEA fleet: UGVs and UAVs

The project involved the design and development of specific new automatic systems (agricultural tools), mobile robots (aerial and ground) and related equipment (machine vision, communications and location equipment, safety systems, human–machine interfaces, fleet management, etc.), as well as their holistic integration, to accomplish the main tasks of observation, decision making and implementation on croplands to reduce the use of pesticides while improving efficiency.

To achieve the main goal, the activities were organized around a number of specific technical and scientific objectives dedicated to the development of the following:

• (1) Automatic weed mapping systems based on computer vision targeted to detect approximately 90 % of weed patches.

  The accuracy of vision systems depends on external conditions, which vary considerably in natural environments. Thus detecting 90 % of weeds was considered a reasonable scientific challenge and a profitable figure from the agricultural stand point.

• (2) Decision-making algorithms comprising those specific for co-ordinating the actions of the robots and making them co-operate and collaborate in achieving the tasks. These algorithms include innovative strategies for efficiently scanning terrain with fleets of robots as well as for re-planning the mission after detecting any contingency.

  The outcomes achieved on these topics are detailed in Conesa-Muñoz et al. (2012) and Emmi et al. (2014).

• (3) Improved agricultural tools for precise real-time treatment to reduce chemical inputs. Three agricultural tools were developed:

  • (i) Patch sprayer—The target was to reduce herbicide use by approximately 75 %.

    Crop fields with weeds covering approximately 25 % of the field area (average-case scenario) need only 25 % of the traditional methods and thus they can save up to 75 % of the herbicide. Nevertheless, this percentage decreases when weed coverage increases.

  • (ii) Canopy sprayer—The target was to reduce approximately 50 % in the use of pesticide in canopy spraying.

    Due to the nature of this sprayer (based on only 4 position-controlled nozzles per side) a reduction of 50 % in the use of pesticide was estimated as an achievable target and economically interesting.

  • (iii) Mechanical/thermal tool—the target was to destroy 90 % of the detected weeds.

    This implement based on liquefied petroleum gas does not leave inputs to the ground and an efficiency for killing 90 % of the detected weeds was selected as an achievable target and attractive enough from the agricultural stand point.

• (4) A fleet of aerial robots (unmanned aerial vehicles, UAVs) equipped to acquire images of the task field for post-flight processing.

• (5) A fleet of ground robots (unmanned ground vehicles, UGVs) equipped to apply physical or chemical treatments for pest and weed control (agricultural tools). The robots were equipped with

  • (i) GNSS.

  • (ii) Ground perception systems to discriminate weeds from crops in real time, allowing the robots to follow crop rows; the aim was to accurately steer the robots to work on wide-row crops (with 0.75 m-spaced rows).

  • (iii) Real-time tree canopy detection systems.

• (6) Graphical user interfaces for monitoring autonomous outdoor vehicles.

• (7) Safety systems for humans and animals near the vehicles and the robots themselves (obstacle detection in the robot’s path).

  Although the main aim of the RHEA project was to check fleets of robots and all software applications were developed to manage a large number of robots, the limited budget and limited duration of the project forced the consortium to build only 3 UGV and 2 UAV. Thus, in this article, the real RHEA robots will be just referred to as a multi-robot system.

Materials and methods

Study site

The RHEA entire system was tested in maize (Zea mays L.), wheat (Triticum aestivum L.) and olive trees (Olea europaea L.), and the experiments were performed in two fields located in Arganda del Rey, Madrid, Spain, (40º18′ 50.241″, −3º29′ 4.653″ for wheat and 40º18′ 57.924″, −3º29′ 3.7134″ for maize and olive trees). These fields are inside the facilities owned by CSIC (RHEA project co-ordinator).

Experimental data were recorded with elemental sensors (ultrasonic sensors, encoders, etc.) and sophisticated devices (GNSS, laser), measuring mainly positions (m) and volumes (l). The recorded data were analyzed based on average values.

System architecture

To accomplish these specific objectives, the RHEA multi-robot system was broken down into seven main systems organized into two parts: stationary equipment and movable equipment (Fig. 2). The stationary equipment contains the systems and devices allocated to fixed positions during the mission, close to the working field. All of these elements (antennas, ethernet switches, routers and receivers) are physically installed in the base station (BS), which consists of a cabin that provides housing for the equipment and shelter for the operator; the BS is the operator’s work station. The BS is supplied with AC voltage and equipped with a computer to which the stationary systems are connected and the relevant software applications are run. The BS computer allows the operator to interact with the pertinent systems and modules through the Graphical User Interface (GUI), to define missions and control the multi-robot system through the Mission Manager. This software module computes and supervises the missions, controlling the main components of the movable parts, i.e., the UGVs, UAVs (through the UAV High-Level Controller) and related equipment, whenever needed. Figure 3 illustrates the BS system architecture.

Fig. 2

RHEA system architecture

Fig. 3

Base station and system architecture

The mobile robots (unmanned vehicles) are the elements of the movable part responsible for providing motion over the entire field for sensing crops and acting on them. Thus, the unmanned vehicles carry the Perception System through the fields. This system is composed of the Aerial Remote Perception System shipped on the UAVs and the Ground Perception System on board the UGVs.

The RHEA UAVs are the result of an innovative design based on a hex-rotor drone, whereas the UGVs are ground robots built on a commercial tractor chassis that is capable of carrying the actuation equipment (agricultural implements), automatic machines that perform physical (mechanical and thermal) and chemical (spraying) pest control in three different crops.

The geographic positions of the UGVs and UAVs are obtained with the Location System based on GNSS technology. Thus, the Location System provides every vehicle with its current position and also provides the Mission Manager and GUI with the positions of all vehicles. Therefore, the Location System has components in both the stationary (base antenna and receiver) and movable (vehicle antenna and receiver) parts.

The stationary and movable equipment are interconnected by the Communication System, which is based on several Wi-Fi technologies. This system is also distributed and provides the BS and the vehicles with the base antenna and the vehicle antennas, respectively.

RHEA is a complex system that requires intensive use to make it economically feasible. This efficiency can be accomplished using the contractor approach, i.e., a specialized company provides the requested services to individual farmers in different fields. Both stationary and movable equipment are moved from field to field using tracks or autonomously if the local legislation allows it. The primary goal of the RHEA project was to design, develop and integrate all of these systems with their relevant modules, as detailed in the following sections.

Unmanned vehicles

Unmanned aerial vehicles

The design of the new UAV was focused on fulfilling the requirement of carrying the Remote Perception System (1.5 kg) for approximately 30 min. The final drone, AR-200, was envisaged as an extension of the commercial quad-rotor AR100-B, manufactured by participant AirRobot, Arnsberg, Germany, (Fig. 1). Table 1 summarizes the drone’s main features.

Table 1 AR-200 main characteristics

The UAV high-level controller was developed in C++ on a Windows operating system to be executed on the BS computer and integrated with the GUI (Figs. 2, 3). This controller drives the drones during their flights through their respective UAV Ground Control Consoles, i.e., devices used by the backup pilot in the remotely controlled flight mode (Valente et al. 2013). The high-level controller translates the mission definition created by the aerial planner (a series of waypoints for each drone) to the specific protocol used by the drones, and, therefore, is responsible for creating the correct commands for the drones to execute the complete mission. This modular solution provides high flexibility in cases of changing drone protocols. The UAV high-level controller is also responsible for performing real-time alarm management, obtaining telemetry data from the aerial vehicles and translating data into meaningful physical variables.

Unmanned ground vehicles

The ground vehicles were based on a commercial vehicle chassis instead of a tailor-made robot structure. The selected chassis was the Boomer-3050-CVT (manufactured by project partner CNHi, Zedelgem, Belgium), illustrated in Fig. 4 along with the rest of the UGV sub-systems. This chassis exhibits the features detailed in Table 2. Using a long-tested, commercial platform to configure a mobile robot ensures:

Fig. 4

UGV and sub-systems on-board

Table 2 Characteristics of the Boomer-3050-CVT chassis

• Greater reliability: these vehicles have been tested and improved over a long period of time, whereas robotic prototypes are always prone to malfunctions and thus require modifications throughout the development of the project.

• Improved robustness: prototypes are weak systems when facing different working conditions not considered during the design phase.

• Easier standardization: the proposed vehicles already fulfilled a large number of standards and had already been homologated for many tasks.

• Earlier availability: the delivery of the UGVs could be advanced by nine months with respect to the initial working plan, allowing the participants responsible for providing the sub-systems to be on-board the UGV to greatly advance their developments, integration tasks and tests.

• Ease of expanding/modifying the system: small mobile robots are designed with a very limited payload, jeopardizing the inclusion of additional sub-systems; however, the proposed vehicle had a large payload/weight ratio, allowing the designers to modify the system without the serious limitation of sub-system weight.

• Ease in facing unforeseen problems: for example, those derived from the adaptation of the vehicles to specific requirements appearing from different sub-systems, such as the type and dimension of the UGV wheels and tires, the length of the wheel shafts (to cover two or three crop rows underneath the vehicle), extra payloads, etc.

Nevertheless, this solution impacts safety: heavier vehicles carrying heavier loads increase the risks linked to a mobile system. Thus, the safety system had to be designed accordingly.

The three commercial vehicles (Fig. 1) were mechanically, electrically and hydraulically modified. A subset of the ISOBUS CAN (ISOBUS 2011) protocol was used to communicate between the vehicle sub-systems (transmission, steering, brakes, valves, etc.) and the UGV controller responsible for handling trajectory control. An electro-hydraulic steering system and some electro-hydraulic valves that power the implements were installed to be controlled by ISOBUS CAN messages. Two DC actuators were installed to control the engine RPM and the height of the three-point hitch (the device where the agricultural tool is attached). A 12 V 200 A alternator and a 24 V 120 A alternator were mounted on the vehicles to supply the required power.

A touchscreen with two CAN ports, one connected to the higher level controller (ISOBUS), the other one to the vehicle CAN bus (SAE-J1939 2013), serves as the gateway. This device controls the handshaking procedure defined in the ISOBUS protocol to hand over the control of vehicle functions to the external controller (ISOBUS 2011). The display also provides a tactile user interface to manually control the different functionalities of the tractor such as steering, three point-hitch position and auxiliary valves.

The transmission controller software was modified to allow electronic control of the transmission instead of the original control using the throttle pedal. A new PID controller was designed to follow the speed set point received from the trajectory controller. This controller was also modified to allow activation of the system that provides the agricultural tool with power [power take-off (PTO)] and selection of the gears by CAN messages.

The High-Level Decision Making System is responsible for co-ordinating all the systems on board the UGV and making decisions on future actions. The hardware architecture for the HLDMS was designed, implemented and assessed relying on a compactRIO 9082 computer (National Instruments Corporation, Austin, Texas, USA) and the software structure consisted of a number of modules and functions, organized as illustrated in Fig. 5. This structure defines the communication (in terms of commands, responses and related parameters) between the HLDMS and every other sub-system connected to it. In particular, the communication between the HLDMS and the Mission Manager, the Weed Detection System, the UGV Controller and the User Portable Device was designed to allow the operator to supervise and control the multi-robot system.

Fig. 5

General scheme of the functions, routines and sub-systems that constitute the high level decision-making system

Several versions of an HLDMS were tested, ranging from a version in which the HLDMS is a slave of the Mission Manager (the one used in the project final demonstration) to a version in which the HLDMS of a given robot controls the whole multi-robot system (Emmi et al. 2014).

Safety system

The safety aspects of both ground and aerial vehicles were evaluated in consecutive phases. First, a general study on current safety ISO and EN standards was conducted to determine possible connections between these standards. Then, the work was focused on each individual type of vehicle, adjusting the safety requirements to their specific characteristics. Finally, a Multi-robot Supervision System (MSS) was built to detect different failures and hazardous situations, such as tractor going out-of-track, incorrect working speeds, inappropriate implement status and potential collisions between robots.

The MSS is arranged as a distributed, multi-level system with two levels. The MSS Low Level consists of the UGV Safety System and the UAV Safety System that operate inside the UGVs and UAVs, respectively and takes care of the most urgent issues. The MSS High Level is responsible for the more complex supervision that involves the entire multi-robot system; this level is performed on an external computer that receives all of the information provided by the robots. An external device allows a human operator to monitor the underlying system and take control if needed. The proposed system performs the three main supervision functions: fault detection, fault diagnosis and fault recovery. The MSS is detailed in Conesa-Muñoz et al. (2015).

UGV safety system

In particular, the intrinsic risk associated with the UGV operations was analyzed, and the conclusions were used to install a safety controller on the UGV to achieve a higher safety integrity level for functions such as shutting down the engine, controlling the brake, controlling and monitoring the safety relays, and reading and monitoring the laser safety fields. This component is an independent controller based on a commercial programmable logic controller (PLC) that is able to stop the vehicle even in the case of failure of other controllers.

For safety, the braking system needed to be changed from a normally non-braking system to a normally braking system. Thus, an electronic brake with control of the brake state by CAN and automatic activation of the brake, in case of CAN communication failure, was installed in the UGV.

There are two safety levels in the UGVs:

• Manual safety system: The manual safety system is responsible for brake activation and engine shut down control of the UGV when any of the three emergency buttons placed on the UGVs or the emergency button on the UGV remote controller is pressed (Fig. 4). When any of these buttons is pressed, the two safety relays of the receiver onboard the UGV connected to the safety controller are switched off, and the safety controller activates the brake and switches off the engine. The remote stop function of that system is SIL3 (Safety Integrity Level 3) certified (SIL3, 2016).

• Proximity safety system: The proximity safety system was based on a range finder (laser) installed on the center of the vehicle’s front in a push–broom configuration (inclination) for the detection of obstacles along the vehicle trajectory (Fig. 4). This sub-system was connected to the PLC of the Safety System to activate the brake and stop the vehicle if an obstacle is detected in the safety zone. Intensive tests were conducted to define safety zones of variable size depending on the vehicle’s actual speed (dynamic size) and crop characteristics (fixed parameters). Additionally, many experiments were conducted to measure the lasers’ precision and adjust their configuration to real agricultural working conditions.

UAV safety system

UAVs do not rely on a proximity detection system since no aerial obstacles are expected during the flights. Thus, aerial multi-robot safety is based on creating safe flight plans that will be carried out under the supervision of the human operator required by the emerging aerial security legislation in Europe.

According to this, and considering that a multi-robot system instead of a single robot has to be managed, the safety has been entrusted to the aerial mission planner, which not only defines safety paths for the UAVs but also maximizes the distance between them during the flights. Maximizing the distance during the operation of the UAV turned out to be essential to reduce the stress of the operator.

Moreover, some operating procedures have been established in order to reduce the crash probabilities during takeoff and landing maneuvers, e.g. to ensure that landing pads are free of obstacles before ordering the landing or commanding different altitudes for approaching or leaving the working areas (where identical altitude should be maintained so as to make image resolution compatible).

Finally, it is worth noting that in addition to the safety margin that six rotors provide (i.e., the UAV is able to fly with only five rotors in a safe manner), the drone controllers have been developed considering robustness requirements in such a manner that they allow flying with no GNSS signal, although the operator is warned about this issue. As a result, this safety packet ensures safe operation of the UAVs during their operation considering human mistakes and mechanical failures.

Managing the multi-robot system

The Mission Manager is a software application developed for handling the multi-robot system. This application runs on the BS computer and is responsible for (a) generating the trajectories and actions for both aerial and ground vehicles, (b) supervising the trajectory and action executions and co-ordinating behaviors among vehicles when a conflicting situation or system malfunction is detected, (c) obtaining data from the remote perception system, and (d) interacting with the operator through the GUI (see Figs. 2, 3).

Two types of missions can be defined: (a) inspection missions carried out by the aerial vehicles and (b) treatment missions performed by the ground vehicles. At the outset of the process, the operator provides information related to the type of mission and the field specifications (crop type, field dimensions, geographical position, crop headings, etc.). This type of information allows the Mission Manager to make decisions on the number of UAVs for inspecting the mission field, to select the number of UGVs needed to accomplish the task and to provide an action plan for each vehicle (Mission Planner) (Fig. 6). During the inspection mission, the Mission Manager uses the Remote Perception System to obtain information to compute a weed distribution map, which is essential for generating the treatment mission plan in narrow-row crops. The Mission Manager also supervises the mission, making real-time decisions when unexpected events occur (Mission Supervisor) using the status information provided by the aerial and ground vehicles (mainly position, orientation, states of different modules, errors, etc.) during the execution of their missions. If the Mission Manager is unable to solve a conflict or manage a potentially dangerous situation, control of the multi-robot system is passed to the operator.

Fig. 6

Examples of plans for a UAVs and b UGVs

The Ground Mission Planner is related to the treatment mission and, for a given crop, determines the configuration of the set of ground vehicles (type and number of vehicles) as well as the plan for each of them to efficiently accomplish the treatment. The approach developed follows several optimization strategies that consider: (1) two possible criteria to be minimized: (a) the task cost, related to the travelled distance and the input costs (e.g., fuel, herbicides, labor), and (b) the time required to perform the task; (2) vehicles with different features (e.g., working speeds –both intra- and inter-field–, turning radii, fuel consumptions, tank capacities and spraying costs); (3) the variability of the field; and (4) the possibility that tanks must be refilled with herbicide or fuel during the task execution. Simulated annealing and basic genetic algorithms are used to find the optimal solution that minimizes either the task cost or the time required cost, while a NSGA-II (non-dominated sorting genetic algorithm) is employed as a proper approach for simultaneously minimizing both criteria. Figure 6 illustrates plans for both types of vehicle (Conesa-Muñoz et al. 2012, 2014).

Perception systems

The Perception System identifies weeds in the field and other elements in the robot’s path relevant to the mission (crop rows, obstacles and persons), and it is divided into the Remote Perception System, placed on-board the UAVs, and the Ground Perception System, placed on-board the UGVs.

Remote perception system

Detecting weeds when crop and weed plants are at early phenological stages and exhibit spectral and appearance similarities is a challenging objective that is overcome only by using high-spatial-resolution imagery (pixel size <0.05 m), which the current vision technology on board UAVs can provide (Lopez-Granados 2011). This high-spatial-resolution generates a high intra-class spectral variability that can be solved using segmentation by dividing the image into multi-pixel spectrally homogeneous regions named objects which are studied as the minimum information units using Object-Based Image Analysis (OBIA). The OBIA approach develops automated and auto-adaptive classification methods by combining the spectral, contextual (position, orientation), morphological and hierarchical information of objects. The detailed OBIA workflows were described by Peña et al. (2013, 2015). Three main activities are considered: first, (a) a set of overlapped images must be acquired to cover the complete field; then, (b) those images must be stitched together and ortho-rectified in a mosaic to finally (c) extract the weed patches using OBIA algorithms and taking into account the relative position of weeds to the crop lines, that is, every plant not located on the crop-row is considered a weed. Finally, a site-specific weed program based on the weed patch map is designed using a grid of 0.5 m. This grid dimension is customisable according to each cropping pattern and the specifications required by the herbicide spraying machinery. Figure 7 illustrates the procedure.

Fig. 7

Aerial unit carrying the remote perception system and sequence of operations performed over the set of aerial images to obtain the weed map of a crop

Image acquisition device

As a first step of weed patch detection, a vegetation-soil discrimination is required. To maximize its robustness under lighting conditions, a multispectral device including visible and near infrared (NIR) channels was designed. A solution that preserves the complete color information was based on the coupling of two commercial still cameras, one of them being modified to provide the NIR channel. A high-end camera combining a low footprint (compact) and very high resolution (4704 × 3136) with no Bayer matrix (Sigma DP2 Merril) was selected as the proper solution. This camera provides one-cm spatial resolution at ground level with a flight elevation of 60 m, each picture covering approximately 40 × 30 m.

Color and NIR images issued from the two cameras must be registered to provide a unique 4-channel picture. An approach based on the Fourier-Mellin (FM) transform was successfully developed and tested. This approach consists of identifying rotation, translation and scale changes between images using Fourier spectrum analysis. To cope with large-sized camera images, which imply non-linear transformations, the original images are partitioned into a set of small image portions, on which the FM identification process is repeated iteratively. A global homographic transformation model is then computed, including lens radial distortion. This procedure was implemented on the Graphics Processing Unit of the BS computer using Compute Unified Device Architecture (CUDA) libraries. A registration accuracy of 0.3 pixels was obtained (Rabatel and Labbé 2016).

Aerial image mosaicking

Due to the unevenness of the field, overlapping source images are necessary so that every point in the parcel can be observed from at least two different points of view, allowing for the creation of a 3D digital surface model. Source images can then be ortho-rectified and combined. Moreover, color targets as ground control points with known geodesic co-ordinates are necessary for the geo-referencing of the resulting mosaic (Fig. 7).

The mosaicking process was implemented using an open-source package (MicMac software, IGN) and a few software modules developed to perform the automation, including picture synchronization with the flight log, automatic detection and labeling of ground targets around the parcel for geo-referencing, communication with the base station, and other modules (Rabatel and Labbé 2015). The resulting geo-referenced, 4-channel image of the parcel with a resolution of one cm is the input for the weed patch detection module.

The resulting georeferencing accuracy mainly depends on the quality of initial georeferencing of ground targets, which is a manual operation (a RTK GPS receiver is successively positioned on each target). According to ground target co-ordinates in the resulting image, an absolute accuracy of a few centimeters is obtained, which is sufficient for further weed patch detection and retrieval. The important point is that this limited accuracy does not jeopardize the quality of multispectral data, because the multispectral registration has been made before (with sub-centimetric accuracy) and does not depend on it.

Weed patch detection

Two OBIA algorithms corresponding to weed patch detection in maize and wheat crops were developed and tested in various real crop situations. Firstly, a multi-temporal study (six UAV flight dates at seven-day intervals) was developed to determine the suitable early phenological stage for weed mapping in both crops being weed and crop plants in the principal stage 1 (leaf development) in the beginning of the experiment, and at the principal stage 2 (tillering for wheat and four true leaves for maize, from the BBCH (Biologische Bundesanstalt, Bundessortenamt und CHemische Industrie) extended scale (Meier 2001). Once the best early growth stage had been assessed, four locations (two wheat fields and two maize fields) were flown over to validate the results by using a systematic on-ground sampling procedure (more details in Peña et al. 2015). One of the first steps to map weeds in both crops was to discriminate bare soil from vegetation fraction (weeds and crop rows) using different vegetation indices and Otsu’s thresholding method (Otsu 1979; Torres-Sánchez et al. 2014, 2015). Once the vegetation objects were discriminated, the crop-row structure was classified mainly using the crop row orientation by an iterative process. Finally, the vegetation objects that were located on the non-crop area were classified as weeds (López-Granados et al. 2016). Figure 8 illustrates the entire process for detecting weed patches in a field 40 m wide and 60 m long with nine 3 × 3 m weed patches equally distributed and located at the locations indicated in Table 3.

Fig. 8

a Image of the crop in standard colors, b false color representation of the previous image including the NIR channel, c weed patch map after applying the weed patch detection algorithm: crop in green, ground in brown and detected weeds in red, and d herbicide treatment map (grid 0.5 m) based on the previous weed patch map indicating the weeds in light green (Color figure online)

Table 3 Location of weed patches

Ground perception system

The Ground Perception System was designed to detect weeds, both inter- and intra-row, on maize fields. This system was based on an SVS-VISTEK camera connected to the HLDMS computer for acquiring images and running the relevant vision algorithms. The system was placed on board the ground vehicles (Fig. 4) and fully integrated with the HLDMS (Romeo et al. 2013).

The UGV operation speed for this application was fixed at 0.83 m/s (3 km/h), and the Region of Interest (ROI) for the Ground Perception System was defined to be 3 m wide and 2 m long and located in front of the UGV. With these parameters, the Ground Perception System has to deliver the data provided by the GNSS for image georeferencing and the data after image processing concerning the identified crop lines and weed coverage in less than 2.4 s, i.e., all required data duly processed and ordered with correct synchronization between them. Experimentally, the final average time used by the perception system was lower than 0.35 s.

Weed detection relies on the spatial identification of crop rows. Thus, determination of crop-row positions with respect to the UGV becomes a crucial task for weed patch detection and UGV guiding.

Crop-row detection

In the last two decades, several strategies have been proposed for crop row detection based on (a) exploration of horizontal strips, (b) Hough transformation, (c) vanishing point-base, (d) stereo-based approach, (e) blob analysis, (f) accumulation of green plant, (g) frequency analysis and (h) linear regression. These methods are described in Guerrero et al. (2013). In this specific case, the crop row positions were determined with respect to the UGV for UGV guiding and weed detection purposes. A vision system has been used to process images to find four crop lines containing two approaches with similar performance. The first is inspired by the Hough transformation and is detailed in Romeo et al. (2013). The second applies linear regression based on the robust Theil-Sen estimator, detailed in Guerrero et al. (2013), where green plants are segmented by the thresholding method proposed in Montalvo et al. (2013). An inertial measurement unit (IMU) provides information regarding external camera parameters, pitch (α) and roll (θ), so that, along with all other parameters (both internal and external), four predictable crop rows are identified, serving as the basis for identifying the four real lines through the two methods mentioned above. The two central crop lines determine the UGV steering correction by computing the deviation of the UGV longitudinal axis with respect to an imaginary vertical line that divides the image into two equal portions. That is, the UGV (Fig. 9a) undergoes a slight deviation from the planned trajectory (approximated by the red line). This misalignment can be corrected by the GMU in the subsequent sampling periods, as illustrated in Fig. 9b. This method provides real-time features for robot guidance and weed detection.

Fig. 9

Crop row detection (yellow lines) and UGV desired path (red line): a path deviation detected and b path deviation corrected (Color figure online)

By using a set of 400 selected images, the corrections to be ordered by the Perception System were established. Two levels of guidance are continuously established during the UGV navigation, one based on the GNSS that guides the UGV following a swath and a second based on the Perception System, which determines small deviations if any, as expressed above, and commands the required correction when required. After each correction, the algorithms confirm that the UGV in the next image in the sequence is positioned correctly. Testing showed that, on average, a correction was required for 30 % of the images and the vehicle was positioned correctly in 89 % of the successive images. For all other images, the algorithm for path tracking based on GNSS assumed full responsibility for the guidance.

Weed detection

For each image, a density matrix containing green density values covering the inter-row lines of weeds associated with the image was computed and stored. The densities were transformed to the following linguistic labels: low, medium and high. Figure 10 illustrates two consecutive images obtained in a sub-path. There are three types of lines limiting the cells required for computing the density matrix defined as follows:

Fig. 10

Sequential images along a path. Detected crop lines in yellow and detected weeds in cyan (Color figure online)

• (1) After identifying the crop lines (in green), they are restricted to the ROI in the image (yellow lines).

• (2) To the left and right of each crop line, parallel lines are drawn in red. These lines divide the inter-crop space into two parts.

• (3) Horizontal lines (in blue) are spaced conveniently in pixels so that each segment is separated from the adjacent lines in the ground by approximately 0.25 m.

The above segments define 8 × 4 trapezoidal cells c _ij (i and j represent rows and columns, respectively). For each c _ij , the total number of pixels, A _ij , and the number of green pixels (represented in cyan), G _ij , are computed. Thus, a weed coverage matrix is computed as w _ij  = G _ij /A _ij . A tolerance margin representing approximately 10 % of the width of a cell is excluded, assuming there are mainly crop plants instead of weeds in this margin. By using approximately 200 images, the weed coverage was classified into three levels (Low, w _ij  ≤ 1/3; Medium, 1/3 < w _ij  ≤ 2/3; and High, w _ij  > 2/3). With these levels (selected by an expert), the system obtained a 91 % success rate. Weed patch position and coverage are the inputs to the Actuation System.

It is worth noting that the Ground Detection System is a real-time procedure that cannot be used in narrow-row crops. Therefore, the Remote Detection System is used to solve this problem in those types of crops, with the shortcoming that it is an off-line procedure. The Remote Detection System can also be used in wide-row crops, but a real time procedure is preferred. The comparison between these procedures was out of the scope of this study.

Actuation systems

The Actuation System is the automated agricultural equipment that acts on the crops directly. Three devices were developed in the RHEA project. Two of them were based on spraying techniques: (a) a weed patch spraying system for herbicide application in cereal fields, (b) an air-blast sprayer for olive-tree canopy treatment; and (c) another that relied on a mechanical/thermal removal tool for weed control in maize. These agricultural tools are illustrated in Fig. 11.

Fig. 11

a Patch sprayer, b Canopy sprayer and c Mechanical and thermal implement

Weed patch spraying system

This system consists of twelve high-speed solenoid valves mounted on a stainless steel sprayer boom with an equidistant spacing of 0.5 m, providing a lateral resolution of 0.5 m. These solenoid valves comprise a brass inlet for incoming liquid, a spray nozzle, a nozzle cap and a LED indicator, which is lit when the solenoid is open (Fig. 11a). The boom sprayer has a fold/unfold mechanism managed by two electric linear actuators. The boom is folded along the sides during transportation between fields and also to allow the UGV to perform rotations in the field headlands and for avoiding obstacles. A commercial central direct injection system was equipped with a water tank (200:1) and a separate container for the herbicide (15:l) to be injected according to the information coming from the High-Level Actuation System. The injection system controller handles the injector pump to turn at the appropriate speed to reach the desired concentration and to stabilize the injected flow afterward. An encoder integrated in the system measures the flow rate of the active ingredient from the injector pump speed. The electronic control device uses the active ingredient flow rate from the pump speed to determine whether a change in active ingredient flow rate is required. The active ingredient flow rate is verified using a mini-flow meter (Perez-Ruiz et al. 2015).

This development was designed for three different herbicide densities corresponding to three weed/crop ratios: low volume application (100 l/ha), standard volume application (200 l/ha) and high volume application (400 l/ha). The injection system controller supplied a variable voltage to the gear motor to power the injector pump. This voltage caused the injector pump to turn at the appropriate speed to generate the desired flow rate of the active ingredient. An encoder integrated into the system measured the flow rate of the active ingredient based on the injector pump speed. The controller used the active ingredient flow rate from the pump speed to determine whether a change in the active ingredient flow rate was needed.

The mixing chamber of the injection system ensures that the flow of the agrochemical that is incorporated into the stream of water will be evenly distributed throughout the resulting volume.

In this test, there was no comparison of weed density (weed m⁻²) to determine the effectiveness of different treatments. The objective of this section was to develop a field sprayer able to operate each nozzle independently commanded by prescription map information.

Realization of individual tests of the High-Level Actuation System and Device System as well as many synchronization and integration tests produced savings of approximately 96 % in the applied liquid for a field infested with weeds covering approximately 3.24 % of the field. For an infested area of approximately 10 % of the total area, the saving fell to 90 %. However, the objective target (75 %) was achieved for a weed infestation area smaller than 25 % of the total area.

Canopy sprayer

A robot airblast sprayer was designed for the application of agricultural pesticides in olive groves, woody crops and orchards. The developed solution was based on a commercial airblast sprayer (Oktopus by Nobili Molinella, Italy), which consists of a high-pressure pump and some spray diffusers to nebulize the mixture, along with a high volume centrifugal flow air fan to distribute the liquid pesticide. The spray diffusers were placed in two vertical steel struts, 4 in each one, and separated by approximately 0.5 m. The design principles were focused on the Real-Time Volumetric Flow Control, i.e., to vary in a separate and independent way the dose, the air flow rate and its orientation related to the target features. The spray optimization relied on (a) a perception system based on ultrasonic sensors, (b) three types of kinematic coupling driven by stepper motors and (c) a solution for liquid management (Fig. 11b).

The perception system, consisting of eight ultrasonic sensors, was also placed in two vertical steel struts, 4 in each one. Every sensor is associated with one spray diffuser and placed at the same vertical height and approximately 1 m in advance in the horizontal plane. Thus, the sensors measure the canopy distance t seconds before the associated spray diffuser reaches the same point (t = 1 m/vehicle speed). The sensing range goes from 0.03 to 2 m with a sensing repeatability of ±0.25 % of the measured distance.

To improve spraying, the tree canopy was virtually divided into four overlapping divisions parallel to the ground. Every division is centered with the horizontal plane defined by a diffuser and its associated ultrasonic sensors (Fig. 12).

Fig. 12

Cases and actions for the actuation of devices in the olive-tree implement

The automation procedure was designed on the following principle: the ultrasonic system detects an object at a distance x, which is sent to the Low-Level Actuation System or implement controller, which, in turn, recognizes the corresponding scenario and generates the signal to (a) determine the opening of the valves and (b) control the air flow rate in the adductor tubes on the partition cap performed by eight on/off small butterfly valves. Finally, the optimization involves addressing the airflow at the tree canopy. This objective was achieved by varying the attitude angle of the terminal spray modules (upper and lower in each strut). This was managed by a motor that permits rotating the spray diffuser in the range of −15° (downward) to 15° (upward) and, thus, orientating the airflow.

To improve the spraying features (air and liquid flow rate), some prescription rules for every actuator were defined according to the olive tree canopy characteristics. Four cases were considered:

Case 1

• Sensing—every ultrasonic sensor measures a distance x ≤ 0.5 m (full canopy detected).

  Actions—upper and lower spray modules parallel to the ground (0°); the air flow rate at maximum; spraying active in each spray module.

Case 2

• Sensing—every ultrasonic sensor measures distance x in the range 0.51 m < x ≤ 1.5 m.

  Actions—the upper and lower spray modules are inclined approximately 15° toward the center of the canopy tree; the airflow rate at maximum; spraying active in each spray module.

Case 3

• Sensing—the higher and lower ultrasonic sensors do not detect canopy, but the central sensors detect a target distance x < 1.5 m.

  Actions—spraying and airflow rate are activated in only the central spray modules.

Case 4

• Sensing—none of the ultrasonic sensors detects canopy.

  Actions—airflow rate and spraying are stopped.

A pressure sensor checks the functioning of the hydraulic plant. If the working pressure is wrong, the Low-Level Actuation System sends a message to the Mission Manager (through the HLDMS) to let the operator take control of the system.

The work carried out with the canopy sprayer relies on the physical application of pesticides rather than on the agrochemical doses. Thus, the system does not change the chemical application rate on the target and then it does not need an injection system to vary the concentration of agrochemical.

The standard physical application on the tree foliage that involve percentage of coverage on surface, or number of droplets per area, and, for more detailed analysis, the specific doses deposited per area were taken into account. The aim is to reproduce the standard application on the target, avoiding the waste due to off-target depositions.

The off-target deposition, or drift internal to the plant or outside the treated field, is caused by continuous spraying where there is no foliage and by the transport by air vector beyond the canopy width when it is thinner than the maximum.

Rules were defined that stop the application of liquid when there is no foliage coverage detected and apply 3 different levels of liquid mixture according to requirement in 3 different steps of canopy width (30, 70, 100 % of conventional application).

The particular cases of the upper and lower nozzles that can be tilted are due to the fact that the upper and lower parts of the tree canopy are not regular and are particularly susceptible to diseases. At the top, there are young shoots that have plant tissues easily attacked by diseases. Similarly, the lower parts, close to the ground are influenced by soil moisture. In a targeted application, these areas need to be sprayed with special attention to obtain the conventional accepted rate of application.

Note that this implementation requires no action from the HLDMS (except a signal to start/finish the mission); therefore, this agricultural implement is autonomous from the control standpoint (Perez-Ruiz et al. 2015).

Mechanical and thermal tools

A physical weed controller was designed to perform mechanical and thermal weed control in maize. The mechanical weed removal implement was designed to remove the inter-row weeds, and it was based on shallow soil tillage intervention, carried out with different end-effector tools tailor-made for robotic applications. The thermal weed removal tools were envisaged to remove the intra-row weeds, and they relied on liquefied petroleum gas (LPG) fed rod burners with an open flame (dry heat). This implement was envisaged to be developed using current robotic technology and consists of a deployed mechanical structure.

This implement is based on an inter-row cultivator structure with a working width of approximately 3 m. The machine consists of five cultivator units, i.e., three complete units and two side half units, enabling treatment of four rows of the crop, including three inter-row spaces of 0.75 m each and two half lateral inter-row spaces of 0.375 m. Each of the three complete units tills a 0.5 m wide strip of soil between the rows using one goose-foot rigid central tool and two L shaped adjustable rigid side sweeps working at a very shallow depth (0.03–0.05 m). The two side-half cultivator units are provided with one goose-foot rigid element and one L shaped sweep to work the two side-inter-row strips (0.25 m wide) of soil. All the rigid elements are adjustable, allowing the tool to work on inter-row spaces ranging from 0.3 to 0.9 m properly.

The mechanical tools for weed removal are mounted on an adjustable articulated parallelogram structure, providing two little pneumatic wheels that, during the treatment, follow the soil surface, which maintains the correct working depth of the rigid elements and the correct distances of the burners from the ground.

The site-specific physical-weed-control (PWC) implement was coupled with a UGV, and the entire system became autonomous. The application of flaming is based on the Weed Detection System attached to the UGV. When this system detects weeds, data with the value of greenness is sent to the High-Level Decision Making System installed on the UGV and then to the Low-Level Actuation System mounted on the implement. Three levels of weed cover (low, medium and high) are supplied by the Weed Detection System to perform non-treatment or treatment with medium or high LPG pressure (and, consequently, null, medium or high LPG biological doses). According to the percentage of weed cover detected by the Weed Detection System, the choice between the three levels of LPG pressure is made in real-time to obtain the proper dose to control weeds. Figure 11c illustrates this implement (Frasconi et al. 2014).

Experiments and a subsequent study on the effect of the LPG doses in maize crops and weeds for different growth stages concluded that the achievable weed killing rate can reach a weed reduction of more than 90 % in terms of both density and biomass at harvest, obtaining maize yields similar or higher than those normally obtained in the conventional way. This confirmed the results obtained in previous experiments on cross flaming application on maize and other row-spaced summer herbaceous crops (Peruzzi and Raffaelli 2000; Perez-Ruiz et al. 2015).

Communication system

The RHEA Network system was realized by using a Linux-based wireless router providing IEEE 802.11b/g, IEEE 802.11a, IEEE 802.14.5 ZigBee PRO and ITU GPRS interfaces. The high-level communication system architecture was based on the multi-technology management approach, where different communication paradigms (publish/subscribe and request/reply) and services (best effort and enhanced reliability) were designed to utilize multiple interfaces. An interface based on TCP/UDP messages, implemented as a Network API, was used by all communication components. Accordingly, an IP addressing scheme for the overall system was defined, unambiguously addressing every component located on board a UGV or on the BS. To assure traffic prioritization and quality of service, the IEEE 802.11e standard was chosen. Finally, the communication system software, implemented as the Middleware, runs on a Linux-based Wireless Router as the execution platform (Hinterhofer et al. 2012).

Location system

The location system accounts for the relative GNSS position of the robots with respect to the BS where the GNSS base antenna is located. For the location system, four RTK-GNSS receivers (Trimble BX982, Sunnyvale, California, US), providing ± 0.02 m accuracy, were used to accurately locate the robots for all field trials. One receiver was used in the BS (reference antenna), and the other three receivers were on board the 3 UGVs (with two GNSS rover antennas). The BX982 GNSS receiver was configured as an autonomous reference station. Streamed outputs from the receiver provide detailed information, including the time, position, heading, quality assurance numbers and the number of satellites in view. The receiver also sends a strobe signal at 1 Hz, which allows the remote devices to synchronize in time precisely. The main task of this GNSS receiver is to provide and transmit the GNSS correction signal to each robot to determine the location of the UGV accurately, obtaining centimeter-scale accuracy on RTK-GNSS receivers and decimeter-scale precision on differential GNSS (DGNSS) receivers. A single connection (RS232, USB, Ethernet or CAN) to the BX982 delivers centimeter accurate positions with better than a tenth of a degree (2 m baseline) heading accuracy.

Graphical user interface

The graphical user interface (GUI) is a software application running in the BS computer (Fig. 3), where the model of the real system is represented in 3D and the user can interact in different ways to obtain information regarding what is currently occurring with the real system or to send commands to the system. The system displays the status of the mission, the position of the vehicles in the field, the sensor measurements, the camera images and other information received from the Mission Supervisor, and it can control the UGVs remotely as well as manage the mission execution. Moreover, the same program allows the operator to execute a real mission in the field or simply run the planned mission in simulation to evaluate its efficiency and test emergency maneuvers.

The core of the GUI consists of the commercial package Webots 7 robot simulator (developed by project partner Cyberbotics, Lausanne, Switzerland) that computes dynamic simulations using a realistic physics engine and represents the robots and relevant equipment in a 3D space. Another independent software module was developed to provide a single interface for the different partners’ modules (for example, the UAV Mission Planner and the UGV Mission Planner). This additional program constituted the entry point for the human operators and allowed them to set the parameters and manage the 3D environment to simulate missions or supervise them.

Some models of the real operational setup were developed to be able to accurately reproduce the setup in the Webots simulation. Figure 13 illustrates what the GUI looks like.

Fig. 13

Graphic User Interface illustrating different scenarios: a wheat crop and weed patches, b maize crop and weed patches, and c olive trees

Experimental results

The sub-systems presented in the last section were individually tested and integrated prior to conducting the relevant multi-robot tests and demonstrations for assessment. This section presents the main demonstrations conducted in the three target crops (some videos are available at http://www.rhea-project.eu/), defining the tests performed and assessing the results.

Patch sprayer in wheat

The preliminary tests and demonstrations for assessing the patch sprayer were held in January 2014 at the appropriate time to apply herbicide treatments on wheat crops. During this time, a UAV carrying an image acquisition device and a UGV with a patch sprayer were used to evaluate the spraying system performance under real working conditions. Additionally, preliminary trials with the complete multi-robot system were carried out.

Experimental field

For conducting the experiment/demonstration, nine 3 × 3 m weed patches were sown following a regular grid (total weed area: 81 m²) in a wheat field of 60 × 40 m (2400 m²) (Fig. 14). Within the UGV-implement mission, the accuracy of herbicide spraying (i.e., on/off time lag, percentage of target sprayed, percentage of target not sprayed and percentage of non-target sprayed), and the UGV trajectory deviation from the planned route were investigated.

Fig. 14

UGV path computed by the Mission Manager for the patch sprayer

Description of the mission

The full mission in wheat was divided into three tasks: inspection, weed mapping and herbicide treatment.

Inspection task

A scouting or inspection task, such as the one illustrated in Fig. 6a, was performed by the UAVs that carry the image acquisition device. Before the flight, color targets were located (Fig. 7) around the parcel and manually geo-referenced using a RTK GPS receiver (0.01 m accuracy). Pictures of the crop were obtained and stored in the camera memory cards during the flight. The complete set of pictures was then uploaded at the BS computer after the landing of the UAVs.

Image mosaicking and weed mapping

The memory card was inserted in the BS computer and the data (geo-referenced pictures) obtained, then aerial image mosaicking of the field was performed by using tailor-made software as described above (Fig. 7). Figure 8a presents a picture of the demo field in standard colors, and Fig. 8b illustrates the image in its false color representation including the NIR channel.

These pictures were the basic inputs to identify and locate weed patches by using a computer application implemented through the commercial software packages described above. The crop rows, soil and a map with the weed patches (after applying the OBIA algorithm) are illustrated in Fig. 8c. Figure 8d presents the herbicide treatment map (grid 0.5 m) based on the previous weed patch map, where the weed patches are represented in green. To better display the image details, the pictures correspond to only one image obtained at a height of 120 m with a 0.02 m spatial resolution.

Treatment task

The information extracted with the weed manager, i.e. positions and dimensions of the weed patches (Fig. 8d), along with the known information from the working field, i.e. dimensions and location of the field, were used as inputs by the Mission Manager to compute the UGV paths and the points in which every nozzle of the patch sprayer attached to the UGV (Fig. 11a) has to be opened and closed. Figure 14 presents a sketch of the working field (input 1) with a superposition of the weed patches detected with the Remote Perception System (input 2) and the path provided by the Mission Manager (output 1). This information was coded and sent to the UGV controller to allow it to steer the UGV and to manage the nozzle valves (output 2) of the patch sprayer.

Assessment of the patch sprayer mission

The evaluations conducted in the demonstration showed that the UGV followed a trajectory with small deviations (less than 0.07 ± 0.01 m from the center of the UGV wheel footprint, measured with a stationary GNSS antenna) with respect to the commanded wheel trajectory. This condition is accurate enough for this type of application.

To detect the areas where the herbicide was applied, water-sensitive paper strips were distributed in the field, both in the weed-infested areas (including the borders) and in the weed-free areas. Over 95 % of the target area (weed-infested) was sprayed. Although none of the paper strips located in the non-target area (far away from the patches) were sprayed, the sprayed area was displaced a few cm ahead of the target area due to a slight dynamic delay in opening and closing the nozzles. Specifically, the nozzles were opened at approximately 0.024 m inside the patches and they were closed at approximately 0.022 m out of the patches, for a vehicle speed of approximately 3 km/h. This lag resulted in a positioning error of approximately 7.6 % for the sizes of the patches used.

Canopy sprayer in olive trees

Experimental field

The RHEA canopy sprayer was tested in an olive grove consisting of four olive-tree rows of approximately 40 m length with trees separated by 1.5 m.

Description of the mission

The mission was generated by the Mission Manager to cover the entire grove with three UGVs to be position-controlled on a GNSS basis. The plan generated for the UGVs, illustrated in Fig. 15a, was executed in May 2014.

Fig. 15

a UGV plan for the demonstration in the olive grove generated by the Mission Manager; b measurements for system assessment

Assessment of the canopy sprayer mission

Some trials to assess the correspondence between target detection and spraying start-up/shutdown at the beginning, during and at the end of each row were carried out. To this end, some video clips were acquired by placing fixed video cameras at those specific points. The results showed a fast system response to the variation of the target features and an equally rapid actuation of the management rules set in the Low-level Actuation System as regards the management of the spraying and air flow rate adjustment. The tests performed showed, at a forward speed of 3.5 ± 0.2 km/h, average actuation times of 0.155 ± 0.055 s (time elapsed between detection and actuation of the devices). This slight delay in actuation produced an average error of 0.16 ± 0.06 m, which is not significant from the agronomic point of view (Fig. 15b).

The preliminary trials, performed to evaluate the reduction of mixture consumption, showed an average reduction of the sprayed dose of 48 %. Recoveries over 50 % are achievable in conventional plants where the canopy variation is higher than the final demo plant, i.e., higher than 3.5 m. Moreover, chemical coverage quality on leaves was similar to conventional treatments. Nevertheless, the following additional indirect benefits can be obtained: (a) 50 % less chemical used (normally wasted also as over-dosage), (b) 50 % less pressure on environment and health, (c) 50 % less cost for chemicals and (d) 50 % more controllable plant area, i.e., yard efficiency 50 % higher.

Mechanical/thermal treatment in maize

The final tests and demonstrations for assessment were conducted in May 2014 at the appropriate time for weed control in maize crops.

Experimental field

For the demonstration in maize, a field of approximately 90 × 45 m = 4050 m² was grown and infested with seven artificial weed patches distributed covering approximately 135 m², as illustrated in Fig. 13b.

Description of the mission

Based on the initial field dimensions, the Mission Manager generated the plan illustrated in Fig. 16 to cover the entire field. The trajectories generated in that plan were roughly followed by the UGVs because they were adapted to the crop rows with the information from the Weed Detection System (crop row positions). During the execution of the mission, the Ground Perception System detected the position of the weed patches in real-time. That position information was used by the High-Level Decision Maker System to command the Low-Level Actuation System to turn on/off the required implement burners (Fig. 11c). Inter-row weeds were removed with the mechanical part of the implement.

Fig. 16

Plan for the demonstration of the mechanical/thermal implement in maize generated by the Mission Manager

Assessment of the mechanical/thermal system mission

The experimental results obtained with the mechanical/thermal system during the final demo showed that all the weeds present in both the inter-row space (where the mechanical tools worked at a depth of approximately 0.05 m) and the intra-row space (where variable rate application, VRA, cross flaming was carried out with different LPG doses according to the detected weed cover rate) were successfully controlled. However, the assessment of this implement was previously based on the individual evaluation of the systems involved. Thus, the spatial accuracy of the Ground Perception System and the spatial accuracy of the implement were evaluated previously to study the effect of flames in crops. During the demo, a qualitative evaluation was made, observing the points where the flames went on and off with respect to the limits of the weed patches. The burners were always switched on at the right pressure, according to the detected weed cover rate, although it always occurred a little before and after the infested patches, to avoid leaving untreated all the weeds detected in the intra-row space. The average error of the burners switching on and off, as determined in the field, was approximately 5 %. This means that the burners were activated approximately 0.5 m before and switched off approximately 0.5 m after artificial weed patches which were approximately 10 m long. These qualitative observations were made in all the artificial weed patches present in the field treated in the final demo. The methodology was the same as that used to assess the precision of herbicide distribution with the RHEA patch sprayer working in wheat and consisted of the positioning of thermal paper strips (able to make easily evident flame presence) 2 m long on the maize rows at the beginning and at the end of any treated patch. These observations allowed confirmation of the correct functioning of the whole system and particularly the degree of integration among the decision making, the perception and the flaming systems (Perez-Ruiz et al. 2015; Peruzzi 2014).

Spatial accuracy of the ground perception system

Experimental tests, based on camera calibration relating pixels and spatial locations, were conducted for spatial accuracy of the vision system in three ways:

• (a) Crop row location was measured as deviations of the detected crop lines with respect to the expected ones. The average error was ± 0.03 m, inducing total errors of ± 0.05 m when following the planned crop rows. These results are accurate enough for applying the Ground Perception System in wide-row crops with rows separated more than 0.34 m (for a 0.2-m wide tractor tire). Note that in ordinary maize fields, the rows are spaced approximately 0.75 m apart.

• (b) As described in the section “Ground Perception System”, concerning weed detection (Fig. 10), the correct classification of weed coverage (Low, Medium and High), with the 200 images used, was quantified as approximately 92 %.

• (c) Longitudinal ROI overlapping is between consecutive ROIs; the mean square error was quantified as ±0.08 m, representing 4 % of the ROI length.

Spatial accuracy of the implement

The implement, parts and components exhibited some delays, especially those related to the ignition/extinction of burners that needed to be identified to apply the pertinent corrections. The delays associated with the Decision-Making System, the Actuation Controller, electro-mechanical parts and igniters were experimentally measured to be, on average, approximately 0.7 s for ignition and 1 s for extinction. Taking these delays into account, some experiments were conducted to assess the accuracy of the entire system. Thus, the positioning error of the weed coverage matrix and the lateral displacement of the path tracking with respect to the center of the crop rows were about ±0.08 and ±0.03 m, respectively. These errors can be acceptable because the burners have a length of 0.25 m, which determines the resolution of the system.

Effect of flame in crops

Maize tolerance to flames was also studied in weed-free conditions. No dead plants were observed after the flaming treatments were carried out at different LPG biological doses. Moreover, the analysis of variance revealed that the maize yield was not affected by flaming treatments at any LPG dose, confirming the results obtained in previous experiments (Martelloni et al. 2016; Peruzzi and Raffaelli 2000).

Future work

After conducting the final demonstrations, different exploitation possibilities were noted. The multi-robot system can be configured to have n UAVs and m UGVs (a fleet of robots) equipped with similar implements (depending on the type of crop) to perform pest control tasks, but individual robots can also be exploited: one tractor and an LPG implement can be used as an independent tandem, and the same is applicable to the patch sprayer working with either a remote perception system as explained above or a ground perception system as the LPG implement does. Regarding the canopy sprayer, it is even easier to be used independently because it works autonomously: it integrates perception, decision and action, and it can just be towed by either an autonomous or a manned tractor easily. Finally, some sub-systems can be commercialized for similar application in different sectors: the safety system is applicable in vehicles for general purposes, the Mission Manager and the GUI can be designed for multi-robot systems based on other types of robots for different applications.

In spite of the fact that the entire system performed properly, there are a number of specific features and a large number of prospective improvements worth mentioning. For example, the Remote Perception System achieved a fully automatic acquisition and processing chain providing a high resolution georeferenced multispectral image of crop parcels from aerial acquisitions; however, image acquisition in automatic exposure mode (easier operation and more robust results in case of variable weather) requires balancing between grey levels of various images based on Exchangeable Image File format (Exif) information, which may improve the process significantly.

The OBIA algorithms for weed patch detection demonstrated the robustness to accurately discriminate and map crop-rows (100 % accuracy) and inter-row weed patches (>85 % accuracy depending on the field studied) and to design a field program of site-specific weed management at the early growth stage according to these weed maps. Current investigations focus on improving the OBIA algorithm when a number of specific field conditions, such as curved crop rows or sowing failures, are present in the fields, undoubtedly a step forward in the application of these techniques in real situations.

The Row Detection Systems exhibited a total error of ±0.05 m when following the crop rows under different growth stages of green plants (for both crops and weeds) in the specific case of maize sown in rows separated by 0.75 m. Accurate location was accomplished even in the case of high weed pressure. The next step is to focus on a flexible system that is valid for detecting crop rows with different row spacing, i.e., valid for different crops (such as tomato and onion). Additionally, a robust system to identify crop rows of non-green plants, i.e., valid for tilling during sowing or other tasks, would extend the number of applications significantly.

The weed patch spraying system performed as expected achieving the objective of reducing up to 75 % of the herbicide use, and thus it can be used in precision farming to implement site-specific management techniques for variable-rate application of cereal production inputs using a map-based treatment. A device for mixing water and herbicide during the treatment operation is essential to avoid disposal of the excess product following the treatment operation. The intelligent sprayer was developed with a Central Direct-Injection Pesticide System, and this technology achieves VRA by spraying the carrier at a predetermined constant flow while varying the concentration of the pesticide as needed. An issue of this system is the distance lag that the sprayer travels before the concentration change reaches the nozzles. The next step is to use electrochemical conductivity sensors on the nozzles to estimate the concentration of an injected spray solution in real-time. This implementation will help to monitor the applied concentration at nozzle level and to generate maps of pesticide use.

The air blast sprayer allowed the spray features to be managed, saving approximately half the conventional application rate while maintaining the quality of the foliage depositions. Nevertheless, some desirable improvements would rely on the adoption of 3D Lidar systems and relative algorithms to better characterize the canopy features, as well as air flow monitoring sensors to achieve a complete air blast sprayer monitoring. An actual need is the development of spray-dose monitoring systems, which would allow real control of the pesticide depositions. These instruments would be basic elements of a commercial air blast sprayer.

The results obtained in the field experiments demonstrated that the use of the LPG system allows site-specific and VRA of flaming to be performed, according to the detected weed cover rates. This saves an important amount of LPG, ranging from 50 % up to more than 90 % (according to the spatial distribution, the number and length of the weed patches) with respect to the application of intra-row cross flaming continuously on the entire cultivated surface. Moreover, this reduces the cost of the thermal interventions to 30–80 Euro/ha. This is in contrast to a major problem to be solved at the European level to commercialize this type of system that concerns the lack of standardization of the LPG tanks and the rules for their commercialization and refilling. Moreover, there are no rules for LPG storage according to homologated provisions and no tax incentives for the use of this clean fuel in agriculture. Therefore, political actions are required regarding the definition of standardization rules for LPG bottles and storage elements, as well as quantifications of benefits for contractors and farmers using flaming for weed control.

The computing equipment on board the Unmanned Ground Vehicles relying on the HLDMS (CompactRIO) and the UGV Controller (PC-based computer) was assessed indirectly by evaluation of the components this system was in charge of. Thus, the information the HLDMS received from the Mission Manager, the Ground Perception System, GPS and other sensing systems, such as the Safety System, was handled without significant delays, as were the decisions made and that influence the features obtained in the application of the three target treatments. The UGV Controller proved to work efficiently in the communication with the HLDMS and the Low-Level Actuation System on-board the implements. Furthermore, the overall system worked reliably at temperatures over 40 °C inside the UGV cabin. The next step in the computing system is focused on the integration of the UGV Controller as a part of the HLDMS. This would reduce hardware and avoid the Ethernet communication link between these two systems, thus improving reliability while reducing costs. One more step forward would be the integration of critical modules in the compactRIO FPGA (Field Programmable Gate Array), improving the speed of some processes and allowing the vision software to be extended.

The GUI has been assessed as customized modular and extensible software able to compute and execute missions on specific agricultural fields and environments, but its application to other types of scenarios and surroundings will require the generalization and extension of rules for automatic mission computation and representations. Moreover, as with other large software applications, intensive tests have to be conducted to improve the GUI’s ease of use, ergonomics and reliability.

Regarding the communication system, implemented communication features contributed to a successful proof of concept of the overall communication in real scenarios. The implemented system was capable of dynamically utilizing the multi-interface communication topology for the transport of time critical commands and non-time critical video over the wireless links and integrating different on-board communication technologies. For potentially extended scenarios including more robots, it would be interesting to identify the potential for further improvements, in particular, providing time-critical wireless services. For this purpose, new protocols for bandwidth management based on TDMA would be proposed (Hinterhofer et al. 2012). Finally, extended features of a communication middleware for multi-robot systems (Drenjanac and Tomic 2013) should be explored, as well as the potential to support more dynamic task assignment in the multi-robot system (Drenjanac et al. 2014).

The main drawback of the UAVs is normally the maximum time of flight, which, in this application (approximately 40 min), is enough because the multi-robot system uses several units and the main task is not to travel a large distance but to cover an area. That means the UAV, even if the distance traveled is large, is always close to the Base Station, and landing and changing the batteries becomes an easy task. Nevertheless, an important logistical and legal problem that needs to be solved is that flight regulations in many countries require a backup pilot for each UAV. This implies a high cost in terms of human resources. A tool for supervision of the entire multi-robot system by only one backup pilot would be desirable, but the target is to perform fully autonomous tasks with UAVs.

Concerning the UGVs, those vehicles based on internal combustion engines exhibit good energetic autonomy and provide the demanded power to carry the payloads and accomplish the tasks; the current GNSS technology provides position accuracy, and modern controllers are reliable enough to configure robust vehicles. Nevertheless, safety for humans, animals and the vehicle itself is an important issue for these types of vehicles, and new standards are being defined by different working groups. For example, the group ISO/IEC TC 23/WG 15 (ISO 2016) is working on the new standard ISO 18947- Safety of Highly Automated Agricultural Machines.

UAVs and UGVs require a significant amount of energy to perform their tasks. Battery innovations will increase the mission time for UAVs, and the improvement of fuel cells will allow, in the near future, configuration of a complete ecological, efficient ground vehicle for carrying heavy equipment.

Regarding the Mission Manager, the supervision modules (distributed and multi-level) greatly facilitate supervision in other applications by replacing, adjusting or adding only the individual pieces of code related to the specific treatment of new alarms. Furthermore, many of the alarms defined and the code developed for monitoring them are basic in the supervision of autonomous robots regardless of the type of application. Thus, the software developed is largely reusable. However, the proposed planning strategies for the mission planners, based on Simulated Annealing, NSGA II or Genetic Algorithms, are applicable to other more general optimization problems, such as the Vehicle Routing Problem. In addition, the Planner can be used in other agricultural activities that require planning, e.g., crop distribution prior to sowing. In fact, the developed planner can work stand-alone, i.e., disconnected from the rest of the modules of the Base Station or the GUI. The Mission Manager could be improved by including mechanisms to define and use different policies for alarm management, even allowing the changing of management strategy during the mission execution if needed. Other highly desirable improvements would be related to the Planner and its ability to develop a plan that simultaneously considers various crops and the transactions among them.

Conclusions

In all, the RHEA project made a step forward for applying automatic and robotic systems to the agricultural industry by showing real possibilities of using a multi-robot system, pointing out further needs and contributing to pave the way to the Fully Automatic Farm as a paradigm of the Fully Automatic Factory. In particular, the project RHEA designed, developed and tested a new generation of automatic tools and robots for effective weed management in agriculture in both the chemical and physical (i.e., mechanical and thermal) domains.

The ground robots (UGV) were based on a commercial mobile platform equipped with sensors, on-board perception systems, controllers and communication systems to configure autonomous robots capable of carrying agricultural implements. The aerial robots (UAV) were based on a new hex-rotor drone designed to carry a two-camera perception system to perform remote detection of weed patches on narrow-row crops. Both types of vehicles were checked and demonstrated complementary features to work jointly.

The implements, or agricultural tools, were focused on the development of (a) a patch sprayer, (b) a mechanical and thermal implement and (c) an air-blast sprayer for tree canopies. The patch sprayer was developed to implement site-specific management techniques for variable-rate application of cereal production inputs using a map-based prescription provided by the remote detection system. The thermal implement performed cross-flaming only if weeds were identified by the perception system on-board the UGVs. Finally, an air-blast sprayer was developed to produce a variable-rate spray application optimized to match the structural variation of the vegetation. These three implements formed a main set of tools to cover an important part of agricultural crops (wide-row crops, narrow-row crops and trees).

Actuation on crops is not possible without previous sensing; thus, RHEA developed two complementary ground and remote perception systems for (a) real-time discrimination of weeds from crops in wide-row crops and (b) off-line weed patch detection on narrow-row crops, respectively.

Carrying out the main tasks of observing, making decisions and actuating on crops to reduce the use of pesticides while improving efficiency is not only a matter of providing motion, sensing and actuation. A comprehensive combination of these sub-systems is needed and it was accomplished through the integration of communications and location equipment, safety systems, human–machine interfaces and multi-robot management algorithms. The RHEA project was a unique opportunity to test all these components as a multi-robot system (in the future fleets of robots) for agricultural activities.