1 Introduction

Agriculture currently faces multiple challenges, characterized by the imperative to optimize land utilization while adhering to stringent demands for ecological sustainability and biodiversity conservation [1]. Crop protection herbicides, integral to conventional agricultural practices, are pivotal in this context, while mechanical weed control plays a minor role [16]. Reliance on chemical herbicides leads to major problems, such as pesticides leaching into neighbouring environmental compartments or weeds becoming resistant [22]. In addition, the elimination of weeds and the resulting loss of biodiversity in agroecosystems has a massive impact on their ecosystem services, such as pollination or prevention of soil erosion [16].

In response to these challenges, the Cognitive Weeding project has undertaken the development of a selective weed management system presented in this paper. This system offers tailored treatment strategies within an expert system framework. To operationalize this system, detailed information of the field is essential, necessitating meticulous sensor data collection and subsequent automated analysis.

This paper outlines an architectural framework that integrates data acquisition, data analysis, and the expert system to enable a selective weed management approach. Empirical findings substantiate the feasibility of implementing such a system. This described system, importantly, holds the potential to directly integrate ecological objectives into the application maps of agricultural machinery, thereby contributing significantly to the environmental and ecological sustainability goals within contemporary agriculture.

2 Related Work

During the past years, plenty of research has been done on the construction and effectiveness of so-called smart sprayers. The main objective is usually reducing the total volume of herbicides sprayed onto the field. Partel et al. [19] show an exemplary approach by combining AI-based plant detection with a spraying system with individually controllable nozzles, detecting and distinguishing weeds from crops, so only weeds are targeted by the sprayer. Similar systems are available for different crop types [13, 18], meanwhile even commercially [4, 6]. Additionally, the effectiveness of selective spraying was tested under multiple conditions, reporting values of reduced herbicide volumes between 23 % - 89 % [3], depending on target weeds or crop [2]. For the detection of weed plants, different sensor data (image data or 3D-information) of ground-based or UAV-based sensors are combined with computer vision algorithms [10]. These systems are quite capable of detecting plants (green on brown) and differating between crops and weeds (green on green). The differentiation between weed species and therefore harmful and non-harmful weeds is still an ongoing research topic [7].

A possible approach is to adapt the damage threshold principle to multiple occurring weed species. While this is still hardly feasible in practical applications, Elstone et al. [9] show that spot spraying using a weed-specific damage threshold is possible in salad cultures.

In contrast to the available spot-spraying solutions, we aim to integrate the differentiation between weed species into the removal process. The goal is not only to target weeds but also classify different weed species. With weeds being categorized, one can utilize a decision process based on prior agricultural knowledge like specific damage thresholds to identify and remove only weeds that are considered harmful. In addition to spot spraying, site-specific mechanical weeding has the potential for more sustainable weed control. This has been done for example by varying the aggressiveness of different harrow types [21], but to the best of our knowledge not with a selective mechanical hoe, which is able to spare specific weeds. In the Cognitive Weeding project, both a spot sprayer and a selective mechanical weeder have been tested for the removal of only harmful weeds. While spot sprayers are able to remove weeds as they are detected, the decision process requires the recorded data to be analysed offline. As a result, an application map is generated, which is known in subdomains like precision agriculture [8].

3 Architecture

In this section, our proposed architectural framework will be described. Initially, the needed expert knowledge and field trials are described in Sect. 3.1. Based on that, the expert system which applies the expert knowledge is presented in Sect. 3.2. Afterwards, the data acquisition using drone based and ground based sensors (Sect. 3.3) as well as the data storage and provisioning using a semantic environment representation (Sect. 3.4) is elaborated. Finally, the selective actuators which do the actual weeding based on an application map are presented in Sect. 3.5. The general interplay of the components is sketched in Fig. 1 and an overview over the processing steps is given in Fig. 2.

Fig. 1
figure 1

Schematic overview of the presented architectural framework

Full size image
Fig. 2
figure 2

Processing steps in the Cognitive Weeding project. Images on the top show ground-based data, images on the bottom UAV data. The right figure shows the application map, spots are indicating the location of harmful weeds

Full size image

3.1 Agricultural Expert Knowledge and Field Trials

To feed the expert system (Sect. 3.2) with needed rules, literature on weed crop interactions was consulted. A rule for categorizing each plant detected by our sensors is necessary. For this, we choose the categories crop, harmful and harmless weeds (Fig. 1). Only weeds categorized as harmful are going to be regulated. The first criterion is a species-specific weed threshold, which is already known since the 1980s. Unfortunately, it did not establish in common agricultural weeding practices [15]. This is most likely due to the labour-intensive acquiring of species-specific plant counts and the concerns about upbuilding soil seed banks. Advances in image acquisition and processing technology facilitate threshold monitoring. Furthermore, sensors acquire data at a very high resolution, which takes patchy distribution of weeds into account, thus avoiding averaging weed distribution over the whole field [11].

Most relevant criteria for the categorization into harmful and harmless weeds after applying species-specific thresholds are the distance between crop plant and weed, and the relative germination time of the weed plants. Most crops have a critical period in which they react with high yield depression as a result of competition. Moving from direct competition to more general effects of weeds, potential seed production posing risk for subsequent years is of interest. Additionally, certain weeds are potential hosts for pests, whereas others are hosts for beneficial insects. Roots of weeds can also reduce soil erosion. Furthermore, rare species can be protected, and invasive species can be regulated strictly upon their first occurrence. [12]

To measure the impacts of this approach to single plant specific weeding, we set up field trials in both maize (Zea mays L.) and beets (Beta vulgaris L.). In this trials, we compared standard whole-area weeding with our site-specific approach. Weeding was done with the selective hoe (Fig. 4a). We created one variant for each image acquisition implement and one control group with no weed regulation at all. During these trials, we evaluated species-specific weed density and coverage. Height, leaf area index and yield were determined for maize.

Existing weed thresholds are developed in single competition between crop and one specific weed species. In the field, multiple weed species coexist simultaneously. Literature indicates that a multi-species weed community has lower yield impact compared to a weed infestation dominated by a single species [1]. To investigate the interspecific competition effects, we conducted a second trial. Since there are no weed thresholds for maize and beets, we also aimed to approximate potential thresholds for the most relevant weed species on our sites.

3.2 Expert System

Based on this expert knowledge from Sect. 3.1, an expert system carries out the decision process, whether a plant is considered harmful or not, using specifically formulated rules.

Using the plant detections from the sensors, the plant instances are stored in a grid map. Now, for each plant, the number of plants of the same species is counted in the surrounding square meter (\(Count/m^2\)). This builds the foundation for the decision process and is assigned to each plant, as described in [20].

For example, in a cluster with 10 plants of Chenopodium album and 3 plants of Sinapsis arvensis, each C. album plant is assigned the value 10, and each S. arvensis plant is assigned the value 3.

During rule inference in the expert system, the \(Count/m^2\) value of each plant is compared with a species-specific threshold. When exceeding the threshold, a plant is categorized as a harmful weed, otherwise it is considered harmless. If a threshold is not available for a species, a threshold for a higher taxon is used.

For example, the species Echium vulgare is detected, but there is no threshold for E. vulgare or its genus, Echium. There is a threshold for the family Boraginaceae, so it is used in the expert system. The highest taxon with an available threshold is class with Magnoliopsida (monocotyl) and Liliopsida (dicotyl), which includes all relevant plants. Additional rules are possible, but currently not implemented.

Also, it is possible to handle more general classifications. Sometimes, species-specific classifications are hard to achieve. In this case, a more general classification i.e., genus-specific can be processed identical to species-specific classifications. This also applies to unspecified weeds, that are distinguished from the crop but are not classified further. Those are treated as monocotyl, since monocotyl plants have a lower damage threshold than dicotyl plants. Furthermore, detected crops are categorized as such and are excluded from the threshold rule inference.

After categorizing all detected plants, an application map is generated based on the position of all harmful weeds, marking the spots, where weed removal is required (Fig. 2).

3.3 Data Acquisition

The expert system mentioned earlier requires information on the plants present in the field. To acquire this information, the plant detection was carried out in field trials using both a ground based multispectral sensor and a multispectral UAV-camera (Fig. 3). Data from both sensors were collected two to three days before the hoeing events, depending on weather conditions.

The ground-based sensor data was recorded using a JAI fusion multispectral sensor covering the wavelength ranges of 400-700 nm (visible RGB spectrum) and 750-900 nm (near infrared). The camera is supported by 10 Hz short-pulsed light source to cope with the influence of sunlight. It was mounted on a sensor carrier in front of a tractor at a height of 0.5 m, resulting in a ground resolution of 0.5 mm per pixel. The images were annotated with position and orientation information from 2 RTK GNSS antennas, and pre-processed to even out lighting conditions before being fed to the classifier. A semantic segmentation approach based on the EfficientNet architecture [24] was used, which outputs the most likely of up to 81 distinct plant species per super pixel. Sensor data and classification results are shown in the top of Fig. 2. On our test set, an F1 score of 0.92 for maize and F1 scores between 0.74 and 0.84 for the weed classes were achieved. An earlier version of the system is described in [5].

Fig. 3
figure 3

Ground based sensor mounted on the sensor carrier in front of a tractor and UAV with multispectral camera during data acquisition on 19th of June 2023

Full size image

UAV imagery was captured using a MicaSense Altum multispectral camera mounted on a DJI Matrice 210. The flight altitude was 10 m with a front and side overlap of 75%, resulting in a ground resolution of 3.5 mm. Images were stitched using Agisoft Metashape (Version 1.7.2), georeferenced with field targets (located with a bi-differential GNSS receiver) and radiometrically calibrated using a white reference panel. The detection of maize and up to six occurring weed species was done with a pixel-based approach using a Support Vector Machine [14]. The classification results were post-processed using an NDVI mask and a majority filter to eliminate misclassifications and vectorized to polygons. Sensor data and classification results are shown in the bottom of Fig. 2. F1 scores between 0.7 and 0.91 were achieved for maize, and mean F1 scores between 0.64 and 0.90 for the weed classes, depending on phenological phase and weed density.

3.4 Data Storage

To standardize the data interface and to enable spatio-temporal-semantic queries, the sensor data as well as the georeferenced plant detections are stored in the semantic environment representation SEEREP [17]. The position of the detections is stored as a 3D point with an attached timestamp and a semantic label. Based on this information, indices are created to enable efficient queries based on the position, the timestamp and the semantic label. Additionally, the semantic label is extended by a unique identifier representing a specific plant instance. If the detections of consecutive images in the RGB stream of the ground based system can be associated with each other, they get the same unique identifier of the plant instances so that only one plant instance is present instead of multiple detections.

These plant instances and their positions can be queried in the next step by the expert system (see Sect. 3.2) to get all needed information for the reasoning process. For this specific application, the images themselves are not needed in the following steps. Thus, it is not strictly necessary to store the images in SEEREP and the storage of the images can be omitted for efficient data storage.

3.5 Selective Weeding Actuators

Finally, the application map created by the expert system is used to control the actual weeding hardware. For the single plant specific regulation, a selective hoe and a precision sprayer from Amazone are developed further and used within this project. The selective hoe (Fig. 4a) can actuate each of the hoe coulters individually. Each coulter has a working area width of 0.65 m at a 0.75 m crop row distance and a resolution of 0.8 m in the driving direction at 7 km/h. The selective hoe was already used in first experiments. The precision sprayer is still under development and aims to have a spot size of 0.1 m by 0.1 m (currently at 4 km/h, higher velocities are under development). The current prototype is shown in Fig. 4b.

Fig. 4
figure 4

The two selective weeding actuators used in the experiments

Full size image

4 Experiments

In 2023, our initial field trials demonstrated the fundamental functionality of all proposed framework modules. During three points in time in the season, we utilized ground-based and UAV-based sensor data to create application maps of different parts of our test field using the framework presented. Using the selective hoe (Fig. 4a) we achieved a reduction of the regulated area between 20 % and 93 % for organic maize.

During these experiments, a few issues became apparent which will be addressed in the remainder of this research project: (1) The ground-based plant detections are generated from an image stream, where a single plant appears in multiple consecutive images. Even though fused plant instances can be represented in the data storage, each appearance is currently considered as a single plant instance, which inflates the total plant count and results in more weeds categorized as harmful. Fusing the plant detections reliably to single instances has been challenging and is part of current work. Until now, an estimate of how often a single plant appears in the data is used to reduce the total number per species, although this is just considered a rough projection. (2) The UAV-based plant classification was done using a pixel-based Support Vector Machine. While this method is fast to apply and does not require a large amount of training data (compared to deep learning methods), it does not generalize well over different phenological phases, weather, and lighting conditions. Therefore, it needs to be re-trained for each classification. Also, the classification may contain some misclassified pixels that are treated as individual plants in the decision process. For the next season, a more efficient instance segmentation classifier will be trained based on the data collected in 2022 and 2023 to make the classification of UAV imagery less time-consuming and more accurate. (3) In the rule inference process, \(n/m^2\)-values are compared with thresholds based on matching taxon names. When different labels are used between detector and the expert system, thresholds can not be applied successfully (i.e. Z. mays and corn refer to the same species). To solve this, an ontology based on the Agrovoc thesaurus [23] will be used as common vocabulary, so different labels are associated with the same plant or taxon. (4) Furthermore, the quality of the classification is critical for the final result. The classifiers often identify weeds as "unspecified weed", which can render the species-specific decision process less effective. Further improvement of the weed detection is therefore necessary. At least, a distinction between monocotyl and dicotyl should be possible reliably.

5 Conclusion

The prototype of the framework developed in Cognitive Weeding is working. As mentioned in Sect. 4, some issues will be optimized for the next season. In general, the results are promising. The approach to use formalized expert knowledge for single plant-specific decision-making on whether to regulate the weed or not is very convincing. The known damage threshold system for weeds could be revived with the use of this concept and iteratively adapted to site-specific conditions. Further rules could be added to incorporate more environmental and biodiversity based considerations, though the transferability is ambiguous.

On the one hand, the sensor system can be exchanged to include further relevant information based on higher resolution, different carrier platforms or spectral bands. As mentioned, rules for the expert system can be adapted and the creation of application maps can be applied to other use cases. For example, weed management of harmful weeds in grain crops or grasslands could easily be implemented. Additionally, the presented architectural framework eases the development of weed thresholds, by regulating weeds in plots so that a given plant density is present, and the competition effects can be evaluated. With bigger alterations, even the generation of fertilizing maps or harvesting maps may be possible.

On the other hand, when using the developed system for weed control on different sites or in subsequent years, limitations may become evident. Changes in soil conditions, a different weather conditions and crop rotation will lead to the emergence of different weed species. Additionally, each crop reacts differently to competition of the same weed species. Also, weather conditions can shift competition between crops and weeds. Therefore, experiments in the next year will be particularly intriguing. Moreover, with the newly available spot sprayer featuring a herbicide spot radius of 5 cm, the area requiring regulation will be further reduced. We expect significant herbicide savings, potentially meeting the goals for GAP 2030 with a 50 % reduction in herbicide usage.