Introduction

Chemical pesticides are used to control insect pests, weeds and diseases (collectively referred to as pests) in most intensive agricultural systems. There are, however, increasing concerns about the direct and indirect effects of their use on public health and the environment (Thompson et al., 2020; Whitmee et al., 2015). The repeated use of pesticides has also led to the development of pest resistance in many instances, rendering products less effective and pest control less manageable (Gould et al., 2018). To promote more sustainable agricultural systems, EU Directive 2009/128/EC established eight key principles to reduce pesticide use, including that users should apply pesticides where and when needed, and that the decision to apply a spray should be based on pest monitoring and thresholds (Barzman et al., 2015).

Decision support systems are tools that guide users in making decisions about when, and possibly where, to apply pesticides. Ideally, by incorporating knowledge of pest dynamics and impacts in a DSS, pesticides can be applied only when they are needed, increasing pesticide use efficiency, maintaining control of the pest without allowing an impact on yield. The use of DSS often leads to a reduction in the amount of pesticide applied (Lázaro et al., 2020), or recommendations for alternative control strategies (e.g. Mensah, 2010; Zhang & Swinton, 2009), reducing the negative impacts on the environment and human health. Indeed, a recent meta-analysis of field experiments testing the effectiveness of DSS demonstrated that they can reduce the amount of pesticide used by half without compromising yield (Lázaro et al., 2021).

Initial methods to guide the need for treatment took the form of control thresholds – pest densities at given crop growth stages above which a pesticide should be applied (e.g. Pedigo et al., 1986). More complex approaches were developed into algorithms and incorporated into decision support systems (DSS) or decision support tools, which can vary considerably in their design and complexity. Some DSS calculate the risk of pest outbreaks based on the environmental suitability for the target pest populations at a location, and use this risk to guide if a treatment should or should not be applied (e.g. te Beest et al., 2009a, 2009b). More complex DSS may incorporate process-based models that simulate crop phenology and pest lifecycles to guide the optimal application time and dose of an individual pesticide product (e.g. Johnen et al., 2010, Parsons, et al., 2004). Recent DSS have included the use of machine learning algorithms (Jabir & Falih, 2022), cloud-based infrastructure (Rupnik et al., 2019), and real-time updating (Rossi et al., 2014).

However, despite an increasing number of DSS being published, uptake has been slow (Gent et al., 2013). There are several reasons suggested for this low uptake, and these vary with region (Marinko et al., 2003), but they include lack of trust in the performance of the DSS, the perceived need for additional training, and a failure to demonstrate positive economic benefits (Rose et al., 2017, Jørgensen et al., 2007, Parker & Campion, 1997, Marinko et al., 2003).

This lack of confidence in DSS may be a consequence of the difficulties involved in validating DSS. To validate a DSS, field experiments need to be designed and carried out that compare the pest control in trials following a DSS to those following a standard application programme for that region. However, such field experiments are typically limited in space (few locations) and time (few years), due to financial and time constraints. As a result, the conditions represented in the validation studies may not be representative of those under which the DSS is used. With limited validation field trials, sites or years that have an abnormal pest intensity are likely to be missed. Several studies have demonstrated that DSS can perform poorly when examined in environments in which they were not developed (e.g. Hijmans et al., 2000). Similarly, field tests of models in the Nordic-Baltic region for control of Z. tritici showed variable accuracy in guiding control decisions, depending on the country they were used in. The models tested were developed in Denmark and showed (as might be expected) the highest accuracy in Danish trials (Jørgensen et al., 2020). The same models also showed a much bigger potential for reducing fungicide treatments in relative dry seasons (when there were fewer opportunities for splash dispersal) compared with more normal seasons during which standard treatments performed similarly to the DSS. For these reasons, it is desirable to demonstrate that DSS are accurate and effective in the environments in which they may potentially be used.

Additionally, evaluations of DSS often report only the accuracy of the DSS predictions whether a DSS correctly predicts the occurrence of a pest outbreak (e.g. Small et al., 2015, Sciaretta et al., 2019), and not report on the yield impact or cost benefit. However, an end user is likely to be more interested in the value obtained from using the DSS, whether economic (the extra profit achieved by using the DSS) or environmental (the reduction in the amount of pesticide used) or both. Potentially more important, users need to be sure that using the DSS is unlikely to result in a loss (either of profit or environmental impact); particularly relevant if the DSS predicts there is no need for pesticide treatment, but a severe pest outbreak subsequently develops (a false negative prediction). Being able to provide users with a cost–benefit analysis of a DSS before they use it could lead to increased trust and therefore uptake in DSS.

Due to the constraints in experimentally validating DSS, what is needed is a suite of methods that are able to quantify the value of DSS using existing datasets. However, existing large data sets are unlikely to have been collected for the purpose of testing DSS. For example, data may have been gathered to measure the efficacy of new pesticide products in a range of environments, comparing them to current ‘standard’ spray programmes and untreated plots, or from research experiments undertaken by various institutions with no shared protocol.

In this paper we present a generic framework for estimating the value of DSS using pre-existing data. We then describe how the methodology can be applied in practice to four case studies of DSS systems with contrasting characteristics.

Method

Theoretical framework

The economic value of a DSS \(\left(V\right)\) is defined as the difference between the cost of not following the DSS, i.e. using a standard pesticide spray programme \(\left(C_\text{S}\right)\), and the cost of following the DSS guidance \(\left(C_\text{DSS}\right)\). The value of a DSS may therefore be stated as:

$$V={C}_{\text{S}}-{C}_{\text{DSS}}$$
(1)

The costs incurred when using a given spray programme \(\left(q\right)\) consist of the cost of the pesticide, the cost of applying the pesticide, and the price of any crop yield lost due to any uncontrolled pest damage, and is given by:

$$C(x,q)=F(q)+\rho L\left(x,q\right)$$
(2)

where \(C(x,q)\) is the cost under a given pest intensity, \(x\), and spray programme, \(q\), \(F(q)\) is the price of the spray programme (including the amount of pesticide applied and the cost of application), and \(L(x,q)\) is the amount of yield lost when the spray programme \(q\) is applied in a situation with pest intensity \(x\), and \(\rho\) is the price of a unit of yield.

When using a DSS, the spray programme \(q\) is generated by the DSS from a prediction of pest intensity \({x}_{\text{pred}}\), and therefore \(q\) is a function of \({x}_{\text{pred}}\):

$${C}_{\text{DSS}}\left(x,q\left({x}_{\text{pred}}\right)\right)=F\left(q\left({x}_{\text{pred}}\right)\right)+\rho L\left(x,q\left({x}_{\text{pred}}\right)\right)$$
(3)

Therefore, the cost associated with using a DSS depends on both the actual pest intensity \(x\), and the pest intensity predicted by the DSS, \({x}_{\text{pred}}\), which then informs the spray programme. The cost of a standard spray programme depends only on the actual pest intensity and the standard spray programme:

$${C}_{\text{S}}\left(x\right)=F\left({q}_{s}\right)+\rho L\left(x,{q}_{s}\right).$$
(4)

Both the actual pest intensity, \(x\), and the pest intensity predicted by the DSS, \({x}_{\text{pred}},\) vary in space and time. This variation can be described by a probability density function, \(f\left(x,{x}_{\text{pred}}\right)\), which describes the joint distribution of the predicted pest intensity and the observed pest intensity. This joint distribution captures both the distribution of the real observed pest intensity, and how well the DSS performs at different pest intensities – an accurate DSS will predict values of \({x}_{\text{pred}}\) that are close to \(x\). Poorly performing DSS may be less accurate and/or less precise. Figure 1 illustrates possible relationships between the observed pest intensity and that predicted by the DSS.

Fig. 1
figure 1

Illustrations of theoretical forms of the join distribution function \(f\left(x,{x}_{\text{pred}}\right)\). White represents the largest density and black represents near zero density. Each distribution is from a DSS that predicts with (left) very good accuracy and precision, (middle) good accuracy but poor precision, and (right) poor accuracy but good precision

Full size image

The value of a DSS can then be estimated by calculating the difference in costs between the DSS and the standard spray programme at all possible combinations of the observed and predicted pest intensity, weighted by their frequency, resulting in a distribution of the value of the DSS. The resulting distribution of the value of the DSS can give an indication of how likely different profit margins are, including the probability of each DSS resulting in an economic loss \(P(V<0)\).

To summarise the value of a DSS, the expected value of the DSS over all pest intensities may be calculated by integrating over the joint probability density function:

$$\text{E}\left(V\right)={\int }_{0}^{{x}_{\text{max}}}{\int }_{0}^{{x}_{\text{max}}}\left\{{C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{x}_{\text{pred}}\right)\right\}f\left(x,{x}_{\text{pred}}\right)\text{d}x\text{d}{x}_{\text{pred}}$$
(5)

where \({x}_{\text{max}}\) is some maximum possible pest intensity.

To implement the above approach, the following steps are required:

  1. (i)

    Derive a distribution describing the joint probability of the actual pest intensity and that predicted by a DSS, \(f\left(x,{x}_{\text{pred}}\right)\)

  2. (ii)

    Obtain the spray programme recommended by the DSS given every predicted pest intensity, i.e. derive \(q({x}_{\text{pred}})\)

  3. (iii)

    Calculate the costs associated with following both the recommendation of the DSS given that the true pest intensity was \(x\), and of following a standard spray programme. That is, calculate \(L(x,q)\), and thus \({C}_{\text{DSS}}(x,{x}_{\text{pred}})\) and \({C}_{S}(x)\) for each possible combination of \(x\) and \({x}_{\text{pred}}\).

Whilst theoretically possible, these parameter estimation steps are rarely achievable due to a lack of data, and/or the complexities associated with the DSS. The more complex the model behind the DSS, the greater the data requirements to parameterise \(f(x,{x}_{\text{pred}})\) and \(C(x,q)\), and with the more complex DSS such data is unlikely to be available. However, by tailoring Eq. 5 to an individual DSS, it is possible to reframe the framework in a manner that requires substantially less data to estimate.

To illustrate the implementation of the framework we consider four case studies, with different types of DSS and/or different data availability; demonstrating how to reframe Eq. 5 for each. The four case studies considered are:

  1. 1.

    DSS that inform a user when it is safe to reduce the applied pesticide dose (e.g. Kudsk, 2007; Schepers et al., 1996).

  2. 2.

    DSS that guide the number of pesticide applications to apply to a crop to protect against a pest. The simplest case is when the DSS determines whether a pesticide application should be applied or not (Shtienberg, 2013). Such DSS often do not explicitly predict pest intensity \(\left(x_\text{pred}\right)\) but instead only provide an indication of risk, for example when humidity has been high for a given number of days (e.g. Hansen et al., 1994), and so surrogates for \({x}_{\text{pred}}\) must be used in the analysis of value.

  3. 3.

    DSS that guide when to start spraying (e.g. Marimon et al., 2020).

  4. 4.

    A case when no data are directly available but data on the accuracy of prediction exists in the form, for example, of a confusion matrix.

The implementation of case studies 1 and 4 are shown in the Supplementary Information.

Results

A DSS developed to predict whether fungicide dose reduction is appropriate

The te Beest DSS

The DSS described in te Beest et al. (2009a) supports decisions on controlling Zymoseptoria tritici (previously Mycosphaerella graminicola; the causal organism of septoria tritici blotch). Zymoseptoria tritici is a fungal pathogen that is a serious problem in wheat (Hagelskjær and Nistrup Jørgensen, 2003), and is typically controlled by fungicide applications at key crop growth stages.

Based on weather data from January to April the DSS predicts whether a damaging epidemic is likely to occur, and when a damaging epidemic is not predicted a reduced fungicide dose may be appropriate. This DSS therefore does not explicitly predict pest intensity \(\left(x_\text{pred}\right)\) but instead provides a binary output indicating whether or not it is safe to lower the dose. In two subsequent papers (te Beest et al., 2009b, 2013) the authors calculated the optimal fungicide dose to maximise the economic return. Their analysis on disease susceptible varieties forms the basis for the example presented here.

Formulating the value calculation for the DSS

As noted above, the prediction of disease intensity is captured as one of two discrete states; a low severity season where a reduced dose rate is appropriate \(\left(\delta_l\right)\), or a season where a damaging epidemic is predicted and a standard dose rate is likely to be required \(\left(\delta_h\right)\). Therefore, the predicted disease severity \({x}_{\text{pred}}\) is represented by the surrogate values \(\{{\delta }_{l},{\delta }_{h}\}\). Equation 5 therefore simplifies to:

$$\text{E}\left(V\right)={\sum }_{\delta }{\int }_{0}^{{x}_{\text{max}}}\left\{{C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,\delta \right)\right\}f\left(x,\delta \right)\text{d}x$$
(6)

The distribution \(f\left(x,\delta \right)\), which describes the relationship between observed uncontrolled disease severity \(x\) and the dose predicted by DSS, simplifies to

$$f\left(x,\delta \right)={\gamma }_{l}{g}_{l}\left(x,\delta \right)+{\gamma }_{h}{g}_{h}\left(x,\delta \right)$$
(7)

where \({g}_{l}\left(x,\delta \right)\) is the observed distribution of uncontrolled disease severity when a low dose is predicted \(\left({\delta=\delta}_l\right)\) and is zero otherwise, and \({g}_{h}\left(x,\delta \right)\) is the observed distribution of disease severity when a standard dose is predicted \(\left({\delta=\delta}_h\right)\) and is zero otherwise. The parameters \({\gamma }_{l}\) and \({\gamma }_{h}\) scale according to the proportions of the total predictions allocated to \({\delta }_{l}\) or \({\delta }_{h}\) (ensuring that \(f\left(x,\delta \right)\) integrates to one). As severity is bounded between 0 and 100%, a beta distribution is an appropriate distribution to model the severity (scaled between 0 and 1).

To derive the yield loss, \(L\left(x,\delta \right)\), associated with applying dose \(\delta\) to disease of severity \(x\), we adopted the equations reported in te Beest et al. (2009b). They assume that the amount of yield lost increases linearly with the disease severity:

$$\left(x,\delta \right)={Y}_{0}\lambda S\left(x,\delta \right)$$
(8)

where \({Y}_{0}\) is the yield potential assuming no Z. tritici infection, \(S(x,\delta )\) is the treated disease severity and \(\lambda\) is the yield loss coefficient per unit severity. The treated disease severity, \(S\), is given by:

$$S(x,\delta )=\left(1-R(1-{e}^{-k\delta })\right)x$$
(9)

where \(R\) defines the maximum proportional reduction at high values of fungicide, \(\delta\) is the total amount of fungicide dose applied across the season and \(k\) determines the shape of the dose–response curve.

If a damaging epidemic is predicted then applying \({\delta }_{h}\) is appropriate hence the difference in cost between using the DSS or following standard practice is zero:

$${C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{\delta }_{h}\right)=0$$
(10)

If low disease severity is predicted then the difference in cost is given by

$${C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{\delta }_{l}\right)=\Delta F+P\theta {Y}_{0}\lambda x$$
(11)

where \(\theta =R\left({\text{e}}^{-{\text{kD}}_{\text{h}}}-{\text{e}}^{-{\text{kD}}_{\text{l}}}\right)\) and \(\Delta F\) is the difference in cost between the two spray programmes. The cost of application remains the same between the standard and DSS spray programmes, and so does not appear in Eq. 11.

Combining Eqs. 6, 10, and 11, the expected value of the DSS can be calculated analytically, by recognising that \(\int xf\left(x\right)dx\) is the expected value of a probability distribution, which for a beta distribution is \(\frac{\alpha }{\alpha +\beta }\):

$$\text{E}\left(V\right)={\int }_{0}^{100}\left({C}_{S}\left(x\right)-{C}_{DSS}\left(x,{\delta }_{l}\right)\right)f\left(x,{\delta }_{l}\right)dx +{\int }_{0}^{100}\left({C}_{S}\left(x\right)-{C}_{DSS}(x,{\delta }_{h}\right)) f\left(x,{\delta }_{h}\right)dx={\int }_{0}^{100}\left\{\Delta F+\rho \theta {Y}_{0}\lambda x\right\}f\left(x,{\delta }_{l}\right)\text{d}x+0 =\Delta F{\int }_{0}^{100}f\left(x,{\delta }_{l}\right)dx+\rho \theta {Y}_{0}\lambda {\int }_{0}^{100}xf\left(x,{\delta }_{l}\right)dx= {\gamma }_{l}\Delta F+ \rho \theta {Y}_{0}\lambda {\gamma }_{l}\left(\frac{{\alpha }_{l}}{{\alpha }_{l}+{\beta }_{l}}\right)\begin{array}{c}={\gamma }_{l}\left\{\Delta F+\frac{\rho \theta {Y}_{0}\lambda {\alpha }_{l} }{{\alpha }_{l}+{\beta }_{l}}\right\}\end{array}$$
(12)

There is an economic loss if \(V<0\). This can only happen when low disease intensity is predicted, and when \(\Delta F+\rho \theta {Y}_{0}\lambda x<0\).

Rearranging for disease intensity, we can see that an economic loss only occurs when

$$\frac{\Delta F}{-\rho \theta {Y}_{0}\lambda }<x$$
(13)

Parameterisation.

We fitted Eq. (7) to fungicide trials data for susceptible varieties of wheat described in te Beest et al. (2009b). The data were from across the UK: 25 site x year combinations spanning 2003 – 2005. To quantify disease intensity \((x)\) we use the disease severity (%) on leaf 2 at growth stage 75, which te Beest et al. (2009b) found to have a good relationship to yield. We fitted separate distributions (Eq. 7) to the untreated severity for each dose level using the `fitdist` function from the `fitdistrplus` package in R. Together, these distributions, scaled by the proportion of trials for which were observed (denoted \({\gamma }_{i}\)), make up the joint distribution function (Fig. 2). The fitted parameters are given in Table 1. As expected, when a severe epidemic was not predicted the distribution is skewed towards the lower end of the severity range, with a mean severity of 8.0%, whereas when a severe epidemic is predicted the distribution skews towards high severity with a mean severity of 16.9%.

Fig. 2
figure 2

Beta distributions (solid lines) fitted to the severity of septoria tritici blotch on leaf 2 of wheat at GS75 from untreated trials on susceptible cultivars in te Beest (2009b), when (left) a severe epidemic is not predicted, or (right) a severe epidemic is predicted by the DSS

Full size image
Table 1 The shape parameters of the Beta distributions describing the severity of septoria tritici blotch on leaf 2 of wheat at GS75, depending on whether te Beest et al. (2009a) predicted high or low disease intensity, and the scaling parameter applied to each distribution
Full size table

To parameterise the yield loss relationship, we used the values given in te Beest et al. (2009a) (\({Y}_{0}=9.15\) t ha−1, \(\lambda =0.0081\), \(R=0.996\), and \(k=2.8\)) with a range of crop prices (100, 200 and 300 £ ha−1) and 45 £ ha−1 per spray dose. Following te Beest et al. (2009a), we assume a standard spray programme adopts a total dose of 1.5 units ha−1. If an epidemic is predicted then the user is guided to follow this standard recommendation, otherwise a more conservative application of 1 unit ha−1 is applied. The results are shown in Table 2. The expected value of this DSS decreases as the wheat price increases, and the probability of incurring a loss when using the DSS increases.

Table 2 The expected value of the te Beest model at three different wheat price points, and the probability of a user using the DSS making a loss
Full size table

Evaluating a DSS that informs on the number of required pesticide applications

The Crop Protection Online DSS

Our second case study concerns the Crop Protection Online (CPO) DSS. This DSS was developed in Denmark and includes models for several pests and pathogens, one of which is Zymoseptoria tritici (Hagleskjær & Jørgensen, 2003). The CPO DSS aims to achieve the best control with the lowest inputs.

For Z. tritici, CPO tracks the cumulative number of days that have greater than 1 mm of rain between wheat growth stages (GS) 32 and 71. When susceptible cultivars are grown, the model suggests that fungicide is applied following four days with more than 1 mm of rain, whereas on resistant cultivars five days are required before a fungicide application is suggested and the model first starts counting at GS 37. Once sprayed, the model assumes protection for ten days, after which a spray is suggested after a further four or five days of rain (Jørgensen et al., 2020).

Formulating the value calculation for the DSS

To estimate the value of the CPO DSS, we evaluate how the number of sprays predicted by CPO related to the observed disease severity and how that in turn affected costs. Therefore, the predicted disease intensity \({x}_{\text{pred}}\) is represented by the surrogate value, \(n\), the number of sprays predicted by the DSS. The expected value of the DSS therefore degenerates from Eq. 5 to:

$$\text{E}(V)={\sum }_{n}{\int }_{0}^{{x}_{\text{max}}}\left\{{C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,n\right)\right\}f\left(x,n\right)\text{d}x$$
(14)

where \(f(x,n)\) describes the distribution between the untreated disease severity (\(x\)) and the number of suggested sprays.

The distribution \(f\left(x,n\right)\), which describes the relationship between untreated disease severity \(x\) and the number of sprays predicted by DSS, simplifies to

$$f\left(x,n\right)=\sum_{j=0}^{n}{\gamma }_{j}{g}_{j}\left(x\right)$$
(15)

where \({g}_{j}\left(x\right)\) is the distribution of untreated disease severity when \(j\) sprays are predicted and is zero otherwise. As above we assume that the form of these distributions can be described by a Beta distribution, and that the parameters \({\gamma }_{j}\) scale according to the proportions of the total predictions allocated to each number of sprays (ensuring that \(f\left(x,n\right)\) integrates to one).

We assume that yield is given by

$$Y\left(x,D\right)={Y}_{0}-\lambda S\left(x,n\right)$$
(16)

where \({Y}_{0}\) is the yield potential assuming no Z. tritici infection, \(S(x,n)\) is the disease severity and \(\lambda\) is the yield loss coefficient per unit severity. The disease severity, \(S\), is given by:

$$S\left(x,n\right)=x{e}^{-\kappa n},$$
(17)

where \(x\) is the untreated severity, and \(\kappa\) is a shape parameter. Therefore, similar to above, yield loss increases linearly with the disease severity, \(L\left(x,n\right)=\lambda S\left(x,n\right)\) and the difference in cost between a standard spray programme and the DSS is given by:

$${C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,n\right)=\Delta F+\rho \lambda \left\{S\left(x,{n}_{s}\right)-S\left(x,n\right)\right\}$$
(18)

where \({n}_{s}\) is the number of sprays applied as standard and \(\Delta F\) is the difference in cost between the two spray programmes.

Parameterisation

Data from 186 field trials were used to estimate both the distribution function and the yield loss relationship. The trials were conducted to assess the efficacy of new fungicide products and were carried out between 2014 and 2018 in eight countries across Europe (Table 3) by Corteva Agriscience (Indianapolis, United States). In each trial, the severity of several foliar diseases on the top six leaves of wheat was recorded at various time points during the growing season, and crop yield was recorded for some of the trials. The crop growth stage was also recorded at various time points in most trials. Each trial consisted of two or more fungicide treatment programmes, including a control where no fungicide was applied (untreated). There were typically four replicates per treatment in a randomised block design. The cultivars used were susceptible or moderately susceptible to Z. tritici.

Table 3 The number of trials in each country in the dataset
Full size table

To quantify disease intensity \((x)\) we use the disease severity on leaf 2 at growth stage 75 (GS75) – the same metric as in the previous section. As noted above, this has been found to be a strong predictor of yield, which was confirmed to be the case for the data used in this case study. Where necessary, the date of GS75 was linearly interpolated from the nearest recorded growth stages, and the severity on leaf two was estimated at that date. For trials in which the severity could not be estimated, either because there wasn’t suitable growth stages or severity reported to allow interpolation, the trial was omitted from the analysis.

To calculate the number of sprays suggested by the CPO DSS at each of the field trials, hourly precipitation data was downloaded from the ERA5 reanalysis (Copernicus Climate Change Service Climate Data Store, 2020), which has a 30-km grid resolution. After coding the rules of the DSS, described in Jørgensen et al. (2020), \(n\) was calculated from the weather data for each trial for which severity data was available (58 of the 186 trials).

Between one and four sprays were predicted for each trial by the CPO DSS, with 4, 24, 10, and 2 trials having 1, 2, 3, and 4 sprays, respectively. To fit the distribution function (Eq. 15), we used the `fitdist` function from the `fitdistrplus` package in R. The fitted distributions are shown in Fig. 3 with parameters reported in Table 4.

Fig. 3
figure 3

The probability density of the untreated severity of septoria leaf blotch on leaf 2 of wheat plants at growth stage 75, when the CPO DSS predicts 1, 2, 3, or 4 sprays. Both the data (represented as a scaled histogram) and the fitted probability distributions (red lines) are shown

Full size image
Table 4 The shape parameters of the Beta distributions, describing the severity of septoria on leaf 2 of wheat at growth stage 75, depending on whether the CPO DSS predicted 1, 2, 3, or 4 sprays for each trial, and the scaling parameter applied to each distribution
Full size table

To fit the parameters for the yield loss relationships (Eqs. 16 and 17) a linear model was fit with the `lm` function in R. The fitted parameter values were \(\kappa =0.43\) with standard error \(0.026, {Y}_{0}=11.36\) and \(\lambda =0.05\) with standard errors \(0.22\) and \(0.005\) respectively, \({R}^{2}=0.45\) (Fig. 4).

Fig. 4
figure 4

Left, the relationship between untreated and treated severity (%) when trials are treated with 0, 1, 2, 3 and 4 fungicide applications. The fitted parameter for \(\kappa =0.43\) with standard error \(0.026\). Right. The relationship between the severity on leaf 2 at GS75 and yield in all treatments. The fitted yield loss relationship had parameters \({Y}_{0}=11.36\) and \(\lambda =0.05\) with standard errors \(0.22\) and \(0.005\) respectively, \({R}^{2}=0.45\)

Full size image

In our examples we consider scenarios with a range of crop prices (100, 200 and 300 £ t−1) and 25 £ ha−1 per spray application. We assume standard practice is to apply two fungicide applications, \(n=2\). The distribution of value of the DSS was calculated by Monte Carlo simulation accounting for the \(f\) distribution and the errors in the fitted yield relationships. Values for the expected yield and probability of loss are given in Table 5 with associated errors propagated from the fitted yield response functions.

Table 5 The expected value of the CPO DSS for various prices of wheat
Full size table

Evaluating DSS that inform when to start a pesticide application programme

The Hutton Criteria DSS

The third DSS we evaluate is the Hutton Criteria, which was developed to identify when potato crops are at high risk of infection by potato late blight, caused by Phytophthora infestans. The DSS uses daily air temperature and humidity to determine risk. A high risk ‘Hutton Criteria’ period occurs when two consecutive days have a minimum temperature of 10 °C, and at least six hours of relative humidity at or above 90%.

While the Hutton Criteria can be used to identify risk periods throughout the potato growing season, it is often used to identify when to start applying a weekly fungicide programme. In this case study, we estimate the value of the Hutton Criteria in identifying the optimal timing for the first spray.

Formulating the value calculation for the DSS

For this case study we reformulate Eqs. 3 – 4 to represent the disease intensity in terms of the number of days away from a theoretical optimal timing of the first spray, denoted \({t}_{0.}\) In this formulation, the variable \({t}_{0}\) is conceptually similar to our true pest intensity \(x.\) In an environment favourable to a pest \({t}_{0}\) would occur earlier in the growing season, whereas in unsuitable environments \({t}_{0}\) would occur later.

Equation 5 is reformulated as:

$$\text{E}\left(V\right)={\int }_{{t}_{\text{start}}}^{{t}_{\text{end}}}{\int }_{{t}_{\text{start}}}^{{t}_{\text{end}}}\left\{{C}_{\text{S}}\left({t}_{0},{t}_{S}\right)-{C}_{\text{DSS}}\left({t}_{0},{t}_{\text{pred}}\right)\right\}f\left({t}_{0},{t}_{\text{pred}}\right)\text{d}{t}_{0}\text{d}{t}_{\text{pred}}$$
(19)

where \({t}_{\text{pred}}\) is the time at which the DSS suggests the first spray should occur, \({t}_{s}\) the time at which it occurs under a standard application programme, and \({t}_{\text{start}}\) and \({t}_{\text{end}}\) can be considered to be the planting date and harvest dates respectively. We measure time as days after planting (\({t}_{\text{start}}=0)\) and assume that the joint distribution \(f\left({t}_{0},{t}_{\text{pred}}\right)\) can be described by a bivariate normal.

The losses due to disease are assumed to occur when the first spray is late. Therefore, we assume that no yield is lost if the spray programme starts earlier than the start of the epidemic, and that the amount of yield lost due to a spray program starting \(\tau\) days after the optimal spray time was assumed to increase linearly after \({t}_{0}\). That is:

$$L\left(\tau \right)=\left\{\begin{array}{c}0 \tau <0\\ m\tau \tau \ge 0\end{array}\right.$$
(20)

where \(\tau =t-{t}_{0}\).

The cost of a spray programme depends on both the ideal spray time, \({t}_{0}\), and the time at which the first spray was applied \(\left(t_f\right)\) and is given by:

$$C\left({t}_{0},{t}_{f}\right)=F\left({t}_{f}\right)+\rho L\left({t}_{0},{t}_{f}\right)$$
(21)

A spray programme based on the DSS may result in fewer or more sprays applied, changing the amount and therefore the cost of the fungicide applied and the application costs.

Parameterisation.

To estimate the distribution function and the yield loss relationship, data from 26 field experiments were used, which included information on the fungicide treatments and severity observations. Experiments typically included several treatments with four replicates per treatment in a randomised block design. Several of the experiments also included yield data.

The time at which the first spray was predicted by the Hutton Criteria DSS, \({t}_{\text{pred}}\), was estimated from weather data retrieved from the Copernicus climate data store for each time period and location of each trial. Risk periods were calculated according to the Hutton Criteria, and the first risk period following crop emergence was taken as the predicted first spray time, \({t}_{\text{pred}}\). Based on Skelsey et al. (2009), we assumed that the value of \({t}_{0}\) was a single latent period 5 days — before the first observation of symptoms.

The joint distribution function, \(f({t}_{0},{t}_{\text{pred}})\), was estimated by fitting a bivariate normal to the paired values of \({t}_{0}\) and \({t}_{\text{pred}}\), calculated for each trial, using the `mvtnorm` package. The fitted parameters were \({\mu }_{{t}_{0}}=34.8\), \({\mu }_{{t}_{\text{pred}}}=38.8\), \({\sigma }_{{t}_{0}}=14.4\), \({\sigma }_{{t}_{\text{pred}}}=9.0\), and \(\rho =0.47\) (Fig. 5).

Fig. 5
figure 5

The fitted joint distribution (contour lines) of the optimal starting spray time \(\left(t_0\right)\) and the predicted starting spray time \(\left(t_p\right)\). Data are depicted by the points

Full size image

The data available to fit the yield loss relationship were highly variable, and so we took a conservative approach and estimated the slope parameter so that the observed yield-loss relationship was an envelope of the data, giving an estimate of \(m=5\) (see Eq. (20)).

Based on the average first spray time in the data, we assume a standard first spray at 40 days after planting, after which sprays are applied weekly until day 110, therefore 10 sprays were applied under a standard spray program, while the number of sprays suggested by the DSS varied. The cost of an individual spray was taken as £45 per application.

Noting that the data we had to parameterise the yield loss relationship were variable and that the yield loss relationship was fitted to be conservative, the expected value of the Hutton Criteria was calculated to be £420 ha−1, but with a 45% chance of a negative value.

Estimating the joint distribution from literature statistics

The AHDB Sclerotinia Stem Rot DSS

The AHDB DSS produces risk alerts for sclerotinia stem rot, which is usually the main disease to consider during the flowering stages of oilseed rape. If spores are present, conducive weather is required for infection to occur. Fungicides for sclerotinia control are protectants and need to be applied prior to infection.

The forecast is weather-based and alerts farmers to the risk of infection (AHDB, 2022). The AHDB highlight the value of the tool for allowing farmers to delay treatment if risks are low, resulting in potentially fewer sprays. We therefore consider this example, assuming that standard practice would be to apply control.

Formulating the value calculation for the DSS

Data are presented on the AHDB website allowing farmers to assess the performance of the system. This takes the form of the probability of true positive \(\left(p_{yy}\right)\), false positive \(\left(p_{ny}\right)\), true negative \(\left(p_{nn}\right)\) and false negative \(\left(p_{yn}\right)\) alerts, where the first index in \({p}_{ij}\) relates to the observation and the second relates to whether an alert was produced by the DSS.

To use this information to estimate value, we simplify the density function \(f\left(x,{x}_{\text{pred}}\right)\) by degenerating it to one of these four discrete states. The expected value becomes:

$$E\left(V\right)=\sum_{x=y,n}\sum_{{x}_{\text{pred}}=y,n}\left\{{C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{x}_{\text{pred}}\right)\right\}{f}_{x, {x}_{\text{pred}}}.$$
(22)

where \({f}_{x,{x}_{pred}}={p}_{ij}\) with \(i\in \{y,n\}\) and \(j\in \{y,n\}\).

As standard practice is to apply a spray, if a risk alert is given by the DSS then farmers using the DSS would also follow standard practice and apply a spray. Therefore, when a risk alert is given, the difference in cost between using the DSS or following standard practice is zero:

$${C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{x}_{pred}\right)=0, \text{when }{x}_{\text{pred}}=y \text{and }x\in \left\{y,n\right\}$$
(23)

If no risk is predicted and no infection occurs then the only difference is that those who follow standard practice apply an unnecessary spray:

$${C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{x}_{\text{pred}}\right)=c, \text{when }{x}_{\text{pred}}=n \text{and }x=n$$
(24)

However, if no risk is predicted but an infection occurs then the farmers following the DSS don’t apply a spray and lose the price of the lost crop:

$${C}_{\text{S}}\left(x\right)-{C}_{\text{DSS}}\left(x,{x}_{\text{pred}}\right)=c-v,\text{ for }x=y, {x}_{\text{pred}}=n$$
(25)

where \(c\) is the cost of treating and \(v\) is the expected difference in value between a healthy and infected crop.

Therefore, the expected value can be simplified to:

$$E\left(V\right)=c{p}_{nn}+\left(c-v\right){p}_{yn}$$
(26)

Parameterisation.

For 2021 the following statistics were recorded on the AHDB website: true positive \(\left(p_{yy}=9.9\%\right)\), a false positive (\(\left(p_{ny}=3.4\%\right)\), a true negative (\(\left(p_{nn}=81.3\%\right)\) and a false negative \(\left(p_{yn}=5.4\%\right)\).

Therefore, for this case study, Eq. 26 becomes:

$$E\left(V\right)=0.813 c+0.054\left(c-v\right)$$
(27)

Therefore, as long as the cost of control \(\left(c\right)\) is greater than 0.06 times the value of crop loss \(\left(v\right)\) in a situation where a Sclerotinia outbreak occurs but the DSS does not give an alert, the expected value of the DSS is positive. The value of the DSS depends on the cost of control, and the value of any crop lost (Fig. 6). Under reasonable assumptions of the value of crop lost and cost of control, the expected value of the DSS is positive.

Fig. 6
figure 6

The expected value of the Sclerotinia Stem Rot DSS varies depending on the cost of applying control, and the value of the potential crop lost when a false negative alert occurs

Full size image

In this case, there is a 5.4% chance that the farmer will suffer a loss of \(v-c\) compared to if they had followed standard practice.

Discussion

In this paper we present a framework (Eq. 5) that can enable researchers to estimate the value of a DSS from data that have been collected for other purposes. This framework requires two key pieces of information: a relationship between the distribution of pest intensity in a region and the degree to which the DSS accurately reflects that pest intensity; and a relationship describing the effectiveness of pest management programmes over the whole distribution of pest intensity.

While the framework is simple in concept, the generic formulation is impractical in most instances as the data requirements are vast. Nevertheless, we have demonstrated that this formulation can be simplified to make it practically useful. Each of the practical implementations use surrogates for the predicted pest or disease intensity, such as the number of sprays suggested by the DSS, the dose suggested, or the timing of the first spray of a pesticide programme. This abstraction from the underlying infection level is particularly important for evaluating DSS that predict the risk of pest damage as opposed to the percentage disease or infestation. While all the case studies presented have focused on the use of fungicides to control fungal or oomycete pathogens, the framework could be applied to the application of insecticides to control invertebrate pests. However, it may be less applicable to systems that advise on weed control, as decisions made in one year can affect the long-term efficacy of weed control programs through effects on seed bank density (Benjamin et al., 2009). Because of this, the value of herbicide DSS would need to consider the long-term dynamics of the seed bank over the course of DSS-informed and standard herbicide application programmes.

The results presented in this paper demonstrate both the potential and the limitations of the framework. In all cases we were able to calculate the expected value of the DSS. In each case the economic value was positive, providing additional profit compared with using a standard spray programme. Where the uncertainties in the underlying distributions were able to be estimated – such as for CPO and the Hutton Criteria – we also calculated the distribution in the value, allowing an estimate of the probability of a negative value compared to a standard spray programme. For each of the DSS for which a distribution of value could be obtained, the probability of a user obtaining negative value was non-negligible, and as high as 45% for the timing of the first application of a potato fungicide programme. This outcome should remind users that DSS are supporting tools, and should not be seen as the sole means of making the final decision. Until more precise prediction models are developed for all relevant pest and diseases, field scouting and monitoring remains a very important element in IPM decision making.

The first case study we explored (te Beest et al., 2009b) was a simple DSS developed to predict whether a less intense fungicide programme could be used without compromising profit. The authors of the paper provided the information that we needed to parameterise the joint distribution describing the relationship between the observed disease intensity and that predicted by the DSS. To ensure consistency, it is important that the data used to derive the joint distribution are from the same (or similar) set of experiments that is used to derive the loss relationship. Therefore, we also adopted the parameters for the empirical yield loss relationship that the authors had derived (from trials conducted in the UK). This case study illustrates that the information necessary to implement the framework can be derived from data reported in the literature, but not without limitations. In particular, the parameters for the yield loss relationship were presented without any metric of uncertainty, and so a fully informed Monte Carlo simulation to derive the distribution of value could not be undertaken.

Our second study used raw secondary data to derive the joint distributions and yield loss relationships. The clear advantage here was that we could quantify more directly the uncertainties associated with the yield loss relationships and consider a wider range of plausible forms. Nonetheless, we were faced with several challenges deriving the relationships for pesticide efficacy, not least because the data used a range of fungicide products and doses, with no clear mechanism to account for differences in their efficacy. In the end we adopted a simple approach and considered only the number of applications made. The variation in response that results from this simplified assumption (on top of environmental variation) is clearly large yet a usable signal prevailed. Indeed, the variation in predictions meant that, although the expected value was positive for CPO, there was a large variation in the predicted response. Despite this, the value of the DSS was very rarely negative.

The third case study, estimating the value of the Hutton Criteria, provides a case study where the data were insufficient to confidently estimate the value of a DSS. The Hutton Criteria, in its basic form, highlights days on which there is likely to be significant pest pressure, and therefore farmers should consider applying a spray. In simplifying the framework to make it amenable to parameter estimation with the available data, several simplifying assumptions had to be made, such as the time point of an optimal spray time, and the effect on yield. As \({t}_{0}\) is an unknown optimal spray timing, there will be uncertainty in its true value, which is captured within the joint distribution function. While this may lead to an imprecise joint distribution function, it should allow for a reasonable estimate of the expected value to be calculated. Even so, the data presented here are likely too sparce to accurately parameterise the joint distribution function, meaning that the distribution of value should be treated with caution.

By far the simplest implementation relates to when data are available to populate a matrix specifying the proportion of times that the DSS predicts accurately or incorrectly, as in our fourth case study. This clear and simple quantification has the benefit that it is straight forward to interpret, but it is only applicable when data on the accuracy of prediction are available.

We have considered four variations of the generic framework, but there are likely to be other formulations. The approaches that we considered were driven by the type of DSS and data available. We have only considered simple DSS, and simplified them further where necessary. It is likely to be difficult to obtain enough data to get good estimates of \(L(x,q)\) when a DSS recommends specific products, doses and/or timings of pesticide applications. As demonstrated, simplifying the framework reduces the data required to characterise and fit the distributions. In doing so, however, the framework does lose some precision.

Ultimately, the accuracy of the framework will depend on the quality and amount of data available, with better data requiring fewer simplifications and enabling a better characterisation of the distribution and relationships that form the basis of the framework. Increasingly, industry and the research community are adopting principles of FAIR data (Findable, Accessible, Interoperable and Reusable). Theoretically it is possible to design experimental protocols to provide data to explicitly parameterise both the joint distribution function and the cost function. Combining data sets from a variety of sources would prove a powerful resource for approaches such as the one we present here.

The assumptions made in establishing the frameworks for the DSS can have profound implications. Indeed, a key difference between case studies 1 and 2, both considering the same pathogen, is that an increase in wheat price leads to the value of the DSS decreasing in case study 1, whereas in case study 2 value increases. The reason for this discrepancy lies with the choice of the standard spray programme. In case study 1, it was assumed that a standard spray programme was highly effective in the UK, and that the DSS informs the user when it is viable to spray less, saving the cost of unnecessary fungicide applications. For case study 2, the assumption is that a standard spray programme provides moderate control. The value of the CPO DSS is largely determined by the value obtained when large epidemics are correctly forecast by the DSS and thereby controlled. When the price of wheat is more expensive, these yield savings are even more valuable. Such nuances highlight the need for locally relevant information on what is meant by a “standard programme” and associated yield prices and costs.

Despite the limitations, the framework provides a method to give developers and users increased knowledge of the potential benefits and risks when using a DSS in a particular region, whether it is in a region similar to that in which the DSS was developed or elsewhere. Testing the DSS for a new environment requires data from that environment.

One aspect that has been touched on only briefly in this paper is an analysis of the variation in the value of the DSS. By capturing the variability in the relationships, it is possible to explore the distribution of the value of each DSS. While we have attempted to capture the distribution in the pest intensity, the framework can also capture the variability in each of the relationships and propagate that variability through to the distribution of value. This is particularly important in assessing the DSS in terms of their benefit for risk-averse users, who may want to be reassured how often the DSS outperforms a standard spray programme. Additionally, the distribution in value would enable researchers to develop DSS that minimise the risk of losses to the users of the system.

Throughout this paper, we have attempted to provide a data driven approach to valuing decision support systems, limiting model-based assumptions to empirical relationships describing yield loss and pesticide efficacy. However, in the absence of new data, expert opinion could be used to adjust any existing distributions to then test the value of DSS in other regions. Expert opinion could also be integrated into the methodology through a Bayesian approach to derive the distributions in the framework using expert informed priors. The benefit of such an approach would be increased flexibility to test over a larger range of environments, however, it is then not straightforward to say whether the DSS has performed well or whether the analysis has reproduced model-based assumptions. Ultimately, we argue that a combination of model-based and data driven approaches should be adopted to derive the value of DSS where direct field testing of a DSS is not possible.

Conclusions

The generic framework presented has the flexibility to estimate the value of DSS in many situations given the availability of sufficient data to quantify the joint distribution between predicted and actual pest intensity and information to quantify the impact of control on a given level of disease. In doing so we hope to allow DSS developers and academics to better illustrate the value of DSS in different environments. It is desirable to present the user with both the expected value of a DSS, as well as a measure of the risk of extreme losses so that they can assess benefits and risks associated with following DSS guidance.