1 Introduction

World crop production over the course of the last 50 years has increased at an unprecedented rate as a result of sustained rises in yield (Slafer et al. 1994; Rondanini et al. 2012; Alston et al. 2009), although recent contributions point at crop yield slowing down or even stagnating (Bruinsma 2011; Calderini and Slafer 1998; Hafner 2003; Ray et al. 2012; Ray et al. 2015; Lin and Huybers 2012). Several of these studies focuses on European crops (Brisson et al. 2010; Finger 2010; Oury et al. 2012; Licker et al. 2013), although the presence of this phenomenon has been reported on a global scale (Grassini et al. 2013). Limited growth in or stagnation of crop yield has attracted considerable attention for obvious policy and environmental implications. Bearing in mind scenarios with a growing population and shifts of developing countries towards OECD countries diets (Alexander et al. 2016), increasing crop productivity helps alleviate food shortages, improve food security and mitigate price increases (Wheeler and von Braun 2013). Stagnation in the crop yield may also influence future migration flows (Feng et al. 2010) and regional political stability and, in the most dramatic cases, trigger violent conflicts (Brinkman and Hendrix 2011). Understanding the historical trend of crop yield is fundamental for the construction of econometric forecasts, but it is also important for models simulating future food scenarios (Bruinsma 2011, Tubiello et al. 2007). Moreover, if trends in weather factors have contributed to the historical pattern of the trend of crop yield, they have to be taken into account when computing baseline yield projections from which the weather variability is eventually subtracted (Tollenaar et al. 2017).

Several authors have examined the trend in the yield of farm crops, i.e. the shape of the observed time pattern over a long time-horizon. Unfortunately, the great majority of the studies implement relatively simplistic analysis with functional forms mainly confined to linear, quadratic and cubic deterministic terms (Brisson et al. 2010; Calderini and Slafer 1998; Finger 2010; Grassini et al. 2013; Hafner 2003; Lin and Huybers 2012; Osborne and Wheeler 2013; Ray et al. 2012, and Rondanini et al. 2012). Slowdown or stagnation in the trend has been modelled through piecewise linear deterministic models or linear models with upper plateau (Brisson et al. 2010; Calderini and Slafer 1998; Grassini et al. 2013; Lin and Huybers 2012; Rondanini et al. 2012), with both type of models imposing a one-off change in the value of the trend coefficient and the latter class of models imposing the further constraint of a zero slope in the second segment.

Existing studies have a good coverage of crops, incusing those most widely cultivated such as maize, rice, soybean and wheat (Brisson et al. 2010, Calderini and Slafer 1998, Finger 2010, Grassini et al. 2013, Hafner 2003, Lin and Huybers 2012, Osborne and Wheeler 2013, Ray et al. 2012 and Rondanini et al. 2012). Trends in the yield of other crops such as barley, oats, rye, rapeseed and triticale have been only rarely examined (Finger 2010; Rondanini et al. 2012). In terms of geographical coverage, detailed analyses can be found for France (Brisson et al. 2010), Switzerland (Finger 2010) and the USA (Andersen et al. 2018), but most studies provide evidence of a slowdown in the long-run yield trend over large regions by implementing deterministic models that allow for a limited number (normally one) of sudden changes in the coefficient of the trend as discussed above. European crops have received considerable attention in relation to the debate on crop yield stagnation, to an extent that it is sometimes concluded that Western Europe contains the majority of countries showing a levelling off in the crop yield (Lin and Huybers 2012).

Surprisingly, applications of more advanced analysis of the trend of crop yield based on structural time series modelling and Kalman filtering are surprisingly sparse and have become even less so in recent years. Kaylen and Koroma (1991), Moss and Shonkwiler (1993) and Myers and Jayne (1997) are three examples of early applications, respectively focusing on the benefits that this methodology brings to the computation of yield distributions, the ability to take into account nonnormal errors and to distinguish between regime shifts with effects diffused over time and more immediate changes influencing the level of the yield. Other applications of the Kalman filtering include Cunha and Richter (2016) in relation to the production of wine. The closest work to ours in terms of methodology and field of application is Michel and Makowski (2013), who use a “Dynamic Linear Model,” among other alternatives. The adopted empirical strategy as well as the results is very different from ours, as they surprisingly conclude for a driftless random walk being the best description of the data.

The goal of this article is to characterize the nature of the long-run trend in crop yield and to assess whether it is influenced by the trend in weather factors. We aim to ‘reintroduce’ the Kalman filter approach in the literature on the long-term evolution of farm yield as a flexible modelling methodology that allows a very general representation of the trend. Our opinion is that the deterministic approach that has dominated the literature (Tolhurst and Ker 2015) is largely inadequate to characterize the long-run trend in crop yield and to offer a rigorous investigation of its apparent slowdown. We focus our analysis on major European producers that are exposed to different climatic conditions, as well as being the object of previous empirical work (e.g. Brisson et al. 2010; Calderini and Slafer 1998; Finger 2010; Grassini et al. 2013; Hafner 2003; Lin and Huybers 2012; Osborne and Wheeler 2013; Ray et al. 2012; Rondanini et al. 2012). Our contributions to the existing understanding of the long-run trend in crop yield are the following: (1) we show that the evolution of the trend of crop yield should be modelled as a gradually changing stochastic phenomenon, rather than being affected by the sudden permanent breaks currently used in the literature; (2) we find that long-run trends in weather factors are a major determinant of the shape and nature of the yield trend in European crops; and (3) we produce a quantitative measure of the impact of weather factors on the long-run growth rate of crop yield.

These contributions are delivered based on a research plan that is divided in two stages. In the first stage, we start analysing the statistical nature of the trend in crop yield and the evidence for an abrupt change in its trajectory. In addition to a function that incorporates a one-time permanent change, which is the typical strategy adopted in the current literature, we also take into account the possibility of changes in trend occurring gradually and in a non-deterministic fashion (a stochastic trend). In the second stage, we explore the hypothesis that the long-run trend in crop yield acquires part of its statistical characteristics from the existing trend in weather factors. We do so by comparing the estimation results on the nature and the size of the trend from the models selected in the first stage to those obtained from a model that includes weather factors (temperature and precipitation) and verifying that these results are robust with respect to the exact specification of the trend chosen in the first stage. We eventually produce an estimate of the extent to which historical improvements in crop yield have been offset by adverse trends in weather. In that regard, the intent of this last item is very similar to the work in Ortiz-Bobea and Tack (2018) although methodologically completely unrelated.

The rest of the article has been structured as follows. After describing the data in Sect. 2, we present our methodological approach in Sect. 3, and explain in detail our results in Sect. 4. The final Sect. 5 offers a discussion of our findings.

2 Data

Crop yield is defined as the harvested production per unit of harvested area with data measured in tonnes per hectare and collected from the online data set of the Food and Agriculture Organization of the United Nations (FAO). These are yearly time series, shown in Fig. 1, at country level for wheat and maize, over the period 1961–2014, for Belgium, France, Germany, Italy, Spain and the UK. The procedure we followed to construct the weather variables, i.e. country-level temperature and precipitation specific to each crop, is based on established practice in the crop yield literature, e.g. Moore and Lobell (2014) and Moore and Lobell (2015). The computed series implies using gridded weather observations for the grid cells where a specific crop is cultivated and the growing season of that crop. Like in Moore and Lobell (2014) and Moore and Lobell (2015) only the 4 months prior to the observed harvest date are used in the computation for winter crops. This procedure implies combining information from three different data sets, namely

  1. 1.

    Monthly average of temperature and precipitation on a grid of 30 min resolution, collected from the Climate Research Unit of the University of East Anglia (Harris et al. 2013);

  2. 2.

    A map of cropland at 5 min resolution (Monfreda et al. 2008) used to select the cells of the grid where harvest takes place; and

  3. 3.

    A crop calendar providing the growing season for each crop on 5 min resolution (Sacks et al. 2010) ,which is used to select the months that are relevant to a specific crop.

Fig. 1
figure 1

Maize and wheat yield. Time series of the yield in the countries analysed in this study. Yield is measured in tonnes per hectare

Full size image

The resulting time series are plotted in Figure S4. Long-term trends in the two weather factors are fairly similar, despite the different growing seasons for maize and wheat. Temperature shows a clear upwards trend, while precipitation tends to fluctuate around a relatively stable mean. This is a relevant aspect, as one can exclude that there are substantial differences in crop-specific weather trends influencing the difference in the results for the two crops discussed below.

3 Methodological approach

The econometric framework we adopt is the general class of Gaussian linear state space models, which we estimate by Maximum Likelihood implemented via the diffuse Kalman filter (Harvey 1989; Durbin and Koopman 2012). These models, also known as Structural Time Series Models, have been employed as forecasting devices in agricultural economics, in climatic sciences mainly in the guise of the Ensemble Kalman filter used to discretize the partial differential equations used in geophysical models (Hannart et al. 2016), and, closer to our approach, also as a way to represent temperature trends (Mills 2010) and trends in the impact of weather-related disasters (Visser et al. 2014). This approach enables us to perform an explicit, rigorous and general analysis of the trend components in crop yield, while capturing the potential role of explanatory variables, the impact of which is allowed to vary across time as coefficients are potentially stochastic.Footnote 1

We perform our empirical analysis in two distinct stages, each devised to tackle a different research question. In the first stage, we examine the nature of the trend in crop yield and we explore the evidence on the existence of sudden permanent changes, i.e. structural breaks, in the slope of the trend. In the second stage, we investigate whether the trend in crop yield is influenced by weather variables and provide a quantitative measure of their impact. We assess the validity of the estimated models by running standard diagnostic checks on the residuals of the Kalman filter, that is the Ljung-Box test for serial correlation, the Jarque-Bera test for normality, and an F-test for heteroskedasticity that compares the variance of the first and last third of the observations in the sample (see Durbin and Koopman 2012). Throughout the estimation, we include pulse dummies to treat occasional outliers identified through the plot of the auxiliary observation residuals. In the following, we describe in more detail each of the two stages of the analysis.

3.1 Stage 1

The aim of the first stage is to assess the nature of the trend and any evidence on the existence of a structural break in its slope. With this aim, we estimate a set of local linear trend models in which the level and the slope are potentially stochastic. The general univariate model for crop yield, i.e. without any explanatory variable, is defined as follows.

$$ \left\{\begin{array}{cc}{y}_t={\mu}_t+{\varepsilon}_t& {\varepsilon}_t\sim N\left(0,{\sigma}_{\varepsilon}^2\right)\\ {}{\mu}_{t+1}={\mu}_t+{v}_t+{\eta}_t& {\eta}_t\sim N\left(0,{\sigma}_{\eta}^2\right)\\ {}{v}_{t+1}={v}_t+{\xi}_t& {\xi}_t\sim N\left(0,{\sigma}_{\xi}^2\right)\end{array}\right. $$
(1)

where the first line defines the crop yield yt, while the remaining two equations define the two states of the trend, the level μt and the slope vt, and where all error terms are assumed to follow independent Gaussian distributions. We assess the evidence for a structural break in the long-run yield trend in two ways: we look for outliers in the plot of the slope auxiliary residuals; and we evaluate the additional fit of modelling a structural break in the slope via information criteria.

In the first case, we estimate the model above assuming a deterministic level and a stochastic slope, i.e. (\( {\sigma}_{\eta}^2=0 \), \( {\sigma}_{\xi}^2\ne 0 \)), and we examine the resulting auxiliary residuals of the slope equation. Following standard practice in the literature, the presence of any outliers exceeding the 95% confidence band in the slope auxiliary residuals is interpreted as a sign of a structural break in the slope (Durbin and Koopman 2012).

In the second case, we assess whether a model that explicitly includes a structural break in the slope provides substantial gains in fit compared with a model without this component. The model we consider is as follows.

$$ \left\{\begin{array}{cc}{y}_t={\mu}_t+{\varepsilon}_t& {\varepsilon}_t\sim N\left(0,{\sigma}_{\varepsilon}^2\right)\\ {}{\mu}_{t+1}={\mu}_t+{v}_t+{\eta}_t& {\eta}_t\sim N\left(0,{\sigma}_{\eta}^2\right)\\ {}{v}_{t+1}={v}_t+{\delta}_t{d}_t+{\xi}_t& {\xi}_t\sim N\left(0,{\sigma}_{\xi}^2\right)\\ {}{\delta}_{t+1}={\delta}_t& \end{array}\right. $$
(2)

which includes a pulse dummy in the slope equation, that is dt = 1(t = t∗), where 1(∙) is an indicator taking the value of 1 if the event in brackets occurs and 0 otherwise, with δt being a deterministic coefficient. As the conclusions on the existence of a break might vary across different assumptions on the nature of the trend—being either deterministic or stochastic—we consider all four possible specifications, that is (1) both level and slope deterministic (\( {\sigma}_{\eta}^2=0 \), \( {\sigma}_{\xi}^2=0 \)); (2) stochastic level and deterministic slope (\( {\sigma}_{\eta}^2\ne 0 \), \( {\sigma}_{\xi}^2=0 \)); (3) deterministic level and stochastic slope (\( {\sigma}_{\eta}^2=0 \), \( {\sigma}_{\xi}^2\ne 0 \)); and (4) both level and slope stochastic (\( {\sigma}_{\eta}^2\ne 0 \), \( {\sigma}_{\xi}^2\ne 0 \)). As we do not have strong a priori beliefs on the timing of the break, we treat the break date as an unknown parameter, which is endogenously determined by the data. To this aim, for each of the four models above, we set t∗ equal to one specific year, calculate the corresponding likelihood and repeat this calculation for all possible years, excluding 10% of the observations at the beginning and at the end of the sample to ensure parameter identification in each subsample. The break date is finally estimated as the year producing the model with the largest likelihood.Footnote 2 Once the break date for each model has been estimated, we perform a model selection by applying information criteria over a total of eight models, that is four specifications with a structural break in the slope, as in Eq. (2), and four without, as in Eq. (1). The corresponding loss functions are defined as follows.

$$ IC=\frac{1}{T}\left[- logL+k(T)\left(2+n+b\right)\right] $$
(3)

where n is the number of stochastic states, b = 3 in models with a break and b = 0 in models without a break and k(T) a function that reflects two different ways in which the number of parameters are penalized, i.e. using the Schwartz criterion from Yao (1988), where k(T) = logT, and using the criterion of Liu et al. (1998), where k(T) = 0.299logT2.1.Footnote 3

Application of the information criterion above enables us to simultaneously determine the nature of the trend (by choosing one of the four specifications) and the existence of a structural break in the slope. The stochastic features of the trend selected by the information criterion are examined in light of three sources of evidenceFootnote 4: the size of the Maximum Likelihood (ML) estimate of the state error variance, along with the corresponding t-statistic, which would give an indication on whether the stochasticity is sizeable or negligible; the plot of the smoothed state to verify the presence of noticeable time-variation; and the outcome of the three standard diagnostic tests based on the standardized innovations, which help to ascertain the overall validity of the model. The combined information from these three pieces of evidence allows us to draw a robust conclusion on the deterministic or stochastic nature of the yield trend.Footnote 5

3.2 Stage 2

We propose a simple approach to verify the extent to which the trend in weather influences the trend in crop yield. Let us assume that the data-generating process for the crop yield yt is as follows.

$$ {y}_t=\sum \limits_{i=1}^k{b}_i{x}_{it}+{u}_t $$
(4)

where \( {u}_t\sim N\left(0,{\sigma}_u^2\right) \) is an error term and x1t, …, xkt are k exogenous variables generated by the following local linear trend models.

$$ \left\{\begin{array}{cc}{x}_{it}={\mu}_{it}+{\varepsilon}_{it}& {\varepsilon}_{it}\sim N\left(0,{\sigma}_{\varepsilon_i}^2\right)\\ {}{\mu}_{it+1}={\mu}_{it}+{v}_{it}+{\eta}_{it}& {\eta}_{it}\sim N\left(0,{\sigma}_{\eta_i}^2\right)\\ {}{v}_{it+1}={v}_{it}+{\xi}_{it}& {\xi}_{it}\sim N\left(0,{\sigma}_{\xi_i}^2\right)\end{array}\right.i=1,\dots, k. $$
(5)

If we use the univariate representation (1) to describe yt, we have that each state in (1) is the linear combination of the corresponding state of xit.

$$ \left\{\begin{array}{c}{\mu}_t=\sum \limits_{i=1}^k{b}_i{\mu}_{it}\\ {}{v}_t=\sum \limits_{i=1}^k{b}_i{v}_{it}\end{array}\right. $$
(6)

and the same thing holds for the error terms

$$ \left\{\begin{array}{c}{\varepsilon}_t={\sum}_{i=1}^k{b}_i{\varepsilon}_{it}+{u}_t\\ {}{\eta}_t={\sum}_{i=1}^k{b}_i{\eta}_{it}\\ {}{\xi}_t={\sum}_{i=1}^k{b}_i{\xi}_{it}\end{array}\right. $$
(7)

If we observe one of the explanatory variables above, say xmt, and we explicitly control for it in the observation equation, the resulting level state captures the trend that is specific to all the other variables xit with i ≠ m

$$ \left\{\begin{array}{c}{y}_t={b}_m{x}_{mt}+{\overline{\mu}}_t+{\overline{\varepsilon}}_t\\ {}{\overline{\mu}}_t={\sum}_{i\epsilon {I}_m}{b}_i{\mu}_{i t}\end{array}\right. $$
(8)

where \( {\overline{\varepsilon}}_t={\sum}_{i\epsilon {I}_m}^k{b}_i{\varepsilon}_{1t}+{u}_t \), and Im = {1, …, k} − m. Thus, one can characterize the contribution of the trend in xmt to the determination of the observed trend in yt by looking at how the estimated trend changes once we control for that variable, that is examining the difference between the level and slope states obtained without and with the inclusion of the variable xmt

$$ \left\{\begin{array}{c}{\mu}_t-{\overline{\mu}}_t={b}_m{\mu}_{mt}\\ {}{v}_t-{\overline{v}}_t={b}_m{v}_{mt}.\end{array}\right. $$
(9)

With regard to our specific empirical application, we consider a model that includes the two trend components as before but also two regressors, the level of temperature and precipitation, constructed as described above. In the case of temperature, we allow for its impact on yield to vary over time in order to capture potential non-linear effects, whereas, for precipitation, we decide for a constant coefficient.Footnote 6

Hence, the model we consider can be written as follows.

$$ \left\{\begin{array}{cc}{y}_t={\overline{\mu}}_t+{\beta}_t{T}_t+\rho {P}_t+{\overline{\varepsilon}}_t& {\varepsilon}_t\sim N\left(0,{\sigma}_{\overline{\varepsilon}}^2\right)\\ {}{\overline{\mu}}_{t+1}={\overline{\mu}}_t+{\overline{v}}_t+{\overline{\eta}}_t& {\eta}_t\sim N\left(0,{\sigma}_{\overline{\eta}}^2\right)\\ {}{\overline{v}}_{t+1}={\overline{v}}_t+{\overline{\xi}}_t& {\xi}_t\sim N\left(0,{\sigma}_{\overline{\xi}}^2\right)\\ {}{\beta}_{t+1}={\beta}_t+{\zeta}_t& {\zeta}_t\sim N\left(0,{\sigma}_{\zeta}^2\right)\end{array}\right. $$
(10)

where βt is the time-varying coefficient on temperature Tt and ρ is the constant coefficient on precipitation Pt.

Once model (10) is estimated, we investigate the statistical characteristics of the resulting yield trend to understand whether there is any change in its stochastic features and in the size of the slope as a result of incorporating weather factors in the model. If any substantial difference emerges compared with the univariate model (1), we attribute it to the influence of weather dynamics, and more specifically to the trend of temperature, given that the latter dominates precipitation as a predictor of crop yield. In particular, we are interested to verify whether any trend stochasticity emerging from the univariate analysis persists in the regression model (10) or whether the trend is effectively deterministic.

Like in Stage 1, we consider four sources of evidence to produce a robust assessment of the statistical nature of the trend, in this case, conditional on the inclusion of our weather variables.

  1. 1)

    We ascertain the gain in fit generated from modelling a stochastic trend by comparing the corresponding value of information criteria with that of a model that has the same specification but features a deterministic trend. Here, we adopt three widely-common information criteria: Akaike (AIC), Schwartz (BIC) and Hannan-Quinn (HQ).

  2. 2)

    We look at the size of the ML estimate of the state error variance and compare it with the one from the univariate model (1) to check whether it has dropped substantively towards zero after the inclusion of the weather regressors.

  3. 3)

    We visually examine the plot of the smoothed state to understand if there are noticeable changes over time, which is a sign of stochastic state.

  4. 4)

    We obtain a final confirmation on the validity of the selected model by running the standard diagnostic tests, built on the standardized innovations of the model. Here, the importance of stochasticity is signalled by a substantial deterioration of any of the three diagnostics when a deterministic trend is imposed.Footnote 7

As the results about the stochasticity of the trend might be dependent on the specific model selected in Stage 1, we also provide a robustness analysis where we use two alternative specifications of a stochastic trend, so that we eventually look at three sets of models:

  1. 1)

    A model with stochastic level and time-varying impact of temperature, i.e. \( {\upsigma}_{\overline{\upeta}}^2\ne 0 \), \( {\upsigma}_{\overline{\upxi}}^2=0 \) and \( {\upsigma}_{\upzeta}^2\ne 0 \);

  2. 2)

    A model with stochastic slope and time-varying impact of temperature, i.e. \( {\upsigma}_{\overline{\upeta}}^2=0 \), \( {\upsigma}_{\overline{\upxi}}^2\ne 0 \) and \( {\upsigma}_{\upzeta}^2\ne 0 \); and

  3. 3)

    A model with stochastic level, stochastic slope and time-varying impact of temperature, i.e. \( {\upsigma}_{\overline{\upeta}}^2\ne 0 \), \( {\upsigma}_{\overline{\upxi}}^2\ne 0 \) and \( {\upsigma}_{\upzeta}^2\ne 0 \).

Finally, we provide a quantitative indicator of the impact of weather on the long-run trend of crop yield by taking the average difference in the smoothed slope obtained from the univariate model (1) and the regression model (10), calculated over the whole period

$$ \frac{1}{T}\sum \limits_{t=1}^T\left({v}_t-{\overline{v}}_t\right) $$

and its relative counterpart

$$ \sum \limits_{t=1}^T\left({v}_t-{\overline{v}}_t\right)/\sum \limits_{t=1}^T{\overline{v}}_t $$

The magnitude of this quantity measures the extent to which the observed long-run annual rate of change in crop yield can be attributed to weather trends, and its sign indicates whether this contribution is positive or negative.

4 Results

In this section, we illustrate the results we obtain from applying the methodology discussed above on the European data. We begin by highlighting some relevant descriptive features of our data set, which suggest the suitability of our approach. Next, we describe our findings about the nature of the trend in crop yield (Stage 1). Then, we present our analysis of the influence of weather factors on the statistical characteristics of the trend (Stage 2), accompanied by a robustness analysis. Finally, we calculate the impact of weather on the size of the observed long-run growth rate of crop yield (Stage 2).

4.1 Heterogeneity in the yield of European crops

The yield of European crops reveals considerable differences in terms of its time patterns, annual growth and variability. Maize in Southern European countries can be as productive as in Northern Europe and recover productivity gaps resulting from specific events in the political and economic history, as is the case of Spain (Fig. 1).Footnote 8 The opposite result is found for wheat, as the productivity gap between the countries with the highest and lowest yield increased by 90% in the last 50 years. From a visual inspection of the data in Fig. 1 and in more detail in Figure S1 in the Supplementary Information, we conclude that yield trends display a different time pattern across countries and crops. The yield of maize seems to have reached a plateau in Italy and perhaps in Belgium, affected by a deceleration in France and increased linearly in Germany and Spain. The yield of wheat seems to have reached a plateau in Germany, France and the UK, affected by a deceleration in Belgium and increased more or less linearly in Spain and Italy. Less clear insights can be drawn from average annual growths in crop yield across subsamples, as conclusions are sometimes influenced by the cut points used to divide our sample (see Figure S2in the Supplementary Information). There is also considerable heterogeneity in the variability of the yield, with Southern European countries showing a much more stable level of the yield, although variability has increased in recent years. A decrease in the variability of yield towards the end of the sample can be seen in Northern European countries (see Figure S3 in the Supplementary Information). Overall, the fact that our dataset shows considerable heterogeneity in the level of the yield, its variability and the dynamics of these two measures,Footnote 9 confirms a flexible time series approach, applied separately on each country and crop, as the most suitable empirical strategy.

4.2 Characterizing the trend in crop yield

Contrary to the great majority of existing empirical investigations favouring a deterministic specification subject to a sudden break at some date, using a more general Structural Time Series approach, we obtain no evidence of a sudden change in the size of the trend. As discussed in Sect. 3, we gather evidence towards this conclusion (1) by assessing the existence of outliers in the state auxiliary residuals of a model that allows for a stochastic slope and (2) by explicitly modelling a structural break in the slope (see Sect. 3.1).

Using the first approach, in no instance one can detect statistically significant outliers (see Figure S5 of the Supplementary Information). Using the second approach, we find that the models with a sudden permanent change in the direction of the trend are never selected by the information criteria. Detailed results are presented in Table SI 1 of the Supplementary Information, with each table showing models with ether a deterministic or a stochastic level, a deterministic or stochastic slope, and comprising a break in the slope or assuming no break, so eight models in total arising from the combination of three sets of binary options.

The final list of selected models, displayed in Table 1, shows extensive evidence not only against sudden breaks in the trend of crop yield, which is never selected, but also in favour of its nature being stochastic rather than deterministic. In the great majority of the cases, eight out of 11, our information criterion selects a specification where one of the two trend components (either the level or the slope, but never both) is stochastic. In all these eight cases, the stochasticity of the trend is evident from the size of the estimated variance of the state error (see Table 1) and the plot of the smoothed state (Fig. 2).Footnote 10 Indeed, when the level is stochastic, its dynamics exhibits an erratic behaviour that clearly distinguishes it from a deterministic linear trend model; when the slope is stochastic, its size changes visibly over time. In all the 11 cases, the validity of the univariate model is confirmed by the three diagnostic tests, with only two exceptions, Maize in Italy and Wheat in Spain, where there is some residual sign of heteroskedasticity.

Table 1 Selected univariate models for the yield of European crops. The column “Variance” presents the ML estimate of the stochastic state error variance (either the level or the slope), while the “t-ratio” is the corresponding t statistic. JB, LB and H indicate the p value for respectively the Jarque-Bera test for normality, the Ljung-Box test for serial correlation and the F test for heteroskedasticity
Full size table
Fig. 2
figure 2

a Maize. Smoothed state for the stochastic level or slope (blue line), along with its 90% confidence band (red lines), for each of the univariate models of the maize yield with stochastic trend. b Wheat. Smoothed state for the stochastic level or slope (blue line), along with its 90% confidence band (red lines), for each of the univariate models of the wheat yield with stochastic trend

Full size image

More in detail, in six cases (maize in Belgium, France and Spain; wheat in Belgium, Germany and UK), the size of the estimated variance is above the 10−2 order of magnitude, which is a value that can be safely considered above zero. Of the two remaining cases, we notice that, for maize in Italy, the estimated variance is far from negligible, 0.0054, with a t-ratio of 1.40, while the plot of the smoothed slope displays considerable time-variation, between a minimum of − 0.092 and a maximum of 0.314 (see Fig. 2a). For the other case, wheat in France, the evidence is less strong if we look at the size of the estimated variance, 0.0003, but not if we consider the t-ratio, 1.24, and especially the plot of smoothed slope, which shows a substantial decreasing dynamics over time, with a value ranging between 0.019 and 0.138 (see Fig.2b).

We conclude that, in the great majority of the cases, the trend in the yield of European crops is stochastic, meaning that its long-run trajectory cannot be characterized using a simple deterministic function of time and that there is not enough evidence of a permanent sudden change in its direction.

4.3 The influence of weather factors

Our method essentially consists in augmenting the selected models in Table 1 with two variables representing crop-specific temperature and precipitation over the growing season and analysing the consequence on the statistical features of the yield trend (see Sect. 3.2).

The main results of applying our methodology is displayed in Table 2, where we present the ML estimate of the error variance of the stochastic state (indicated in the second column), the corresponding t-ratio, the value of the HQ information criterion for the model including the stochastic trend (IC) and for the same model after imposing a deterministic trend (IC*), the value of the diagnostic tests of the final model in which temperature coefficient is set as deterministic when any stochasticity can be discarded and our conclusion as to whether the inclusion of weather factors remove stochastic trends which were identified in Stage 1. The full comparison in terms of information criteria using AIC, BIC and HQ can be found in Table SI 2. Figure 3 plots the smoothed states for the stochastic trend component of the model after the introduction of the weather factors, which can be usefully compared with those from the univariate models of Stage 1, shown in Fig. 2. The overall conclusion is that, in seven of the eight cases displaying a stochastic trend in the univariate analysis, the stochastic nature of the trend is for the most part explained by weather factors.

Table 2 Results from estimation of the regression model (10), obtained by augmenting the models in Table 1 by weather factors. Under “Variance” we list the ML estimate of the error variance associated with the trend component that is stochastic. By “IC” and “IC*,” we indicate the HQ information criterion for the regression model featuring a stochastic and a deterministic trend respectively. JB, LB and H are the diagnostics of the final model obtained after setting to zero any insignificant variance. More precisely, they indicate the p value for respectively the Jarque-Bera test for normality, the Ljung-Box test for serial correlation and the F test for heteroskedasticity. “TVP” denotes a time-varying parameter associated to temperature
Full size table
Fig. 3
figure 3

a Maize. Smoothed state for the stochastic level or slope (blue line), along with its 90% confidence band (red lines), for each of the regression models of the maize yield with stochastic trend. b Wheat. Smoothed state for the stochastic level or slope (blue line), along with its 90% confidence band (red lines), for each of the regression models of the wheat yield with stochastic trend

Full size image

In five cases (maize in Belgium, France, Italy and Spain and wheat in Germany) all four sources of evidence discussed in Sect. 3 points at the trend changing from stochastic to deterministic once we control for weather. The ML estimate of the state error variance is below 10−4, all information criteria prefer the model with deterministic trend after including weather factors, the plot of the smoothed state indicates absence of erratic components, and all three diagnostics confirm the validity of a model with deterministic trend.Footnote 11

In the remaining two cases, wheat in France and the UK, the magnitude of the error variance is not negligible on its own, but overall the evidence suggests weather factors playing a dominant role in the stochastic trend detected in the univariate models of Stage 1. In the case of France, the variance is only one third of the value found in the univariate model, where t-ratio has decreased by almost 80%. Some residual stochasticity can be observed in the level of the trend (see Fig. 3), although it is considerably lower compared with the plot in Fig. 2, while all information criteria suggest a deterministic trend, which is corroborated by the three diagnostic tests. In the case of wheat in the UK, the estimated state error variance is still sizeable, 0.0055, but an order of magnitude smaller than the value found in the univariate model with a t-ratio changing from 1.92 to 0.02, and the plot of the smoothed state in Fig. 2b resembling a deterministic linear trend model.

Finally, only for wheat in Belgium, the inclusion of weather variables does not remove the features typical of a stochastic level. Indeed, we observe an increase in the estimated state error variance, with t-ratio being largely unaltered and the plot of the smoothed level, shown in Fig.2b, exhibiting the clear signs of an erratic component.

4.4 Robustness analysis

As the analysis performed in Stage 2 is conditional on the model selected in Stage 1, it is informative to explore whether the main conclusions on the stochastic features of the trend in the regression models would have been the same if a different trend specification had been used. We perform this robustness analysis by looking at the size of the ML estimate of the state error variance of the stochastic trend component, in comparison with that of the coefficient on temperature, which is time-varying, in three sets of models: stochastic level only (Table 3), stochastic slope only (Table 4) and both level and slope stochastic (Table 5).

Table 3 Models with stochastic level and time-varying coefficient on temperature. The columns “level variance” and “temperature variance” contain the ML estimates of the state error variance associated with the level and the coefficient on temperature, respectively. JB, LB and H indicate the p value for respectively the Jarque-Bera test for normality, the Ljung-Box test for serial correlation and the F test for heteroskedasticity, all tests being discussed in Sect. 4
Full size table
Table 4 Models with stochastic slope and time-varying coefficient on temperature. The columns “level variance” and “temperature variance” contain the ML estimates of the state error variance associated with the level and the coefficient on temperature, respectively. JB, LB and H indicate the p value for respectively the Jarque-Bera test for normality, the Ljung-Box test for serial correlation and the F test for heteroskedasticity, all tests being discussed in Sect. 4
Full size table
Table 5 Models with both level and slope stochastic and time-varying coefficient on temperature. The columns “level variance” and “temperature variance” contain the ML estimates of the state error variance associated with the level and the coefficient on temperature, respectively. JB, LB and H indicate the p value for respectively the Jarque-Bera test for normality, the Ljung-Box test for serial correlation and the F test for heteroskedasticity, all tests being discussed in Sect. 4
Full size table

It turns out that our conclusion on the importance of weather factors in explaining the stochastic nature of the observed yield trend is robust to the exact specification of the trend selected in Stage 1. Using the same cut-off value of 10−4 to decide if the estimated error variance is small enough to be described as approximately zero, we obtain that the number of instances in which the stochasticity of the trend is negligible is twice those of the temperature coefficient. More precisely, the cases in which the error variance associated with the trend state is approximately zero against those where the temperature coefficient can be considered constant are 8 against 4 when only the level is stochastic (Table 3); 10 against 4 when only the slope is stochastic (Table 4); 8 and 10 against 5 when both level and slope are stochastic (Table 5).

4.5 The impact of weather on the size of the trend

The results from estimation of model (1) and model (10) can be used to build a quantitative measure of the effect of weather on the long-run growth rate of crop yield by simply calculating the difference in the estimated slope from the two models. A negative difference will indicate a detrimental effect of weather on the size of the long-run growth rate of crop yield. As we explained in Sect. 3, we obtain a relative measure of this effect by calculating the percentage change in the slope of the long-run trend that we can attribute to weather (Table 6).

Table 6 Average size of the smoothed slope and percentage difference due to weather factors
Full size table

Two features emerges from this calculation, one is the variety of impact magnitudes that we obtain across countries, likely reflecting the heterogeneous agricultural and environmental conditions, the other being the overall difference between the two crops, with the negative impact being much more pronounced in the case of wheat. In the case of maize, we estimate a small positive impact of weather on the size of the yield trend in Belgium, France and Germany, a moderately negative effect in Spain and a large negative impact in Italy. This latter result is reflected by Italy being the only country showing clear signs of stagnation in the time series of its yield (Fig. 1). On the contrary, the impact of weather on the size of the yield trend for wheat has been negative in all but one country analysed in this study, although with a different magnitude. The negative impact ranges from almost − 7% in Spain to approximately − 30% in Germany, whereas weather benefitted the UK by increasing its long-run growth rate by two thirds.

5 Discussion

Motivated by the wide variety of historical dynamics in the crop yield across European countries, we used a general and flexible Structural Time Series approach to study its long-run trend. We built a novel empirical strategy that allows to explore the nature of the trend, the evidence for the presence of a slowdown in its dynamics and the potential determinants of its magnitude. First, we found substantial evidence that the trend in yield is most of the time stochastic and not reducible to a simple deterministic function of time. Second, the data rejected the hypothesis of a discrete decrease in the yield trend as a way to describe the apparent slowdown of recent years. Third, using our new methodology, we found that the trends of weather and, in particular, temperature influence both the stochastic nature and the magnitude of the trend in crop yield. We now discuss each of these points more in depth.

Our finding that most of the crops in this study exhibit a stochastic trend is in clear contradiction with the typical approach in the literature that is based on deterministic representations of the trend, but it is far from surprising if one considers the multitude of factors likely to play a role in the determination of the long-run growth rate of crop yield. Examples of relevant factors are genetic progress, technological innovation, mechanization, irrigation, soil and crop management, CO2 concentration, changing stress tolerance and adaptation.Footnote 12 All these factors are likely to influence the historical evolution of the yield, sometimes in opposite directions, with varying intensity across time and in many instances diffusing slowly through the agronomic system.

A number of contributions in the literature have highlighted the existence of a recent slowdown in the long-run growth rate of crop yield in some regions of the world and in Western Europe in particular. Our comprehensive statistical analysis showed that a discontinuous change in the trend, which is the typical way to model stagnation and deceleration in the long-run growth rate of yield, is not a statistically grounded description of the data, which instead favour models that represent more gradual and uncertain changes in the yield through a stochastic trend.

There is large empirical evidence on the impact of year-to-year fluctuations of weather on crop yield, but the research on the impact of weather on the historical long-run trend of crop yield is surprisingly scarce. Our study shows that long-run trends in weather factors play an important role in the determination of the long-run trend in crop yield, in addition to the short-run fluctuations already highlighted in previous studies. Weather factors are important in shaping both the stochastic nature observed in the yield trend, as well as the magnitude of its long-run growth rate. To obtain these results we developed a methodology that improves on several aspects of the existing practice to calculate the impact of weather on crop yield trend. We did not impose an arbitrary detrending technique on the data, which could mislead the whole analysis; we avoided the uncertainty deriving from preliminarily testing the stationarity of the variables, with the ensuing risk of misspecification; we proposed a simple method to study the general influence of one variable on the long-run trend of another variable.

We obtained that the stochastic trend in European crop yields can be largely explained by the influence of weather dynamics, an interesting result considering that there are many other factors potentially important for crop yields. Moreover, we estimated that up to 30% of the long-run increase over time in the yield of European crops has been cancelled out by the adverse impact of weather, although this impact varies markedly across countries and crops. Given that weather is only one of many potential determinants of the long-run growth rate of crop yield, it is natural to ask how likely it is that other factors have been important. It turns out that all other alternatives are unlikely to have been predominant in determining either the stochastic nature of the trend or its recent slowdown (see the Supplementary Information for a detailed discussion).