A hierarchical Bayesian model for forecasting state-level corn yield

Abstract

Historically, the National Agricultural Statistics Service crop forecasts and estimates have been determined by a group of commodity experts called the Agricultural Statistics Board (ASB). The corn yield forecasts for the “speculative region,” ten states that account for approximately 85 % of corn production, are based on two sets of monthly surveys, a farmer interview survey and a field measurement survey. The members of the ASB subjectively determine a forecast on the basis of a discussion of the survey data and auxiliary information about weather, average planting dates, and crop maturity. The ASB uses an iterative procedure, where initial state estimates are adjusted so that the weighted sum of the final state estimates is equal to a previously-determined estimate for the speculative region. Deficiencies of the highly subjective ASB process are lack of reproducibility and a measure of uncertainty. This paper describes the use of Bayesian methods to model the ASB process in a way that leads to objective forecasts and estimates of the corn yield. First, we use small area estimation techniques to obtain state-level forecasts. Second, we describe a way to adjust the state forecasts so that the weighted sum of the state forecasts is equal to a previously-determined regional forecast. We use several diagnostic techniques to assess the goodness of fit of various models and their competitors. We use Markov chain Monte Carlo methods to fit the models to both historic and current data from the two monthly surveys. Our results show that our methodology can provide reasonable and objective forecasts of corn yields for states in the speculative region.

Appendix: full conditionals for Metropolis-Hastings

Notation used in this section: $\varvec{1}_{d}$ is a $d$—dimensional column vector of ones, $\varvec{I}_{d}$ is a $d$—dimensional identity matrix, $\varvec{\varOmega }$ denotes the vector of parameters excluding the parameter being generated, and $\varvec{Z}$ is the matrix with rows $\varvec{z}_{t\ell }^{\prime }$ listed in the order with states grouped together.

1.1 Unconstrained state model

Looking at the joint posterior density in (5), we can obtain the conditional posterior densities (CPD) to run the Metropolis-Hastings sampler.

• For state$\times$year combinations where data from all surveys are available, the conditional posterior distribution (CPD) of $\mu _{t\ell }\,|\,\varvec{\varOmega }$ is

  $$\begin{aligned} \mu _{t\ell }\,|\,\varvec{\varOmega } \sim \hbox {N}(\varDelta _{1,t\ell }^{-1}\varDelta _{2,t\ell }, \varDelta _{1,t\ell }^{-1}), \end{aligned}$$

  where

  $$\begin{aligned} \varDelta _{1,t\ell }&= [\varvec{1}_{5}^{\prime }(\varvec{\varSigma }_{O} + \varvec{S}_{t\ell O})^{-1}\varvec{1}_{5}\nonumber \\&\quad +\, \varvec{1}_{4}^{\prime }(\varvec{\varSigma }_{A} + \varvec{S}_{t\ell A})^{-1}\varvec{1}_{4} + 1/S^{2}_{Dt\ell } + 1/\sigma ^{2}_{\eta }],\end{aligned}$$

  (10)
  $$\begin{aligned} \varDelta _{2,t\ell }&= \varvec{1}_{5}^{\prime }(\varvec{\varSigma }_{O}+\varvec{S}_{t\ell O})^{-1}(\varvec{Y}_{t\ell O} - \varvec{b}_{O})\nonumber \\&\quad +\, \varvec{1}_{4}^{\prime }(\varvec{\varSigma }_{A}+ \varvec{S}_{t\ell A})^{-1}(\varvec{Y}_{t\ell A} - \varvec{b}_{A}) + S^{-2}_{Dt\ell }Y_{Dt\ell } + \varvec{z}_{t\ell }^{\prime }\varvec{\beta }/\sigma ^{2}_{\eta }. \end{aligned}$$

  (11)

  For combinations of states and years where at least one survey indication is not available, the terms in $\varDelta _{1,t\ell }$ and $\varDelta _{2,t\ell }$ associated with the missing survey indications are omitted.

• The CPD of $\varvec{b}_{k} \,|\,\varvec{\varOmega }$ is $\varvec{b}_{k} \,|\,\varvec{\varOmega } \sim \hbox {N}(\varvec{\varDelta }_{1,k}^{-1}\varvec{\varDelta }_{2,k}, \varvec{\varDelta }_{1,k}^{-1})$, where

  $$\begin{aligned} \varvec{\varDelta }_{1,k}&= \sum _{\ell =1}^{L}\sum _{t=s(\ell , k)}^{T} (\varvec{\varSigma }_{k} + \varvec{S}_{t\ell k})^{-1} + 10^{-6}\varvec{I}_{M(k)},\\ \varvec{\varDelta }_{2,k}&= \sum _{\ell =1}^{L}\sum _{t=s(\ell , k)}^{T} (\varvec{\varSigma }_{k} + \varvec{S}_{t\ell k})^{-1}(\varvec{Y}_{t\ell k} - \varvec{1}_{M(k)}\mu _{t\ell }). \end{aligned}$$

• Letting $p$ denote the number of covariates including the intercept, the CPD of $\varvec{\beta }\,|\,\varvec{\varOmega }$ is $\varvec{\beta }\,|\,\varvec{\varOmega } \sim \hbox {N}(\varvec{\varDelta }_{1}^{-1}\varvec{\varDelta }_{2}, \varvec{\varDelta }_{1}^{-1})$, where

  $$\begin{aligned} \varvec{\varDelta }_{1} = \varvec{Z}^{\prime }\varvec{Z}/\sigma ^{2}_{\eta } + 10^{-6}\varvec{I}_{p}, \varvec{\varDelta }_{2} = \varvec{Z}^{\prime }\varvec{\mu }/\sigma ^{2}_{\eta }. \end{aligned}$$

• The CPD of $\sigma ^{2}_{\eta }\,|\,\varvec{\varOmega }$ is $\sigma ^{2}_{\eta }\,|\,\varvec{\varOmega } \sim \hbox {Inverse-gamma}(a, b)$, where

  $$\begin{aligned} a = 0.001 + 0.5\sum _{t=1}^{T}L(t), b = 0.001 + \sum _{t = 1}^{T} \sum _{\ell =1}^{L(t)}0.5(\varvec{z}_{t\ell }^{\prime }\varvec{\beta } - \mu _{t\ell })^{2}. \end{aligned}$$

• The CPD of $\sigma ^{2}_{km}$ and $\rho _{k}$ are not proportional to known distributions, so we use Metropolis-Hastings with the vector of transformed variables,

  $$\begin{aligned} \varvec{\gamma } = (\varvec{\gamma }_{1}^{\prime },\varvec{\gamma }_{2}^{\prime })^{\prime }, \end{aligned}$$

where

$$\begin{aligned} \varvec{\gamma }_{1} = \left( \hbox {log}(\sigma ^{2}_{O,1}),\ldots , \hbox {log}(\sigma ^{2}_{A,4})\right) ^{\prime }, \end{aligned}$$

and

$$\begin{aligned} \varvec{\gamma }_{2} = \left( \hbox {log}\left[ (1 - \rho _{O})^{-1}(1+\rho _{O})\right] , \hbox {log}\left[ (1 - \rho _{A})^{-1}(1+\rho _{A})\right] \right) ^{\prime }. \end{aligned}$$

The proposal distribution is a multivariate normal distribution with a fixed covariance matrix and mean equal to the current value.

1.2 Constrained state model

For the constrained state model, $\mu _{tL(t)} = w_{tL(t)}^{-1}(\theta _{t}^{R} - \sum _{\ell =1}^{L(t)-1} w_{t\ell }\mu _{t\ell })$. We sample from the joint posterior density (9) using Metropolis-Hastings. The full conditional distributions for the biases, regression coefficients, and variance parameters are the same in the unconstrained and constrained state models. To define the distribution used to generate $\mu _{t\ell }$ for $\ell = 1,\ldots L(t) - 1$, we introduce some notation. Let $\varvec{\mu }_{t}^{(1)} = (\mu _{t1},\ldots ,\mu _{t,L(t)-1})^{\prime }$,

$$\begin{aligned} \varvec{V}_{t,1}&= \left( \begin{array}{cc} \hbox {diag}(\varDelta _{1,t1}^{-1}, \ldots , \varDelta _{1,t(L(t)-1)}^{-1}) &{} \varvec{\omega }_{t,1}^{\prime } \\ \varvec{\omega }_{t,1} &{} \sum \nolimits _{\ell =1}^{L(t)}w_{t\ell }^{2}\varDelta _{1,t\ell }^{-1} \end{array} \right) ,\\ \varvec{\omega }_{t,1}&= (w_{t1}\varDelta _{1,t1}^{-1}, \ldots ,w_{t(L(t)-1)} \varDelta _{1,t(L(t)-1)}^{-1})^{\prime }, \end{aligned}$$

and

$$\begin{aligned} \varvec{m}_{t,1} = \left( \varDelta _{1,t1}^{-1}\varDelta _{2,t1},\ldots , \varDelta _{1,t(L(t)-1)}^{-1}\varDelta _{2,t(L(t)-1)}, \sum _{\ell =1}^{L(t)} w_{t\ell }\varDelta _{1,t\ell }^{-1}\varDelta _{2,t\ell } - \theta _{t}\right) ^{\prime }, \end{aligned}$$

where $\varDelta _{1,t\ell }$ and $\varDelta _{2,t\ell }$ are defined in the specification of the full conditional distribution for $\mu _{t\ell }$ in the unconstrained state model in Eqs. (10) and (11). Partition $\varvec{V}_{t,1}$ and $\varvec{m}_{t,1}$ above as,

$$\begin{aligned} \varvec{V}_{t,1} = \left( \begin{array}{cc} \varvec{\varSigma }_{t}^{(11)} &{} (\varvec{\sigma }_{t}^{(12)})^{\prime } \\ \varvec{\sigma }_{t}^{(12)} &{} \sigma _{t}^{(22)} \end{array} \right) , \end{aligned}$$

and $\varvec{m}_{t,1} = (\varvec{m}_{t,1}^{(1)}, m_{t}^{(2)})^{\prime }$, respectively, where $\varvec{m}_{t}^{(1)}$ is of dimension $L(t) - 1,\,\varvec{\varSigma }_{t}^{(11)}$ is of dimension $(L(t)-1) \times (L(t) - 1)$, and $\varvec{\sigma }_{t}^{(12)}$ is of dimension $(L(t) -1) \times 1$.

• The CPD of $\varvec{\mu }_{t}^{(1)} \,|\,\varvec{\varOmega }$ is $\varvec{\mu }_{t}^{(1)}\,|\,\varvec{\varOmega } \sim \hbox {N}(\varvec{\delta }_{t}, \varvec{\varDelta }_{t})$, where

  $$\begin{aligned} \varvec{\delta }_{t} = \varvec{m}_{t}^{(1)} - \varvec{\sigma }_{t}^{(12)}/ \sigma _{t}^{(22)} m_{t}^{(2)}, \quad \varvec{\varDelta }_{t} = \varvec{\varSigma }_{t}^{(11)} - \varvec{\sigma }_{t}^{(12)}/ \sigma _{t}^{(22)}(\varvec{\sigma }_{t}^{(12)})^{\prime }. \end{aligned}$$