Simultaneous estimation of cross-validation errors in least squares collocation applied for statistical testing and evaluation of the noise variance components

Abstract

The cross-validation technique is a popular method to assess and improve the quality of prediction by least squares collocation (LSC). We present a formula for direct estimation of the vector of cross-validation errors (CVEs) in LSC which is much faster than element-wise CVE computation. We show that a quadratic form of CVEs follows Chi-squared distribution. Furthermore, a posteriori noise variance factor is derived by the quadratic form of CVEs. In order to detect blunders in the observations, estimated standardized CVE is proposed as the test statistic which can be applied when noise variances are known or unknown. We use LSC together with the methods proposed in this research for interpolation of crustal subsidence in the northern coast of the Gulf of Mexico. The results show that after detection and removing outliers, the root mean square (RMS) of CVEs and estimated noise standard deviation are reduced about 51 and 59%, respectively. In addition, RMS of LSC prediction error at data points and RMS of estimated noise of observations are decreased by 39 and 67%, respectively. However, RMS of LSC prediction error on a regular grid of interpolation points covering the area is only reduced about 4% which is a consequence of sparse distribution of data points for this case study. The influence of gross errors on LSC prediction results is also investigated by lower cutoff CVEs. It is indicated that after elimination of outliers, RMS of this type of errors is also reduced by 19.5% for a 5 km radius of vicinity. We propose a method using standardized CVEs for classification of dataset into three groups with presumed different noise variances. The noise variance components for each of the groups are estimated using restricted maximum-likelihood method via Fisher scoring technique. Finally, LSC assessment measures were computed for the estimated heterogeneous noise variance model and compared with those of the homogeneous model. The advantage of the proposed method is the reduction in estimated noise levels for those groups with the fewer number of noisy data points.

Appendices

Appendix A

1.1 A lemma in linear algebra

Notation For an arbitrary matrix $\mathbf{D}=\left\{ {d_{ij} } \right\}$, $\mathbf{d}_{i,-i}$ is the ith row of D whose ith element is removed and ${{\varvec{D}}}_{-i,-i}$ is the same matrix whose ith row and column are removed.

Lemma

If $\mathbf{A}$ represents an arbitrary symmetric positive definite matrix and $\mathbf{B}=\mathbf{A}^{-1}$, then

$$\begin{aligned} \mathbf{a}_{i,-i} \mathbf{A}_{-i,-i}^{-1} =-\frac{1}{b_{ii} }\mathbf{b}_{i,-i} \end{aligned}$$

(A1)

Proof

we define the vector $\mathbf{d}^{(i)}$ by

$$\begin{aligned} \mathbf{d}^{(i)}=\mathbf{b}_i \mathbf{A} \end{aligned}$$

(A2)

where $\mathbf{b}_i$ is the ith row of $\mathbf{B}$, the kth element of $\mathbf{d}^{(i)}$ is simply derived

$$\begin{aligned} d_k^{(i)} =\sum _{j} {b_{ij} a_{jk}} =\delta _{ik} \end{aligned}$$

(A3)

where $\delta _{ik}$ is the Kronecker delta. Considering the arbitrary vector $\mathbf{e}^{(i)}$ that is defined by

$$\begin{aligned} \mathbf{e}^{(i)}=\mathbf{b}_{i,-i} \mathbf{A}_{-i,-i} \end{aligned}$$

(A4)

and using Eq. (A3), one can conclude that:

$$\begin{aligned} e_k^{(i)}= & {} \sum _{j\ne i} {b_{ij} a_{jk} } =d_k^{(i)} -b_{ii} a_{ik}\nonumber \\= & {} -b_{ii} a_{ik} \quad \forall k\ne i \end{aligned}$$

(A5)

Finally, the following relations are deduced from Eq. (A5)

$$\begin{aligned} a_{ik}= & {} \frac{-1}{b_{ii} }e_k^{(i)} \nonumber \\= & {} \frac{-1}{b_{ii} }\sum _{j\ne i} {b_{ij} a_{jk} } \quad \forall k\ne i \end{aligned}$$

(A6)

Therefore,

$$\begin{aligned} \mathbf{a}_{i,-i} =-\frac{1}{b_{ii} }{} \mathbf{b}_{i,-i} \mathbf{A}_{-i,-i} \end{aligned}$$

(A7)

It has to be mentioned here that any principal submatrix of a positive definite matrix is also positive definite (Harville 1997, p. 214). Therefore, for the positive definite matrix $\mathbf{A}$, $\mathbf{A}_{-i,-i}$ is always invertible. $\square$

Appendix B

1.1 LSC prediction errors and noise estimation

LSC prediction error at an unobserved point $p_0$ is computed by (Moritz 1972, p. 47; Mikhail and Ackermann 1976, p. 422)

$$\begin{aligned} \sigma _{\hat{{y}}_0 }^2= & {} c_{s_{0} s_{0} } -\mathbf{c}_{{s}_{0} \mathbf{s}} \mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{c}_{s_0 \mathbf{s}}^T \nonumber \\&+\left( {\mathbf{c}_{s_0 \mathbf{s}} \mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{A}-\mathbf{a}_0 } \right) \mathbf{C}_{{\hat{\mathbf{x}}\hat{\mathbf{x}}}} \left( {\mathbf{c}_{s_0 \mathbf{s}} \mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{A}-\mathbf{a}_0 } \right) ^{T} \end{aligned}$$

(B1)

where $\hat{{y}}_0$ is prediction of y at $p_0$ and $c_{s_0 s_0 }$ is the signal variance, $\mathbf{c}_{s_0 \mathbf{s}}$ is the cross-covariance vector of the predicted point and the vector of data points, $\mathbf{a}_0$ is the vector of trend for predicted point, and $\mathbf{C}_{{\hat{\mathbf{x}}\hat{\mathbf{x}}}}$ denotes the covariance matrix of estimated trend parameters which is computed by the following formula

$$\begin{aligned} \mathbf{C}_{{\hat{\mathbf{x}}\hat{\mathbf{x}}}} =\left( {\mathbf{A}^{T}\mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{A}} \right) ^{-1} \end{aligned}$$

(B2)

LSC internal error (adopted from Darbeheshti and Featherstone 2009) is LSC prediction error at an observed point $p_i$

$$\begin{aligned} \sigma _{\hat{{y}}_i }^2= & {} c_{s_i s_i } -\mathbf{c}_{s_i \mathbf{s}} \mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{c}_{s_i \mathbf{s}}^T \nonumber \\&+\left( {\mathbf{c}_{s_i \mathbf{s}} \mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{A}-\mathbf{a}_i } \right) \mathbf{C}_{{\hat{\mathbf{x}}\hat{\mathbf{x}}}} \left( {\mathbf{c}_{s_i \mathbf{s}} \mathbf{C}_{\mathbf{ww}}^{-1} \mathbf{A}-\mathbf{a}_i } \right) ^{T} \end{aligned}$$

(B3)

where $\hat{{y}}_i$ is prediction of y at $p_i$ and $c_{s_i s_i }$ is the signal variance, $\mathbf{c}_{s_i \mathbf{s}}$ is the cross-covariance vector of the predicted point and the vector of data points, $\mathbf{a}_i$ is the ith row of $\mathbf{A}$.

Noise of the observations in Eq. (1) is always unknown. It can be estimated by the following formula (Moritz 1980, p. 119)

$$\begin{aligned} {\hat{\mathbf{n}}}=\mathbf{C}_{\mathbf{nn}} \mathbf{C}_{\mathbf{ww}}^{-1} \left( {\mathbf{y}-\mathbf{A}{\hat{\mathbf{x}}}} \right) \end{aligned}$$

(B4)