1 Introduction

Total least squares (TLS) adjustment referring to Errors-in-Variables (EIV) models has a wide mathematical literature, e.g., Golub van Loan (1980), van Huffel and Vandewalle (1991), and Rao and Toutenburg (1999). It has also been extensively explored by researchers in the field of geodesy. There are a number of contributions analyzing the relationships between the EIV models and the standard iteratively linearized models, well established in geodesy, and simultaneously proposing suitable algorithms for the rigorous evaluation of parameters in nonlinear EIV models (e.g., Schaffrin and Wieser 2008; Schaffrin and Felus 2008; Neitzel 2010).

The present contribution is focussed entirely on the problem of response-based reliability analysis for TLS adjustment. It should be noted that analyses of this type are usually carried out at the design stage when one wants to evaluate the reliability properties of the originally nonlinear adjustment model under consideration. In such a priori analyses, the nonlinearity problems may be overcome by using approximate values of the parameters when observation results are lying sufficiently close to the true values, or, practically, by using nominal values of these quantities.

In an attempt to generalize the EIV model for the purpose of the response-based reliability analysis, the most reasonable approach, backed by an appropriate proof, appeared to this author to take, as a basis, a nonlinear stochastic model containing two types of quantities, namely, the error-free unknown parameters to be determined and the observations as random variables of well-known values and accuracy characteristics. This led to the use of the so-called combined case of least-squares adjustment (Krakiwsky 1975), being termed a method of condition equations with unknowns, also known as the Gauss–Helmert model. This equivalent approach to TLS adjustment as a specific least-squares problem turned out to be consistent with that discussed in Schaffrin et al. (2006), Schaffrin and Snow (2010), and Neitzel (2010), and it is followed here since it seems to be most suitable for the purpose of the response-based reliability analysis along the lines of the approach as in Prószyński (2010).

However, since such an approach requires the use of the linear relationship between the observations and residuals, restrictions to a general G–H model had to be made confining the considerations to its quasi-linear form only. Such a form means here a nonlinear G–H model that is linear with respect to the observation vector formed of both the dependent and the independent variables.

To establish a link between this paper and publications that do not use the term reliability, but are concerned with similar properties of over-determined linear models (e.g. Chatterjee and Hadi 1988), the domain of this paper could as well be expressed as the “sensitivity” analysis of orthogonal regression.

2 Generalized EIV model and its linearized form for the purpose of reliability analysis

We shall first show that the TLS adjustment problem referring to a nonlinear EIV model is, with respect to response-based reliability analysis, equivalent to the LS problem referring to a linearized form of this model.

Let us thus consider a (quasi-linear) EIV model for homoscedastic and uncorrelated observations, having the form

$$\begin{aligned} (\mathbf{A}_\mathrm{obs} -\mathbf{E}_\mathrm{A})\mathbf{x}=\mathbf{y}_\mathrm{obs} -{\varvec{\upvarepsilon }}_{ y} \end{aligned}$$
(1)

where \(\mathbf{A}_\mathrm{obs}\) is the \(n\times u\) matrix of observed coefficients, rank \(\mathbf{A}_\mathrm{obs} =u, \mathbf{E}_\mathrm{A}\) is the \(n\times u\) matrix of unknown random errors in observed coefficients, \(\mathbf{y}_\mathrm{obs}\) is the \(n \times 1\) vector of observations, \({\varvec{\upvarepsilon }}_{ y}\) is the \(n \times 1\) vector of unknown random errors in observations, and x is the \(u\times 1\) vector of unknown parameters.

To follow the notation as in (Prószyński 2010), we shall use the form (1) putting \(\mathbf{V}_\mathrm{A} =-\mathbf{E}_\mathrm{A} , \mathbf{v}_{ y} =-{\varvec{\upvarepsilon }}_{ y} \), i.e.

$$\begin{aligned} (\mathbf{A}_\mathrm{obs} +\mathbf{V}_\mathrm{A} )\mathbf{x}=\mathbf{y}_\mathrm{obs} +\mathbf{v}_{ y} \end{aligned}$$
(2)

In the homoscedastic cases, the TLS problem is defined as finding \( \mathbf{x}_\mathrm{{TLS}}\) for the nonlinear system (2), such that

$$\begin{aligned} \Vert {[ {{\begin{array}{l@{\quad }l} {\mathbf{V}_\mathrm{A} }&{\mathbf{v}_{ y} } \\ \end{array} }}]} \Vert _\mathrm{F}^2 =\min \end{aligned}$$
(3)

where \(\Vert {\cdot }\Vert _\mathrm{F}\) denotes the Frobenius norm, avoiding the linearization of the model.

Since \(\Vert {[{{\begin{array}{l@{\quad }l} {\mathbf{V}_\mathrm{A} }&{\mathbf{v}_{ y} } \\ \end{array} }}]}\Vert _\mathrm{F}^2 =\Vert {\text{ vec}\mathbf{V}_\mathrm{A} } \Vert _2^2 +\Vert {\mathbf{v}_{ y} }\Vert _2^2 =\left\Vert {{\begin{array}{c} {\text{ vec}\mathbf{V}_\mathrm{A} } \\ {\mathbf{v}_{ y} } \\ \end{array} }} \right\Vert_2^2 \), where \(\text{ vec}\mathbf{V}_\mathrm{A}\) is the \((un \times 1)\) vector formed by stacking the columns of the matrix \(\mathbf{V}_\mathrm{A}\) underneath each other, we obtain the TLS condition in equivalent form to (3) for the EIV model (2), as

$$\begin{aligned} \left\Vert {{\begin{array}{c} {\text{ vec}\mathbf{V}_\mathrm{A} } \\ {\mathbf{v}_{ y} } \\ \end{array} }} \right\Vert_2^2 =\min \end{aligned}$$
(4)

which is the LS condition for this model.

The equivalence between the conditions (3) and (4) as applied to the EIV model (2) makes it possible to formulate the TLS problem for correlated observations, using a suitably modified condition (3).

For the response-based reliability analysis of any adjustment model, we need a linear relationship between the vector of observations and the vector of LS residuals. To obtain such a relationship for the EIV model (2), we find its linearized form, being first-order Taylor approximation obtained at a point \((\mathbf{x}_\mathrm{o} , \mathbf{A}_\mathrm{obs} )\), and transform it, so that it contains aggregated vectors of observations and unknown random errors. Coming through an intermediate step in derivations after neglecting the second-order term \(\mathbf{V}_\mathrm{A} {d}\mathbf{x}\), we get

$$\begin{aligned} \mathbf{A}_\mathrm{o} {d}\mathbf{x}+\mathbf{V}_\mathrm{A} \mathbf{x}_\mathrm{o} -\mathbf{v}_{ y} +\mathbf{A}_\mathrm{obs} \mathbf{x}_\mathrm{o} -\mathbf{y}_\mathrm{obs} =\mathbf{0} \end{aligned}$$

where \(\mathbf{A}_\mathrm{o} \) is a non-random matrix, obtained from \(\mathbf{A}_\mathrm{obs} \) by subtracting random zeros as in (Schaffrin and Snow 2010).

After regrouping the terms, we obtain finally the Gauss–Helmert model in linearized form

$$\begin{aligned} \mathbf{A}_\mathrm{o} {d}\mathbf{x}+[{{\begin{array}{ll} \mathbf{K}&{-\mathbf{I}_{ n} } \\ \end{array} }}]\left[ {{\begin{array}{c} {\text{ vec} \mathbf{V}_\mathrm{A} } \\ {\mathbf{v}_{ y} } \\ \end{array} }} \right]+[ {{\begin{array}{ll} \mathbf{K}&{-\mathbf{I}_{ n} } \\ \end{array} }}]\left[ {{\begin{array}{c} {\text{ vec} \mathbf{A}_\mathrm{obs} } \\ {\mathbf{y}_\mathrm{obs} } \\ \end{array} }} \right]=\mathbf{0}\nonumber \\ \end{aligned}$$
(5)

where K is the \((n \times nu)\) matrix; \(\mathbf{K}=\mathbf{I}_{ n} \otimes \mathbf{x}_\mathrm{o}^\mathrm{T} ; \text{ rank} [{{\begin{array}{ll} \mathbf{K}&{-\mathbf{I}_{ n} } \\ \end{array} }}]\!=n\). Finding \({d}\mathbf{x}\) that minimizes the LS condition (4) subject to the linearized Gauss–Helmert model (5), we shall consider as an approximation of the TLS problem for the purposes of response-based reliability analysis. Unlike in seeking the solution to the original TLS problem, in reliability analysis that is usually carried out at a design stage, there is no problem of getting approximate values of parameters \((\mathbf{x}_\mathrm{o} )\), as we may directly use the nominal values of x. The same applies to approximate values of independent random variables \((\mathbf{A}_\mathrm{obs})\).

In order to generalize the EIV model for the purposes of response-based reliability analysis, we shall consider the following nonlinear Gauss–Helmert model

$$\begin{aligned} \mathbf{f}(\mathbf{u}, \mathbf{r}_\mathrm{obs} -{\varvec{\upvarepsilon }})=\mathbf{0} \qquad {\varvec{\upvarepsilon }}\sim (\mathbf{0},\mathbf{C}) \end{aligned}$$
(6)

obtained by combining a nonlinear functional model \(\mathbf{f}(\mathbf{u}, \mathbf{r}){=} \mathbf{0}\) with a stochastic observation model (as in the method of condition equations with unknowns, Krakiwsky 1975). Where f is the \(n\times 1\) vector of condition equations, u is the \(u \times 1\) vector of unknown parameters \((n >u)\), \(\mathbf{r}_\mathrm{obs}\) is the \(r \times 1\) vector of random variables \((r\ge n)\) with \(\mathbf{r}={E}(\mathbf{r}_\mathrm{obs} )\), \({\varvec{\upvarepsilon }}\) is the \(r\times 1\) vector of unknown random observation errors; later we shall be using \(\mathbf{v}=-{\varvec{\upvarepsilon }}\), \(\mathbf{C}\) is the \(r\times r\) (p.d.) covariance matrix for the vector \({\varvec{\upvarepsilon }}\) as well the vector \( \mathbf{r}_\mathrm{obs}\), and E is the expectation operator.

We assume that the random variables in the vector \(\mathbf{r}_\mathrm{obs}\) can be network observations, directly observed parameters or observed coefficients. Considering the need for a response-based reliability analysis, we shall require that the functions in \(\mathbf{f}(\mathbf{u}, \mathbf{r})\) are confined to those that are linear with respect to the vector r (thus termed quasi-linear), what can be formally expressed as

$$\begin{aligned} \frac{\partial ^{2}\mathbf{f}(\mathbf{u},\mathbf{r})}{\partial \mathbf{r}^{2}}=\mathbf{0} \end{aligned}$$
(7)

Here are the examples of characteristic EIV models that, together with the model (1), satisfy the above requirement, i.e.

  1. (a)

    \(\mathbf{y}_\mathrm{obs} +\mathbf{v}_{ y} =(\mathbf{G}_\mathrm{obs} +\mathbf{E}_\mathrm{G })\mathbf{x}+\mathbf{z}\), with x and z being the vectors of unknown parameters, the aggregated vectors are \(\mathbf{u}=\left[ {{\begin{array}{c} \mathbf{x} \\ \mathbf{z} \\ \end{array} }} \right], \mathbf{r}=\left[ {{\begin{array}{c} {\text{ vec} \mathbf{G}_\mathrm{obs} } \\ {\mathbf{y}_\mathrm{obs} } \\ \end{array} }} \right]\) The TLS condition and the equivalent LS condition will have the form as for the EIV model (2), i.e.

    $$\begin{aligned} \Vert {[{{\begin{array}{l@{\quad }l} {\mathbf{V}_\mathrm{G} }&{\mathbf{v}_{ y} } \\ \end{array} }}]} \Vert _\mathrm{F}^2 =\min \equiv \left\Vert {{\begin{array}{c} {\text{ vec}\mathbf{V}_\mathrm{G} } \\ {\mathbf{v}_{ y} } \\ \end{array} }} \right\Vert_2^2 =\min \end{aligned}$$
  2. (b)

    \(\mathbf{y}_\mathrm{obs} +\mathbf{v}_{ y} =\mathbf{G}(\mathbf{t})\cdot (\mathbf{x}_\mathrm{obs} +\mathbf{v}_{ x} )+\mathbf{z}\), with t and z being the vectors of unknown parameters, the aggregated vectors are \(\mathbf{u}=\left[ {{\begin{array}{c} \mathbf{t} \\ \mathbf{z} \\ \end{array} }} \right],\mathbf{r}=\left[ {{\begin{array}{c} {\mathbf{x}_\mathrm{obs} } \\ {\mathbf{y}_\mathrm{obs} } \\ \end{array} }} \right]\) The TLS condition and the equivalent LS condition will have the form

    $$\begin{aligned} \Vert {[ {{\begin{array}{l@{\quad }l} {\mathbf{v}_{ x} }&{\mathbf{v}_{ y} } \\ \end{array} }}]}\Vert _\mathrm{F}^2 =\min \equiv \left\Vert {{\begin{array}{c} {\mathbf{v}_{ x} } \\ {\mathbf{v}_{ y} } \\ \end{array} }} \right\Vert_2^2 =\min \end{aligned}$$

    which is consistent with the approach for the model (2), since \(\mathbf{v}_{ x} \) can be interpreted as a one-column matrix of residuals, i.e. \(\text{ vec}\mathbf{v}_{ x} =\mathbf{v}_{ x} \).

Let the linearized form of the model (6), obtained in a similar way as (5), i.e. with the expansion point \((\mathbf{u}_\mathrm{o} , \mathbf{r}_\mathrm{obs} )\), be denoted as

$$\begin{aligned} \mathbf{A}{d}\mathbf{u}+\mathbf{Bv}+\mathbf{w}=\mathbf{0} \qquad \mathbf{v}\sim (\mathbf{0},\mathbf{C}) \end{aligned}$$
(8)

where \(\mathbf{A}=\frac{\partial \mathbf{f}}{\partial \mathbf{u}}\left| {_{(\mathbf{u}_\mathrm{o},\mathbf{r}_\mathrm{o} )} } \right.\!\!; \mathbf{B}=\frac{\partial \mathbf{f}}{\partial \mathbf{r}}\left| {_{(\mathbf{u}_\mathrm{o},\mathbf{r}_\mathrm{obs} )} } \right.\!\!; \mathbf{w}=\mathbf{f}(\mathbf{u}_\mathrm{o} , \mathbf{r}_\mathrm{obs} ); \mathbf{A}(n\times u)\), rank \(\mathbf{A} = u\), \(\mathbf{B}(n\times r)\), rank \(\mathbf{B} = n, r\ge n; \mathbf{w}(n\times 1); \mathbf{B}\) corresponds to the matrix \([{{\begin{array}{ll} \mathbf{K}&{-\mathbf{I}_{ n}}\\ \end{array} }}]\) as in the model (5); \(\mathbf{r}_\mathrm{o}\) is a non-random vector obtained from \(\mathbf{r}_\mathrm{obs} \) like \(\mathbf{A}_\mathrm{o}\) in the model (5); for the quasi-linear G–H model, we have

$$\begin{aligned} \mathbf{w}=\mathbf{Br}_\mathrm{obs} +\mathbf{g},\, \text{ where}\quad \mathbf{g}=\mathbf{f}(\mathbf{u}_\mathrm{o} , \mathbf{0}). \end{aligned}$$

This can be derived in the following way:

For the models (6) that satisfy (7) we have

$$\begin{aligned} \mathbf{B}&= \frac{\partial \mathbf{f}(\mathbf{u},\mathbf{r})}{\partial \mathbf{r}}\Big |{_{(\mathbf{u}_\mathrm{o},\mathbf{r}_\mathrm{obs} )}} =\frac{\partial \mathbf{f}(\mathbf{u}_\mathrm{o} ,\mathbf{r}_\mathrm{obs})}{\partial \mathbf{r}_\mathrm{obs} }\Big |{_{(\mathbf{u}_\mathrm{o},\mathbf{r}_\mathrm{obs} )} } \\&= \frac{\partial \mathbf{f}(\mathbf{u}_\mathrm{o} ,\mathbf{r}_\mathrm{obs} )}{\partial \mathbf{r}_\mathrm{obs} }\Big | {_{(\mathbf{u}_\mathrm{o},\mathbf{0})} } \end{aligned}$$

and hence,

$$\begin{aligned} \mathbf{w}&= \mathbf{f}(\mathbf{u}_\mathrm{o} , \mathbf{0}+\mathbf{r}_\mathrm{obs} ) =\mathbf{f}(\mathbf{u}_\mathrm{o} , \mathbf{0})+\frac{\partial \mathbf{f}(\mathbf{u}_\mathrm{o} ,\mathbf{r}_\mathrm{obs} )}{\partial \mathbf{r}_\mathrm{obs} }\Big |{_{(\mathbf{u}_\mathrm{o} , \mathbf{0})} \cdot } \mathbf{r}_\mathrm{obs}\\&= \mathbf{f}(\mathbf{u}_\mathrm{o} , \mathbf{0})+\mathbf{Br}_\mathrm{obs} \end{aligned}$$

The model (8) enables one to easily handle the case of heteroscedastic and correlated observations, by applying the LS condition \(\mathbf{v}^\mathrm{T}\mathbf{C}^{-1}\mathbf{v}=\min \), but at the cost of linearizing the Gauss–Helmert model (6).

3 Derivation of disturbance/response relationship for quasi-linear EIV models

In contrast to Schaffrin (1997), the approach to “reliability analysis” for systems with correlated observations according to Prószyński (2010) requires the use of the observation model with random variables which are correlated, dimensionless variables of equal accuracy. We thus have to modify the model (8), rescaling the random errors so that instead of the vector v we operate with the vector \(\mathbf{v}_\mathrm{s} ={\varvec{\Sigma }}^{-1}\mathbf{v}\), where \({\varvec{\Sigma }}=(\text{ diag}\;\mathbf{C})^{1/2}\). This naturally results in that the covariance matrix of the rescaled random errors coincides with the original correlation matrix.

So, using the matrix \({\varvec{\Sigma }}\), we present the model (8) in the equivalent form

$$\begin{aligned}&\mathbf{A}{d}\mathbf{u}+\mathbf{B}{\varvec{\Sigma }}\cdot {\varvec{\Sigma }}^{-1}\mathbf{v}+\mathbf{B}{\varvec{\Sigma }}\cdot {\varvec{\Sigma }}^{-1}\mathbf{r}_\mathrm{obs} +\mathbf{g}=\mathbf{0}\\ &\quad {\varvec{\Sigma }}^{-1}\mathbf{v}\nonumber \sim (\mathbf{0},{\varvec{\Sigma }}^{-1}\mathbf{C}{\varvec{\Sigma }}^{-1}) \end{aligned}$$
(9)

and introducing the notation

$$\begin{aligned}&\mathbf{r}_\mathrm{{obs,s}} ={\varvec{\Sigma }}^{-1}\mathbf{r}_\mathrm{obs} ,\quad \mathbf{v}_\mathrm{s} ({\text{ as} \text{ above}}),\quad \mathbf{B}_\mathrm{s} =\mathbf{B}{\varvec{\Sigma }};\nonumber \\&\quad \mathbf{w}_\mathrm{s} =\mathbf{B}_\mathrm{s} \mathbf{r}_\mathrm{{obs,s}} +\mathbf{g};\quad \mathbf{C}_\mathrm{s} ={\varvec{\Sigma }}^{-1}\mathbf{C}{\varvec{\Sigma }}^{-1} \end{aligned}$$
(10)

we obtain a modified form of the model (8)

$$\begin{aligned} \mathbf{A}{d}\mathbf{u}+\mathbf{B}_\mathrm{s} \mathbf{v}_\mathrm{s} +\mathbf{w}_\mathrm{s} =\mathbf{0} \qquad \mathbf{v}_\mathrm{s} \sim (\mathbf{0},\mathbf{C}_\mathrm{s} ) \end{aligned}$$
(11)

To get the relationship between \({\hat{\mathbf{v }}}_\mathrm{s} \) (i.e. the LS estimate for \(\mathbf{v}_\mathrm{s})\) and \(\mathbf{r}_\mathrm{{obs,s}}\), necessary for response-based reliability analysis, we use the formulas given in (Krakiwsky 1975) adopting them to the notation in (11), i.e.

$$\begin{aligned} {\hat{\mathbf{v}}}_\mathrm{s} =-\mathbf{Mw}_\mathrm{s} \end{aligned}$$
(12)

where:

$$\begin{aligned} \mathbf{M}&= \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} (\mathbf{B}_\mathrm{s} \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}\\&\quad \times \,\{{\mathbf{I}-\mathbf{A}[{\mathbf{A}^\mathrm{T}(\mathbf{B}_\mathrm{s} \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}\mathbf{A}}]^{-1}\mathbf{A}^\mathrm{T}(\mathbf{B}_\mathrm{s} \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}}\} \end{aligned}$$

Substituting into (12) the vector \(\mathbf{w}_\mathrm{s} \) as in (10) (i.e. for quasi-linear models) and denoting \(\mathbf{H}=\mathbf{MB}_\mathrm{s}\), we obtain (12) in the form

$$\begin{aligned} {\hat{\mathbf{v}}}_\mathrm{s} =-\mathbf{Hr}_\mathrm{obs,s} -\mathbf{Mg} \end{aligned}$$
(13)

where

$$\begin{aligned} \mathbf H&= \mathbf C _{s}\mathbf B _\mathrm{s}^\mathrm{T} (\mathbf{B}_\mathrm{s} \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}\\&\times \,\{ {\mathbf{I}-\mathbf{A}[{\mathbf{A}^\mathrm{T}(\mathbf{B}_\mathrm{s} \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}\mathbf{A}}]^{-1}\mathbf{A}^\mathrm{T}(\mathbf{B}_\mathrm{s} \mathbf{C}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}} \}\mathbf{B}_\mathrm{s} \end{aligned}$$

We easily can check that the matrix H as in (13) is an operator of oblique projection since it is idempotent and asymmetric.

The rank of H, which is crucial for internal reliability analysis, is

$$\begin{aligned} \text{ rank} \mathbf{H}=n-u \end{aligned}$$
(14)

The proof, based on trace properties (Rao 1973), is immediate

$$\begin{aligned} \text{ rank} \,\mathbf{H}&= \text{ Tr} \mathbf{H}=\text{ Tr} \{ {\mathbf{I}_{ n} -\mathbf{A}[ {\mathbf{A}^\mathrm{T}(\mathbf{BCB}^\mathrm{T})^{-1}\mathbf{A}}]^{-1}\mathbf{A}^\mathrm{T}(\mathbf{BCB}^\mathrm{T})^{-1}}\}\\&= {n}-\text{ Tr}\mathbf{I}_{ u} ={n-u} \end{aligned}$$

With \(\Delta \mathbf{r}_\mathrm{obs} \) representing the vector of standardized observation gross errors, and \(\Delta { \hat{\mathbf{v}}}_\mathrm{s} \) the vector of induced incremental changes in the corresponding observation corrections, we may formulate on the basis of (13) the so called “disturbance/response” relationship for the model (11), i.e.

$$\begin{aligned} \Delta {\hat{\mathbf{v}}}_\mathrm{s} =-\mathbf{H}\cdot \Delta \mathbf{r}_\mathrm{{obs,s}} \end{aligned}$$
(15)

For the original model (8) we would get

$$\begin{aligned} \Delta {\hat{\mathbf{v}}}=-\mathbf{R}\cdot \Delta \mathbf{r}_\mathrm{obs} \end{aligned}$$

where

$$\begin{aligned} \mathbf{R}&= \mathbf{CB}^\mathrm{T}(\mathbf{BCB}^\mathrm{T})^{-1}\\&\times \,\{{\mathbf{I}-\mathbf{A}[{\mathbf{A}^\mathrm{T}(\mathbf{BCB}^\mathrm{T})^{-1}\mathbf{A}}]^{-1}\mathbf{A}^\mathrm{T}(\mathbf{BCB}^\mathrm{T})^{-1}}\}\mathbf{B} \end{aligned}$$

It is straightforward to show that the operators H and R are similar matrices, i.e. \(\mathbf{H}={\varvec{\Sigma }}^{-1}\mathbf{R}{\varvec{\Sigma }}\).

Listed below are specific cases covered by the disturbance/response relationship (15) :

$$\begin{aligned}&\mathbf{B}_\mathrm{s} (n\times r), \mathbf{r}>n; \mathbf{C}_\mathrm{s} =\mathbf{I} \quad \text{ EIV,} \text{ uncorrelated} \text{ observations}\\&\mathbf{H}=\mathbf{B}_\mathrm{s}^\mathrm{T} (\mathbf{B}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}\{ {\mathbf{I}-\mathbf{A}[{\mathbf{A}^\mathrm{T}(\mathbf{B}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}\mathbf{A}} ]^{-1}\mathbf{A}^\mathrm{T}(\mathbf{B}_\mathrm{s} \mathbf{B}_\mathrm{s}^\mathrm{T} )^{-1}}\}\mathbf{B}_\mathrm{s}\\&\mathbf{B}_\mathrm{s} (n\times n),\mathbf{B}= -\mathbf{I}; \mathbf{C}_\mathrm{s} \ne \mathbf{I}\quad \text{ GM,} \text{ correlated} \text{ observations}\\&\mathbf{H}=\mathbf{I}-\mathbf{A}_\mathrm{s} (\mathbf{A}_\mathrm{s}^\mathrm{T} \mathbf{C}_\mathrm{s}^{-1} \mathbf{A}_\mathrm{s} )^{-1}\mathbf{A}_\mathrm{s}^\mathrm{T} \mathbf{C}_\mathrm{s}^{-1} , \quad \text{ where}\; \mathbf{A}_\mathrm{s} ={\varvec{\Sigma }}^{-1}\mathbf{A}\\&\mathbf{B}_\mathrm{s} (n\times n),\mathbf{B}=-\mathbf{I}; \mathbf{C}_\mathrm{s} =\mathbf{I} \quad \text{ GM,} \text{ uncorrelated} \text{ observations}\\&\mathbf{H}=\mathbf{I}-\mathbf{A}_\mathrm{s} (\mathbf{A}_\mathrm{s}^\mathrm{T} \mathbf{A}_\mathrm{s} )^{-1}\mathbf{A}_\mathrm{s}^\mathrm{T} , \, \text{ where} \, \mathbf{A}_\mathrm{s} ={\varvec{\Sigma }}^{-1}\mathbf{A} \end{aligned}$$

We shall add a commentary on the advantages of operating in reliability analysis with the standardized model (11) instead of the original, non-standardized one (8). The basic advantage is that the standardized observations, being dimensionless variables of equal variances, are more readily comparable with one another within the whole model. This enables one to formulate consistent and interpretable reliability criteria, which would not be possible in the original non-standardized model where observations are, in general, mutually uncomparable quantities. Moreover, the correlation matrix \(\mathbf{C}_\mathrm{s}\) appears in the operator in explicit form. Hence, we get a clear discrimination between the case of uncorrelated observations (H being an operator of orthogonal projection) and the case of correlated observations (H being an operator of oblique projection).

4 Indices for response-based reliability of quasi-linear EIV models

Since, for the EIV models with correlated observations, the matrix H is an oblique projector (see formula (13)), we shall be using a two-parameter reliability measure for the \(i\)th observation as proposed for GM models with standardized correlated observations (Prószyński 2010)

$$\begin{aligned} h_{(i)} =(h_{ii}, w_{ii} ) \end{aligned}$$
(16)

where \(h_{ii} \) is the \(i\)th diagonal element of H, and \(w_{ii}\) is the asymmetry index for the \(i\)th row and the \(i\)th column of H. The index \(h_{ii}\), denoted also as \(L_{i(i)} \), is called a “local response of the model”, i.e. the response in the \(i\)th residual to a potential gross error in that observation.

It also proved advantageous to use as a reliability measure the pair of indices \((h_{ii},k_i )\), where \(k_i\) is the ratio of the squared quasi-global response \(Q_{(i)}\) to the squared local response \(L_{i(i)} \) of the residuals to a potential gross error in the \(i\)th observation, i.e.

$$\begin{aligned} k_i \!=\!\frac{Q_{(i)}^2 }{L_{i(i)}^2 }\!=\!\frac{h_{ii} -h_{ii}^2 -w_{ii} }{h_{ii}^2 }\!=\!\frac{h_{ii} -w_{ii} }{h_{ii}^2 }-1\quad ({\text{ for}}\;h_{ii} \ne 0)\nonumber \\ \end{aligned}$$
(17)

where the quasi-global response \(Q_{(i)}\) means the global response after stripping it from the local response.

In the numerical examples that will follow, the results of such a response-based reliability analysis for EIV and GM models will be shown in a tabular and/or a graphic form. To distinguish the case of uncorrelated observations, we shall replace \(h_{ii} \) by the index \(\bar{{h}}_{ii} \), as in (Prószyński 2010).

The method of reliability analysis applied in the present paper does not follow the traditional approach of Baarda, since it does not lead to specifying the minimal detectable biases for individual observations. It is based entirely on the model responses to gross errors, and therefore is termed here a “response-based” reliability analysis. This approach offers “reliability criteria” interpretable in terms of model responses to observation disturbances.

We recall here the criteria proposed for GM models, i.e.

$$\begin{aligned} ({a})\;\bar{{h}}_{ii}&> 0.5;\quad ({b})\;0.5<h_{ii} \le 1.5;\quad \!\! h_{ii} -2.2h_{ii}^2 \\&<w_{ii} <h_{ii} -h_{ii}^2\nonumber \end{aligned}$$
(18)

for uncorrelated (a) and correlated (b) observations, respectively.

Since the above criteria are derived from the following requirements:

  • the response in the individual observation (i.e. a local response) should compensate for at least half of the disturbance residing in that observation;

  • the local response with its absolute value should surpass the quasi-global response,

with networks that satisfy them, we may expect better detectability of outliers, and hence, smaller values of MDBs obtained along the lines of Baarda.

5 Formulas for reliability analysis of specific cases of quasi-linear EIV models

We shall discuss specific cases of quasi-linear EIV models assuming the systems with correlated observations with given positive-definite covariance matrix. The cases themselves are very important in geodetic technologies, since they represent the observation systems frequently met in practice that fall into the class of EIV models.

5.1 Multiple linear regression

Let us consider a functional model

$$\begin{aligned} {a}_\mathrm{1} {x}_{ {i}\mathrm{1}} +\cdots +{a}_\mathrm{s} {x}_{{ i}\mathrm{s}} +{b}={y}_{ i} \qquad i= \text{1},\ldots ,n \end{aligned}$$
(19)

or, in a matrix form,

$$\begin{aligned} \left[{{\begin{array}{c} {\mathbf{x}_1^\mathrm{T}} \\ {\ldots } \\ {\mathbf{x}_{ n}^\mathrm{T}} \\ \end{array} }} \right]\cdot \mathbf{a}+{b}\cdot \mathbf{1}_{(n)} =\mathbf{y} \end{aligned}$$
(20)

where \(\mathbf{a} ({s}\times 1), \mathbf{x}_{ i} (\mathbf{s}\times 1), i\!=\!1,\ldots ,n,\mathbf{1}_{(n)}^\mathrm{T}\!=\![{{\begin{array}{llll} 1&1&\ldots&1 \\ \end{array} }} ], u\!=\!s+1, n>u\).

With \(\mathbf{x}_1 ,\ldots ,\mathbf{x}_n ,\mathbf{y}\) being vectors of random variables, and \({a}_\mathrm{1},{a}_\mathrm{2}, \ldots ,{a}_\mathrm{s} \), b the unknown parameters, the linearized form of (20) will be

$$\begin{aligned} \left[ {{\begin{array}{c} {\mathbf{x}_\mathrm{{1,obs}}^\mathrm{T}} \\ {\ldots } \\ {\mathbf{x}_{n,\mathrm {obs}}^\mathrm{T}} \\ \end{array} }} \right]\cdot \mathbf{a}_\mathrm{o}&+ \left[{{\begin{array}{c} {\mathbf{x}_\mathrm{{1,o}}^\mathrm{T}} \\ {\ldots } \\ {\mathbf{x}_{n,\mathrm {o}}^\mathrm{T}} \\ \end{array} }} \right]\cdot {d}\mathbf{a}+\left[{{\begin{array}{c} {\mathbf{v}_{{x},1}^\mathrm{T} } \\ {\ldots } \\ {\mathbf{v}_{{x},{n}}^\mathrm{T} } \\ \end{array} }} \right]\nonumber \\&\cdot \ \mathbf{a}_\mathrm{o} +{b}\cdot \mathbf{1}_{(n)} =\mathbf{y}_\mathrm{obs} +\mathbf{v}_{ y} \end{aligned}$$
(21)

After regrouping terms to get the form (8), we obtain

$$\begin{aligned}&\left[{{\begin{array}{l@{\quad }l} \mathbf{x}_\mathrm{1,o}^\mathrm{T}&1\\ \ldots&\ldots \\ \mathbf{x}_{n,\mathrm {o}}^\mathrm{T}&1\\ \end{array} }} \right]\cdot \left[{{\begin{array}{c} {d}\mathbf{a}\\ {b}\\ \end{array}}}\right]+[{{\begin{array}{lll} \mathbf{I}_{(n)}&\bigotimes \mathbf{a}_\mathrm{o}^\mathrm{T}&-\mathbf{I}_{(n)}\\ \end{array}}}]\cdot \left[ {{\begin{array}{c} \mathbf{v}_{x,1}\\ \ldots \\ \mathbf{v}_{x,n}\\ \mathbf{v}_{y}\\ \end{array}}}\right]\nonumber \\&\qquad +\left[ {{\begin{array}{lll} \mathbf{I}_{(n)}&\bigotimes \mathbf{a}_\mathrm{o}^\mathrm{T}&-\mathbf{I}_{(n)}\\ \end{array}}}\right] \left[{{\begin{array}{c} \mathbf{x}_\mathrm{1,obs}\\ \ldots \\ \mathbf{x}_{n,\mathrm {obs}}\\ \mathbf{y}_\mathrm{obs}\\ \end{array}}}\right]=\mathbf{0}\nonumber \\&\quad \mathbf{v}-(\mathbf{0},\mathbf{C}) \end{aligned}$$
(22)

where \(\mathbf{I}_{(n)} \) is a unit matrix, v represents an aggregated vector of residuals.

Hence, the matrices A and B are defined by

$$\begin{aligned} \mathbf{A}=\left[ {{\begin{array}{l@{\quad }l} {\mathbf{x}_{1,\mathrm{o}}^\mathrm{T} }&1 \\ {\ldots }&{\ldots } \\ {\mathbf{x}_{n,\mathrm{o}}^\mathrm{T} }&1 \\ \end{array} }} \right]\quad \mathbf{B}=[ {{\begin{array}{ll} {\mathbf{I}_{(n)} \otimes \mathbf{a}_\mathrm{o}^\mathrm{T} }&{-\mathbf{I}_{(n)} } \\ \end{array} }}] \end{aligned}$$
(23)

which, together with the given covariance matrix C, are necessary for the reliability analysis of this case of an EIV model.

We omit discussion of the structure of C, since it will depend on the properties of the observations used in a particular task.

We can check that putting \({b}={b}_\mathrm{o} +{db}\) into (21), we would obtain approximation (8) for the model (20) with the same matrices A and B as in (23), but with

$$\begin{aligned} \mathbf{w}=\mathbf{Br}_\mathrm{obs} +\mathbf{g}, \quad \text{ where}\;\mathbf{g}={b}_\mathrm{o} \cdot \mathbf{1}_{(n)} \end{aligned}$$

5.2 Similarity transformation (2D)

Let us consider a functional model

$$\begin{aligned} \begin{array}{l} {X}_{ i} =\upmu \cos \alpha \cdot x_{ i} -\upmu \sin \alpha \cdot {y}_{ i} +{a} \\ {Y}_{ i} =\upmu \sin \alpha \cdot x_{ i} +\upmu \cos \alpha \cdot {y}_{ i} +{b} \qquad i=1,\ldots ,k\\ \end{array} \end{aligned}$$
(24)

where \(k\) is the number of points involved,

or, in a matrix form,

$$\begin{aligned} \left[ {{\begin{array}{c} {\mathbf{X}_1 } \\ {\mathbf{X}_2 } \\ {\ldots } \\ {\mathbf{X}_k } \\ \end{array} }} \right]=\upmu \cdot \left[ {{\begin{array}{llll} {\mathbf{T}_{\alpha }}&\mathbf{0}&{\ldots }&\mathbf{0} \\ \mathbf{0}&{\mathbf{T}_{\alpha }}&{\ldots }&\mathbf{0} \\ {\ldots }&{\ldots }&{\ldots }&{\ldots } \\ \mathbf{0}&\mathbf{0}&{\ldots }&{\mathbf{T}_\alpha } \\ \end{array} }} \right]\left[ {{\begin{array}{c} {\mathbf{x}_\mathrm{1} } \\ {\mathbf{x}_\mathrm{2} } \\ {\ldots } \\ {\mathbf{x}_{k} } \\ \end{array} }} \right]+\left[ {{\begin{array}{c} \mathbf{a} \\ \mathbf{a} \\ {\ldots } \\ \mathbf{a} \\ \end{array} }} \right] \end{aligned}$$
(25)

where

$$\begin{aligned} \mathbf{X}_{ i} =\left[ {{\begin{array}{c} {{X}_{ i} } \\ {{Y}_{ i} } \\ \end{array} }} \right] \mathbf{x}_{ i} =\left[ {{\begin{array}{c} {{x}_{ i} } \\ {{y}_{ i} } \\ \end{array} }} \right] \mathbf{a}=\left[ {{\begin{array}{c} {a} \\ {b} \\ \end{array} }} \right] \mathbf{T}_{\alpha } =\left[ {{\begin{array}{l@{\quad }l} {\cos \alpha }&{-\sin \alpha } \\ {\sin \alpha }&{\cos \alpha } \\ \end{array} }} \right] \end{aligned}$$

With \(\mathbf{X}_{ i} ,\mathbf{x}_{ i} \,\,(i = 1,\ldots ,{k})\) being vectors of observations, thus random variables, and \(\upmu \), \(\alpha \), a, b being the unknown parameters (see Problem 3 of Neitzel 2010), the linearized form of (25), rearranged to obtain the form (8), will be as follows

$$\begin{aligned} \mathbf{A}\cdot \left[ {{\begin{array}{c} {{d}}\upmu \\ {{d}}\alpha \\ {{da}} \\ {{db}} \\ \end{array} }} \right]&+ \mathbf{B}\cdot \left[ {{\begin{array}{c} {\mathbf{v}_{x,1} } \\ {\ldots } \\ {\mathbf{v}_{x,k} } \\ {\mathbf{v}_{X,1} } \\ {\ldots } \\ {\mathbf{v}_{X,k} } \\ \end{array} }} \right]+\mathbf{B}\cdot \left[ {{\begin{array}{c} {\mathbf{x}_\mathrm{1,obs} } \\ {\ldots } \\ {\mathbf{x}_{{k},\mathrm{obs}} } \\ {\mathbf{X}_\mathrm{1,obs} } \\ {\ldots } \\ {\mathbf{X}_{{k},\mathrm{obs}} } \\ \end{array} }} \right]\nonumber \\&+ \left[{{\begin{array}{c} {\mathbf{a}_\mathrm{o}} \\ {\mathbf{a}_\mathrm{o}} \\ {\ldots } \\ {\mathbf{a}_\mathrm{o}} \\ \end{array} }} \right]=\mathbf{0},\quad \mathbf{v}-(\mathbf{0},\mathbf{C}) \end{aligned}$$
(26)

where \(\mathbf{a}_\mathrm{o} =\left[{{\begin{array}{c} {{a}_\mathrm{o} } \\ {{b}_\mathrm{o} } \\ \end{array} }} \right]\), and with

$$\begin{aligned} \mathbf{M}_{ i}&= \left[ {{\begin{array}{l@{\quad }l} {\text{ cos}\alpha \cdot {x}_{i,\mathrm {o}} -\sin \alpha \cdot {y}_{i,\mathrm {o}} }&{-\upmu \sin \alpha \cdot {x}_{i,\mathrm {o}} -\upmu \cos \alpha \cdot {y}_{i,\mathrm {o}} } \\ {{sin} \alpha \cdot {x}_{{i,\mathrm {o}}} +\cos \alpha \cdot {y}_{i,\mathrm {o}} }&{\upmu \cos \alpha \cdot {x}_{i,\mathrm {o}} -\upmu \sin \alpha \cdot {y}_{i,\mathrm {o}}} \\ \end{array} }} \right]\nonumber \\ \mathbf{A}&= \left[ {{\begin{array}{l@{\quad \;}l} \mathbf{M}_1&\mathbf{I}_{(2)} \\ \mathbf{M}_2&\mathbf{I}_{(2)} \\ \ldots&\ldots \\ {\mathbf{M}_{ k} }&{\mathbf{I}_{(2)} } \\ \end{array} }} \right]\quad \mathbf{B}=[{{\begin{array}{l@{\quad }l} {\mathbf{I}_{(k)} \otimes \upmu \mathbf{T}_{\alpha } }&{-\mathbf{I}_{(2k)} } \\ \end{array} }}] \end{aligned}$$
(27)

Using the substitution \({p}=\upmu \cos \alpha , {q}=\upmu \sin \alpha \), as in (Neitzel 2010), the functional model (24) will take the form

$$\begin{aligned} {X}_{ i}&= {p}\cdot x_{ i} -{q}\cdot {y}_{ i} +{a} \nonumber \\ {Y}_{ i}&= {q}\cdot x_{ i} +{p}\cdot {y}_{ i} +{b} \qquad i= \text{1},\ldots ,k \end{aligned}$$
(28)

Denoting A, B, and g for this model by \(\mathbf{A}_{*} , \mathbf{B}_{*} \), and \(\mathbf{g}_{*} \) respectively and omitting the derivations, we show the final results, i.e.

$$\begin{aligned}&\mathbf{A}_{*} =\left[ {{\begin{array}{l@{\quad \;}l} {\mathbf{N}_1 }&{\mathbf{I}_{(2)} } \\ {\mathbf{N}_2 }&{\mathbf{I}_{(2)} } \\ {\ldots }&{\ldots } \\ {\mathbf{N}_{ k} }&{\mathbf{I}_{(2)} } \\ \end{array} }} \right] \quad \text{ where}\;\mathbf{N}_{ i} =\left[ {{\begin{array}{l@{\quad }r} {{x}_{i,\mathrm{{o}}} }&{{-y}_{i,\mathrm{{o}}} } \\ {{y}_{i,\mathrm{{o}}} }&{{x}_{i,\mathrm{{o}}} } \\ \end{array} }} \right];\nonumber \\&\quad \mathbf{B}_{*} =\mathbf{B}; \mathbf{g}_{*} =\mathbf{g} \end{aligned}$$
(29)

Since we can prove the equality \(\mathbf{A}_{*} {d}\mathbf{u}_{*} =\mathbf{A}\cdot {d}\mathbf{u}\), we obtain the same values of the reliability indices when using \(\mathbf{A}_{*} \) instead of A. The matrix \(\mathbf{A}_{*} \), which has a simpler form, could be a better choice.

5.3 Affine transformation (3D)

Let us consider a functional model

$$\begin{aligned} {y}_{1,i}&= {a}_{11} {x}_{1,i} +{a}_{12} {x}_{2,i} +{a}_{13} {x}_{3,i} +{a}_{1}\nonumber \\ {y}_{2,i}&= {a}_{21} {x}_{1,i} +{a}_{22} {x}_{2,i} +{a}_{23} {x}_{3,i} +{a}_{2} \qquad i=1,\ldots ,k \nonumber \\ {y}_{3,i}&= {a}_{31} {x}_{1,i} +{a}_{32} {x}_{2,i} +{a}_{33} {x}_{3,i} +{a}_\mathrm{3} \end{aligned}$$
(30)

or, in a matrix form,

$$\begin{aligned} \left[ {{\begin{array}{c} {\mathbf{y}_1 } \\ {\mathbf{y}_2 } \\ {\ldots } \\ {\mathbf{y}_{ k}} \\ \end{array} }} \right]=\left[ {{\begin{array}{llll} \mathbf{G}&\mathbf{0}&{\ldots }&\mathbf{0} \\ \mathbf{0}&\mathbf{G}&{\ldots }&\mathbf{0} \\ {\ldots }&{\ldots }&{\ldots }&{\ldots } \\ \mathbf{0}&\mathbf{0}&{\ldots }&\mathbf{G} \\ \end{array} }} \right]\left[ {{\begin{array}{c} {\mathbf{x}_1 } \\ {\mathbf{x}_2 } \\ {\ldots } \\ {\mathbf{x}_{ k}} \\ \end{array} }} \right]+\left[ {{\begin{array}{c} \mathbf{a} \\ \mathbf{a} \\ {\ldots } \\ \mathbf{a} \\ \end{array} }} \right] \end{aligned}$$
(31)

where \(\mathbf{y}_{ i} (3\times 1), \mathbf{x}_{ i} (3\times 1), i = 1,{\ldots },k\) (being the number of points), \(\mathbf{G}(3\times 3), \mathbf{a}(3\times 1)\).

With \(\mathbf{x}_1 ,\ldots ,\mathbf{x}_{ k} ,\mathbf{y}_1 ,\ldots ,\mathbf{y}_{ k} \) being observations, thus random variables, and \(\text{ vec}\mathbf{G}, \mathbf{a}\) being the unknown parameters, the linearized form of (31), rearranged to obtain the form (8), will be as follows

$$\begin{aligned} \mathbf{A}\cdot \left[ {{\begin{array}{c} {{da}_{11} } \\ {{da}_{12} } \\ {\ldots } \\ {{da}_{33} } \\ {{da}_1 } \\ {{da}_2 } \\ {{da}_3 } \\ \end{array} }} \right]&+ \mathbf{B}\cdot \left[ {{\begin{array}{c} {\mathbf{v}_{x,1}} \\ {\ldots } \\ {\mathbf{v}_{{x},{ k}} } \\ {\mathbf{v}_{y,1} } \\ {\ldots } \\ {\mathbf{v}_{{y},{ k}} } \\ \end{array} }} \right]+\mathbf{B}\cdot \left[ {{\begin{array}{c} {\mathbf{x}_\mathrm{1,obs} } \\ {\ldots } \\ {\mathbf{x}_{k,\mathrm{obs}} } \\ {\mathbf{y}_\mathrm{1,obs} } \\ {\ldots } \\ {\mathbf{y}_{k,\mathrm{obs}} } \\ \end{array} }} \right]\nonumber \\&+\left[ {{\begin{array}{c} {\mathbf{a}_\mathrm{o} } \\ {\mathbf{a}_\mathrm{o} } \\ {\ldots } \\ {\mathbf{a}_\mathrm{o} } \\ \end{array} }} \right]=\mathbf{0}, \quad \mathbf{v}-(\mathbf{0},\mathbf{C}) \end{aligned}$$
(32)

where

$$\begin{aligned} \mathbf{A}&= \left[ {{\begin{array}{l@{\quad }l@{\quad }l@{\quad }l} {{x}_\mathrm{{11,o}} \mathbf{I}_{(3)} }&{{x}_\mathrm{{21,o}} \mathbf{I}_{(3)} }&{{x}_\mathrm{{31,o}} \mathbf{I}_{(3)} }&{\mathbf{I}_{(3)} } \\ {{x}_\mathrm{{12,o}} \mathbf{I}_{(3)} }&{{x}_\mathrm{{22,o}} \mathbf{I}_{(3)} }&{{x}_\mathrm{{32,o}} \mathbf{I}_{(3)} }&{\mathbf{I}_{(3)} } \\ {\ldots }&{\ldots }&{\ldots }&{\ldots } \\ {{x}_{1k,\mathrm{o}} \mathbf{I}_{(3)} }&{{x}_{2k,\mathrm{o}} \mathbf{I}_{(3)} }&{{x}_{3k,\mathrm{o}} \mathbf{I}_{(3)} }&{\mathbf{I}_{(3)} } \\ \end{array} }} \right]\nonumber \\&\mathbf{B}=\left[ {{\begin{array}{l@{\quad }l} {\mathbf{I}_{(k)} \otimes \mathbf{G}_\mathrm{o} }&{-\mathbf{I}_{(3k)} }\\ \end{array} }} \right]\\ \mathbf{G}_\mathrm{o}&= \left[ {{\begin{array}{l@{\quad }l@{\quad }l} {{a}_\mathrm{{11,o}} }&{{a}_\mathrm{{12,o}} }&{{a}_\mathrm{{13,o}} } \\ {{a}_\mathrm{{21,o}} }&{{a}_\mathrm{{22,o}} }&{{a}_\mathrm{{23,o}} } \\ {{a}_\mathrm{{31,o}} }&{{a}_\mathrm{{32,o}} }&{{a}_\mathrm{{33,o}} } \\ \end{array} }} \right]\quad \mathbf{a}_\mathrm{o} =\left[ {{\begin{array}{c} {{a}_\mathrm{{1,o}} } \\ {{a}_\mathrm{{2,o}} } \\ {{a}_\mathrm{{3,o}} } \\ \end{array} }} \right] \end{aligned}$$

6 Specific properties of quasi-linear EIV models concerning the average reliability indices

The following properties are discussed:

  1. i.

    the relationship between average reliability indices in quasi-linear EIV models versus those in GM models

  2. ii.

    the relationship between average reliability indices for dependent and independent variables in quasi-linear EIV models with homoscedastic and uncorrelated observations

  1. ad i.

    Let us compare the average reliability indices \(\bar{{h}}_{ii} \) for the EIV and GM models. Introducing an auxiliary coefficient \(\gamma =n/r\), where due to \(r>n,\) it is always \(\gamma <1\), we shall write

    $$\begin{aligned} {\bar{{h}}}_\mathrm{{avr}} \text{(EIV)}&= \frac{\text{ Tr} \mathbf{H}}{\dim \mathbf{H}}=\frac{\text{ rank} \mathbf{H}}{\dim \mathbf{H}}=\frac{n-u}{r} =\gamma (1-\frac{u}{n})\nonumber \\&=\gamma \cdot {\bar{{h}}}_\mathrm{{avr}} (\text{ GM}) \end{aligned}$$
    (33)

    and hence

    $$\begin{aligned} {\bar{{h}}}_\mathrm{{avr}} (\text{ EIV})<{\bar{{h}}}_\mathrm{{avr}} (\text{ GM}) \end{aligned}$$

    The values of the coefficient \(\gamma \) as in (33) for specific cases of quasi-linear EIV models will be as follows:

    $$\begin{aligned}&\text{ multiple} \text{ regression}\quad \gamma =\frac{n}{r}=\frac{n}{ns+n}=\frac{1}{1+s}\\&\text{ similarity} \text{ transformation} \text{(2D,} \text{3D;} d \text{=} \text{2,} \text{3)} \\&\quad \gamma =\frac{n}{r}=\frac{{d}k}{{2d}k}=\frac{1}{2}\\&\text{ affine} \text{ transformation} \text{(2D,} \text{3D;} d \text{=} \text{2,} \text{3)}\\&\gamma =\frac{n}{r}=\frac{{d}k}{{2d}k}=\frac{1}{2} \end{aligned}$$

    As shown above, the value of \(\gamma \) reaches 0.5 for similarity and affine transformation and is smaller than that for multiple regression with \(s>1\). For instance, with \(s = 4\) we have \(\gamma = 0.2\), which implies a very low level of reliability. As could be expected, in terms of the response-based reliability the EIV models are weaker than the corresponding GM models. It follows from (33) that no matter how high the redundancy level of the EIV model is, we will have \({\bar{{h}}}_\mathrm{{avr}} \text{(EIV)}<0.5\). Thus, the reliability criteria proposed for GM models (see Sect. 4) are too rigorous for EIV models, and should be weakened. The decrease in average internal reliability between the GM and EIV models that have the same number of parameters and observation equations can be explained by a specific property of EIV models. The explanation of the property can be that the independent variables being treated as observed quantities do not cause the increase in the rank of the operator H, as it is the case when adding equations for the new observed dependent variables both in GM and EIV models. Hence, in EIV models the sum of reliability indices being equal to the rank of H depends upon the number (n) of condition equations, but not on the number (r) of observed variables \((r>n)\). Therefore, in EIV models the sum of reliability indices must be shared by a greater number of observed variables than in GM models.

  2. ad ii.

    For such models the reliability matrix H as in (13) will take the form

    $$\begin{aligned} \mathbf{H}=\mathbf{B}_\mathrm{s}^\mathrm{T} \mathbf{UB}_\mathrm{s} =\sigma ^{2}\mathbf{B}^\mathrm{{T}}\mathbf{UB} \end{aligned}$$
    (34)

    where \(\sigma ^{2}\) is the common variance and U is the \((n\times n)\) central matrix. Substituting \(\mathbf{B}=[{{\begin{array}{ll} \mathbf{K}&{-\mathbf{I}_{ n} } \\ \end{array} }}]\) (see (8)) into (34) and after simple manipulations we obtain

    $$\begin{aligned} \mathbf{H}=\sigma ^{2}\left[ {{\begin{array}{l@{\quad }c} {\mathbf{K}^\mathrm{T}\mathbf{UK}}&{-\mathbf{K}^\mathrm{T}\mathbf{U}} \\ {-\mathbf{UK}}&\mathbf{U} \\ \end{array} }} \right] \end{aligned}$$
    (35)

    Denoting by \(\text{ Tr} \mathbf{H}_\mathrm{{ind}} \) and \(\text{ Tr} \mathbf{H}_\mathrm{{dep}} \) the traces for blocks of H corresponding to independent and dependent variables and by \({\bar{{h}}}_\mathrm{{avr}} (\text{ ind})\) and \({\bar{{h}}}_\mathrm{{avr}} (\text{ dep})\) the average reliability indices for independent and dependent variables, we shall introduce a coefficient \(\upeta \) defined as

    $$\begin{aligned} {\upeta }=\frac{{\bar{{h}}}_\mathrm{{avr}} (\text{ ind})}{{\bar{{h}}}_\mathrm{{avr}} (\text{ dep})}&= \frac{\text{ Tr} \mathbf{H}_\mathrm{{ind}} /({r-n)}}{\text{ Tr} \mathbf{H}_\mathrm{{dep}} /{n}}\nonumber \\&= \frac{{n}}{{r-n}}\cdot \frac{\text{ Tr} \mathbf{U}\mathbf{KK}^\mathrm{{T}}}{\text{ Tr} \mathbf{U}} \end{aligned}$$
    (36)

    For multiple regression we have r = ns + n

    $$\begin{aligned} \mathbf{KK}^\mathrm{{T}}&= (\mathbf{I}_{ n} \otimes \mathbf{a}^\mathrm{{T}})(\mathbf{I}_{ n} \otimes \mathbf{a}^\mathrm{{T}})^\mathrm{{T}}=(\mathbf{I}_\mathrm{n} \otimes \mathbf{a}^\mathrm{{T}})(\mathbf{I}_\mathrm{n} \otimes \mathbf{a})\\&=\mathbf{I}_{ n} \otimes \mathbf{a}^\mathrm{{T}}\mathbf{a}=\left\Vert \mathbf{a} \right\Vert^{2}\cdot \mathbf{I}_{ n} \end{aligned}$$

    and hence

    $$\begin{aligned} {\upeta }=\frac{{n}}{{ns}}\cdot \frac{\Vert \mathbf{a} \Vert ^\mathrm{{2}}\text{ Tr} \mathbf{U}}{\text{ Tr} \mathbf{U}}=\frac{\Vert \mathbf{a}\Vert ^\mathrm{{2}}}{{s}} \end{aligned}$$
    (37)

    For similarity transformation (2D, 3D) we have: n = dk, r = 2dk, where \(d = 2\) or 3.

    $$\begin{aligned} \mathbf{KK}^\mathrm{{T}}&= (\mathbf{I}_{k} \otimes \upmu \mathbf{T}_\alpha )(\mathbf{I}_{k} \otimes \upmu \mathbf{T}_\alpha )^\mathrm{{T}}=(\mathbf{I}_{k} \otimes \upmu \mathbf{T}_\alpha )\\&\times \,(\mathbf{I}_{k} \otimes \upmu \mathbf{T}_\alpha ^\mathrm{T} )=\mathbf{I}_\mathrm{k} \otimes \upmu ^{2}\mathbf{I}_{d} =\upmu ^{2}\cdot \mathbf{I}_{{dk}} \end{aligned}$$

    and hence

    $$\begin{aligned} \upeta =\frac{{dk}}{{dk}}\cdot \frac{\upmu ^\mathrm{{2}}\text{ Tr} \mathbf{U}}{\text{ Tr} \mathbf{U}}=\upmu ^\mathrm{{2}} \end{aligned}$$
    (38)

    For isometric transformation \((\upmu = 1)\) we get \(\upeta = 1\). For affine transformation it was not possible to reduce the formula (36) to a simple form as was done for the cases above.

7 Numerical examples of reliability analysis for EIV versus GM modelling

We will consider the models of similarity transformation and multiple regression. For each model we shall compare the reliability indices for EIV, resp. GM modelling.

Example 1

Similarity transformation

$$\begin{aligned} {X}_{i}&= \upmu \cos \alpha \cdot x_{i} -\upmu \sin \alpha \cdot {y}_{i} +{a} \\ {Y}_{i}&= \upmu \sin \alpha \cdot x_{i} +\upmu \cos \alpha \cdot {y}_{i} +{b} \qquad i=1,\ldots ,6 \end{aligned}$$

The matrices A and B will have the form as in (27) and the dimensions \((12\times 4)\) and \((12\times 24)\), respectively.

The location of the observation points is shown in Fig. 1 and Table 1.

Fig. 1
figure 1

Observation points in the old and the new coordinate system

Full size image
Table 1 Observed coordinates and approximate transformation parameters
Full size table

The other data for the response-based reliability analysis are as follows:

  • uncorrelated observations : \(\mathbf{C}_{x, \mathrm{obs}} =\mathbf{C}_{{y},\mathrm{obs}} =\mathbf{C}_{X,\mathrm {obs}} =\mathbf{C}_{Y,\mathrm {obs}} =\sigma ^{2}\cdot \mathbf{I}; \sigma =0.005\)

  • correlated observations : \(\mathbf{C}_{x,\mathrm{obs}} =\sigma ^{2}\cdot \mathbf{C}_{\mathrm{{s}},x} ; \mathbf{C}_{y,\mathrm{obs}} =\sigma ^{2}\cdot \mathbf{C}_{\mathrm{{s}},y} ; \mathbf{C}_{X,\mathrm {obs}} =\sigma ^{2}\cdot \mathbf{C}_{\mathrm{{s,}}{X}} ; \mathbf{C}_{Y,\mathrm {obs}} =\sigma ^{2}\cdot \mathbf{C}_{\mathrm{{s,}}{Y}} ; \mathbf{C}_{\mathrm{{s}},{x}} ,\mathbf{C}_{\mathrm{{s}},{y}} ; \mathbf{C}_{\mathrm{{s}},{X}} ,\mathbf{C}_{\mathrm{{s}},{Y}}\) are independently generated correlation matrices, each such that \(\left| {\left\{ {\mathbf{C}_\mathrm{s} } \right\} _{ {ij}} } \right|\le 0.5\) \((j\ne i)\). There is no correlation between the vectors \(\mathbf{x}_\mathrm{obs} ,\mathbf{y}_\mathrm{obs} , \mathbf{X}_\mathrm{obs} ,\mathbf{Y}_\mathrm{obs}\).

Figure 2 shows the effect of observational surrounding upon the model’s reliability. The highest level of controllability between the observations (and hence the highest reliability index) is shown for the central point No. 6, whereas the second in turn is point No. 5, being closer to the gravity centre of the group than any of the remaining points Nos. 1 to 4. The value of the coefficient \(\gamma \) is 0.5 (see Fig. 2).

Fig. 2
figure 2

Analysis results for uncorrelated observations—similarity transformation

Full size image

For uncorrelated observations, all the reliability indices for the GM model satisfy the criteria \(({\hat{h}}_{ii} > 0.5)\), whereas those for the EIV model do not. This confirms the need for specifying a separate acceptance area, being an extension of the acceptance area for the GM model. Correlation slightly changes the situation, as several reliability indices for the GM model fall outside the acceptance region (i.e. shaded area in Fig. 3). Careful study of the reliability indices listed in Table 2 may be helpful in improving the adjustment model.

Fig. 3
figure 3

Analysis results for correlated observations—similarity transformation

Full size image
Table 2 Reliability indices for GM and EIV modelling—similarity transformation
Full size table

The coefficient \(\upeta \) as defined in (36), is \({\upeta }={\upmu }^{2}=1.1^{2}=1.21\)

We can check that \(\upeta =\frac{{\bar{{h}}}_\mathrm{{avr}} (\text{ ind})}{{\bar{{h}}}_\mathrm{{avr}} (\text{ dep})}=\frac{0.365}{0.302}=1.21\)

Since \(\upeta > 1\) the average reliability index for independent variables (i.e. coordinates in the old system) is greater than that for dependent variables (i.e. coordinates in the new system). We can see it in Fig. 2, where the line EIV(x) \(\equiv \) EIV(y) runs above the line EIV(X) \(\equiv \) EIV(Y). The separation between both lines is not great, since the scale coefficient \({\upmu }\) does not differ much from 1.

Example 2

Multiple regression

We shall consider the model (19) where s = 4 and n = 8. The following variants will be analyzed:

  • C = \(\mathbf{C}_{\mathrm{s}} =\mathbf{I}\) and \(\mathbf{C}_{\mathrm{s}}\ne \mathbf{I}\), where \(| {\{{\mathbf{C}}_{s} \}_{ij} }|\le 0.5, j\ne i\)

  • \(\mathbf{a}_{\mathrm{o}}^{\mathrm{T}} (1)=[{{\begin{array}{l@{\quad }l@{\quad }l@{\quad }l} 2&{-3}&1&4 \\ \end{array} }}]\) and \(\mathbf{a}_{\mathrm{o}}^{\mathrm{T}} (2) =[{{\begin{array}{l@{\quad }l@{\quad }l@{\quad }l} {-0.43}&{-0.20}&{0.59}&{-0.49} \end{array}}}]\)

To save space in this article, the analysis results will be presented in graphical form only, i.e. for the variant \(\mathbf{a}_\mathrm{o} (1)\)—in Figs. 4 and 5, and for \(\mathbf{a}_\mathrm{o} (2)\)—in Figs. 6 and 7. In each case the two variants of the correlation matrix will be taken into consideration.

The coefficient \(\gamma \) as defined in (33), is common for all the variants and takes the value 0.20. We can check that \(\gamma =\frac{{\bar{{h}}}_\mathrm{{avr}} (\text{ EIV})}{{\bar{{h}}}_\mathrm{{avr}} (\text{ GM})}=\frac{0.075}{0.375}=0.20\).

Fig. 4
figure 4

Analysis results for correlated and uncorrelated observations—multiple regression; \(\mathbf{a}_\mathrm{o} (1)\); symbols “avr” in the indices \(\bar{{h}}(\cdot )\) are deliberately omitted

Full size image
Fig. 5
figure 5

Analysis results for correlated observations—multiple regression; \(\mathbf{a}_\mathrm{o} (1)\)

Full size image
Fig. 6
figure 6

Analysis results for correlated and uncorrelated observations—multiple regression; \(\mathbf{a}_\mathrm{o} (2)\); symbols “avr” in the indices \(\bar{{h}}(\cdot )\) are deliberately omitted

Full size image
Fig. 7
figure 7

Analysis results for correlated observations—multiple regression; \(\mathbf{a}_o (2)\)

Full size image

The coefficient \(\upeta \) as defined in (36) and denoted by \(\upeta \)(1), is

$$\begin{aligned} \upeta (1)=\frac{ \left\Vert \mathbf{a}_{\mathrm{o}} (1) \right\Vert_{2}^{2}}{\mathrm{s}}=\frac{30.0}{4}=7.5 \end{aligned}$$

We can check that \(\upeta (1)=\frac{{\bar{{h}}}_\mathrm{{avr}} (\text{ ind})}{{\bar{{h}}}_\mathrm{{avr}} (\text{ dep})}=\frac{0.0907}{0.0121}=7.5\)

The coefficient \(\upeta \) denoted here by \(\upeta \)(2), is \(\upeta (2)=\frac{\Vert \mathbf{a}_\mathrm{o} (2)\Vert _{2}^{2} }{\mathrm{s}}=\frac{0.81}{4}=0.20\).

We can check that \(\upeta (2)=\frac{{\bar{{h}}}_\mathrm{{avr}} (\text{ ind})}{{\bar{{h}}}_\mathrm{{avr}} (\text{ dep})}=\frac{0.0420}{0.207}=0.20\).

The analysis shows (Figs. 5, 7) that the investigated GM model, except for one observation, does not satisfy the reliability criteria (\({h}_\mathrm{{ii}} >0.5)\). According to the theory we have \(\gamma =\frac{\bar{{h}}_{avr} (EIV)}{\bar{{h}}_{avr} (GM)}=\frac{0.075}{0.375}=0.2\), which means that the average value of the reliability index being in GM model equal to 0.375, drops down in the EIV model to 0.075. We observe significant differences in the values of the reliability indices \({h}_\mathrm{{ii}} \) both for uncorrelated and the correlated observations. Some values reach 0.03 or even 0.01.

For the EIV model with uncorrelated observations, in the variant \(\mathbf{a}_\mathrm{o} (1)\) (see Fig. 4) all the y-observations and in the variant \(\mathbf{a}_\mathrm{o} (2)\) (see Fig. 6) most of the x-observations are practically uncontrolled by the other observations in the model, and hence, potential gross errors residing in them are practically undetectable. This example of multiple regression confirms the theory that the distribution of the response-based reliability indices between the independent and dependent variables is dependent on the norm of the vector of regression coefficients (a).

In the case a \(_\mathrm{o}\)(1), the coefficient \({\upeta }\) is much greater than 1 and the independent variables x display better average reliability than the dependent variables y. For the case a \(_\mathrm{o}\)(2), where \({\upeta }\) is much smaller than 1, we have the opposite relation, i.e. the dependent variables \(y\) show better average reliability than the independent variables \(x\).

8 Conclusions

The response-based reliability of EIV models can be analyzed in an analogous way as for the corresponding GM models. The theoretical derivations showed that in terms of average reliability indices EIV models are at least two times weaker than the GM models. This can be simply explained by the fact that the coefficients are treated as error-free (deterministic) quantities in GM models, whereas they are considered as random variables in the EIV models. This confirms that the EIV models are subject to a greater number of sources of observation errors than GM models, which results in the lower level of their response-based reliability. Therefore, the reliability criteria for EIV models should be set at a lower level than for GM models. Such criteria are not proposed in this paper and require separate research.

Taking into account the empirically confirmed connection between the level of reliability indices and effectiveness of outlier detection in GM models, we have grounds to conclude that the relatively low response-based reliability of EIV models may indicate lower effectiveness of outlier detection than in GM models.

The a priori reliability analysis proposed within this paper is only one particular aspect of EIV models. Other aspects, obviously of greater importance when considering a full scope of practical problems, include numerical algorithms for parameter estimation and the associated outlier detection procedures (see e.g., Schaffrin 2011). It seems, however, that the revealed reliability properties of EIV models can be helpful in constructing the outlier detection procedures. For doing so, the research findings of geodesists in the area of hypotheses testing (eg. Teunissen 1996) can be a valuable theoretical basis. On the grounds of this theory, one might also undertake the task of deriving a generalized formula for minimal detectable biases (MDBs) of observed quantities in EIV models. The testing-based approach to reliability measures (Schaffrin 1997; Knight et al. 2010) might be helpful in carrying out that task.

The equality \(r=n\) as a specific case of EIV models being equivalent to GM models, has been proposed in this paper only for the needs of the response-based reliability analysis. Therefore, it does not have a general character. At any rate, it is commonly known that both EIV and GM models can be treated by the classical method of least-squares adjustment.

A more forward-looking approach to reliability analysis, however, has already been undertaken by Schaffrin and Uzun (2011) who applied the TLS-techniques within EIV models. It would be interesting to see any correspondence to the approach presented. However, despite differences in the assumptions, both the approaches are important to the development of geodetic technologies, as they are extending the methods of reliability analyses upon the observation systems that fall into the class of EIV models.