Asymptotic Distributions

Abstract

This chapter focuses on the asymptotic distribution of the parameter estimates for the case when the number of measured data grows without bound. In general terms, the parameter estimates are then asymptotically Gaussian distributed. It is shown for all the main methods treated in the book, what the covariance matrix of that Gaussian distribution looks like. It thus gives a measure of the accuracy of the obtained parameter estimates. The chapter includes also some results on Cramér–Rao lower bounds (CRLB) on this covariance matrix. Explicit algorithms to compute the covariance matrix for different estimation methods and for the CRLB are given.

Appendices

Appendix

14.A Asymptotic Distribution of CFA Estimates

14.1.1 14.A.1 Proof of Lemma 14.1

First note that ${\mathbf C}_r$ can be written as a block matrix, where an arbitrary partition ($\mu , \nu = 1, \dots , n_{\varvec{\varphi }}$) can be written as follows

$$\begin{aligned} \left( {\mathbf C}_r \right) _{\mu , \nu }= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \left[ \frac{1}{N} \sum _{t=1}^N \varvec{\varphi }(t) \varvec{\varphi }_{\mu } (t) - {\mathbf R}{\mathbf e}_{\mu } \right] \left[ \frac{1}{N} \sum _{s=1}^N \varvec{\varphi }^T(s) \varvec{\varphi }_{\nu } (s) - {\mathbf e}^T_{\nu } {\mathbf R}\right] \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \left[ r_{\varvec{\varphi }_{\mu } \varvec{\varphi }_{\nu } } (t-s) {\mathbf R}_{\varvec{\varphi }} (t-s) + r_{\varvec{\varphi }\varvec{\varphi }_{\nu } } (t-s) {\mathbf r}_{\varvec{\varphi }_{\mu } \varvec{\varphi }} (t-s) \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{\tau =-N}^N \left( N - | \tau | \right) \left[ r_{\varvec{\varphi }_{\mu } \varvec{\varphi }_{\nu } } (\tau ) {\mathbf R}_{\varvec{\varphi }} (\tau ) + r_{\varvec{\varphi }\varvec{\varphi }_{\nu } } (\tau ) {\mathbf r}_{\varvec{\varphi }_{\mu } \varvec{\varphi }} ^T (\tau ) \right] \nonumber \\= & {} \sum _{\tau =-\infty }^{\infty } \left[ r_{\varvec{\varphi }_{\mu } \varvec{\varphi }_{\nu } } (\tau ) {\mathbf R}_{\varvec{\varphi }} (\tau ) + r_{\varvec{\varphi }\varvec{\varphi }_{\nu } } (\tau ) {\mathbf r}_{\varvec{\varphi }_{\mu } \varvec{\varphi }} (\tau ) \right] \nonumber \\= & {} \sum _{\tau =-\infty }^{\infty } \left[ r_{ \varvec{\varphi }_{\mu } \varvec{\varphi }_{\nu } } (\tau ) {\mathbf R}_{\varvec{\varphi }} (\tau ) + {\mathbf R}_{ \varvec{\varphi }} (\tau ) {\mathbf e}_{\nu } {\mathbf e}^T _{\mu } {\mathbf R}_{ \varvec{\varphi }} (\tau ) \right] \; . \end{aligned}$$

(14.311)

This means that the total matrix ${\mathbf C}_r$ can be written as

$$\begin{aligned} {\mathbf C}_r&= \sum _{\tau =-\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }} (\tau ) \otimes {\mathbf R}_{\varvec{\varphi }} (\tau ) + \left( {\mathbf I}\otimes {\mathbf R}_{\varvec{\varphi }} (\tau ) \right) \left( \begin{array}{ccc} {\mathbf e}_1 {\mathbf e}_1^T &{} \dots &{} {\mathbf e}_{n_{\varvec{\varphi }} } {\mathbf e}_1^T \\ \vdots \\ {\mathbf e}_1 {\mathbf e}_{n_{\varvec{\varphi }} }^T &{} \dots &{} {\mathbf e}_{n_{\varvec{\varphi }} } {\mathbf e}_{n_{\varvec{\varphi }} }^T \end{array}\right) \left( {\mathbf I}\otimes {\mathbf R}_{\varvec{\varphi }} (\tau ) \right) \right] \nonumber \\&{\mathop {=}\limits ^{\varDelta }} \sum _{\tau =-\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }} (\tau ) \otimes {\mathbf R}_{\varvec{\varphi }} (\tau ) + \left( {\mathbf I}\otimes {\mathbf R}_{\varvec{\varphi }} (\tau ) \right) {\mathbf K}\left( {\mathbf I}\otimes {\mathbf R}_{\varvec{\varphi }} (\tau ) \right) \right] \; , \end{aligned}$$

(14.312)

which is (14.53).

14.1.2 14.A.2 Evaluation of ${\mathbf R}_{\varvec{\varphi }}(\tau )$

To proceed it is relevant to evaluate the covariance function ${\mathbf R}_{\varvec{\varphi }}(\tau )$. Here it is done when CFA is applied to dynamic models as in Sect. 8.5. Then it is useful to write

$$\begin{aligned} \varvec{\varphi }(t)= & {} \varvec{\varphi }_0(t) + \tilde{\varvec{\varphi }}(t) \nonumber \\= & {} \varvec{\varGamma }{\mathbf z}(t) + \tilde{\varvec{\varphi }}(t) \nonumber \\= & {} \varvec{\varGamma }\left( \begin{array}{c} z(t) \\ \vdots \\ z(t-k) \end{array}\right) + \left( \begin{array}{c} \tilde{y}(t) \\ \vdots \\ \tilde{y}(t-n_a-p_y) \\ \tilde{u}(t-1) \\ \vdots \\ \tilde{u}(t-n_b-p_u) \end{array}\right) \; . \end{aligned}$$

(14.313)

The two terms in (14.313) are uncorrelated.

Next, introduce the notation ${\mathbf J}_m$ as the ‘shift matrix’ of dimension $m \times m$,

$$\begin{aligned} {\mathbf J}_m = \left( \begin{array}{cccc} 0 &{} 1 &{} 0 &{} \dots \\ 0 &{} 0 &{} 1 &{} {\mathbf 0}\\ \vdots \\ {\mathbf 0}&{} &{} &{} 1 \\ {\mathbf 0}&{} &{} &{} 0 \end{array}\right) \; . \end{aligned}$$

(14.314)

With some abuse of notation, set also

$$\begin{aligned} {\mathbf J}_m^{-1} = {\mathbf J}_m^T, \ \ \ {\mathbf J}_m^{-k} = \left( {\mathbf J}_m^T \right) ^k \; . \end{aligned}$$

(14.315)

One can then write

$$\begin{aligned} {\mathbf R}_{\tilde{\varvec{\varphi }}}(\tau ) = \left( \begin{array}{cc} \lambda _y {\mathbf J}_{n_a+p_y+1}^{\tau } &{} {\mathbf 0}\\ {\mathbf 0}&{} \lambda _u {\mathbf J}_{n_b+p_u}^{ \tau } \end{array}\right) \; . \end{aligned}$$

(14.316)

Note that

• Due to the convention (14.315) the result (14.316) holds for all (positive as well as negative) values of $\tau$.

• The expression in (14.316) will in fact be zero, if

  $$\begin{aligned} | \tau |> n_a + p_y +1 \ \mathrm{and \ } | \tau | > n_b + p_u \; , \end{aligned}$$

  (14.317)

  that is, as soon as

  $$\begin{aligned} | \tau | > \min ( n_a + p_y +1 , n_b + p_u ) {\mathop {=}\limits ^{\varDelta }} \tau _0 \; . \end{aligned}$$

  (14.318)

Therefore

$$\begin{aligned} {\mathbf R}_{\varvec{\varphi }}(\tau ) = \varvec{\varGamma }{\mathbf R}_{{\mathbf z}}(\tau ) \varvec{\varGamma }^T + {\mathbf R}_{\tilde{\varvec{\varphi }}}(\tau ) \; . \end{aligned}$$

(14.319)

It also holds

$$\begin{aligned} {\mathbf R}_{{\mathbf z}}(\tau )= & {} \sum _{i=1}^{k+1} \sum _{j=1}^{k+1} {\mathbf e}_i {\mathbf e}_j^T \left( {\mathbf R}_{{\mathbf z}} (\tau ) \right) _{ij} \nonumber \\= & {} \sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T \mathsf{E} \left\{ z(t-i+1 +\tau ) z(t-j+1) \right\} \nonumber \\= & {} \sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \; . \end{aligned}$$

(14.320)

Using (14.311), (14.318), and (14.320) one can now write

$$\begin{aligned}\left( {\mathbf C}_r \right) _{\mu \nu }= & {} \sum _{\tau = - \infty }^{\infty } \left[ {\mathbf e}_{\mu }^T {\mathbf R}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } {\mathbf R}_{\varvec{\varphi }}(\tau ) + {\mathbf R}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } {\mathbf e}_{\mu }^T {\mathbf R}_{\varvec{\varphi }}(\tau ) \right] \nonumber \\= & {} \sum _{\tau = - \tau _0}^{\tau _0} \left[ {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) + \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) \right] \nonumber \\&+ \sum _{\tau = - \tau _0}^{\tau _0} \left[ {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T \right. \nonumber \\&+ {\mathbf e}_{\mu }^T \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T {\mathbf e}_{\nu } \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) \nonumber \\&+ \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } {\mathbf e}_{\mu }^T \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T \nonumber \\&\left. + \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T {\mathbf e}_{\nu } {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) \right] \nonumber \\&+ \sum _{\tau = - \infty }^{\infty } \left[ {\mathbf e}_{\mu }^T \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T {\mathbf e}_{\nu } \varvec{\varGamma }\sum _k \sum _{\ell } {\mathbf e}_k {\mathbf e}_{\ell }^T r_z(\tau -k+\ell ) \varvec{\varGamma }^T \right. \nonumber \\&\left. + \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T {\mathbf e}_{\nu } {\mathbf e}_{\mu }^T \varvec{\varGamma }\sum _k \sum _{\ell } {\mathbf e}_k {\mathbf e}_{\ell }^T r_z(\tau -k+\ell ) \varvec{\varGamma }^T \right] \nonumber \\= & {} \sum _{\tau = - \tau _0}^{\tau _0} \left[ {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) + \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) \right] \nonumber \\&+ \sum _{\tau = - \tau _0}^{\tau _0} \left[ {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T \right. \nonumber \\&+ \sum _i \sum _j \left( \varvec{\varGamma }_{\mu i} \varvec{\varGamma }_{\nu j} r_z(\tau -i+j) \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) + \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) {\mathbf e}_{\nu } \varvec{\varGamma }_{\mu i} {\mathbf e}_j^T r_z(\tau -i+j) \varvec{\varGamma }^T \right) \nonumber \\&\left. + \varvec{\varGamma }\sum _i \sum _j {\mathbf e}_i \varvec{\varGamma }_{\nu j} r_z(\tau -i+j) {\mathbf e}_{\mu }^T \tilde{{\mathbf R}}_{\varvec{\varphi }}(\tau ) \right] \nonumber \end{aligned}$$

$$\begin{aligned}&+ \sum _i \sum _j \varvec{\varGamma }_{\mu i} \varvec{\varGamma }_{\nu j} \varvec{\varGamma }\sum _k \sum _{\ell } {\mathbf e}_k {\mathbf e}_{\ell }^T \varvec{\varGamma }^T \sum _{\tau = - \infty }^{\infty } r_z(\tau -i+j) r_z(\tau -k+\ell ) \nonumber \\&+ \sum _i \sum _j \sum _k \sum _{\ell } \varvec{\varGamma }{\mathbf e}_i \varvec{\varGamma }_{\nu j} \varvec{\varGamma }_{\mu k} {\mathbf e}_{\ell }^T \varvec{\varGamma }^T \sum _{\tau = - \infty }^{\infty } r_z(\tau -i+j) r_z(\tau -k+\ell ) \; . \end{aligned}$$

(14.321)

To proceed it would be convenient to have an algorithm for computing sums of the form

$$\begin{aligned} \alpha _i {\mathop { =}\limits ^{\varDelta }} \sum _{\tau = -\infty }^{\infty } r_z(\tau +i) r_z(\tau ) \end{aligned}$$

(14.322)

for an arbitrary (fixed) integer i.

This can be done using Lemma A.10. Assume that the noise-free input is an ARMA process

$$\begin{aligned} u_0(t) = \lambda _v \frac{C(q^{-1})}{D(q^{-1})} v(t) \; , \end{aligned}$$

(14.323)

where v(t) is white noise of unit variance. Then it holds

$$\begin{aligned} z(t) = \frac{1}{A(q^{-1})} u_0(t) = \lambda _v \frac{C(q^{-1})}{A(q^{-1})D(q^{-1})} v(t) \; , \end{aligned}$$

(14.324)

and it follows that

$$\begin{aligned} \alpha _i = \mathsf{E} \left\{ s(t+i) s(t) \right\} = r_s(i) \; , \end{aligned}$$

(14.325)

where

$$\begin{aligned} s(t) = \lambda _v^2 \frac{ C^2(q^{-1})}{A^2(q^{-1})D^2(q^{-1})} v(t) \; . \end{aligned}$$

(14.326)

14.B Asymptotic Distribution for IV Estimates

14.1.1 14.B.1 Proof of Lemma 14.2

Using the assumption of joint Gaussian distribution one can apply the general rule for product of Gaussian variables, see Lemma A.9,

$$\begin{aligned} \mathsf{E} \left\{ x_1 x_2 x_3 x_4 \right\} = \mathsf{E} \left\{ x_1 x_2 \right\} \mathsf{E} \left\{ x_3 x_4 \right\} + \mathsf{E} \left\{ x_1 x_3 \right\} \mathsf{E} \left\{ x_2 x_4 \right\} + \mathsf{E} \left\{ x_1 x_4 \right\} \mathsf{E} \left\{ x_2 x_3 \right\} \,\, . \end{aligned}$$

(14.327)

Using the result (14.327) in (14.68) leads to

$$\begin{aligned} {\mathbf C}= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{t=1}^N \sum _{s=1}^N \left\{ \left[ \mathsf{E} \left\{ {\mathbf z}(t) {\mathbf z}^T(s) \right\} \right] \left[ \mathsf{E} \left\{ v(t) v(s) \right\} \right] \right. \nonumber \\&+ \left. \left[ \mathsf{E} \left\{ {\mathbf z}(t) v(s) \right\} \right] \left[ \mathsf{E} \left\{ {\mathbf z}^T(s) v(t) \right\} \right] \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } \sum _{\tau = - N } ^N \left( 1 - \frac{|\tau |}{N} \right) \left\{ {\mathbf R}_{{\mathbf z}}(\tau ) r_v(\tau ) + {\mathbf r}_{{\mathbf z}v} (\tau ) {\mathbf r}_{{\mathbf z}v}^T(-\tau ) \right\} .\; \end{aligned}$$

(14.328)

The assumption (14.70) implies that

$$\begin{aligned} {\mathbf r}_{{\mathbf z}v}(\tau ) {\mathbf r}_{{\mathbf z}v}^T(-\tau ) = {\mathbf 0}\ \ \ \forall \ \tau \end{aligned}$$

(14.329)

as at least one of the factors is zero.

Now use the conventions

$$\begin{aligned} h_0 = 1, \ \ \ h_i = 0 \ \ \mathrm{\ for \ } i < 0 \; . \end{aligned}$$

(14.330)

Recall that the covariance function $r_v(\tau )$ decays exponentially with $\tau$. Therefore it holds

$$\begin{aligned} \parallel \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{\tau =-N}^N | \tau | {\mathbf R}_{{\mathbf z}}(\tau ) r_v(\tau ) \parallel \le \lim _{N \rightarrow \infty } \frac{2}{N} \sum _{\tau =0}^N \tau C \alpha ^{\tau } = 0 \end{aligned}$$

(14.331)

for some $| \alpha | < 1$. Using this result, one gets from (14.328)

$$\begin{aligned} {\mathbf C}= & {} \sum _{\tau = -\infty }^{\infty } {\mathbf R}_{{\mathbf z}}(\tau ) r_v(\tau ) = \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{{\mathbf z}}(\tau ) \lambda \sum _{i= 0}^{\infty } h_i h_{i+\tau } \right] \nonumber \\= & {} \lambda \sum _{\tau = -\infty }^{\infty } \sum _{i= -\infty }^{\infty } h_i h_{i+\tau } \mathsf{E} \left\{ {\mathbf z}(t+\tau ) {\mathbf z}^T(t) \right\} \nonumber \\= & {} \lambda \sum _{\tau = -\infty }^{\infty } \sum _{i= -\infty }^{\infty } h_i h_{i+\tau } \mathsf{E} \left\{ {\mathbf z}(t-i) {\mathbf z}^T(t-i-\tau ) \right\} \nonumber \\= & {} \lambda \mathsf{E} \left\{ \sum _{i= -\infty }^{\infty } h_i {\mathbf z}(t-i) \sum _{\tau = -\infty }^{\infty } h_{i+\tau } {\mathbf z}^T(t-i-\tau ) \right\} \nonumber \\= & {} \lambda \mathsf{E} \left\{ \left( \sum _{i= -\infty }^{\infty } h_i {\mathbf z}(t-i) \right) \left( \sum _{k = -\infty }^{\infty } h_{k} {\mathbf z}^T(t-k) \right) \right\} \nonumber \\= & {} \lambda \mathsf{E} \left\{ \left[ H(q^{-1}) {\mathbf z}(t) \right] \left[ H(q^{-1}) {\mathbf z}(t) \right] ^T \right\} \; , \end{aligned}$$

(14.332)

which is (14.72).

14.1.2 14.B.2 Proof of Lemma 14.6

Use the definition (14.77) of K(z) and introduce the notations

$$\begin{aligned} \varvec{\alpha }(t)= & {} {\mathbf R}^T {\mathbf W}\sum _{i=0} ^{\infty } K_i {\mathbf z}(t+i) \; , \end{aligned}$$

(14.333)

$$\begin{aligned} \varvec{\beta }(t)= & {} H^{-1} (q^{-1}) \varvec{\varphi }_0(t) \; . \end{aligned}$$

(14.334)

Then it holds

$$\begin{aligned} {\mathbf R}^T {\mathbf W}{\mathbf R}= & {} {\mathbf R}^T {\mathbf W}\mathsf{E} \left\{ {\mathbf z}(t) F(q^{-1}) \varvec{\varphi }_0^T(t) \right\} \nonumber \\= & {} {\mathbf R}^T {\mathbf W}\mathsf{E} \left\{ {\mathbf z}(t) K(q^{-1}) H^{-1}(q^{-1}) \varvec{\varphi }_0^T(t) \right\} \nonumber \\= & {} {\mathbf R}^T {\mathbf W}\mathsf{E} \left\{ {\mathbf z}(t) \sum _{i=0}^{\infty } K_i H^{-1}(q^{-1}) \varvec{\varphi }_0^T(t-i) \right\} \nonumber \\= & {} \mathsf{E} \left\{ \varvec{\alpha }(t) \varvec{\beta }^T (t) \right\} \; . \end{aligned}$$

(14.335)

Using (14.78)

leads to

$$\begin{aligned} \lambda {\mathbf R}^T {\mathbf W}{\mathbf C}{\mathbf W}{\mathbf R}= \mathsf{E} \left\{ \varvec{\alpha }(t) \varvec{\alpha }^T (t) \right\} \; . \end{aligned}$$

(14.336)

The stated inequality (14.81) then reads

$$\begin{aligned} {\mathbf P}_\mathrm{IV}= & {} \lambda \left( \mathsf{E} \left\{ \varvec{\alpha }(t) \varvec{\beta }^T(t) \right\} \right) ^{-1} \left( \mathsf{E} \left\{ \varvec{\alpha }(t) \varvec{\alpha }^T(t) \right\} \right) \left( \mathsf{E} \left\{ \varvec{\beta }(t) \varvec{\alpha }^T(t) \right\} \right) ^{-1} \nonumber \\\ge & {} \lambda \left( \mathsf{E} \left\{ \varvec{\beta }(t) \varvec{\beta }^T(t) \right\} \right) ^{-1} \; . \end{aligned}$$

(14.337)

Now, (14.337) is equivalent to

$$\begin{aligned} \left( \mathsf{E} \left\{ \varvec{\beta }(t) \varvec{\alpha }^T(t) \right\} \right) \left( \mathsf{E} \left\{ \varvec{\alpha }(t) \varvec{\alpha }^T(t) \right\} \right) ^{-1} \left( \mathsf{E} \left\{ \varvec{\alpha }(t) \varvec{\beta }^T(t) \right\} \right) \le \left( \mathsf{E} \left\{ \varvec{\beta }(t) \varvec{\beta }^T(t) \right\} \right) \; , \end{aligned}$$

(14.338)

which follows from the theory of partitioned matrices, cf Lemma A.4 in Söderström and Stoica (1989), as

$$\begin{aligned} \mathsf{E} \left\{ \left( \begin{array}{cc} \varvec{\beta }(t) \varvec{\beta }^T(t) &{} \varvec{\beta }(t) \varvec{\alpha }^T(t) \\ \varvec{\alpha }(t) \varvec{\beta }^T(y) &{} \varvec{\alpha }(t) \varvec{\alpha }^T(t) \end{array}\right) \right\} = \mathsf{E} \left\{ \left( \begin{array}{c} \varvec{\beta }(t) \\ \varvec{\alpha }(t) \end{array}\right) \left( \begin{array}{cc} \varvec{\beta }^T(t)&\varvec{\alpha }^T (t) \end{array}\right) \right\} \ge {\mathbf 0}\; . \end{aligned}$$

(14.339)

Further, for the specific choice

$$\begin{aligned} {\mathbf z}(t) = H^{-1}(q^{-1}) \varvec{\varphi }_0(t), \ \ \ F(q^{-1}) = H^{-1}(q^{-1}) \; , \end{aligned}$$

(14.340)

(${\mathbf W}$ has then no influence) it holds that

$$\begin{aligned} {\mathbf R}= \mathsf{E} \left\{ [ H^{-1}(q^{-1}) \varvec{\varphi }_0(t) ] [ H^{-1}(q^{-1}) \varvec{\varphi }_0^T(t) ] \right\} = \mathsf{E} \left\{ \varvec{\beta }(t) \varvec{\beta }^T(t) \right\} = {\mathbf C}\; , \end{aligned}$$

(14.341)

from which equality in (14.81) follows.

14.C Asymptotic Distribution for GIVE

14.1.1 14.C.1 The Sensitivity Matrix ${\mathbf S}$ for the SISO Case

In this section the sensitivity matrix ${\mathbf S}$ will be analyzed in more detail. In addition, expressions for the covariance function $r_{\varepsilon } (\tau )$ and the cross-covariance functions ${\mathbf r}_{{\mathbf z}\varepsilon } (\tau )$ are provided.

Use generally the conventions

$$\begin{aligned} \begin{array}{rcl} a_i &{} = &{} 0, \ \mathrm{if\ } i< 0 \ \mathrm{or \ } i> n_a \; , \\ a_0 &{} = &{} 1 \; , \\ b_i &{} = &{} 0, \ \mathrm{if\ } i < 1 \ \mathrm{or \ } i > n_b \; . \end{array} \end{aligned}$$

(14.342)

The two cases of white and correlated output noises are treated in the next two subsections, respectively.

14.1.1.1 14.C.1.1 Expressions for White Output Noise

Using the conventions (14.342), the auto-covariance function of the residuals is easily found to be

$$\begin{aligned} r_{\varepsilon } (\tau ) = \lambda _y \sum _i a_i a_{i + \tau } + \lambda _u \sum _i b_i b_{i + \tau } \; . \end{aligned}$$

(14.343)

Similarly, the covariance vector ${\mathbf r}_{{\mathbf z}\varepsilon }(\tau ) = \mathsf{E} \left\{ {\mathbf z}(t+\tau ) \varepsilon (t) \right\}$ is calculated as follows. As the generalized IV vector is given by (7.97), it holds

$$\begin{aligned} {\mathbf r}_{{\mathbf z}\varepsilon }(\tau )= & {} \mathsf{E} \left\{ \left( \begin{array}{c} y(t) \\ \vdots \\ y(t-n_a-p_y) \\ u(t-1) \\ \vdots \\ u(t-n_b -p_u) \end{array}\right) \right. \nonumber \\&\times \left. \left[ \sum _i a_i \tilde{y}(t-\tau - i) - \sum _i b_i \tilde{u}(t- \tau - i) \right] \right\} \nonumber \\= & {} \left( \begin{array}{c} \lambda _y \left( \begin{array}{c} a_{-\tau } \\ \vdots \\ a_{n_a+p_y-\tau } \end{array}\right) \\ - \lambda _u \left( \begin{array}{c} b_{1-\tau } \\ \vdots \\ b_{n_b+p_u-\tau } \end{array}\right) \end{array}\right) \; . \end{aligned}$$

(14.344)

The general expression of the matrix ${\mathbf S}$, (14.90), reads

$$\begin{aligned} {\mathbf S}= - \left( \begin{array}{cc} {\mathbf R}_{{\mathbf z}\varvec{\varphi }} + {\mathbf r}_{\varvec{\theta }}&{\mathbf r}_{\varvec{\rho }} \end{array}\right) \; . \end{aligned}$$

(14.345)

When the output noise is white, the noise parameter vector is

$$\begin{aligned} \varvec{\rho }= \left( \begin{array}{c} \lambda _y \\ \lambda _u \end{array}\right) \; , \end{aligned}$$

(14.346)

and using (14.344) as well as $\bar{{\mathbf a}} = \left( 1 \ a_1 \ \dots \ a_{n_a} \right) ^T, \ {\mathbf b}= \left( b_1 \ \dots \ b_{n_b} \right) ^T$

$$\begin{aligned} {\mathbf r}(\varvec{\theta }, \varvec{\rho })= & {} {\mathbf r}_{{\mathbf z}\varepsilon }(0) = \left( \begin{array}{c} \lambda _y {\bar{\mathbf {a}} } \\ {\mathbf 0}_{p_y \times 1} \\ - \lambda _u {\mathbf b}\\ {\mathbf 0}_{p_u \times 1} \end{array}\right) \; . \end{aligned}$$

(14.347)

It follows from (14.347) that

$$\begin{aligned} {\mathbf r}_{\varvec{\theta }} = \frac{ \partial {\mathbf r}_{{\mathbf z}\varepsilon } }{ \partial \varvec{\theta }} = \left( \begin{array}{c} {\mathbf 0}_{1 \times (n_a + n_b) } \\ \begin{array}{cc} \lambda _y {\mathbf I}_{n_a} &{} {\mathbf 0}_{n_a \times n_b } \end{array}\\ {\mathbf 0}_{p_y \times (n_a + n_b) } \\ \begin{array}{cc} {\mathbf 0}_{n_b \times n_a} &{} - \lambda _u {\mathbf I}_{n_b } \end{array}\\ {\mathbf 0}_{p_u \times (n_a + n_b) } \end{array}\right) \; , \end{aligned}$$

(14.348)

and

$$\begin{aligned} {\mathbf r}_{\varvec{\rho }} = \frac{ \partial {\mathbf r}_{{\mathbf z}\varepsilon } }{ \partial \varvec{\rho }} = \left( \begin{array}{c} \begin{array}{cc} {\bar{\mathbf {a}} } &{} {\mathbf 0}_{ (n_a+1) \times 1 } \end{array}\\ {\mathbf 0}_{p_y \times 2} \\ \begin{array}{cc} {\mathbf 0}_{ n_b \times 1 } &{} -{\mathbf b}\end{array}\\ {\mathbf 0}_{p_u \times 2} \end{array}\right) \; . \end{aligned}$$

(14.349)

A closer evaluation of the first part of ${\mathbf S}$ gives

$$\begin{aligned} {\mathbf R}_{{\mathbf z}\varvec{\varphi }} + {\mathbf r}_{\varvec{\theta }} = {\mathbf R}_{{\mathbf z}_0 \varvec{\varphi }_0} + {\mathbf R}_{\tilde{{\mathbf z}} \tilde{\varvec{\varphi }}} + {\mathbf r}_{\varvec{\theta }} = {\mathbf R}_{{\mathbf z}_0 \varvec{\varphi }_0} \; , \end{aligned}$$

(14.350)

as ${\mathbf R}_{\tilde{{\mathbf z}} \tilde{\varvec{\varphi }}} + {\mathbf r}_{\varvec{\theta }} = {\mathbf 0}$, using, say, (14.347).

14.1.1.2 14.C.1.2 Expressions for Correlated Output Noise

Now consider the case of correlated output noise. The instrumental vector ${\mathbf z}(t)$ is in this case taken as in (7.97) with $p_y = 0$; that is,

$$\begin{aligned} {\mathbf z}(t) = \left( \begin{array}{c} y(t) \\ \vdots \\ y(t-n_a) \\ u(t-1) \\ \vdots \\ u(t-n_b-p_u) \end{array}\right) \; . \end{aligned}$$

(14.351)

The noise parameter vector $\varvec{\rho }$ is given by, see (5.27),

$$\begin{aligned} \varvec{\rho }= \left( \begin{array}{c} r_{\tilde{y}}(0) \\ \vdots \\ r_{\tilde{y}}(n_a) \\ \lambda _u \end{array}\right) \; . \end{aligned}$$

(14.352)

The vector ${\mathbf r}_{{\mathbf z}\varepsilon }(\varvec{\theta })$ becomes

$$\begin{aligned} {\mathbf r}_{{\mathbf z}\varepsilon } (\varvec{\theta })= & {} \mathsf{E} \left\{ \left[ {\mathbf z}_0(t) + \tilde{{\mathbf z}}(t) \right] \left[ \varepsilon _0 (t,\varvec{\theta }) + \tilde{\varepsilon }(t,\varvec{\theta }) \right] \right\} \nonumber \\= & {} {\mathbf r}_{{\mathbf z}_0 \varepsilon _0}(\varvec{\theta }) + {\mathbf r}_{\tilde{{\mathbf z}} \tilde{\varepsilon }}(\varvec{\theta }) \; . \end{aligned}$$

(14.353)

Note that in particular $\varepsilon _0 (t,\varvec{\theta }_0) = 0$.

The first part of the matrix ${\mathbf S}$, (14.90), will now be

$$\begin{aligned} {\mathbf R}_{{\mathbf z}\varvec{\varphi }} + \left. {\mathbf r}_{\varvec{\theta }} \right| _{\varvec{\theta }= \varvec{\theta }_0}= & {} \mathsf{E} \left\{ \left[ {\mathbf z}_0(t) + \tilde{{\mathbf z}}(t) \right] \left[ \varvec{\varphi }_0^{{T}} (t) + \tilde{\varvec{\varphi }}^{{T}}(t) \right] \right\} \nonumber \\&- \mathsf{E} \left\{ \left[ {\mathbf z}_0(t) + \tilde{{\mathbf z}}(t) \right] \left[ \tilde{\varvec{\varphi }}^{{T}}(t) \right] \right\} \nonumber \\= & {} \mathsf{E} \left\{ {\mathbf z}_0(t) \varvec{\varphi }_0^{{T}} (t) \right\} = {\mathbf R}_{{\mathbf z}_0 \varvec{\varphi }_0} \; . \end{aligned}$$

(14.354)

This is the same relation as in (14.350).

Furthermore, the vector ${\mathbf r}_{{\mathbf z}\varepsilon }( \varvec{\theta }_0)$ is given by

$$\begin{aligned} {\mathbf r}_{{\mathbf z}\varepsilon }( \varvec{\theta }_0)= & {} \mathsf{E} \left\{ \left( \begin{array}{c} y(t) \\ \vdots \\ y(t-n_a) \\ u(t-1) \\ \vdots \\ u(t-n_b-p_u) \end{array}\right) \left[ \sum _i a_i \tilde{y}(t-i) - \sum _i b_i \tilde{u}(t-i) \right] _{\varvec{\theta }= \varvec{\theta }_0} \right\} \nonumber \\= & {} \left( \begin{array}{cc} {\mathscr {A}} &{} {\mathbf 0}_{(n_a +1) \times 1} \\ {\mathbf 0}_{ n_b \times (n_a + 1)} &{} - {\mathbf b}\\ {\mathbf 0}_{p_u \times (n_a + 1)} &{} {\mathbf 0}_{p_u \times 1 } \end{array}\right) \left( \begin{array}{c} r_{\tilde{y}}(0) \\ \vdots \\ r_{\tilde{y}}(n_a) \\ \lambda _u \end{array}\right) \ . \end{aligned}$$

(14.355)

The upper left part ${\mathscr {A}}$ of the matrix in (14.355) is an $(n_a+1) \times (n_a+1)$ matrix, given in (14.114).

Trivially, the derivative ${\mathbf r}_{\varvec{\rho }}$ is precisely the matrix appearing in (14.355).

14.1.2 14.C.2 Computation of the Matrix ${\mathbf C}$

An efficient tool for computing the elements of the matrix ${\mathbf C}$ without using explicit summation over $\tau$ is presented here, for the case when both $\tilde{y}(t)$ and the noise-free input $u_0(t)$ are ARMA processes. Details for how to compute such sums of products of covariances are described in Lemma A.10.

When applying Lemma A.10 for evaluation of ${\mathbf C}$, it is convenient to first decompose the variables as

$$\begin{aligned} {\mathbf z}(t) = {\mathbf z}_0(t) + \tilde{{\mathbf z}}(t),&\varepsilon (t) = \varepsilon _y(t) + \varepsilon _u(t) \; , \end{aligned}$$

(14.356)

$$\begin{aligned} \varepsilon _y(t) = A(q^{-1}) \tilde{y}(t),&\varepsilon _u(t) = B(q^{-1}) \tilde{u}(t) \; . \end{aligned}$$

(14.357)

It then holds

$$\begin{aligned} {\mathbf C}= & {} \sum _{\tau = -\infty }^{\infty } \left[ \left( {\mathbf R}_{{\mathbf z}_0}(\tau ) + {\mathbf R}_{\tilde{{\mathbf z}}}(\tau ) \right) \left( r_{\varepsilon _y}(\tau ) + r_{\varepsilon _u}(\tau ) \right) \right. \nonumber \\&\left. + \left( {\mathbf r}_{ \tilde{{\mathbf z}}\varepsilon _y}(\tau ) + {\mathbf r}_{\tilde{{\mathbf z}}\varepsilon _u}(\tau ) \right) \left( {\mathbf r}_{\varepsilon _y \tilde{{\mathbf z}}}(\tau ) + {\mathbf r}_{\varepsilon _u \tilde{{\mathbf z}}}(\tau ) \right) \right] \; . \end{aligned}$$

(14.358)

Introduce also the following model assumptions and notations, where $e_0(t)$ and v(t) are white noise signals,

$$\begin{aligned}&\tilde{y}(t) = F(q^{-1}) e_0(t), \ \ \mathsf{E} \left\{ e_0 ^2 (t) \right\} = \lambda _y \; , \end{aligned}$$

(14.359)

$$\begin{aligned}&u_0(t) = H(q^{-1}) v(t), \ \ \mathsf{E} \left\{ v ^2 (t) \right\} = \lambda _v \; , \end{aligned}$$

(14.360)

$$\begin{aligned}&G(q^{-1}) = \frac{B(q^{-1})}{A(q^{-1})} \; . \end{aligned}$$

(14.361)

Further observe:

• 1. When the output noise $\tilde{y}(t)$ is white, it holds

  $$\begin{aligned} F (q^{-1}) = 1 \; . \end{aligned}$$

  (14.362)

• 2. When the output noise is correlated, it holds

  $$\begin{aligned} p_y = 0 \; . \end{aligned}$$

  (14.363)

It will be convenient though to derive expressions for the general case.

The evaluation of ${\mathbf C}$ has to be done blockwise. Set

$$\begin{aligned} {\mathbf C}= \left( \begin{array}{cc} {\mathbf C}_{11} &{} {\mathbf C}_{12} \\ {\mathbf C}_{21} &{} {\mathbf C}_{22} \end{array}\right) \; , \end{aligned}$$

(14.364)

where the block ${\mathbf C}_{11}$ has dimension $n_a + p_y +1$, and the block ${\mathbf C}_{22}$ has dimension $n_b + p_u$. The three blocks in (14.364) can be evaluated using repeatedly Lemma A.10.

14.1.2.1 14.C.2.1 Evaluation of the Block ${\mathbf C}_{11}$

Consider the $\mu , \nu$ element where it is practical to number the elements such that $\mu , \nu = 0, \dots n_a + p_y$. From (14.358)

$$\begin{aligned} \left( {\mathbf C}_{11} \right) _{\mu ,\nu }= & {} \sum _{\tau = -\infty }^{\infty } \left[ \left( r_{y_0}(\tau - \mu + \nu ) + r_{\tilde{y}}(\tau - \mu + \nu ) \right) \left( r_{\varepsilon _y}(\tau ) + r_{\varepsilon _u}(\tau ) \right) \right. \nonumber \\&+ \left. r_{ \tilde{y}\varepsilon _y}(\tau -\mu ) r_{\varepsilon _y \tilde{y}}(\tau + \nu ) \right] \; . \end{aligned}$$

(14.365)

Note that due to the assumptions and notions above

$$\begin{aligned} \begin{array}{ll} y_0(t) = G(q^{-1}) H(q^{-1}) v(t), &{} \tilde{y}(t) = F(q^{-1}) e_0(t) \; , \\ \varepsilon _y(t) = A(q^{-1})F(q^{-1}) e_0(t), &{} \varepsilon _u(t) = B(q^{-1}) \tilde{u}(t) \; . \end{array} \end{aligned}$$

(14.366)

Using Lemma A.10, one now gets, with e(t) being white noise of unit variance,

$$\begin{aligned} \left( {\mathbf C}_{11} \right) _{\mu ,\nu }= & {} \lambda _v \lambda _y \mathsf{E} \left\{ [q^{-\nu } G(q^{-1})H(q^{-1}) A(q^{-1})F(q^{-1}) e(t) ] \right. \nonumber \\&\left. \ \ \times [q^{-\mu } G(q^{-1})H(q^{-1}) A(q^{-1})F(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _v \lambda _u \mathsf{E} \left\{ [q^{-\nu } G(q^{-1})H(q^{-1}) B(q^{-1}) e(t) ] \right. \nonumber \\&\left. \ \ \times [q^{-\mu } G(q^{-1})H(q^{-1}) B(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _y^2 \mathsf{E} \left\{ [q^{-\nu } A(q^{-1})F^2(q^{-1}) e(t) ] [q^{-\mu } A(q^{-1})F^2(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _y \lambda _u \mathsf{E} \left\{ [q^{-\nu } F(q^{-1}) B(q^{-1}) e(t) ] [q^{-\mu } F(q^{-1}) B(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _y^2 \mathsf{E} \left\{ [ A^2(q^{-1})F^2(q^{-1}) e(t) ] [q^{-\mu -\nu } F^2(q^{-1}) e(t) ] \right\} \; . \end{aligned}$$

(14.367)

14.1.2.2 14.C.2.2 Evaluation of the Block ${\mathbf C}_{22}$

Consider the $\mu , \nu$ element where $\mu , \nu = 1, \dots n_b + p_u$. From (14.358)

$$\begin{aligned} \left( {\mathbf C}_{22} \right) _{\mu ,\nu }= & {} \sum _{\tau = -\infty }^{\infty } \left[ \left( r_{u_0}(\tau - \mu + \nu ) + r_{\tilde{u}}(\tau - \mu + \nu ) \right) \left( r_{\varepsilon _y}(\tau ) + r_{\varepsilon _u}(\tau ) \right) \right. \nonumber \\&+ \left. r_{ \tilde{u}\varepsilon _u}(\tau -\mu ) r_{\varepsilon _u \tilde{u}}(\tau + \nu ) \right] \nonumber \\= & {} \lambda _v \lambda _u \mathsf{E} \left\{ [q^{-\nu } B(q^{-1})H(q^{-1}) e(t) ] [q^{-\mu } B(q^{-1})H(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _v \lambda _y \mathsf{E} \left\{ [q^{-\nu } A(q^{-1}) F(q^{-1}) H(q^{-1}) e(t) ] \right. \nonumber \\&\left. \ \ \times [q^{-\mu } A(q^{-1}) F(q^{-1}) H(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _u^2 \mathsf{E} \left\{ [q^{-\nu } B(q^{-1}) e(t) ] [q^{-\mu } B(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _y \lambda _u \mathsf{E} \left\{ [q^{-\nu } A(q^{-1}) F(q^{-1}) e(t) ] [q^{-\mu } A(q^{-1}) F(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _u^2 \mathsf{E} \left\{ [ B^2(q^{-1}) e(t) ] [q^{-\mu -\nu } e(t) ] \right\} \; . \end{aligned}$$

(14.368)

14.1.2.3 14.C.2.3 Evaluation of the Block ${\mathbf C}_{12}$

Consider the $\mu , \nu$ element where $\mu = 0, \dots , n_a + p_y, \ \nu = 1, \dots n_b + p_u$. From (14.358)

$$\begin{aligned} \left( {\mathbf C}_{12} \right) _{\mu ,\nu }= & {} \sum _{\tau = -\infty }^{\infty } \left[ r_{y_0 u_0} (\tau - \mu + \nu ) \left( r_{\varepsilon _y}(\tau ) + r_{\varepsilon _u}(\tau ) \right) \right. \nonumber \\&+ \left. r_{ \tilde{y}\varepsilon _y}(\tau -\mu ) r_{\varepsilon _u \tilde{u}}(\tau + \nu ) \right] \nonumber \\= & {} \lambda _v \lambda _y \mathsf{E} \left\{ [q^{-\nu } H(q^{-1}) A(q^{-1})F(q^{-1}) e(t) ] \right. \nonumber \\&\left. \ \ \times [q^{-\mu } G(q^{-1})H(q^{-1}) A(q^{-1})F(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _v \lambda _u \mathsf{E} \left\{ [q^{-\nu } H(q^{-1}) B(q^{-1}) e(t) ] [q^{-\mu } G(q^{-1})H(q^{-1}) B(q^{-1}) e(t) ] \right\} \nonumber \\&+ \lambda _y \lambda _u \mathsf{E} \left\{ [ A(q^{-1}) B(q^{-1}) F(q^{-1}) e(t) ] [q^{-\mu -\nu } F(q^{-1}) e(t) ] \right\} \; . \end{aligned}$$

(14.369)

14.1.3 14.C.3 Non-Gaussian Distributed Data. Proof of Lemma 14.7

It follows by construction that an arbitrary element of ${\mathbf C}$ in (14.97) has the form of ${\mathbf C}$ in (A.80). One thus can write ${\mathbf C}$ as in (14.104) where the elements of the ‘non-Gaussian contribution’ ${\mathbf C}^\mathrm{NG}$ are given by the last term in (A.80). Reflecting the structure of the vector ${\mathbf z}(t)$ it is found that ${\mathbf C}^\mathrm{NG}$ will always have the form given by (14.108).

Consider first the block ${\mathbf C}^\mathrm{NG}_{33}$. For its $\mu , \nu$ element (with $\mu , \nu = 1, \dots , n_b$), apply Lemma A.12 using

$$\begin{aligned} \begin{array}{l} x_1(t) = \tilde{u}(t-\mu ), \ \ \ x_2(t) = B(q^{-1}) \tilde{u}(t) \; , \\ x_3(t) = \tilde{u}(t-\nu ), \ \ \ x_4(t) = B(q^{-1}) \tilde{u}(t) \; , \end{array} \end{aligned}$$

(14.370)

and therefore the expressions for ${\mathbf C}^\mathrm{NG}_{33}$ in (14.109) and (14.112) follow.

Next consider the block ${\mathbf C}^\mathrm{NG}_{11}$. For its $\mu , \nu$ element ($\mu , \nu = 0, \dots n_a$), apply Lemma A.12 using

$$\begin{aligned} \begin{array}{l} x_1(t) = \tilde{y}(t-\mu ), \ \ \ x_2(t) = A(q^{-1}) \tilde{y}(t) \; , \\ x_3(t) = \tilde{y}(t-\nu ), \ \ \ x_4(t) = A(q^{-1}) \tilde{y}(t) \; . \end{array} \end{aligned}$$

(14.371)

If $\tilde{y}(t)$ is white noise, the expression (14.109) for ${\mathbf C}^\mathrm{NG}_{11}$ is obtained. In the more general case of $\tilde{y}(t)$ being correlated noise, one gets instead the expression in (14.112) that holds with

$$\begin{aligned} {\mathbf r}_x= & {} \mathsf{E} \left\{ \left( \begin{array}{c} \tilde{y}(t) \\ \vdots \\ \tilde{y}(t-n_a) \end{array}\right) \sum _{i = 0} ^{n_a} a_i \tilde{y}(t-i) \right\} \nonumber \\= & {} {\mathscr {A}} \left( \begin{array}{c} r_{\tilde{y}} (0) \\ \vdots \\ r_{\tilde{y}} (n_a) \end{array}\right) \; , \end{aligned}$$

(14.372)

with $\mathscr {A}$ given by (14.114). This proves (14.113).

14.1.4 14.C.4 Proof of Lemma 14.8

First note that by construction, see (7.130), and (7.132) it holds

$$\begin{aligned} \overline{{\mathbf f}} = \hat{{\mathbf r}}_{ \varvec{\varepsilon }\otimes {\mathbf z}} (\varvec{\theta }) - {{\mathbf r}}_{ \varvec{\varepsilon }\otimes {\mathbf z}} (\varvec{\theta }, \varvec{\rho }) \; . \end{aligned}$$

(14.373)

The definition (14.121) of ${\mathbf C}$ gives

$$\begin{aligned} {\mathbf C}= & {} \lim _ {N \rightarrow \infty } N \mathsf{E} \left\{ \overline{{\mathbf f}} \ \overline{{\mathbf f}} ^{{T}} \right\} \nonumber \\= & {} \lim _ {N \rightarrow \infty } N \mathsf{E} \left\{ \left( \frac{1}{N} \sum _{t=1}^N \varvec{\varepsilon }(t) \otimes {\mathbf z}(t) \right) \left( \frac{1}{N} \sum _{s=1}^N \varvec{\varepsilon }^{{T}} (s) \otimes {\mathbf z}^{{T}} (s) \right) - {\mathbf r}_{ \varvec{\varepsilon }\otimes {\mathbf z}} {\mathbf r}^{{T}}_{ \varvec{\varepsilon }\otimes {\mathbf z}} \right\} \; . \nonumber \\ \end{aligned}$$

(14.374)

The arguments $\varvec{\theta }$ and $\varvec{\rho }$ were omitted for simplicity.

Consider now an arbitrary block, say block j, k of ${\mathbf C}$, and apply the assumption on Gaussian distributed data, using Lemma A.9

$$\begin{aligned} \left( {\mathbf C}\right) _{j, k}= & {} \lim _{N \rightarrow \infty } N \left( \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \left[ {\mathbf r}_{\varepsilon _j {\mathbf z}} {\mathbf r}_{\varepsilon _k {\mathbf z}}^{{T}} + \mathsf{E} \left\{ \varepsilon _j(t) \varepsilon _k(s) \right\} \mathsf{E} \left\{ {\mathbf z}(t) {\mathbf z}^{{T}} (s) \right\} \right. \right. \nonumber \\&+ \left. \left. \mathsf{E} \left\{ \varepsilon _k(s) {\mathbf z}(t) \right\} \mathsf{E} \left\{ \varepsilon _j(t) {\mathbf z}^{{T}} (s) \right\} \right] - {\mathbf r}_{ \varepsilon _j {\mathbf z}} {\mathbf r}_{ \varepsilon _k {\mathbf z}}^{{T}} \right) \nonumber \\= & {} \lim _{N \rightarrow \infty } N \frac{1}{N^2} \sum _{\tau = -N}^N \left( N - | \tau | \right) \left[ r_{\varepsilon _j \varepsilon _k} (\tau ) {\mathbf R}_{{\mathbf z}} (\tau ) + {\mathbf r}_{{\mathbf z}\varepsilon _k} (\tau ) {\mathbf r}_{ \varepsilon _j {\mathbf z}} (\tau ) \right] \nonumber \\= & {} \sum _{\tau = -\infty }^{\infty } \left[ r_{\varepsilon _j \varepsilon _k} (\tau ) {\mathbf R}_{{\mathbf z}} (\tau ) + {\mathbf r}_{{\mathbf z}\varepsilon _k} (\tau ) {\mathbf r}_{ \varepsilon _j {\mathbf z}} (\tau ) \right] \; . \end{aligned}$$

(14.375)

The last equality in (14.375) follows as in the scalar IV case, see Sect. 14.B.1, using arguments that all the involved covariance functions do decay exponentially as $|\tau |$ increases to infinity.

Going back to the whole matrix ${\mathbf C}$ it turns out from (14.375) that it can be written as

$$\begin{aligned} {\mathbf C}= & {} \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varepsilon }} (\tau ) \otimes {\mathbf R}_{{\mathbf z}} (\tau ) + \left( \begin{array}{c} {\mathbf r}_{{\mathbf z}\varepsilon _1} (\tau ) \\ \vdots \\ {\mathbf r}_{{\mathbf z}\varepsilon _{n_y} } (\tau ) \end{array}\right) \left( \begin{array}{ccc} {\mathbf r}_{{\mathbf z}\varepsilon _1} ^{{T}} (-\tau )&\dots&{\mathbf r}_{{\mathbf z}\varepsilon _{n_y} } ^{{T}} (-\tau ) \end{array}\right) \right] \nonumber \\= & {} \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varepsilon }} (\tau ) \otimes {\mathbf R}_{{\mathbf z}} (\tau ) + \mathrm{vec} \left( {\mathbf R}_{{\mathbf z}\varvec{\varepsilon }}(\tau ) \right) \left( \mathrm{vec} \left( {\mathbf R}_{{\mathbf z}\varvec{\varepsilon }} (- \tau ) \right) \right) ^{{T}} \right] \; , \end{aligned}$$

(14.376)

which completes the proof.

14.D Asymptotic Accuracy for Models Obtained under Linear Constraints

Consider the problem of minimizing

$$\begin{aligned} V_N(\varvec{\vartheta }) = \frac{1}{2} {\mathbf f}_N^T(\varvec{\vartheta }) {\mathbf f}_N(\varvec{\vartheta }) \end{aligned}$$

(14.377)

under the constraint

$$\begin{aligned} {\mathbf g}_N(\varvec{\vartheta }) = {\mathbf 0}\; . \end{aligned}$$

(14.378)

Here, ${\mathbf f}$ and ${\mathbf g}$ are smooth functions of the parameter vector $\varvec{\vartheta }$. They depend also on the number of data points, N, and converge uniformly to ${\mathbf f}_{\infty }$ and ${\mathbf g}_{\infty }$, respectively, as $N \rightarrow \infty$. Further, in the limiting case for the true parameter vector it holds

$$\begin{aligned} {\mathbf f}_{\infty } (\varvec{\vartheta }_0) = {\mathbf 0}, \ \ {\mathbf g}_{\infty } (\varvec{\vartheta }_0) = {\mathbf 0}\; . \end{aligned}$$

(14.379)

The solution to the optimization problem will be denoted by $\hat{\varvec{\vartheta }}_N$. The issue to be considered here is how to express the estimation error $\hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0$ (for large enough values of N) in terms of ${\mathbf f}_N$ and ${\mathbf g}_N$.

To treat the general optimization problem, introduce the Lagrangian

$$\begin{aligned} L(\varvec{\vartheta }, \varvec{\lambda }) = \frac{1}{2} {\mathbf f}_N^T(\varvec{\vartheta }) {\mathbf f}_N(\varvec{\vartheta }) + \varvec{\lambda }^T {\mathbf g}_N(\varvec{\vartheta }) \; . \end{aligned}$$

(14.380)

Setting the gradient of L to zero gives the necessary conditions for the optimum to be

$$\begin{aligned} {\mathbf 0}= & {} {\mathbf f}_N^T ( \hat{\varvec{\vartheta }} ) \frac{ \partial {\mathbf f}_N(\hat{\varvec{\vartheta }}_N) }{\partial \varvec{\vartheta }} + \varvec{\lambda }^T \frac{ \partial {\mathbf g}_N(\hat{\varvec{\vartheta }}_N) }{\partial \varvec{\vartheta }} \; , \end{aligned}$$

(14.381)

$$\begin{aligned} {\mathbf 0}= & {} {\mathbf g}_N(\hat{\varvec{\vartheta }}_N) \; . \end{aligned}$$

(14.382)

To this aim introduce the sensitivity matrices

$$\begin{aligned} {\mathbf F}= & {} \frac{ \partial {\mathbf f}(\varvec{\vartheta }) }{\partial \varvec{\vartheta }} |_{\varvec{\vartheta }= \varvec{\vartheta }_0} \; , \end{aligned}$$

(14.383)

$$\begin{aligned} {\mathbf G}= & {} \frac{ \partial {\mathbf g}(\varvec{\vartheta }) }{\partial \varvec{\vartheta }} |_{\varvec{\vartheta }= \varvec{\vartheta }_0} \; . \end{aligned}$$

(14.384)

As the estimate $\hat{\varvec{\vartheta }}_N$ will be close to the true parameter vector $\varvec{\vartheta }_0$ for large N, one can now use the linearized expressions

$$\begin{aligned} {\mathbf f}_N(\hat{\varvec{\vartheta }}_N)\approx & {} {\mathbf f}_N({\varvec{\vartheta }}_0) + {\mathbf F}\left( \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \right) \; , \end{aligned}$$

(14.385)

$$\begin{aligned} {\mathbf g}_N(\hat{\varvec{\vartheta }}_N)\approx & {} {\mathbf g}_N( \varvec{\vartheta }_0 ) + {\mathbf G}\left( \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \right) \; . \end{aligned}$$

(14.386)

Making use of (14.385) and (14.386) in (14.381) and (14.382) now leads to

$$\begin{aligned}{\mathbf 0}\approx & {} \left[ {\mathbf f}_N^T( {\varvec{\vartheta }}_0 ) + \left( \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \right) ^T {\mathbf F}^T \right] {\mathbf F}+ \varvec{\lambda }^T {\mathbf G}\; , \nonumber \\ {\mathbf 0}\approx & {} {\mathbf g}_N( \varvec{\vartheta }_0 ) + {\mathbf G}\left( \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \right) \; , \nonumber \end{aligned}$$

which is easily rewritten as

$$\begin{aligned} {\mathbf 0}= & {} {\mathbf F}^T {\mathbf f}_N + \left( {\mathbf F}^T {\mathbf F}\right) \left( \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \right) + {\mathbf G}^T \varvec{\lambda }\; , \end{aligned}$$

(14.387)

$$\begin{aligned} {\mathbf 0}\approx & {} {\mathbf g}_N + {\mathbf G}\left( \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \right) \; , \end{aligned}$$

(14.388)

where, for convenience only, the arguments of the functions are skipped. Note that (14.387), (14.388) is a linear system of equations with $\hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0$ and $\varvec{\lambda }$ as unknowns. The solution can be written in the form

$$\begin{aligned}\left( \begin{array}{c} \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0 \\ \varvec{\lambda }\end{array}\right)= & {} - \left( \begin{array}{cc} {\mathbf F}^T {\mathbf F}&{} {\mathbf G}^T \\ {\mathbf G}&{} {\mathbf 0}\end{array}\right) ^{-1} \left( \begin{array}{c} {\mathbf F}^T {\mathbf f}_N \\ {\mathbf g}_N \end{array}\right) \nonumber \\= & {} - \left[ \left( \begin{array}{cc} \left( {\mathbf F}^T {\mathbf F}\right) ^{-1} &{} {\mathbf 0}\\ {\mathbf 0}&{} {\mathbf 0}\end{array}\right) - \left( \begin{array}{c} - \left( {\mathbf F}^T {\mathbf F}\right) ^{-1} {\mathbf G}^T \\ {\mathbf I}\end{array}\right) \right. \nonumber \\&\ \ \ \left. \times \left[ {\mathbf G}\left( {\mathbf F}^T {\mathbf F}\right) ^{-1}{\mathbf G}^T \right] ^{-1} \left( \begin{array}{cc} - {\mathbf G}\left( {\mathbf F}^T {\mathbf F}\right) ^{-1}&{\mathbf I}\end{array}\right) \right] \left( \begin{array}{c} {\mathbf F}^T {\mathbf f}_N \\ {\mathbf g}_N \end{array}\right) \; , \nonumber \end{aligned}$$

leading to

$$\begin{aligned} \hat{\varvec{\vartheta }}_N - \varvec{\vartheta }_0= & {} - \left( {\mathbf F}^T {\mathbf F}\right) ^{-1}{\mathbf F}^T {\mathbf f}_N + \left( {\mathbf F}^T {\mathbf F}\right) ^{-1}{\mathbf G}^T \nonumber \\&\times \left[ {\mathbf G}\left( {\mathbf F}^T {\mathbf F}\right) ^{-1}{\mathbf G}^T \right] ^{-1} \left[ - {\mathbf G}\left( {\mathbf F}^T {\mathbf F}\right) ^{-1}{\mathbf F}^T {\mathbf f}_N + {\mathbf g}_N \right] \; . \end{aligned}$$

(14.389)

14.E Asymptotic Distribution for the Covariance Matching Method

14.1.1 14.E.1 Covariance Matrix of the Extended Parameter Vector

In this section the results of Theorem 14.1 are extended to give the covariance matrix not only of the estimate $\hat{\varvec{\theta }}$ but also of $\hat{{\mathbf r}}_z$ and of $\hat{\varvec{\rho }} = \left( \hat{\lambda }_y \ \hat{\lambda }_u \right) ^T$. How $\hat{\varvec{\rho }}$ can be found is described in Remark 8.6.

In fact, the augmented model then used has precisely the same algebraic form as in (8.21):

$$\begin{aligned} \hat{ \bar{{\mathbf r}}}= & {} \left( \begin{array}{c} \hat{{\mathbf r}} \\ \hat{{\mathbf r}}_2 \end{array}\right) \; , \end{aligned}$$

(14.390)

$$\begin{aligned} \bar{{\mathbf F}}(\varvec{\theta })= & {} \left( \begin{array}{cc} {\mathbf F}(\varvec{\theta }) &{} {\mathbf 0}\\ {\mathbf F}_2(\varvec{\theta }) &{} {\mathbf I}\end{array}\right) \ \ \overline{{\mathbf W}} = \left( \begin{array}{cc} {\mathbf W}_{11} &{} {\mathbf W}_{12} \\ {\mathbf W}_{21} &{} {\mathbf W}_{22} \end{array}\right) \; , \end{aligned}$$

(14.391)

$$\begin{aligned} \bar{{\mathbf r}}_{{\mathbf z}}= & {} \left( \begin{array}{c} {\mathbf r}_{{\mathbf z}} \\ \varvec{\rho }\end{array}\right) \ \ {\mathbf r}_2 = \left( \begin{array}{c} r_y(0) \\ r_u(0) \end{array}\right) \ \ \varvec{\rho }= \left( \begin{array}{c} \lambda _y \\ \lambda _u \end{array}\right) \; . \end{aligned}$$

(14.392)

In particular, assume there is no cross-weighting, so ${\mathbf W}_{12} = {\mathbf 0}$ in (14.391). Then the estimate of the noise variances can be treated separately and is indeed given by

$$\begin{aligned} \hat{\varvec{\rho }} = \left( \begin{array}{c} \hat{r}_y(0) \\ \hat{r}_u(0) \end{array}\right) - {\mathbf F}_2(\hat{\varvec{\theta }}) \hat{{\mathbf r}}_{{\mathbf z}} \; . \end{aligned}$$

(14.393)

Concerning the accuracy of the estimate (14.393) one needs to relate it to the covariance matrix of $\hat{\overline{{\mathbf r}}}$ in (14.390). To this aim introduce

$$\begin{aligned} \overline{{\mathbf R}} = \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \left[ \hat{\overline{{\mathbf r}}} - \overline{{\mathbf r}} \right] \left[ \hat{\overline{{\mathbf r}}} - \overline{{\mathbf r}} \right] ^T \right\} = \left( \begin{array}{cc} {\mathbf R}&{} {\mathbf R}_{12} \\ {\mathbf R}_{21} &{} {\mathbf R}_2 \end{array}\right) \end{aligned}$$

(14.394)

and note that it can be computed using the algorithm for ${\mathbf R}$, see Sect. 14.5.2. (The modification is only that a few running indices, such as $\mu , \nu$, have to start at 0 instead of 1.)

The following result applies.

Lemma 14.15

Consider the joint estimate $\hat{\varvec{\theta }}$, $\hat{{\mathbf r}}_{{\mathbf z}}$, and $\hat{\varvec{\rho }}$. The normalized asymptotic joint covariance matrix can be written as

$$\begin{aligned}&\lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \left( \begin{array}{c} \hat{\varvec{\theta }} -\varvec{\theta }\\ \hat{{\mathbf r}}_{{\mathbf z}} - {\mathbf r}\\ \hat{\varvec{\rho }} - \varvec{\rho }\end{array}\right) \left( \begin{array}{ccc} \left( \hat{\varvec{\theta }} -\varvec{\theta }\right) ^T&\left( \hat{{\mathbf r}}_{{\mathbf z}} - {\mathbf r}\right) ^T&\left( \hat{\varvec{\rho }} - \varvec{\rho }\right) ^T \end{array}\right) \right\} \nonumber \\&\ \ \ = \left( \begin{array}{ccc} {\mathbf P}_{\varvec{\theta }} &{} {\mathbf P}_{\varvec{\theta }{\mathbf z}} &{} {\mathbf P}_{\varvec{\theta }\varvec{\rho }} \\ {\mathbf P}_{{\mathbf z}\varvec{\theta }} &{} {\mathbf P}_{ {\mathbf z}} &{} {\mathbf P}_{{\mathbf z}\varvec{\rho }} \\ {\mathbf P}_{\varvec{\rho }\varvec{\theta }} &{} {\mathbf P}_{\varvec{\rho }{\mathbf z}} &{} {\mathbf P}_{ \varvec{\rho }} \end{array}\right) \; , \end{aligned}$$

(14.395)

where

$$\begin{aligned} {\mathbf P}_{\varvec{\theta }}= & {} {\mathbf L}_1 {\mathbf R}{\mathbf L}_1^T \; , \end{aligned}$$

(14.396)

$$\begin{aligned} {\mathbf P}_{\varvec{\theta }{\mathbf z}}= & {} {\mathbf L}_1 {\mathbf R}{\mathbf L}_2^T \; , \end{aligned}$$

(14.397)

$$\begin{aligned} {\mathbf P}_{\varvec{\theta }\varvec{\rho }}= & {} {\mathbf L}_1 \left( {\mathbf R}_{12} - {\mathbf R}{\mathbf L}_1^T {\mathbf S}_2^T - {\mathbf R}{\mathbf L}_2^T {\mathbf F}_2^T \right) \; , \end{aligned}$$

(14.398)

$$\begin{aligned} {\mathbf P}_{{\mathbf z}}= & {} {\mathbf L}_2 {\mathbf R}{\mathbf L}_2^T \; , \end{aligned}$$

(14.399)

$$\begin{aligned} {\mathbf P}_{{\mathbf z}\varvec{\rho }}= & {} {\mathbf L}_2 \left( {\mathbf R}_{12} - {\mathbf R}{\mathbf L}_1^T {\mathbf S}_2^T - {\mathbf R}{\mathbf L}_2^T {\mathbf F}_2^T \right) \; , \end{aligned}$$

(14.400)

$$\begin{aligned} {\mathbf P}_{\varvec{\rho }}= & {} \left( {\mathbf S}_2 {\mathbf L}_1 + {\mathbf F}_2 {\mathbf L}_2 \right) {\mathbf R}\left( {\mathbf S}_2 {\mathbf L}_1 + {\mathbf F}_2 {\mathbf L}_2 \right) ^T + {\mathbf R}_2 \nonumber \\&\ \ - \left( {\mathbf S}_2 {\mathbf L}_1 + {\mathbf F}_2 {\mathbf L}_2 \right) {\mathbf R}_{12} - {\mathbf R}_{21} \left( {\mathbf S}_2 {\mathbf L}_1 + {\mathbf F}_2 {\mathbf L}_2 \right) ^T \; , \end{aligned}$$

(14.401)

$$\begin{aligned} {\mathbf L}_1= & {} \left( {\mathbf S}^T {\mathbf M}{\mathbf S}\right) ^{-1} {\mathbf S}^T {\mathbf M}\; , \end{aligned}$$

(14.402)

$$\begin{aligned} {\mathbf L}_2= & {} \left( {\mathbf F}^T {\mathbf M}_{{\mathbf z}} {\mathbf F}\right) ^{-1} {\mathbf F}^T {\mathbf M}_{{\mathbf z}} \; , \end{aligned}$$

(14.403)

$$\begin{aligned} {\mathbf S}_2= & {} \left( \begin{array}{ccc} \frac{ \partial {\mathbf F}_2}{\partial \varvec{\theta }_1} {\mathbf r}_{{\mathbf z}}&\dots&\frac{ \partial {\mathbf F}_2}{\partial \varvec{\theta }_{n_a+n_b} } {\mathbf r}_{{\mathbf z}} \end{array}\right) \; , \end{aligned}$$

(14.404)

$$\begin{aligned} {\mathbf M}_{{\mathbf z}}= & {} {\mathbf W}- {\mathbf W}{\mathbf S}\left( {\mathbf S}^T {\mathbf W}{\mathbf S}\right) ^{-1} {\mathbf S}^T {\mathbf W}\; . \end{aligned}$$

(14.405)

Further, ${\mathbf M}$ is given by (14.142), ${\mathbf S}$ by (14.128), and ${\mathbf F}_2$ by (8.29).

Proof

Note from (14.8) and (14.132), (14.133) that asymptotically (for large N)

$$\begin{aligned} \tilde{\varvec{\theta }} = \left( {\mathbf S}^T {\mathbf M}{\mathbf S}\right) ^{-1} {\mathbf S}^T {\mathbf M}\tilde{{\mathbf r}} = {\mathbf L}_1 \tilde{{\mathbf r}} \; . \end{aligned}$$

(14.406)

Similarly to (14.8), cf. also (14.126),

$$\begin{aligned} \tilde{{\mathbf r}}_{{\mathbf z}}= & {} \left( \begin{array}{cc} {\mathbf 0}&{\mathbf I}\end{array}\right) \left( {\mathbf L}^T {\mathbf W}{\mathbf L}\right) ^{-1} {\mathbf L}^T {\mathbf W}\tilde{{\mathbf r}} \nonumber \\= & {} \left( \begin{array}{cc} {\mathbf 0}&{\mathbf I}\end{array}\right) \left( \begin{array}{cc} {\mathbf S}^T {\mathbf W}{\mathbf S}&{} {\mathbf S}^T {\mathbf W}{\mathbf F}\\ {\mathbf F}^T {\mathbf W}{\mathbf S}&{} {\mathbf F}^T {\mathbf W}{\mathbf F}\end{array}\right) ^{-1} \left( \begin{array}{c} {\mathbf S}^T {\mathbf W}\\ {\mathbf F}^T {\mathbf W}\end{array}\right) \tilde{{\mathbf r}} \nonumber \\= & {} \left( \begin{array}{cc} {\mathbf 0}&{\mathbf I}\end{array}\right) \left( \begin{array}{c} - \left( {\mathbf S}^T {\mathbf W}{\mathbf S}\right) ^{-1} {\mathbf S}^T {\mathbf W}{\mathbf F}\\ {\mathbf I}\end{array}\right) \nonumber \\&\times \left( {\mathbf F}^T {\mathbf W}{\mathbf F}- {\mathbf F}^T {\mathbf W}{\mathbf S}\left( {\mathbf S}^T {\mathbf W}{\mathbf S}\right) ^{-1} {\mathbf S}^T {\mathbf W}{\mathbf F}\right) ^{-1} \nonumber \\&\times \left( \begin{array}{ccc} - {\mathbf F}^T {\mathbf W}{\mathbf S}\left( {\mathbf S}^T {\mathbf W}{\mathbf S}\right) ^{-1}&\,&{\mathbf I}\end{array}\right) \left( \begin{array}{c} {\mathbf S}^T {\mathbf W}\\ {\mathbf F}^T {\mathbf W}\end{array}\right) \tilde{{\mathbf r}} \nonumber \\= & {} \left( {\mathbf F}^T {\mathbf M}_{{\mathbf z}} {\mathbf F}\right) ^{-1} \left( {\mathbf F}^T {\mathbf W}- {\mathbf F}^T {\mathbf W}{\mathbf S}\left( {\mathbf S}^T {\mathbf W}{\mathbf S}\right) ^{-1} {\mathbf S}^T {\mathbf W}\right) \tilde{{\mathbf r}} \nonumber \\= & {} \left( {\mathbf F}^T {\mathbf M}_{{\mathbf z}} {\mathbf F}\right) ^{-1} {\mathbf F}^T {\mathbf M}_{{\mathbf z}} \tilde{{\mathbf r}} = {\mathbf L}_2 \tilde{{\mathbf r}} \; . \end{aligned}$$

(14.407)

Linearizing (14.393) around the true values leads to

$$\begin{aligned} \tilde{\varvec{\rho }} = \tilde{{\mathbf r}}_2 - {\mathbf S}_2 \tilde{\varvec{\theta }} - {\mathbf F}_2 \tilde{ {\mathbf r}}_{{\mathbf z}} \; . \end{aligned}$$

(14.408)

Combining (14.406)–(14.408) leads to

$$\begin{aligned} \left( \begin{array}{c} \tilde{\varvec{\theta }} \\ \tilde{{\mathbf r}}_{{\mathbf z}} \\ \tilde{\varvec{\rho }} \end{array}\right)= & {} \left( \begin{array}{c} {\mathbf L}_1 \tilde{{\mathbf r}} \\ {\mathbf L}_2 \tilde{{\mathbf r}} \\ - \left( {\mathbf S}_2 {\mathbf L}_1 + {\mathbf F}_2 {\mathbf L}_2 \right) \tilde{ {\mathbf r}} + \tilde{ {\mathbf r}}_2 \end{array}\right) \nonumber \\= & {} \left( \begin{array}{cc} {\mathbf L}_1 &{} {\mathbf 0}\\ {\mathbf L}_2 &{} {\mathbf 0}\\ - \left( {\mathbf S}_2 {\mathbf L}_1 + {\mathbf F}_2 {\mathbf L}_2 \right) &{} {\mathbf I}\end{array}\right) \left( \begin{array}{c} \tilde{{\mathbf r}} \\ \tilde{{\mathbf r}}_2 \end{array}\right) \; , \end{aligned}$$

(14.409)

and the statement follows directly by evaluating the covariance matrix of the vector in (14.409).        $\blacksquare$

14.1.2 14.E.2 Proof of Theorem 14.2

Consider first the block matrix ${\mathbf R}_{11}$. One can write

$$\begin{aligned} \hat{{\mathbf r}}_y = \frac{1}{N} \sum _{t=1}^N \varvec{\varphi }(t) y(t) \; , \end{aligned}$$

(14.410)

where, in this particular case,

$$\begin{aligned} \varvec{\varphi }(t) = \left( \begin{array}{c} y(t) \\ \vdots \\ y(t-p_y) \end{array}\right) \; . \end{aligned}$$

(14.411)

Furthermore, let ${\mathbf r}_y$ denote the true (and expected) value of $\hat{{\mathbf r}}_y$:

$$\begin{aligned} {\mathbf r}_y = \mathsf{E} \left\{ \varvec{\varphi }(t) y(t) \right\} = \left( \begin{array}{c} r_y(0) \\ \vdots \\ r_y(p_y) \end{array}\right) \; . \end{aligned}$$

(14.412)

Consider the asymptotic normalized covariance matrix of $\hat{{\mathbf r}}$:

$$\begin{aligned} {\mathbf R}_{11} = \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ [ \hat{{\mathbf r}}_y - {\mathbf r}_y ][ \hat{{\mathbf r}}_y - {\mathbf r}_y ]^T \right\} \; . \end{aligned}$$

(14.413)

Using (14.412)

$$\begin{aligned} {\mathbf R}_{11}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \hat{{\mathbf r}}_y \hat{{\mathbf r}}^T_y - {\mathbf r}_y {\mathbf r}^T_y \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \left[ \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ \varvec{\varphi }(t) y(t) \varvec{\varphi }^T (s) y(s) \right\} - {\mathbf r}_y {\mathbf r}_y^T \right] \; . \end{aligned}$$

(14.414)

Use Assumptions AN2 and AI3 and apply the general rule for product of Gaussian variables, as given in Lemma A.9. Using the result (A.68) in (14.414) leads to

$$\begin{aligned} {\mathbf R}_{11}= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{t=1}^N \sum _{s=1}^N \left( \left[ \mathsf{E} \left\{ \varvec{\varphi }(t) \varvec{\varphi }^T(s) \right\} \right] \left[ \mathsf{E} \left\{ y(t) y(s) \right\} \right] \right. \nonumber \\&+ \left. \left[ \mathsf{E} \left\{ \varvec{\varphi }(t) y(s) \right\} \right] \left[ \mathsf{E} \left\{ \varvec{\varphi }^T(s) y(t) \right\} \right] \right) \; . \end{aligned}$$

(14.415)

Due to Assumption AS1 the covariance function $r_y(\tau )$ decays exponentially with $\tau$. Therefore one can write

$$\begin{aligned} \parallel \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{\tau =-N}^N | \tau | {\mathbf R}_{\varvec{\varphi }}(\tau ) r_y(\tau ) \parallel \le \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{\tau =0}^N 2 \tau C \alpha ^{\tau } = 0 \end{aligned}$$

(14.416)

for some $| \alpha | < 1$. Using this result leads to

$$\begin{aligned} {\mathbf R}_{11}= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{\tau = -N}^N \left( N - | \tau | \right) \left[ {\mathbf R}_{\varvec{\varphi }}(\tau ) r_y(\tau ) + {\mathbf r}_{\varvec{\varphi }y}(\tau ) {\mathbf r}_{\varvec{\varphi }y}^T(-\tau ) \right] \nonumber \\= & {} \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }}(\tau ) r_y(\tau ) + {\mathbf r}_{\varvec{\varphi }y}(\tau ) {\mathbf r}_{\varvec{\varphi }y}^T(-\tau ) \right] \; . \end{aligned}$$

(14.417)

In order to proceed one will need a technique for computing sums of the form

$$\begin{aligned} \sum _{\tau = -\infty }^{\infty } r_y(\tau ) r_y(\tau + k) \; , \end{aligned}$$

(14.418)

where k is an arbitrary integer. The measured output signal can be written as

$$\begin{aligned} y(t) = y_0(t) + \tilde{y} (t) = \frac{B(q^{-1})}{A(q^{-1})} \frac{C(q^{-1})}{D(q^{-1})} e(t) + \tilde{y} (t) \end{aligned}$$

(14.419)

and Lemma A.10 can be applied: Write the generic element of the upper left partition ${\mathbf R}_{11}$ as follows, using (14.419):

$$\begin{aligned} ({\mathbf R}_{11})_{\mu \nu }= & {} \sum _{\tau = - \infty } ^ {\infty } \left[ r_y(\tau + \mu - \nu ) r_y(\tau ) + r_y(\tau -\mu ) r_y(\tau + \nu ) \right] \nonumber \\= & {} \sum _{\tau = - \infty } ^ {\infty } \left[ \left\{ r_{y_0}(\tau + \mu - \nu ) + \lambda _y \delta _{\tau +\mu -\nu , 0} \right\} \right. \left\{ r_{y_0} (\tau ) + \lambda _y \delta _{\tau , 0} \right\} \nonumber \\&+ \left\{ r_{y_0} (\tau -\mu ) + \lambda _y \delta _{\tau -\mu , 0} \right\} \left. \left\{ r_{y_0} (\tau +\nu ) + \lambda _y \delta _{\tau +\nu , 0} \right\} \right] \nonumber \\= & {} \lambda _y^2 \left[ \delta _{\mu ,\nu } + \delta _{\mu , 0} \delta _{\nu , 0} \right] + \lambda _y \left[ r_{y_0} (\mu -\nu ) + r_{y_0} (\nu -\mu ) \nonumber \right. \\&+ \left. r_{y_0} (\mu +\nu ) + r_{y_0} (-\nu -\mu ) \right] \nonumber \\&+ \sum _{\tau = - \infty } ^ {\infty } \left[ r_{y_0}(\tau +\mu -\nu ) r_{y_0}(\tau ) + r_{y_0} (\tau -\mu ) r_{y_0} (\tau +\nu ) \right] \; .\nonumber \\ \end{aligned}$$

(14.420)

Then it follows from Lemma A.10 that the sought matrix element can be computed as

$$\begin{aligned} \left( {\mathbf R}_{11} \right) _{\mu \nu } = \lambda _y^2 \left[ \delta _{\mu ,\nu } + \delta _{\mu , 0} \delta _{\nu , 0} \right] + 2 \lambda _y \left[ r_{y_0} (\mu -\nu ) + r_{y_0} (\mu +\nu ) \right] + \beta _{\mu -\nu } + \beta _{\mu +\nu } \; , \end{aligned}$$

(14.421)

which is (14.145).

Next consider the remaining matrix blocks of ${\mathbf R}$. As ${\mathbf R}$ is symmetric, it remains to find the elements of the block matrices ${\mathbf R}_{12},\ {\mathbf R}_{13},\ {\mathbf R}_{22}, {\mathbf R}_{23}$, and ${\mathbf R}_{33}$.

The block ${\mathbf R}_{12}$ can be written as

$$\begin{aligned} {\mathbf R}_{12}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ [ \hat{{\mathbf r}}_y - {\mathbf r}_y ][ \hat{{\mathbf r}}_u - {\mathbf r}_u ]^T \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \left[ \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ \varvec{\varphi }_y(t) y(t) \varvec{\varphi }^T_u (s) u(s) \right\} - {\mathbf r}_y {\mathbf r}^T_u \right] \; , \end{aligned}$$

(14.422)

where $\varvec{\varphi }_y(t)$ is as given by (14.411) and

$$\begin{aligned} \varvec{\varphi }_u(t) = \left( \begin{array}{ccc} u(t)&\dots&u(t-p_u) \end{array}\right) ^T \; . \end{aligned}$$

(14.423)

Proceeding as before one obtains

$$\begin{aligned} {\mathbf R}_{12} = \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }_y \varvec{\varphi }_u}(\tau ) r_{yu} (\tau ) + {\mathbf r}_{\varvec{\varphi }_y u}(\tau ) {\mathbf r}_{\varvec{\varphi }_u y}^T(-\tau ) \right] \; , \end{aligned}$$

(14.424)

compare (14.417). In contrast to the previous developments there will in this case be no contribution involving the measurement noise variances. A generic element ($0 \le \mu \le p_y,\ 0 \le \nu \le p_u$) of the matrix ${\mathbf R}_{12}$ can be written as

$$\begin{aligned} \left( {\mathbf R}_{12} \right) _{\mu \nu } = \sum _{\tau = -\infty }^{\infty } \left[ r_{y u} (\tau - \mu + \nu ) r_{y u} (\tau ) + r_{y u} (\tau - \mu ) r_{y u} (\tau + \nu ) \right] \; . \end{aligned}$$

(14.425)

Invoking Lemma A.10 and (14.148) one can write

$$\begin{aligned} \left( {\mathbf R}_{12} \right) _{\mu \nu } = \beta ^{(2)} _{-\mu + \nu } + \beta ^{(2)} _{\mu + \nu } \; , \end{aligned}$$

(14.426)

which is (14.147).

In a similar fashion one obtains for the block ${\mathbf R}_{13}$

$$\begin{aligned} {\mathbf R}_{13}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ [ \hat{{\mathbf r}}_y - {\mathbf r}_y ][ \hat{{\mathbf r}}_{yu} - {\mathbf r}_{yu} ]^T \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \left[ \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ \varvec{\varphi }_y(t) y(t) \varvec{\varphi }^T_{yu} (s) y(s) \right\} - {\mathbf r}_y {\mathbf r}^T_{yu} \right] \; , \nonumber \\ \end{aligned}$$

(14.427)

where this time $\varphi _y(t)$ is given by (14.411) and it holds

$$\begin{aligned} \varvec{\varphi }_{yu}(t) = \left( \begin{array}{ccc} u(t-p_1)&\dots&u(t-p_2) \end{array}\right) ^T \; . \end{aligned}$$

(14.428)

This leads, as above, to

$$\begin{aligned} {\mathbf R}_{13} = \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }_y \varvec{\varphi }_{yu} }(\tau ) r_{y} (\tau ) + {\mathbf r}_{\varvec{\varphi }_y y}(\tau ) {\mathbf r}_{\varvec{\varphi }_{yu} y}^T(-\tau ) \right] \; , \end{aligned}$$

(14.429)

compare (14.417). The generic element of ${\mathbf R}_{13}$ ($0 \le \mu \le p_y,\ p_1 \le \nu \le p_2$) can be written as

$$\begin{aligned} \left( {\mathbf R}_{13} \right) _{\mu \nu }= & {} \lambda _y r_{yu}(-\mu +\nu ) + \lambda _y r_{yu}( \mu +\nu ) \nonumber \\&+ \sum _{\tau = -\infty }^{\infty } \left[ r_{y u} (\tau - \mu + \nu ) r_{y_0} (\tau ) + r_{y u} ( \tau + \nu ) r_{y_0} (\tau - \mu ) \right] \nonumber \\= & {} \lambda _y [ r_{yu}(- \mu +\nu ) + r_{yu}( \mu + \nu ) ] + \beta ^{(3)}_{\nu -\mu } + \beta ^{(3)}_{\mu +\nu } \; , \end{aligned}$$

(14.430)

which is (14.149).

For the block ${\mathbf R}_{22}$ it holds

$$\begin{aligned} {\mathbf R}_{22}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ [ \hat{{\mathbf r}}_u - {\mathbf r}_u ][ \hat{{\mathbf r}}_u - {\mathbf r}_u ]^T \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \left[ \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ \varvec{\varphi }_u(t) u(t) \varvec{\varphi }^T_u (s) u(s) \right\} - {\mathbf r}_u {\mathbf r}^T_u \right] \; , \nonumber \\ \end{aligned}$$

(14.431)

where $\varvec{\varphi }_u(t)$ is as given by (14.423). Therefore

$$\begin{aligned} {\mathbf R}_{22} = \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }_u }(\tau ) r_{u} (\tau ) + {\mathbf r}_{\varvec{\varphi }_u u}(\tau ) {\mathbf r}_{\varvec{\varphi }_u u}^T(-\tau ) \right] \; . \end{aligned}$$

(14.432)

The generic element ($0 \le \mu \le p_u,\ 0 \le \nu \le p_u$) of the matrix ${\mathbf R}_{22}$ becomes

$$\begin{aligned} \left( {\mathbf R}_{22} \right) _{\mu \nu }= & {} \sum _{\tau = -\infty }^{\infty } \left[ \left( \lambda _u \delta _{\mu , \tau +\nu } + r_{ u_0}(\tau +\mu -\nu ) \right) \left( \lambda _u \delta _{\tau , 0} + r_{ u_0} (\tau ) \right) \right. \nonumber \\&+ \left. \left( \lambda _u \delta _{\tau -\mu , 0} + r_{ u_0} (\tau - \mu ) \right) \left( \lambda _u \delta _{-\tau -\nu , 0} + r_{u_0} (-\tau -\nu ) \right) \right] \nonumber \\= & {} \lambda _u^2 \left[ \delta _{\mu ,\nu } + \delta _{\mu , 0} \delta _{\nu , 0} \right] + 2 \lambda _u \left[ r_{u_0} (\mu -\nu ) + r_{u_0} (\mu +\nu ) \right] \nonumber \\&+ \sum _{\tau = -\infty }^{\infty } \left[ r_{ u_0}(\tau +\mu -\nu ) r_{ u_0} (\tau ) + r_{ u_0} (\tau - \mu ) r_{u_0} (-\tau -\nu ) \right] \nonumber \\= & {} \lambda _u^2 \left[ \delta _{\mu ,\nu } + \delta _{\mu , 0} \delta _{\nu , 0} \right] + 2 \lambda _u \left[ r_{u_0} (\mu -\nu ) + r_{u_0} (\mu +\nu ) \right] + \beta ^{(4)}_{\mu -\nu } + \beta ^{(4)}_{\mu +\nu } \; , \nonumber \\ \end{aligned}$$

(14.433)

which is (14.151) and where $\beta _k^{(4)}$ is given by (14.152).

The block ${\mathbf R}_{23}$ can be written as

$$\begin{aligned} {\mathbf R}_{23}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ [ \hat{{\mathbf r}}_u - {\mathbf r}_u ] [ \hat{{\mathbf r}}_{yu} - {\mathbf r}_{yu} ]^T \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \left[ \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ \varvec{\varphi }_u(t) u(t) \varvec{\varphi }^T_{yu} (s) y(s) \right\} - {\mathbf r}_u {\mathbf r}^T_{yu} \right] \; , \end{aligned}$$

(14.434)

where $\varvec{\varphi }_u (t)$ is as given by (14.423) and $\varvec{\varphi }_{yu}(t)$ is defined in (14.428).

Proceeding as before one obtains

$$\begin{aligned} {\mathbf R}_{23} = \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }_u \varvec{\varphi }_{yu}}(\tau ) r_{uy} (\tau ) + {\mathbf r}_{\varvec{\varphi }_u y}(\tau ) {\mathbf r}_{\varvec{\varphi }_{yu} u}^T(-\tau ) \right] \; . \end{aligned}$$

(14.435)

A generic element ($0 \le \mu \le p_u,\ p_1 \le \nu \le p_2$) of the matrix ${\mathbf R}_{23}$ can be written as

$$\begin{aligned} \left( {\mathbf R}_{23} \right) _{\mu \nu } = \sum _{\tau = -\infty }^{\infty } \left[ r_{u} (\tau - \mu + \nu ) r_{y u} ( -\tau ) + r_{y u} ( -\tau + \mu ) r_{ u} (\tau + \nu ) \right] \; . \end{aligned}$$

(14.436)

Invoking Lemma A.10 and (14.436) one can write

$$\begin{aligned} \left( {\mathbf R}_{23} \right) _{\mu \nu }= & {} \lambda _u \left[ r_{yu}(\nu -\mu ) + r_{yu}(\nu +\mu ) \right] \nonumber \\&+ \sum _{\tau = - \infty }^{\infty } \left[ r_{u_0} (\tau - \mu + \nu ) r_{yu}(-\tau ) + r_{yu}(-\tau + \mu ) r_{u_0} (\tau + \nu ) \right] \nonumber \\= & {} \lambda _u \left[ r_{yu}(\nu -\mu ) + r_{yu}(\nu +\mu ) \right] + \beta ^{(5)} _{-\mu + \nu } + \beta ^{(5)} _{\mu + \nu } \; , \end{aligned}$$

(14.437)

which is (14.153).

Finally, the block ${\mathbf R}_{33}$ can be written as

$$\begin{aligned} {\mathbf R}_{33}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ [ \hat{{\mathbf r}}_{yu} - {\mathbf r}_{yu} ][ \hat{{\mathbf r}}_{yu} - {\mathbf r}_{yu} ]^T \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } N \left[ \frac{1}{N^2} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ \varvec{\varphi }_{yu}(t) y(t) \varvec{\varphi }^T_{yu} (s) y(s) \right\} - {\mathbf r}_{yu} {\mathbf r}^T_{yu} \right] \; , \end{aligned}$$

(14.438)

where $\varvec{\varphi }_{yu}(t)$ is as given by (14.428). Proceeding again as before one obtains

$$\begin{aligned} {\mathbf R}_{33} = \sum _{\tau = -\infty }^{\infty } \left[ {\mathbf R}_{\varvec{\varphi }_{yu} }(\tau ) r_{y } (\tau ) + {\mathbf r}_{\varvec{\varphi }_{yu} y}(\tau ) {\mathbf r}_{\varvec{\varphi }_{yu} y}^T(-\tau ) \right] \; , \end{aligned}$$

(14.439)

compare (14.417). A generic element ($p_1 \le \mu \le p_2,\ p_1 \le \nu \le p_2$) of the matrix ${\mathbf R}_{33}$ can be written as

$$\begin{aligned} \left( {\mathbf R}_{33} \right) _{\mu \nu }= & {} \sum _{\tau = -\infty }^{\infty } \left[ r_{ u} (\tau - \mu + \nu ) r_{y } (\tau ) + r_{u y} (\tau - \mu ) r_{y u} (\tau + \nu ) \right] \nonumber \\= & {} \lambda _y \lambda _u \delta _{\mu ,\nu } + \lambda _u r_{y_0}(\mu -\nu ) + \lambda _y r_{u_0}(\nu -\mu ) \nonumber \\&+ \sum _{\tau = -\infty }^{\infty } \left[ r_{ u_0} (\tau - \mu + \nu ) r_{y_0 } (\tau ) + r_{u y} (\tau - \mu ) r_{y u} (\tau + \nu ) \right] \; . \nonumber \\ \end{aligned}$$

(14.440)

Invoking Lemma A.10, (14.156) and (14.157) one can write

$$\begin{aligned} \left( {\mathbf R}_{33} \right) _{\mu \nu } = \lambda _y \lambda _u \delta _{\mu ,\nu } + \lambda _u r_{y_0}(\mu -\nu ) + \lambda _y r_{u_0}(\nu -\mu ) + \beta ^{(6)} _{-\mu + \nu } + \gamma ^{(6)} _{\mu + \nu } \; , \end{aligned}$$

(14.441)

which is (14.155). This observation completes the proof.

14.1.3 14.E.3 Proof of Lemma 14.9

The key tool to use in the proof is Lemma A.12. As that lemma applies directly for any specific element of the matrix ${\mathbf R}$, the result follows directly from equation (A.80).

14.F Asymptotic Distribution for PEM and ML Estimates

14.1.1 14.F.1 Asymptotic Covariance Matrix of the Parameter Estimates

14.1.1.1 14.F.1.1 Proof of Lemma 14.11

It holds

$$ V_{\infty }'(\varvec{\vartheta }) = \mathsf{E} \{ -\ell _{\varvec{\varepsilon }}(\varvec{\varepsilon },\varvec{\vartheta },t) \varvec{\psi }^T(t,\varvec{\vartheta }) + \ell _{\varvec{\vartheta }} (\varvec{\varepsilon },\varvec{\vartheta }, t) \} \; . $$

Noting that $\varvec{\psi }(t,\varvec{\vartheta })$ depends on old data, has zero mean, and is independent of $\varvec{\varepsilon }(t,\varvec{\vartheta })$, and using (9.27) one finds that

$$\begin{aligned} V_{\infty }''(\varvec{\vartheta }_0) = \mathsf{E} \{ \varvec{\psi }(t,\varvec{\vartheta }_0) \ell _{\varvec{\varepsilon }\varvec{\varepsilon }} \varvec{\psi }^T(t,\varvec{\vartheta }_0) + \ell _{\varvec{\vartheta }\varvec{\vartheta }} \} \; , \end{aligned}$$

(14.442)

which is (14.168). Similarly,

$$\begin{aligned} {\mathbf P}_0= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{t=1}^N \sum _{s=1}^N \mathsf{E} \left\{ [ -\varvec{\psi }(t,\varvec{\vartheta }_0) \ell _{\varvec{\varepsilon }}^T(\varvec{\varepsilon }(t,\varvec{\vartheta }_0),\varvec{\vartheta }_0,t) + \ell ^T_{\varvec{\vartheta }} (\varvec{\varepsilon }(t,\varvec{\vartheta }_0),\varvec{\vartheta }_0,t) ] \right. \nonumber \\&\left. \times [ - \ell _{\varvec{\varepsilon }}(\varvec{\varepsilon }(s,\varvec{\vartheta }_0),\varvec{\vartheta }_0,s) \varvec{\psi }^T(s,\varvec{\vartheta }_0) + \ell _{\varvec{\vartheta }} (\varvec{\varepsilon }(s,\varvec{\vartheta }_0),\varvec{\vartheta }_0,s) ] \right\} \; . \end{aligned}$$

(14.443)

Note that the factors $\ell _{\varvec{\vartheta }}(t)$ and $\ell _{\varvec{\varepsilon }}(t)$ are uncorrelated in time and independent of $\varvec{\psi }(s)$ for $s \le t$. This gives that in the double sum (14.443) only terms with $t=s$ contribute. Cf also (9.27). Therefore

$$\begin{aligned} {\mathbf P}_0 = \mathsf{E} \left\{ \varvec{\psi }(t,\varvec{\vartheta }_0) \{ \mathsf{E} \ell _{\varvec{\varepsilon }} ^T \ell _{\varvec{\varepsilon }} \} \varvec{\psi }^T (t, \varvec{\vartheta }_0) \right\} + \mathsf{E} \{ \ell _{\varvec{\vartheta }} ^ T \ell _{\varvec{\vartheta }} \} \; , \end{aligned}$$

(14.444)

which is (14.169).

14.1.1.2 14.F.1.2 Proof of Corollary 14.2

Consider the criterion (9.28):

$$\begin{aligned} \ell (\varvec{\varepsilon },\varvec{\vartheta }, t) = \frac{1}{2} \log \det {\mathbf Q}(\varvec{\vartheta }) + \frac{1}{2} \varvec{\varepsilon }^T (t,\varvec{\vartheta }) {\mathbf Q}^{-1} (\varvec{\vartheta }) \varvec{\varepsilon }(t,\varvec{\vartheta }) \; . \end{aligned}$$

(14.445)

One needs to evaluate the expectations

$$ \mathsf{E} \left\{ \ell _{\varvec{\varepsilon }\varvec{\varepsilon }} \right\} ,\ \ \mathsf{E} \left\{ \ell _{\varvec{\vartheta }\varvec{\vartheta }} \right\} , \ \ \mathsf{E} \left\{ \ell _{\varvec{\varepsilon }} ^T \ell _{\varvec{\varepsilon }} \right\} , \ \ \mathsf{E} \left\{ \ell _{\varvec{\vartheta }} ^T \ell _{\varvec{\vartheta }} \right\} \; . $$

Set

$$\begin{aligned} {\mathbf Q}_{jk} = \frac{ \partial ^2}{\partial \varvec{\vartheta }_j \partial \varvec{\vartheta }_k} {\mathbf Q}\; . \end{aligned}$$

(14.446)

In this case

$$\begin{aligned}\ell _{\varvec{\varepsilon }}= & {} \varvec{\varepsilon }^T {\mathbf Q}^{-1} \; , \nonumber \\ \ell _{\varvec{\varepsilon }\varvec{\varepsilon }}= & {} {\mathbf Q}^{-1} \; , \nonumber \\ \ell _{\varvec{\vartheta }_j}= & {} \frac{1}{2} {\mathrm{tr}}[ {\mathbf Q}^{-1} {\mathbf Q}_j ] - \frac{1}{2} \varvec{\varepsilon }^T {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} \varvec{\varepsilon }\; , \nonumber \\ \ell _{\varvec{\vartheta }_j \varvec{\vartheta }_k}= & {} \frac{1}{2} {\mathrm{tr}}[ -{\mathbf Q}^{-1} {\mathbf Q}_k {\mathbf Q}^{-1} {\mathbf Q}_j + {\mathbf Q}^{-1} {\mathbf Q}_{jk} ] \\&- \frac{1}{2} \varvec{\varepsilon }^T [ -{\mathbf Q}^{-1} {\mathbf Q}_k {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} + {\mathbf Q}^{-1} {\mathbf Q}_{jk} {\mathbf Q}^{-1} -{\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} {\mathbf Q}_k {\mathbf Q}^{-1} ] \varvec{\varepsilon }\; . \nonumber \end{aligned}$$

Apparently,

$$\begin{aligned} \mathsf{E} \left\{ \ell _{\varvec{\varepsilon }\varvec{\varepsilon }} \right\}= & {} {\mathbf Q}^{-1} \; , \end{aligned}$$

(14.447)

$$\begin{aligned} \mathsf{E} \left\{ \ell _{\varvec{\varepsilon }}^T \ell _{\varvec{\varepsilon }} \right\}= & {} \mathsf{E} \left\{ {\mathbf Q}^{-1} \varvec{\varepsilon }\varvec{\varepsilon }^T {\mathbf Q}^{-1} \right\} = {\mathbf Q}^{-1} \; . \end{aligned}$$

(14.448)

Recall that

$$ \mathsf{E} \left\{ \varvec{\varepsilon }^T {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} \varvec{\varepsilon }\right\} = \mathsf{E} \left\{ \ {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} \varvec{\varepsilon }\varvec{\varepsilon }^T ) \right\} = {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_j ) \; , $$

so that $\mathsf{E} \left\{ \ell _{\varvec{\vartheta }_j} \right\} = {\mathbf 0}$ as it should. Next one finds

$$\begin{aligned} \mathsf{E} \left\{ \ell _{\varvec{\vartheta }_j} \ell _{ \varvec{\vartheta }_k} \right\}= & {} - \frac{1}{4} \left( {\mathrm{tr}}[ {\mathbf Q}^{-1} {\mathbf Q}_j ] \right) \left( {\mathrm{tr}}[ {\mathbf Q}^{-1} {\mathbf Q}_k ] \right) \nonumber \\&+ \frac{1}{4} \mathsf{E} \left\{ [ \varvec{\varepsilon }^T {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} \varvec{\varepsilon }] [ \varvec{\varepsilon }^T {\mathbf Q}^{-1} {\mathbf Q}_k {\mathbf Q}^{-1} \varvec{\varepsilon }] \right\} \nonumber \\= & {} \frac{1}{2} {\mathrm{tr}}[ {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} {\mathbf Q}_k ] \; , \end{aligned}$$

(14.449)

where Lemma A.9 is used.

Further,

$$\begin{aligned} \mathsf{E} \left\{ \ell _{\varvec{\vartheta }_j \varvec{\vartheta }_k} \right\}= & {} - \frac{1}{2} {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} {\mathbf Q}_k ) + \frac{1}{2} {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_{jk} ) \nonumber \\&+ {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} {\mathbf Q}_k ) - \frac{1}{2} {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_{jk} )\nonumber \\= & {} \frac{1}{2} {\mathrm{tr}}( {\mathbf Q}^{-1} {\mathbf Q}_j {\mathbf Q}^{-1} {\mathbf Q}_k ) \; . \end{aligned}$$

(14.450)

Finally, inserting (14.447)–(14.450) into (14.168), (14.169) gives (14.177).

14.1.2 14.F.2 Asymptotic Distribution for Frequency Domain ML Estimates

To fully describe the matrix ${\mathbf P}_{\varvec{\theta }}$, it is sufficient to have explicit expressions for ${\mathbf P}_1$ and ${\mathbf P}_2$, see (14.198), (14.199).

Introduce the notations

$$\begin{aligned} \begin{array}{l} A^{\mu } = \frac{ \partial A}{ \partial \varvec{\theta }_u}, \ \ B^{\mu } = \frac{ \partial G}{ \partial \varvec{\theta }_u}, \ \ F^{\mu } = \frac{ \partial F}{ \partial \varvec{\theta }_u} \; , \\ \mu = 1, \dots , n_a+n_b \; . \end{array} \end{aligned}$$

(14.451)

To proceed, differentiate (14.185) to get

$$\begin{aligned} r A^{\mu } A^* + r A A^{* \mu } + B^{\mu } B^* + B B^{* \mu } = F^{\mu } F^* + F F^{* \mu } \; . \end{aligned}$$

(14.452)

For the second-order derivatives, introduce the notation

$$\begin{aligned} F^{\mu \nu } = \frac{\partial }{\partial \varvec{\theta }_{\nu }} F^{\mu } = \frac{\partial ^2}{\partial \varvec{\theta }_{\nu } \partial \varvec{\theta }_{\mu } } F \; , \end{aligned}$$

(14.453)

and note that $A^{\mu \nu } = 0, \ B^{\mu \nu } = 0$, as $A^{\mu }$ and $B^{\mu }$ by construction are constant polynomials and do not depend on $\varvec{\theta }$. Differentiation of (14.452) with respect to $\varvec{\theta }_{\nu }$ gives

$$\begin{aligned} r A^{\mu } A^{* \nu } + r A^{\nu } A^{* \mu } + B^{\mu } B^{* \nu } + B^{\mu } B^{* \nu } = F^{\mu \nu } F^* + F^{\nu } F^{* \mu } + F^{\mu } F^{* \nu } + F F^{* \mu \nu } \; . \end{aligned}$$

(14.454)

Note that (14.452) and (14.454) can be seen as linear Diophantine equations for the unknown polynomials $F^{\mu }$ and $F^{\mu \nu }$, respectively. Equating the different powers of z leads to linear systems of equations in the unknown polynomial coefficients of $F^{\mu }$ and $F^{\mu \nu }$.

To compute ${\mathbf P}_1$ and ${\mathbf P}_2$, proceed as follows. First find F by the spectral factorization (14.185). Next find $F^{\mu }$ by solving the Diophantine equation (14.452) for $\mu = 1, \dots , n_a+n_b$ and $F^{\mu \nu }$ by solving the Diophantine equation (14.454) for $\mu , \nu = 1, \dots , n_a + n_b$.

Next examine the correlation between $\varepsilon (t)$ and $\varvec{\psi }(s)$, which appears in the third terms of (14.193). Differentiating equation (14.187) gives

$$\begin{aligned} \frac{ \partial }{\partial \varvec{\theta }_{\mu } } \varepsilon (t, \varvec{\theta })&= \frac{ A^{\mu }}{F} y(t) - \frac{ B^{\mu }}{F} u(t) - \frac{ F^{\mu }}{F^2} \left( A y(t) - B u(t) \right) , \nonumber \\&= \frac{ A^{\mu } F - A F^{\mu } }{F^2} \left( \frac{B}{A} u_0(t) + \tilde{y} (t) \right) - \frac{ B^{\mu } F - B F^{\mu } }{F^2} \left( u_0(t) + \tilde{u} (t) \right) \nonumber \\&= \frac{ A^{\mu }B - A B^{\mu } }{A F} u_0(t) + \frac{ A^{\mu } F - A F^{\mu } }{F^2} \tilde{y} (t) - \frac{ B^{\mu } F - B F^{\mu } }{F^2} \tilde{u} (t) \nonumber \\&{\mathop {=}\limits ^{\varDelta }} \frac{ A^{\mu } B - A B^{\mu } }{A F} u_0(t) + \frac{ \alpha ^{\mu } }{F^2} \tilde{y}(t) - \frac{ \beta ^{\mu } }{F^2} \tilde{u}(t) \; . \end{aligned}$$

(14.455)

By construction, the first term is always uncorrelated with $\varepsilon (t +\tau ,\varvec{\theta }_0)$ for any value of $\tau$.

Now, let specifically $\tau \ge 0$. Equations (14.195), (14.455) lead to

$$\begin{aligned} \mathsf{E} \left\{ \varepsilon (t+\tau , \varvec{\theta }_0) \frac{ \partial }{\partial \varvec{\theta }_{\mu } } \varepsilon (t, \varvec{\theta }_0) \right\}= & {} r \lambda _u \frac{1}{2 \pi i} \oint z^{-\tau } \frac{ A^*}{F^*} \frac{ A^{\mu } F - A F^{\mu } }{ F^2} \frac{dz}{z} \nonumber \\&\ \ \ + \lambda _u \frac{1}{2 \pi i} \oint z^{-\tau } \frac{ B^*}{F^*} \frac{ B^{\mu } F - B F^{\mu } }{ F^2} \frac{dz}{z} \; . \nonumber \\ \end{aligned}$$

(14.456)

Using (14.452) in (14.456) now leads to

$$ \mathsf{E} \left\{ \varepsilon (t+\tau , \varvec{\theta }_0) \frac{ \partial }{\partial \varvec{\theta }_{\mu } } \varepsilon (t, \varvec{\theta }_0) \right\} $$

$$\begin{aligned}= & {} \lambda _u \frac{1}{2 \pi i} \oint z^{-\tau } \frac{ 1 }{F^* F} \left( r A^* A^{\mu } + B^* B^{\mu } \right) \frac{dz}{z} \nonumber \\&\ \ + \lambda _u \frac{1}{2 \pi i} \oint z^{-\tau } \frac{ F^{\mu } }{F^* F^2} \underbrace{\left( - r A^* A - B^* B \right) } _{-F F^*} \frac{dz}{z} \nonumber \\= & {} \lambda _u \frac{1}{2 \pi i} \oint z^{-\tau } \frac{ 1}{F^* F} \left( - r A A^{* \mu } - B B^{* \mu } + F F^{* \mu } \right) \frac{dz}{z} \nonumber \\= & {} - \lambda _u \frac{1}{2 \pi i} \oint z^{\tau } \frac{ 1}{F^* F} \left( r A^* A^{ \mu } + B^* B^{ \mu } - F^* F^{ \mu } \right) \frac{dz}{z} \nonumber \\= & {} - \mathsf{E} \left\{ \varepsilon (t-\tau , \varvec{\theta }_0) \frac{\partial }{\partial \varvec{\theta }_{\mu } } \varepsilon (t, \varvec{\theta }_0) \right\} \; . \end{aligned}$$

(14.457)

When considering (14.457) specifically for $\tau = 0$ one finds that

$$\begin{aligned} \mathsf{E} \left\{ \varepsilon (t, \varvec{\theta }_0) \frac{\partial }{\partial \varvec{\theta }_{\mu } } \varepsilon (t, \varvec{\theta }_0) \right\} = 0 \; , \end{aligned}$$

(14.458)

which is though something already known (as $\varvec{\theta }_0$ is the minimizing point of $V_{31} (\varvec{\theta })$), see (14.189).

Introduce also the square Sylvester matrix of dimension $n_a + n_b$, cf. Sect. A.1.5

$$\begin{aligned} {\mathscr {S}}(B,-A) = \left( \begin{array}{cccccc} 0 &{} b_1 &{} \dots &{} b_{n_b} &{} &{} {\mathbf 0}\\ &{} &{} \ddots &{} &{} \ddots \\ &{} {\mathbf 0}&{} &{} b_1 &{} \dots &{} b_{n_b} \\ -1 &{} -a_1 &{} \dots &{} -a_{n_a} &{} &{} {\mathbf 0}\\ &{} \ddots &{} &{} &{} \ddots \\ &{} {\mathbf 0}&{} -1 &{} &{} \dots &{} -a_{n_a} \end{array}\right) \; , \end{aligned}$$

(14.459)

and the matrices

$$\begin{aligned} {\mathbf A}^{\mu } = \left( \begin{array}{c} \alpha ^1 \\ \vdots \\ \alpha ^{n_a+n_b} \end{array}\right) , \ \ \ {\mathbf B}^{\mu } = \left( \begin{array}{c} \beta ^1 \\ \vdots \\ \beta ^{n_a+n_b} \end{array}\right) \; , \end{aligned}$$

(14.460)

where, with some abuse of notations, the row vectors consist of the individual polynomial coefficients. Then it holds

$$\begin{aligned} \mathsf{E} \left\{ \varvec{\psi }(t) \varvec{\psi }^T(t) \right\}= & {} {\mathscr {S}}(B,-A) \ \ \mathrm{cov} \left( \begin{array}{c} \frac{1}{AF} u_0(t-1) \\ \vdots \\ \frac{1}{AF} u_0(t-n_a-n_b) \end{array}\right) {\mathscr {S}}^T (B,-A) \nonumber \\&+ r \lambda _u {\mathbf A}^{\mu } \mathrm{cov} \left( \begin{array}{c} \frac{1}{F^2} v(t-1) \\ \vdots \\ \frac{1}{F^2} v(t-n_a-n_b) \end{array}\right) {\mathbf A}^{\mu T} \nonumber \\&+ \lambda _u {\mathbf B}^{\mu } \mathrm{cov} \left( \begin{array}{c} \frac{1}{F^2} v(t-1) \\ \vdots \\ \frac{1}{F^2} v(t-n_a-n_b) \end{array}\right) {\mathbf B}^{\mu T} \; , \end{aligned}$$

(14.461)

where v(t) denotes a white noise process of unit variance.

To find a way to compute the second term in (14.198), differentiate (14.455), evaluate it for $\varvec{\theta }= \varvec{\theta }_0$, and neglect the part consisting of filtered $u_0(t)$ (recall that $\varepsilon (t,\varvec{\theta }_0)$ which is uncorrelated with any filtered version of $u_0(t)$). This procedure gives

$$\begin{aligned} \frac{\partial ^2}{\partial \varvec{\theta }_{\nu } \partial \varvec{\theta }_{\mu } } \varepsilon (t, \varvec{\theta })&= - \frac{A^{\mu } F^{\nu } }{ F^2} y(t) + \frac{B^{\mu } F^{\nu } }{ F^2} u(t) - \frac{F^{\mu \nu } }{ F^2} \left( A y(t) - B u(t) \right) \nonumber \\&\quad + \frac{ 2 F^{\mu } F^{\nu } }{ F^3} \left( A y(t) - B u(t) \right) - \frac{ F^{\mu } }{ F^2} \left( A^{\nu } y(t) - B^{\nu } u(t) \right) \nonumber \\&= \left( - \frac{A^{\mu } F^{\nu } }{ F^2} - \frac{A^{\nu } F^{\mu } }{ F^2} - \frac{A F^{\mu \nu } }{ F^2} + 2 \frac{A F^{\mu } F^{\nu } }{ F^3} \right) \tilde{y}(t) \nonumber \\&\quad + \left( \frac{B^{\mu } F^{\nu } }{ F^2} + \frac{B^{\nu } F^{\mu } }{ F^2} + \frac{B F^{\mu \nu } }{ F^2} - 2 \frac{B F^{\mu } F^{\nu } }{ F^3} \right) \tilde{u}(t) \nonumber \\&{\mathop {=}\limits ^{\varDelta }} \frac{ \alpha ^{\mu \nu } }{F^3} \tilde{y}(t) - \frac{ \beta ^{\mu \mu } }{F^3} \tilde{u}(t) \; . \end{aligned}$$

(14.462)

It is clear that the expression in (14.462) can be interpreted as the sum of two independent ARMA processes, driven by the white noise processes $\tilde{y}(t)$ and $\tilde{u}(t)$, respectively. To compute the second term in (14.198) is then a standard problem for computing cross-covariances.

The term $\mathsf{E} \left\{ \varepsilon (t) \varepsilon ^{''}(t) \right\}$ can thus be evaluated componentwise as

$$\begin{aligned} \mathsf{E} \left\{ \varepsilon (t) \varepsilon ^{''} (t) _{\mu \nu } \right\}= & {} r \lambda _u \mathsf{E} \left\{ \frac{A}{F} v(t) \frac{ \alpha ^{\mu \nu }}{F^3} v(t) \right\} + \lambda _u \mathsf{E} \left\{ \frac{B}{F} v(t) \frac{ \beta ^{\mu \nu }}{F^3} v(t) \right\} \; , \nonumber \\ \end{aligned}$$

(14.463)

where again v(t) is a white noise process of unit variance.

Next consider the sum in (14.199). As $\varepsilon (t)$ is uncorrelated of $u_0(s)$ for all t and s, it holds

$$\begin{aligned} \mathsf{E} \left\{ \varepsilon (t+\tau ) \varvec{\psi }_{\mu } (t) \right\}= & {} \mathsf{E} \left\{ \frac{A}{F} \tilde{y}(t+ \tau ) \frac{\alpha ^{\mu }}{F^2} \tilde{y}(t) \right\} + \mathsf{E} \left\{ \frac{B}{F} \tilde{u}(t+ \tau ) \frac{\beta ^{\mu }}{F^2} \tilde{u}(t) \right\} \; . \nonumber \\ \end{aligned}$$

(14.464)

Lemma A.10 can be conveniently applied for the evaluation of an infinite sum of cross-covariances. Using that lemma leads to

$$\begin{aligned}&\left[ \sum _{\tau = -\infty }^{\infty } \mathsf{E} \left\{ \varepsilon (t+\tau ) \varvec{\psi }(t) \right\} \mathsf{E} \left\{ \varepsilon (t-\tau ) \varvec{\psi }^T (t) \right\} \right] _{\mu \nu } \nonumber \\= & {} \sum _{\tau = -\infty }^{\infty } r_{\varepsilon \varvec{\psi }_{\mu } }(\tau ) r_{\varepsilon \varvec{\psi }_{\nu }^T } (-\tau ) \nonumber \\= & {} r^2 \lambda _u^2 \mathsf{E} \left\{ \frac{ \alpha ^{\mu } \alpha ^{\nu } }{F^4} v(t) \frac{ A^2}{F^2} v(t) \right\} + \lambda _u^2 \mathsf{E} \left\{ \frac{ \beta ^{\mu } \beta ^{\nu } }{F^4} v(t) \frac{ B^2}{F^2} v(t) \right\} \nonumber \\&+ r \lambda _u^2 \mathsf{E} \left\{ \frac{ \alpha ^{\mu } \beta ^{\nu } }{F^4} v(t) \frac{ A B }{F^2} v(t) \right\} + r \lambda _u^2 \mathsf{E} \left\{ \frac{ \alpha ^{\nu }\beta ^{\mu } }{F^4} v(t) \frac{ A B}{F^2} v(t) \right\} \; , \nonumber \\ \end{aligned}$$

(14.465)

with v(t) being a white noise of unit variance.

14.1.3 14.F.3 Asymptotic Distribution for the Extended ML Approach

The asymptotic distribution is characterized by the matrices ${\mathbf F}$ and ${\mathbf H}$ introduced in (14.206) and (14.208), respectively. To evaluate them, introduce the notations

$$\begin{aligned} W_k&{\mathop {=}\limits ^{\varDelta }} Y_k - G_k U_k = \tilde{Y}_k - G_k \tilde{U}_k \; , \end{aligned}$$

(14.466)

$$\begin{aligned} N_k&{\mathop {=}\limits ^{\varDelta }} r + |G_k|^2 \; , \end{aligned}$$

(14.467)

$$\begin{aligned} G_k^{\mu }&{\mathop {=}\limits ^{\varDelta }} \frac{ \partial G_k}{\partial \varvec{\theta }_{\mu } } \; . \end{aligned}$$

(14.468)

It then holds, cf. also Example A.3

$$\begin{aligned} V_{31}= & {} \frac{1}{N} \sum _k \frac{ |W_k|^2 }{ N_k } \; , \end{aligned}$$

(14.469)

$$\begin{aligned} \left( {\mathbf f}_1 \right) _{\mu }= & {} \frac{ \partial V_{31} }{\partial \varvec{\theta }_{\mu } } = \frac{1}{N} \sum _k \frac{1}{N_k} \left( -G_k^{\mu *} U_k^* W_k - W_k^* G_k^{\mu } U_k \right) \nonumber \\&- \frac{1}{N} \sum _k \frac{ |W_k|^2 }{N_k^2} \left( G_k^{\mu *} G_k + G_k^* G_k^{\mu } \right) \; , \end{aligned}$$

(14.470)

$$\begin{aligned} {\mathbf f}_2= & {} \frac{1}{N} \sum _k \frac{1}{N_k} - \frac{1}{N \lambda _u} \sum _k \frac{ |W_k|^2 }{ N_k^2 } \; , \end{aligned}$$

(14.471)

$$\begin{aligned} {\mathbf f}_3= & {} \lambda _u - V_{31} = \lambda _u - \frac{1}{N} \sum _k \frac{ |W_k|^2 }{N_k} \; , \end{aligned}$$

(14.472)

$$\begin{aligned} \mathsf{E} \left\{ W_k \right\}= & {} 0 \; , \end{aligned}$$

(14.473)

$$\begin{aligned} \mathsf{E} \left\{ |W_k|^2 \right\}= & {} \lambda _u N_k \; , \end{aligned}$$

(14.474)

$$\begin{aligned} \mathsf{E} \left\{ |W_j|^2 |W_k|^2 \right\}= & {} \left\{ \begin{array}{lll} \lambda _u^2 N_j N_k &{} &{} j \ne k \\ 2 \lambda _u^2 N_k^2 &{} &{} j = k \end{array}\right. \; , \end{aligned}$$

(14.475)

$$\begin{aligned} \mathsf{E} \left\{ W_k^2 \right\}= & {} 0 \; . \end{aligned}$$

(14.476)

Here the extension of Lemma A.9 to circular complex Gaussian variables was used. Another useful result is Lemma A.13 which implies

$$\begin{aligned} \mathsf{E} \left\{ U_j^* U_k \right\} = \left( \lambda _u + \phi _k \right) \delta _{j, k}, \ \ \ \phi _k = \phi ({\text {e}}^{{\mathrm{i}}\omega _k}) \; , \end{aligned}$$

(14.477)

where $\phi$ is the spectral density of the noise-free input $u_0(t)$. It thus also holds

$$\begin{aligned} \mathsf{E} \left\{ U_k W_k \right\}= & {} 0 \; , \end{aligned}$$

(14.478)

$$\begin{aligned} \mathsf{E} \left\{ U_k^* W_k \right\}= & {} -G_k \lambda _u \; . \end{aligned}$$

(14.479)

The two matrices ${\mathbf F}$ and ${\mathbf H}$ can be expressed in partitioned form as

$$\begin{aligned} {\mathbf F}= & {} \lim _{N \rightarrow \infty } \left( \begin{array}{ccc} \frac{ \partial {\mathbf f}_1}{ \partial \varvec{\theta }} &{} \frac{ \partial 2 {\mathbf f}_1}{ \partial r } &{} \frac{ \partial {\mathbf f}_1}{ \partial \lambda _u} \\ \\ \frac{ \partial {\mathbf f}_2}{ \partial \varvec{\theta }} &{} \frac{ \partial {\mathbf f}_2}{ \partial r } &{} \frac{ \partial {\mathbf f}_2}{ \partial \lambda _u} \\ \\ \frac{ \partial {\mathbf f}_3}{ \partial \varvec{\theta }} &{} \frac{ \partial {\mathbf f}_3}{ \partial r } &{} \frac{ \partial {\mathbf f}_3}{ \partial \lambda _u} \end{array}\right) {\mathop {=}\limits ^{\varDelta }} \left( \begin{array}{ccc} {\mathbf F}_{11} &{} {\mathbf F}_{12} &{} {\mathbf F}_{13} \\ {\mathbf F}_{21} &{} {\mathbf F}_{22} &{} {\mathbf F}_{23} \\ {\mathbf F}_{31} &{} {\mathbf F}_{32} &{} {\mathbf F}_{33} \end{array}\right) \; , \end{aligned}$$

(14.480)

$$\begin{aligned} {\mathbf H}= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \left( \begin{array}{c} {\mathbf f}_1 \\ {\mathbf f}_2 \\ {\mathbf f}_3 \end{array}\right) \left( \begin{array}{ccc} {\mathbf f}_1^T&{\mathbf f}_2&{\mathbf f}_3 \end{array}\right) \right\} {\mathop {=}\limits ^{\varDelta }} \left( \begin{array}{ccc} {\mathbf H}_{11} &{} {\mathbf H}_{12} &{} {\mathbf H}_{13} \\ {\mathbf H}_{21} &{} {\mathbf H}_{22} &{} {\mathbf H}_{23} \\ {\mathbf H}_{31} &{} {\mathbf H}_{32} &{} {\mathbf H}_{33} \end{array}\right) \; . \end{aligned}$$

(14.481)

Set

$$\begin{aligned} L_k^{\mu } = G_k^{\mu *}G_k + G_k^* G_k^{\mu } \; . \end{aligned}$$

(14.482)

The blocks of the matrix ${\mathbf F}$ can now be evaluated as follows.

$$\begin{aligned} \left( {\mathbf F}_{11} \right) _{\mu \nu }= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial \varvec{\theta }_{\nu } } \left( {\mathbf f}_1 \right) _{\mu } \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{N_k} \left( - G_k^{\mu \nu *} U_k^* W_k + G_k^{\mu *} U_k^* G_k^{\nu } U_k + G_k^{\nu *} U_k^* G_k^{\mu } U_k - W_k^* G_k^{\mu \nu } U_k \right) \nonumber \\&- \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{L_k^{\nu } }{N_k^2} \left( -G_k^{\mu *} U_k^* W_k - W_k^* G_k^{\mu } U_k \right) \nonumber \\&+ \lim _{N \rightarrow \infty } \frac{2}{N} \sum _k \frac{|W_k|^2 }{N_k^3} L_k^{ \nu }L_k^{ \mu } - \lim _{N \rightarrow \infty } \frac{\lambda _u}{N} \sum _k \frac{ L_k^{\mu } L_k^{\nu } }{N_k^2} - \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{ |W_k|^2 }{N_k^2} \frac{ \partial }{ \partial \varvec{\theta }_{\nu } } L_k^{ \mu } \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \left[ \right. \frac{1}{N_k} \left( G_k^{\mu \nu *} G_k \lambda _u + G_k^{\mu *} G_k^{\nu } \left( \lambda _u + \phi _k \right) + G_k^{\nu *} G_k^{\mu } \left( \lambda _u + \phi _k \right) \right. \nonumber \\&\left. + G_k^* G_k^{\mu \nu } \lambda _u \right) - \frac{L_k^{\nu } }{N_k^2} \left( G_k^{\mu *}G_k \lambda _u + G_k^* G_k^{\mu } \lambda _u \right) \left. + L_k^{\mu } L_k^{\nu } \frac{\lambda _u}{N_k^2} - \frac{\lambda _u}{N_k} \frac{\partial }{\partial \varvec{\theta }_{\nu } } L_k^{\mu } \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{\lambda _u}{N_k} \left[ \left( G_k^{\mu \nu *} G_k + G_k^{\mu *} G_k^{\nu } + G_k^{\nu *} G_k^{\mu } + G_k^* G_k^{\mu \nu } \right) - \frac{\lambda _u}{N_k} \frac{\partial }{\partial \varvec{\theta }_{\nu } } L_k^{\mu } \right] \nonumber \\&+ \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{\phi _k}{N_k} \left( G_k^{\mu *} G_k^{\nu } + G_k^{\nu *} G_k^{\mu } \right) \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{N_k} \left[ \left( G_k^{\mu \nu *} G_k \lambda _u + G_k^{\mu *} G_k^{\nu } \lambda _u + G_k^{\nu *} G_k^{\mu } \lambda _u + G_k^* G_k^{\mu \nu } \lambda _u \right) \right. \nonumber \\&- \left. \lambda _u \left( G_k^{\mu \nu *} G_k + G_k^{\mu *} G_k^{\nu } + G_k^{\nu *} G_k^{\mu } + G_k^* G_k^{\mu \nu } \right) \right] \nonumber \\&+ \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{\phi _k}{N_k} \left( G_k^{\mu *} G_k^{\nu } + G_k^{\nu *} G_k^{\mu } \right) \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{\phi _k}{N_k} \left( G_k^{\mu *} G_k^{\nu } + G_k^{\nu *} G_k^{\mu } \right) \; , \end{aligned}$$

(14.483)

$$\begin{aligned} \left( {\mathbf F}_{12} \right) _{\mu }= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial r} \left( {\mathbf f}_1 \right) _{\mu } \nonumber \\= & {} - \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{ 1 }{N_k^2} \left( - G_k^{\mu *} U_k^* W_k - W_k^* G_k^{\mu } U_k \right) + \lim _{N \rightarrow \infty } \frac{2}{N} \sum _k \frac{ |W_k|^2 }{N_k^3} L_k^{ \mu } \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \left[ - \frac{1}{N_k^2} \left( G_k^{\mu *} G_k \lambda _u + G_k^* G_k^{\mu } \lambda _u \right) + \frac{2}{N_k^3} \lambda _u N_k L_k^{\mu } \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{\lambda _u}{N} \sum _k \frac{L_k^{\mu } }{N_k^2} \; , \end{aligned}$$

(14.484)

$$\begin{aligned} \left( {\mathbf F}_{13} \right) _{\mu }= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial \lambda _u} \left( {\mathbf f}_1 \right) _{\mu } = 0 \; , \end{aligned}$$

(14.485)

$$\begin{aligned} \left( {\mathbf F}_{21} \right) _{\nu }= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial \varvec{\theta }_{\nu } } {\mathbf f}_2 \nonumber \\= & {} - \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{N_k^2} L_k^{ \nu } + \lim _{N \rightarrow \infty } \frac{2}{N \lambda _u} \sum _k \frac{ |W_k|^2}{ N_k^3} L_k^{ \nu } \nonumber \\&+ \lim _{N \rightarrow \infty } \frac{1}{N \lambda _u} \sum _k \frac{ 1}{ N_k^2} \left( U_k^* G_k^{\nu *} W_k + W_k^* G_k^{\nu } U_k \right) \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \left[ - \frac{L_k^{\nu }}{N_k^2} + \frac{2}{\lambda _u} \frac{\lambda _u N_k}{N_k^3} L_k^{\nu } + \frac{\lambda _u}{\lambda _u N_k^2} \left( - G_k^{\nu *} G_k - G_k^* G_k^{\nu } \right) \right] \nonumber \\= & {} 0 \; , \end{aligned}$$

(14.486)

$$\begin{aligned} {\mathbf F}_{22}= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial r } {\mathbf f}_2 \nonumber \\= & {} \lim _{N \rightarrow \infty } \left( - \frac{1}{N} \sum _k \frac{1}{N_k^2} + \frac{2}{N \lambda _u} \sum _k \frac{ |W_k|^2}{ N_k^3} \right) \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k - \frac{1}{N_k^2} + \frac{2}{\lambda _u} \frac{\lambda _u N_k}{N_k^3} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{N_k^2} \; , \end{aligned}$$

(14.487)

$$\begin{aligned} {\mathbf F}_{23}= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial \lambda _u } {\mathbf f}_2 \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N \lambda _u^2} \sum _k \frac{ |W_k|^2}{ N_k^2} = \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{\lambda _u^2} \frac{\lambda _u N_k}{N_k^2} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{\lambda _u N_k} \; , \end{aligned}$$

(14.488)

$$\begin{aligned} \left( {\mathbf F}_{31} \right) _{\nu }= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial \varvec{\theta }_{\nu } } {\mathbf f}_3 = 0 \; , \end{aligned}$$

(14.489)

$$\begin{aligned} {\mathbf F}_{32}= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial r } {\mathbf f}_3 \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{ |W_k|^2}{ N_k^2} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{\lambda _u}{N_k} \; , \end{aligned}$$

(14.490)

$$\begin{aligned} {\mathbf F}_{33}= & {} \lim _{N \rightarrow \infty } \frac{ \partial }{\partial \lambda _u } {\mathbf f}_3 = 1 \; . \end{aligned}$$

(14.491)

Hence the matrix ${\mathbf F}$ gets the structure

$$\begin{aligned} {\mathbf F}= \left( \begin{array}{ccc} {\mathbf F}_{11} &{} {\mathbf F}_{12} &{} {\mathbf 0}\\ {\mathbf 0}&{} {\mathbf F}_{22} &{} {\mathbf F}_{23} \\ {\mathbf 0}&{} {\mathbf F}_{32} &{} 1 \end{array}\right) \; . \end{aligned}$$

(14.492)

To proceed the analysis of the matrix ${\mathbf F}$, introduce the unique polynomial F as the stable, but not monic, factor through the spectral factorization

$$\begin{aligned} F F^* = r A A^* + B B^* \; . \end{aligned}$$

(14.493)

Then obviously

$$\begin{aligned} N_k = F_k F_k^* / (A_k A_k^*) \; . \end{aligned}$$

(14.494)

Further, as $G_k = B_k/A_k$ it holds

$$\begin{aligned} G_k^{\mu } = \frac{ B_k^{\mu } A_k - A_k^{\mu } B_k }{A_k^2} \; . \end{aligned}$$

(14.495)

Using (14.494) and (14.495) in (14.483) one gets

$$\begin{aligned} \left( {\mathbf F}_{11} \right) _{\mu \nu }= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{\phi _k}{ |F_k|^2} |A_k|^2 \left[ \left( \frac{ B_k^{\mu } A_k - A_k^{\mu } B_k }{ A_k^2} \right) ^* \left( \frac{ B_k^{\nu } A_k - A_k^{\nu } B_k }{ A_k^2} \right) \right. \nonumber \\&\left. + \left( \frac{ B_k^{\nu } A_k - A_k^{\nu } B_k }{ A_k^2} \right) ^* \left( \frac{ B_k^{\mu } A_k - A_k^{\mu } B_k }{ A_k^2} \right) \right] \; . \end{aligned}$$

(14.496)

Invoking Parseval’s relation and using the Sylvester matrix

$$\begin{aligned} {\mathscr {S}}(-B, A) = \left( \begin{array}{cccccc} 0 &{} -b_1 &{} \dots &{} -b_{n_b} &{} &{} {\mathbf 0}\\ &{} \ddots \\ &{} {\mathbf 0}&{} 0 &{} -b_1 &{} \dots &{} -b_{n_b} \\ 1 &{} a_1 &{} \dots &{} a_{n_a} &{} &{} {\mathbf 0}\\ &{} \ddots &{} \\ {\mathbf 0}&{} &{} 1 &{} a_1 &{} \dots &{} a_{n_a} \end{array}\right) \end{aligned}$$

(14.497)

it follows that

$$\begin{aligned} {\mathbf F}_{11} = 2 {\mathscr {S}}(-B, A)\ \mathrm{cov} \left( \frac{1}{ A(q^{-1}) F(q^{-1}) } \left( \begin{array}{c} u_0(t-1) \\ \vdots \\ u_0(t-n_a-n_b) \end{array}\right) \right) {\mathscr {S}}^T(-B, A) \; . \end{aligned}$$

(14.498)

The block matrix ${\mathbf F}_{11}$ is positive definite and hence non-singular. To verify that ${\mathbf F}$ itself is non-singular, it is therefore enough to ensure that

$$ {\mathbf F}_{22}-{\mathbf F}_{23} {\mathbf F}_{32} \ne 0 \; , $$

see (14.492). However,

$$\begin{aligned} {\mathbf F}_{22} {\mathbf F}_{33} - {\mathbf F}_{23} {\mathbf F}_{32} = \lim _{N \rightarrow \infty } \left[ \left( \frac{1}{N} \sum _k \frac{1}{N_k^2} \right) \left( \frac{1}{N} \sum _k 1 \right) - \left( \frac{1}{N} \sum _k \frac{1}{N_k} \right) ^2 \right] \; , \end{aligned}$$

(14.499)

which is positive due to Cauchy–Schwarz inequality. Thus the matrix ${\mathbf F}$ has been proved to be non-singular.

By calculations similar to going from (14.496)–(14.498) one can also show

$$\begin{aligned} {\mathbf F}_{12}= & {} 2 \lambda _u {\mathscr {S}}(-B, A) \mathsf{E} \left\{ \frac{1}{F^2(q^{-1})} \left( \begin{array}{c} v(t-1) \\ \vdots \\ v(t-n_a - n_b) \end{array}\right) \frac{A(q^{-1})B(q^{-1})}{F^2 (q^{-1})} v(t) \right\} \; , \nonumber \\ \end{aligned}$$

(14.500)

$$\begin{aligned} {\mathbf F}_{22}= & {} \mathsf{E} \left\{ \left[ \frac{A^2(q^{-1})}{F^2 (q^{-1})} v(t) \right] ^2 \right\} \; , \end{aligned}$$

(14.501)

$$\begin{aligned} {\mathbf F}_{23}= & {} \frac{1}{\lambda _u} \mathsf{E} \left\{ \left[ \frac{A(q^{-1})}{F (q^{-1})} v(t) \right] ^2 \right\} \; , \end{aligned}$$

(14.502)

$$\begin{aligned} {\mathbf F}_{32}= & {} \lambda _u \mathsf{E} \left\{ \left[ \frac{A(q^{-1})}{F (q^{-1})} v(t) \right] ^2 \right\} \; , \end{aligned}$$

(14.503)

where v(t) is white noise of zero mean and unit variance.

When evaluating the symmetric matrix ${\mathbf H}$, note that all the factors ${\mathbf f}_1$, ${\mathbf f}_2$, ${\mathbf f}_3$ have zero mean (for finite N), and this holds of course also when they are multiplied with a deterministic factor.

In order to evaluate the different blocks of ${\mathbf H}$, a number of auxiliary results are needed. It holds

$$\begin{aligned} \mathsf{E} \left\{ U_k^* W_k U_j^* W_j \right\}= & {} G_k G_j \lambda _u^2 (1 + \delta _{j, k}) \; , \end{aligned}$$

(14.504)

$$\begin{aligned} \mathsf{E} \left\{ U_k^* W_k U_j W_j^* \right\}= & {} G_k G_j^* \lambda _u^2 + \delta _{j, k} N_k \lambda _u (\phi _k + \lambda _u) \; , \end{aligned}$$

(14.505)

$$\begin{aligned} \mathsf{E} \left\{ U_k^* W_k W_j^* W_j \right\}= & {} -G_k \lambda _u^2 N_j - \delta _{j, k} N_j \lambda _u^2 G_j \; . \end{aligned}$$

(14.506)

Using (14.504)–(14.506) repeatedly, one gets

$$\begin{aligned} \left( {\mathbf H}_{11} \right) _{\mu \nu }= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \sum _j \mathsf{E} \left\{ \left[ \frac{1}{N_k} \left( -G_k^{\mu *} U_k^* W_k - W_k^* G_k^{\mu } U_k \right) - \frac{ |W_k|^2 }{N_k^2} L_k^{ \mu } \right] \right. \nonumber \\&\times \left. \left[ \frac{1}{N_j} \left( -G_j^{\nu *} U_j^* W_j - W_j^* G_j^{\nu } U_j \right) - \frac{ |W_j|^2 }{N_j^2} L_j^{ \nu } \right] \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \sum _j \left[ \right. \nonumber \\&\frac{1}{N_k N_j} \left[ G_k^{\mu *} G_j^{\nu *} G_k G_j \lambda _u^2 ( 1 + \delta _{j, k}) + G_k^{\mu } G_j^{\nu } G_k^* G_j^* \lambda _u^2 ( 1 + \delta _{j, k}) \right. \nonumber \\&\ + G_k^{\mu *} G_j^{\nu } \left( G_k G_j^* \lambda _u^2 + \delta _{j, k} N_k \lambda _u (\phi _k + \lambda _u ) \right) \nonumber \\&\left. + G_k^{\mu } G_j^{\nu * } \left( G_j G_k^* \lambda _u^2 + \delta _{j, k} N_j \lambda _u (\phi _j + \lambda _u ) \right) \right] \nonumber \\&- \frac{L_j^{\nu } }{N_k N_j^2} \left[ G_k^{\mu *} G_k \lambda _u^2 N_j \left( 1 + \delta _{j, k} \right) + G_k^{\mu } \left( G_k^* \lambda _u^2 N_j + \delta _{j, k} N_j \lambda _u^2 G_j^* \right) \right] \nonumber \\&- \frac{L_k^{\mu } }{N_k^2 N_j} \left[ G_j^{\nu *} G_j \lambda _u^2 N_k \left( 1 + \delta _{j, k} \right) + G_j^{\nu } \left( G_j^* \lambda _u^2 N_k + \delta _{j, k} N_k \lambda _u^2 G_k^* \right) \right] \nonumber \\&+ \left. \frac{ L_k^{ \mu } L_j^{ \nu } }{N_k^2 N_j^2 } \lambda _u^2 N _j N_k \left( 1 + \delta _{j, k} \right) \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \sum _j \left[ \frac{1}{N_k N_j}\right. \left[ L_k^{\mu } L_j^{\nu } \lambda _u^2 \left( 1 + \delta _{j,k} \right) \right. \nonumber \\&+ \left. \delta _{j, k} \phi _k \lambda _u N_k \left( G_k^{\mu *}G_k^{\nu } + G_k^{\mu } G_k^{\nu *} \right) \right] \nonumber \\&- \frac{L_j^{\nu }}{N_k N_j^2} \lambda _u^2 N_j L_k^{\mu } ( 1 + \delta _{j, k}) - \frac{L_k^{\mu }}{N_k^2 N_j} \lambda _u^2 N_k L_j^{\nu } ( 1 + \delta _{j, k}) \nonumber \\&+ \left. \frac{ L_k^{ \mu } L_j^{ \nu } }{N_k^2 N_j^2 } \lambda _u^2 N _j N_k \left( 1 + \delta _{j, k} \right) \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{ \lambda _u \phi _k}{N_k} \left( G_k^{\mu *} G_k^{\nu } + G_k^{\nu *} G_k^{\mu } \right) \; . \end{aligned}$$

(14.507)

By the procedure leading to (14.498) one then finds

$$\begin{aligned} {\mathbf H}_{11}= & {} {\mathscr {S}}(-B,A) {\mathbf P}{\mathscr {S}}^T(-B, A) \end{aligned}$$

(14.508)

$$\begin{aligned} {\mathbf P}= & {} 2 \lambda _u \mathrm{cov} \left( \frac{1}{A(q^{-1}) F(q^{-1})} \left( \begin{array}{c} u_0(t-1) \\ \vdots \\ u_0(t-n_a - n_b) \end{array}\right) \right) \; . \end{aligned}$$

(14.509)

Next examine the block ${\mathbf H}_{12}$:

$$\begin{aligned} \left( {\mathbf H}_{12} \right) _{\mu }= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \left[ \frac{1}{N} \sum _k \frac{1}{N_k} \left( -G_k^{\mu *} U_k^* W_k - W_k^* G_k^{\mu } U_k \right) - \frac{1}{N} \sum _k \frac{ |W_k|^2 }{N_k^2} L_k^{ \mu } \right] \right. \nonumber \\&\times \left. \left[ \frac{1}{N} \sum _j \frac{1}{N_j} - \frac{1}{N \lambda _u} \sum _j \frac{ |W_j|^2 }{ N_j^2 } \right] \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N \lambda _u} \sum _k \sum _j \left[ \frac{1}{N_k N_j^2} G_k^{\mu *} \left( - G_k \lambda _u^2 N_j ( 1 + \delta _{j, k} ) \right) \right. \nonumber \\&\left. + \frac{1}{N_k N_j^2 } G_k^{\mu } \left( - G_k^* \lambda _u^2 N_j ( 1 + \delta _{j, k} ) \right) + \frac{1}{N_k^2 N_j^2} L_k^{\mu } \lambda _u^2 N_k N_j ( 1 + \delta _{j, k} ) \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N \lambda _u} \sum _k \sum _j \frac{1}{N_k N_j} \lambda _u^2 ( 1 + \delta _{j, k} ) \left[ - G_k^{\mu *} G_k - G_k^{\mu } G_k^* + L_k^{\mu } \right] \nonumber \\= & {} 0 \; . \end{aligned}$$

(14.510)

This leads to

$$\begin{aligned} {\mathbf H}_{12} = {\mathbf 0}\; . \end{aligned}$$

(14.511)

The block ${\mathbf H}_{13}$ can be evaluated using very similar calculations as follows:

$$\begin{aligned} \left( {\mathbf H}_{13} \right) _{\mu }= & {} \lim _{N \rightarrow \infty } N \mathsf{E} \left\{ \left[ \frac{1}{N} \sum _k \frac{1}{N_k} \left( -G_k^{\mu *} U_k^* W_k - W_k^* G_k^{\mu } U_k \right) - \frac{1}{N} \sum _k \frac{ |W_k|^2 }{N_k^2} L_k^{ \mu } \right] \right. \nonumber \\&\times \left. \left[ \lambda _u - \frac{1}{N } \sum _j \frac{ |W_j|^2 }{ N_j } \right] \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N } \sum _k \sum _j \left[ \frac{1}{N_k N_j} G_k^{\mu *} \left( - G_k \lambda _u^2 N_j ( 1 + \delta _{j, k} ) \right) \right. \nonumber \\&\left. + \frac{1}{N_k N_j} G_k^{\mu } \left( - G_k^* \lambda _u^2 N_j ( 1 + \delta _{j, k} ) \right) + \frac{1}{N_k^2 N_j} L_k^{\mu } \lambda _u^2 N_k N_j ( 1 + \delta _{j, k} ) \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N } \sum _k \sum _j \frac{1}{N_k } \lambda _u^2 ( 1 + \delta _{j, k} ) \left[ - G_k^{\mu *} G_k - G_k^{\mu } G_k^* + L_k^{\mu } \right] \nonumber \\= & {} 0 \; , \end{aligned}$$

(14.512)

leading to

$$\begin{aligned} {\mathbf H}_{13} = {\mathbf 0}\; . \end{aligned}$$

(14.513)

Further calculations lead to

$$\begin{aligned} {\mathbf H}_{22}= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \left[ \frac{1}{N_k^2} - 2 \frac{\lambda _u N_k}{ \lambda _u N_k^3} + 2 \frac{\lambda _u^2 N_k^2}{ \lambda _u^2 N_k^4} \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \frac{1}{N_k^2} \ \ = \ \ \mathsf{E} \left\{ \left[ \frac{ A^2 (q^{-1} )}{F^2 (q^{-1}) } v(t) \right] ^2 \right\} \; , \end{aligned}$$

(14.514)

$$\begin{aligned} {\mathbf H}_{23}= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \mathsf{E} \left\{ \left[ \frac{\lambda _u}{N_k} - \frac{ |W_k|^2 }{ N_k^2 } - \frac{ \lambda _u |W_k|^2 }{ \lambda _u N_k^2 } + \frac{ |W_k|^4}{ \lambda _u N_k^3} \right] \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \left[ \frac{\lambda _u}{N_k} - \frac{\lambda _u N_k}{N_k^2} - \frac{\lambda _u^2 N_k}{\lambda _u N_k^2} + 2 \frac{\lambda _u^2 N_k^2}{\lambda _u N_k^3} \right] \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{\lambda _u}{N} \sum _k \frac{1}{N_k} \ \ = \ \ \mathsf{E} \left\{ \left[ \frac{ A (q^{-1} )}{F (q^{-1}) } v(t) \right] ^2 \right\} \; , \end{aligned}$$

(14.515)

$$\begin{aligned} {\mathbf H}_{33}= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \mathsf{E} \left\{ \lambda _u^2 - \frac{2 \lambda _u |W_k|^2 }{ N_k} + \frac{ |W_k|^4}{ N_k^2} \right\} \nonumber \\= & {} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _k \left( \lambda _u^2 - \frac{2 \lambda _u^2 N_k}{N_k} + \frac{ 2 \lambda _u^2 N_k^2}{N_k^2} \right) \ \ = \ \ \lambda _u^2 \; . \end{aligned}$$

(14.516)

14.G Asymptotic Distribution Results for Periodic Data

14.1.1 14.G.1 Proof of Lemma 14.12

It follows from the general theory of instrumental variable estimation for multivariable system, see Lemma 14.3, Söderström and Stoica (1983), Söderström and Stoica (1989), that the estimation error is asymptotically Gaussian distributed as

$$\begin{aligned} \sqrt{MN}(\hat{\varvec{\theta }}_\mathrm{EIV} -\varvec{\theta }_0) {\mathop {\longrightarrow }\limits ^\mathrm{dist}} \mathscr {N}({\mathbf 0}, {\mathbf P}_\mathrm{EIV}) \; , \end{aligned}$$

(14.517)

where

$$\begin{aligned} {\mathbf P}_\mathrm{EIV}= & {} {\mathbf P}({\mathbf W}) = ({\mathbf R}^{T} {\mathbf W}{\mathbf R})^{-1} {\mathbf R}^{T}{\mathbf W}{\mathbf C}_\mathrm{EIV} {\mathbf W}{\mathbf R}({\mathbf R}^{T} {\mathbf W}{\mathbf R})^{-1} \; , \end{aligned}$$

(14.518)

and

$$\begin{aligned} {\mathbf C}_\mathrm{EIV}= & {} \mathsf{E} \left\{ \left[ \sum _{j=0}^{\infty }{\mathbf Z}(t+j) {\mathbf H}_j\right] \varvec{\varLambda }\left[ \sum _{k=0}^{\infty } {\mathbf H}_k^{T} {\mathbf Z}^{T}(t+k) \right] \right\} \; . \end{aligned}$$

(14.519)

In (14.519) ${\{{\mathbf H}_{j}\}}_{j= 0}^{\infty }$ and $\varvec{\varLambda }$ are defined by a spectral factorization:

$$\begin{aligned} \varvec{\varPhi }_{{\mathbf V}}( \omega ) = {\mathbf H}({\text {e}}^{{\mathrm{i}}\omega }) \varvec{\varLambda }{\mathbf H}^{*}( {\text {e}}^{ - {\mathrm{i}}\omega }) \end{aligned}$$

(14.520)

along with the condition that ${\mathbf H}_0 = {\mathbf I},\ {\mathbf H}(q^{-1}) = \sum _{j=0}^{\infty } {\mathbf H}_{j} q^{-j}$ and ${\mathbf H}^{-1}(q^{-1})$ being asymptotically stable. In (14.520), $\varvec{\varPhi }_{V}(\omega )$ denotes the spectral density matrix of the vector ${\mathbf V}(t)$, see (12.20). Note that all quantities in (14.520) as well as ${\{{\mathbf H}_j\}}$ are $M \times M$ matrices.

Due to Assumption AI4, the measurement noise sequences in different periods are uncorrelated. Hence the different components $v_{j}(t)$ of ${\mathbf V}(t)$ are uncorrelated. Therefore the spectral density matrix $\varvec{\varPhi }_{{\mathbf V}}(\omega )$ is diagonal, and, in fact, its diagonal elements are equal. Now write this as

$$\begin{aligned} \varvec{\varPhi }_{{\mathbf V}}(\omega ) = \phi _{v}(\omega ) {\mathbf I}_M \; , \end{aligned}$$

(14.521)

and it follows that the spectral factorization of (14.520) can be substituted by a scalar spectral factorization:

$$\begin{aligned} \phi _{v}(\omega )= & {} H({\text {e}}^{ {\mathrm{i}}\omega }) \lambda H({\text {e}}^{-{\mathrm{i}}\omega }) \; , \end{aligned}$$

(14.522)

$$\begin{aligned} H(q^{-1})= & {} \sum _{k=0}^{\infty }h_{k}q^{-k}, \qquad h_0 =1 \; , \end{aligned}$$

(14.523)

and it follows that

$$\begin{aligned} \begin{array}{rcl} {\mathbf H}_k &{}=&{} h_{k} {\mathbf I}_M \; , \\ \varvec{\varLambda }&{}=&{} \lambda {\mathbf I}_M \; . \end{array} \end{aligned}$$

Therefore, the matrix ${\mathbf C}_\mathrm{EIV}$ in (14.519) can be simplified:

$$\begin{aligned} {\mathbf C}_\mathrm{EIV}= & {} \mathsf{E} \left\{ \left[ \sum _{j=0}^{\infty }h_{j} {\mathbf Z}(t+j)\right] \lambda {\mathbf I}_M \left[ \sum _{k=0}^{\infty }h_k {\mathbf Z}^{T}(t+k)\right] \right\} \nonumber \\= & {} \mathsf{E} \left\{ \lambda \sum _{j=0}^{\infty }\sum _{k=0}^{\infty } \left[ h_k {\mathbf Z}(t'-k)h_{j}{\mathbf Z}^{T}(t'-j)\right] \right\} \nonumber \\= & {} \lambda \mathsf{E} \left\{ \left[ H(q^{-1}){\mathbf Z}(t)\right] \left[ H(q^{-1}){\mathbf Z}(t)\right] ^{T} \right\} \; , \end{aligned}$$

(14.524)

which is (14.224).

14.1.2 14.G.2 Proof of Lemma 14.13

First recall that

$$\begin{aligned} {\mathbf C}_\mathrm{EIV}= & {} \lambda \mathsf{E} \left\{ \left[ H(q^{-1})\left[ {\mathbf z}_{1}(t) \dots {\mathbf z}_{M}(t)\right] \right] \right. \left. \left[ H(q^{-1}) \left[ {\mathbf z}_{1}(t) \dots {\mathbf z}_{M}(t)\right] \right] ^{T} \right\} \; . \nonumber \\ \end{aligned}$$

(14.525)

Split the instrumental vector ${\mathbf z}_{j}(t)$ into a noise-free part and a noise contribution as, see (12.8),

$$\begin{aligned} {\mathbf z}_{j}(t)= & {} {\mathbf z}_{j}^{0}(t) + \tilde{{\mathbf z}}_{j}(t) \nonumber \\= & {} {\mathbf g}_{L} \otimes \varvec{\varphi }_{0}(t) + \left( \begin{array}{c} \tilde{\varvec{\varphi }}_{j+1}(t) \\ \vdots \\ \tilde{\varvec{\varphi }}_{j+L}(t) \end{array} \right) \; . \end{aligned}$$

(14.526)

Using Assumption AI4 leads to

$$\begin{aligned} {\mathbf C}_\mathrm{EIV}= & {} \lambda \mathsf{E} \left\{ \left[ H(q^{-1}) \left( {\mathbf g}_{L} \otimes \varvec{\varphi }_0(t) \right) \right] \left[ H(q^{-1}) \left( {\mathbf g}_{L}^T \otimes \varvec{\varphi }_0^T(t) \right) \right] \right\} \nonumber \\&+ \lambda \mathsf{E} \left\{ \frac{1}{M} \sum _{j=1}^M \left[ H(q^{-1}) \left( \begin{array}{c} \tilde{\varvec{\varphi }}_{j+1} (t) \\ \vdots \\ \tilde{\varvec{\varphi }}_{j+L} (t) \\ \end{array}\right) \right] \right. \left. \left[ H(q^{-1}) \left( \begin{array}{ccc} \tilde{\varvec{\varphi }}_{j+1}^T (t)&\dots&\tilde{\varvec{\varphi }}_{j+L}^ T (t) \end{array}\right) \right] \right\} \nonumber \\= & {} \lambda {\mathbf g}_{L} {\mathbf g}_{L}^T \otimes {\mathbf C}_1 + \lambda {\mathbf I}_{L} \otimes {\mathbf C}_2 \; , \end{aligned}$$

(14.527)

which proves (14.228).

By using properties of the Kronecker product and the matrix inversion lemma

$$\begin{aligned} {\mathbf C}_\mathrm{EIV}^{-1}= & {} \left[ \lambda {\mathbf g}_{L}{\mathbf g}_{L}^{T}\otimes {\mathbf C}_1 +\lambda {\mathbf I}_{L}\otimes {\mathbf C}_2 \right] ^{-1} \nonumber \\= & {} \frac{1}{\lambda }[ {\mathbf I}_{L} \otimes {\mathbf C}_2 + ({\mathbf g}_{L}\otimes {\mathbf I}){(1\otimes {\mathbf C}_1)({\mathbf g}_{L}^T \otimes {\mathbf I})]^{-1}} \nonumber \\= & {} \frac{1}{\lambda } \left[ ( {\mathbf I}_{L}\otimes {\mathbf C}_2^{-1}) -({\mathbf I}_{L}\otimes {\mathbf C}_2^{-1})({\mathbf g}_{L}\otimes {\mathbf I}) \right. \nonumber \\&\times [(1\otimes {\mathbf C}_1)^{-1} +({\mathbf g}_{L}^T \otimes {\mathbf I}) ({\mathbf I}_{L}\otimes {\mathbf C}_2^{-1}) ({\mathbf g}_{L}\otimes {\mathbf I})]^{-1} \left. ({\mathbf g}_{L}^T \otimes {\mathbf I}) ({\mathbf I}_{L}\otimes {\mathbf C}_2^{-1}) \right] \nonumber \\= & {} \frac{1}{\lambda } \left[ ({\mathbf I}_{L}\otimes {\mathbf C}_2^{-1}) -({\mathbf g}_{L}\otimes {\mathbf C}_2^{-1}) [{\mathbf C}_1^{-1}+ L {\mathbf C}_2^{-1}]^{-1} ({\mathbf g}_{L}^{T}\otimes {\mathbf C}_2^{-1}) \right] \nonumber \\= & {} \frac{1}{\lambda } \left[ {\mathbf I}_{L}\otimes {\mathbf C}_2^{-1} - {\mathbf g}_{L}{\mathbf g}_{L}^T \otimes {\mathbf C}_2^{-1} [{\mathbf C}_1^{-1}+ L {\mathbf C}_2^{-1}]^{-1}{\mathbf C}_2^{-1} \right] \nonumber \\= & {} \frac{1}{\lambda } \left[ {\mathbf I}_L \otimes {\mathbf C}_2^{-1} - {\mathbf g}_{L}{\mathbf g}_{L}^{T} \otimes [{\mathbf C}_2 {\mathbf C}_1^{-1} {\mathbf C}_2 + L {\mathbf C}_2]^{-1} \right] \; . \end{aligned}$$

(14.528)

Using (14.528) in (14.226) gives

$$\begin{aligned} ({\mathbf P}_\mathrm{EIV}^{ \mathrm opt})^{-1}= & {} ({\mathbf g}_{L}^T \otimes {\mathbf R}_0) {\mathbf C}_\mathrm{EIV}^{-1} ({\mathbf g}_{L}\otimes {\mathbf R}_0) \nonumber \\= & {} \frac{1}{\lambda } \left[ {\mathbf g}_{L}^T {\mathbf g}_{L} \otimes ({\mathbf R}_0 {\mathbf C}_2^{-1}{\mathbf R}_0)-{\mathbf g}_{L}^{T} {\mathbf g}_{L}{\mathbf g}_{L}^{T}{\mathbf g}_{L} \right. \nonumber \\&\left. \otimes {\mathbf R}_0 [{\mathbf C}_2 {\mathbf C}_1^{-1} {\mathbf C}_2 + L {\mathbf C}_2]^{-1}{\mathbf R}_0 \right] \nonumber \\= & {} \frac{1}{\lambda } \left[ L ({\mathbf R}_0 {\mathbf C}_2^{-1}{\mathbf R}_0) - L ^2 {\mathbf R}_0 [{\mathbf C}_2 {\mathbf C}_1^{-1} {\mathbf C}_2 + L {\mathbf C}_2]^{-1}{\mathbf R}_0 \right] \nonumber \\= & {} \frac{L}{\lambda }{\mathbf R}_0 \left[ {\mathbf C}_2^{-1} - L [{\mathbf C}_2 {\mathbf C}_1^{-1} {\mathbf C}_2 + L {\mathbf C}_2]^{-1} \right] {\mathbf R}_0 \nonumber \\= & {} \frac{1}{\lambda }{\mathbf R}_0 \left[ {\mathbf C}_1 +\frac{{\mathbf C}_2}{L}\right] ^{-1}{\mathbf R}_0 \; . \end{aligned}$$

(14.529)

By inverting this (14.230) is finally obtained.

14.1.3 14.G.3 Proof of Corollary 14.3

For the IV variant of Example 12.3, when ${\mathbf W}= {\mathbf I}$, using (12.30) and (14.228), equation (14.223) shows

$$\begin{aligned} {\mathbf P}_\mathrm{EIV}({\mathbf I})= & {} \left( {\mathbf R}^T {\mathbf R}\right) ^{-1} {\mathbf R}^T {\mathbf C}{\mathbf R}\left( {\mathbf R}^T {\mathbf R}\right) ^{-1} \nonumber \\= & {} \left[ ({\mathbf g}_{L}\otimes {\mathbf R}_0)^T ({\mathbf g}_{L}\otimes {\mathbf R}_0)\right] ^{-1}({\mathbf g}_{L}\otimes {\mathbf R}_0)^{T} \nonumber \\&\times \lambda ({\mathbf g}_{L}{\mathbf g}_{L}^{T}\otimes {\mathbf C}_1 + {\mathbf I}_{L}\otimes {\mathbf C}_2) ({\mathbf g}_{L}\otimes {\mathbf R}_0)\left[ ({\mathbf g}_{L} \otimes {\mathbf R}_0)^T ({\mathbf g}_{L}\otimes {\mathbf R}_0)\right] ^{-1} \nonumber \\= & {} \lambda \left[ ({\mathbf g}_{L}^{T}{\mathbf g}_{L})^{-1}{\mathbf g}_{L}^{T}\otimes ({\mathbf R}_0^T {\mathbf R}_0)^{-1} {\mathbf R}_0^T\right] \nonumber \\&\times ({\mathbf g}_{L}{\mathbf g}_{L}^{T}\otimes {\mathbf C}_1 + {\mathbf I}_{L} \otimes {\mathbf C}_2) \left[ {\mathbf g}_{L}({\mathbf g}_{L}^{T}{\mathbf g}_{L})^{-1} \otimes {\mathbf R}_0({\mathbf R}_0^T {\mathbf R}_0)^{-1}\right] \nonumber \\= & {} \frac{\lambda }{L^2}[{\mathbf g}_{L}^{T}{\mathbf g}_{L}{\mathbf g}_{L}^{T} {\mathbf g}_{L}\otimes {\mathbf R}_0^{-1}{\mathbf C}_1 {\mathbf R}_0^{-1} + {\mathbf g}_{L}^{T}{\mathbf I}_{L}{\mathbf g}_{L}\otimes {\mathbf R}_0^{-1}{\mathbf C}_2 {\mathbf R}_0^{-1}] \nonumber \\= & {} \frac{\lambda }{L^2}[L^2 {\mathbf R}_0^{-1}{\mathbf C}_1 {\mathbf R}_0^{-1} + L {\mathbf R}_0^{-1}{\mathbf C}_2 {\mathbf R}_0^{-1}] \nonumber \\= & {} \lambda {\mathbf R}_0^{-1} \left( {\mathbf C}_1 + \frac{{\mathbf C}_2}{L}\right) {\mathbf R}_0^{-1} \; . \end{aligned}$$

(14.530)

Hence, (14.231) is proved.

14.1.4 14.G.4 Proof of Lemma 14.14

In order to compute the covariance matrix ${\mathbf P}_\mathrm{FML}$ one needs to find the first- and second -order derivatives of the loss function. Note that in (14.234) the parameter vector $\varvec{\theta }$ appears only in the factors $G_j$. A direct differentiation gives

$$\begin{aligned} \frac{ \partial V_N}{ \partial \varvec{\theta }_k}= & {} \frac{4 \pi }{N} \sum _{j = 0}^{N-1} \frac{1}{ \left( \lambda _y + G_j^* G_j \lambda _u \right) ^2 } \left[ \left( \lambda _y + G_j^* G_j \lambda _u \right) \left( -U_j^* \frac{ \partial G_j^* }{ \partial \varvec{\theta }_k } ( Y_j - G_j U_j) \right) \right. \nonumber \\&\ \ \ \left. - \frac{ \partial G_j^* }{ \partial \varvec{\theta }_k } G_j \lambda _u \left( Y_j^* - U_j^* G_j^* \right) \left( Y_j - G_j U_j \right) \right] \nonumber \\= & {} \frac{4 \pi }{N} \sum _{j = 0}^{N-1} \frac{1}{ \left( \lambda _y + G_j^* G_j \lambda _u \right) ^2 } \frac{ \partial G_j^* }{ \partial \varvec{\theta }_k } \left( Y_j - G_j U_j \right) \nonumber \\&\ \ \times \left[ - \left( \lambda _y + G_j^* G_j \lambda _u \right) U_j^* - G_j \lambda _u \left( Y_j^* - U_j^* G_j^* \right) \right] \nonumber \\= & {} \frac{4 \pi }{N} \sum _{j = 0}^{N-1} \beta _j(k) w_j \; , \end{aligned}$$

(14.531)

where

$$\begin{aligned} \beta _j(k)= & {} \frac{1}{ \left( \lambda _y + G_j^* G_j \lambda _u \right) ^2 } \frac{ \partial G_j^* }{ \partial \varvec{\theta }_k } \; , \end{aligned}$$

(14.532)

$$\begin{aligned} w_j= & {} \left( Y_j - G_j U_j \right) \left( - \lambda _y U_j^* - \lambda _u G_j Y_j^* \right) \; . \end{aligned}$$

(14.533)

Note that $\beta _j(k)$ is a deterministic quantity and that $w_j$ is random and has zero mean.

It holds that

$$\begin{aligned} \mathsf{E} \left\{ w_j w_j^* \right\}= & {} \mathsf{E} \left\{ \left( \tilde{Y}_j - G_j \tilde{U}_j \right) \left( \tilde{Y}_j^* - \tilde{U}_j^* G_j^* \right) \right. \nonumber \\&\left. \times \left( - \lambda _y U_j^* - \lambda _u G_j Y_j^* \right) \left( - \lambda _y U_j - \lambda _u Y_j G_j^* \right) \right\} \nonumber \\= & {} \mathsf{E} \left\{ \left( \tilde{Y}_j - G_j \tilde{U}_j \right) \left( \tilde{Y}_j^* - \tilde{U}_j^* G_j^* \right) \right\} \nonumber \\&\times \mathsf{E} \left\{ \left( - \lambda _y U_j^* - \lambda _u G_j Y_j^* \right) \left( - \lambda _y U_j - \lambda _u Y_j G_j^* \right) \right\} \nonumber \\= & {} \left( \lambda _y + G_j G_j^* \lambda _u \right) \left( \lambda _y^2 \phi _{0j} + \lambda _y^2 \lambda _u + \lambda _u^2 G_j G_j^* \left( G_j G_j ^* \phi _{0j} + \lambda _y \right) \right. \nonumber \\&\left. + \lambda _u \lambda _y G_j^* G_j \phi _{0j} + \lambda _u \lambda _y G_j G_j^* \phi _{0j} \right) \nonumber \\= & {} \left( \lambda _y + G_j G_j^* \lambda _u \right) \left[ \lambda _y \lambda _u \left( \lambda _y + G_j G_j^* \lambda _u \right) + \phi _{0j} \left( \lambda _y + G_j G_j^* \lambda _u \right) ^2 \right] \nonumber \\= & {} \left( \lambda _y + G_j G_j^* \lambda _u \right) ^2 \left[ \lambda _y \lambda _u + \phi _{0j} \left( \lambda _y + G_j G_j^* \lambda _u \right) \right] \; . \end{aligned}$$

(14.534)

Therefore

$$\begin{aligned} \mathsf{E} \left\{ \frac{ \partial V_N }{ \partial \varvec{\theta }_k} \frac{ \partial V_N }{ \partial \varvec{\theta }_{\ell } } \right\}= & {} \left( \frac{4 \pi }{N} \right) ^2 \sum _i \sum _j \mathsf{E} \left\{ \beta _i (k) w_i w_j^* \beta _j^* (\ell ) \right\} \nonumber \\= & {} \left( \frac{4 \pi }{N} \right) ^2 \sum _j \beta _j (k) \beta _j^* (\ell ) \mathsf{E} \left\{ w_j w_j ^* \right\} \nonumber \\= & {} \left( \frac{4 \pi }{N} \right) ^2 \sum _j \frac{1}{ \left( \lambda _y + G_j G_j^* \lambda _u \right) ^2 } \frac{ \partial G_j}{ \partial \varvec{\theta }_k} \frac{ \partial G_j^*}{ \partial \varvec{\theta }_{\ell } } \nonumber \\&\times \left[ \lambda _y \lambda _u + \phi _{0j} \left( \lambda _y + G_j G_j^* \lambda _u \right) \right] \; . \end{aligned}$$

(14.535)

Taking further derivatives in (14.531),

$$\begin{aligned} \frac{ \partial ^2 V_N}{ \partial \varvec{\theta }_k \partial \varvec{\theta }_{\ell } } = \frac{ 4 \pi }{N} \sum _j \left( \frac{ \partial \beta _j(k)}{ \partial \varvec{\theta }_{\ell } } w_j + \beta _j(k) \frac{ \partial w_j}{ \partial \varvec{\theta }_{\ell } } \right) \end{aligned}$$

(14.536)

leads to

$$\begin{aligned} \lim _{N \rightarrow \infty } \mathsf{E} \left\{ \frac{ \partial ^2 V_N}{ \partial \varvec{\theta }_k \partial \varvec{\theta }_{\ell } } \right\} = \lim _{N \rightarrow \infty } \frac{ 4 \pi }{N} \sum _j \beta _j(k) \mathsf{E} \left\{ \frac{ \partial w_j}{ \partial \varvec{\theta }_{\ell } } \right\} \; . \end{aligned}$$

(14.537)

The expectation in (14.537) can be evaluated as follows:

$$\begin{aligned} \mathsf{E} \left\{ \frac{ \partial w_j}{ \partial \varvec{\theta }_{\ell } } \right\}= & {} \mathsf{E} \left\{ - \frac{ \partial G_j}{ \partial \varvec{\theta }_{\ell } } U_j \left( - \lambda _y U_j^* - \lambda _u G_ j Y_j ^* \right) \right. \nonumber \\&\ \ \ + \left. \left( Y_j - G_j U_j \right) \left( - \lambda _u \frac{ \partial G_j}{ \partial \varvec{\theta }_{\ell } } Y_j ^* \right) \right\} \nonumber \\= & {} \frac{ \partial G_j}{ \partial \varvec{\theta }_{\ell } } \mathsf{E} \left\{ - U_j \left( - \lambda _y U_j^* - \lambda _u G_ j Y_j ^* \right) + \left( \tilde{Y}_j - G_j \tilde{U}_j \right) \left( - \lambda _u Y_j ^* \right) \right\} \nonumber \\= & {} \frac{ \partial G_j}{ \partial \varvec{\theta }_{\ell } } \left[ \lambda _y \left( \phi _{0j} + \lambda _u \right) + \lambda _u G_j G_j^* \phi _{0j} - \lambda _u \lambda _y \right] \nonumber \\= & {} \phi _{0j} \left( \lambda _y + G_j G_j^* \lambda _u \right) \frac{ \partial G_j}{ \partial \varvec{\theta }_{\ell } } \; . \end{aligned}$$

(14.538)

This gives

$$\begin{aligned} \lim _{N \rightarrow \infty } \mathsf{E} \left\{ \frac{ \partial ^2 V_N}{ \partial \varvec{\theta }_k \partial \varvec{\theta }_{\ell } } \right\}= & {} 2 \int _{-\pi }^{\pi } \frac{ 1 }{ \left( \lambda _y + G G^* \lambda _u \right) ^2 } \frac{ \partial G^*}{ \partial \varvec{\theta }_{k} } \frac{ \partial G}{ \partial \varvec{\theta }_{\ell } } \phi _{u_0} \left( \lambda _y + G G^* \lambda _u \right) {\mathrm{d}}\omega \nonumber \\= & {} 2 \int _{-\pi }^{\pi } \frac{ \phi _{u_0} }{ \left( \lambda _y + G G^* \lambda _u \right) } \frac{ \partial G^*}{ \partial \varvec{\theta }_{k} } \frac{ \partial G}{ \partial \varvec{\theta }_{\ell } } {\mathrm{d}}\omega \; . \end{aligned}$$

(14.539)

Combining the results (14.235), (14.535), and (14.539) gives the results (14.236)–(14.238).

14.H The Cramér–Rao Lower Bound for the Frequency Domain ML Problem

Applying the results (14.274)–(14.276) gives

$$\begin{aligned} {\mathbf J}_{1,1}= & {} \mathsf{E} \left\{ \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Real} \left( \tilde{{\mathbf U}} \right) \right] \right. \nonumber \\&\times \left. \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Real} \left( \tilde{{\mathbf U}} \right) \right] ^T \right\} \nonumber \\= & {} \frac{4}{r^2 \lambda _u^2} \mathsf{E} \left\{ \frac{1}{2} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \tilde{{\mathbf Y}}^* {\mathbf G}\right) \right\} + \frac{4}{\lambda _u^2} \mathsf{E} \left\{ \frac{1}{2} \mathrm{Real} \left( \tilde{{\mathbf U}} \tilde{{\mathbf U}}^* \right) \right\} \nonumber \\= & {} \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* {\mathbf G}\right) + \frac{2}{\lambda _u} {\mathbf I}\; , \end{aligned}$$

(14.540)

$$\begin{aligned} {\mathbf J}_{1,2}= & {} \mathsf{E} \left\{ \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Real} \left( \tilde{{\mathbf U}} \right) \right] \right. \nonumber \\&\times \left. \left[ \frac{2}{r \lambda _u} \mathrm{Imag} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Imag} \left( \tilde{{\mathbf U}} \right) \right] ^T \right\} \nonumber \\= & {} \frac{4}{r^2 \lambda _u^2} \mathsf{E} \left\{ -\frac{1}{2} \mathrm{Imag} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \tilde{{\mathbf Y}}^* {\mathbf G}\right) \right\} + \frac{4}{\lambda _u^2} \mathsf{E} \left\{ -\frac{1}{2} \mathrm{Imag} \left( \tilde{{\mathbf U}} \tilde{{\mathbf U}}^* \right) \right\} \nonumber \\= & {} - \frac{2}{r \lambda _u} \mathrm{Imag} \left( {\mathbf G}^* {\mathbf G}\right) - \frac{2}{\lambda _u} \mathrm{Imag}({\mathbf I}) = {\mathbf 0}\; , \end{aligned}$$

(14.541)

$$\begin{aligned} \left( {\mathbf J}_{1,3} \right) _{:,\mu }= & {} - \mathsf{E} \left\{ \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Real} \left( \tilde{{\mathbf U}} \right) \right] \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf U}_0^* {\mathbf G}^*_{\mu } \tilde{{\mathbf Y}} \right) \right] \right\} \nonumber \\= & {} - \frac{4}{r^2 \lambda _u^2} \mathsf{E} \left\{ \frac{1}{2} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \tilde{{\mathbf Y}}^* {\mathbf G}_{\mu } {\mathbf U}_0 \right) \right\} = - \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* {\mathbf G}_{\mu } {\mathbf U}_0 \right) \; , \nonumber \\ \end{aligned}$$

(14.542)

$$\begin{aligned} {\mathbf J}_{2,2}= & {} \mathsf{E} \left\{ \left[ \frac{2}{r \lambda _u} \mathrm{Imag} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Imag} \left( \tilde{{\mathbf U}} \right) \right] \right. \nonumber \\&\times \left. \left[ \frac{2}{r \lambda _u} \mathrm{Imag} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Imag} \left( \tilde{{\mathbf U}} \right) \right] ^T \right\} \nonumber \\= & {} \frac{4}{r^2 \lambda _u^2} \mathsf{E} \left\{ \frac{1}{2} \mathrm{Real} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \tilde{{\mathbf Y}}^* {\mathbf G}\right) \right\} + \frac{4}{\lambda _u^2} \mathsf{E} \left\{ \frac{1}{2} \mathrm{Real} \left( \tilde{{\mathbf U}} \tilde{{\mathbf U}}^* \right) \right\} \nonumber \\= & {} \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf G}^* {\mathbf G}\right) + \frac{2}{\lambda _u} {\mathbf I}\; , \end{aligned}$$

(14.543)

$$\begin{aligned} \left( {\mathbf J}_{2,3} \right) _{:,\mu }= & {} - \mathsf{E} \left\{ \left[ \frac{2}{r \lambda _u} \mathrm{Imag} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \right) + \frac{2}{\lambda _u} \mathrm{Imag} \left( \tilde{{\mathbf U}} \right) \right] \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf U}_0^* {\mathbf G}^*_{\mu } \tilde{{\mathbf Y}} \right) \right] \right\} \nonumber \\= & {} - \frac{4}{r^2 \lambda _u^2} \mathsf{E} \left\{ - \frac{1}{2} \mathrm{Imag} \left( {\mathbf G}^* \tilde{{\mathbf Y}} \tilde{{\mathbf Y}}^* {\mathbf G}_{\mu } {\mathbf U}_0 \right) \right\} = \frac{2}{r \lambda _u} \mathrm{Imag} \left( {\mathbf G}^* {\mathbf G}_{\mu } {\mathbf U}_0 \right) \; , \nonumber \\ \end{aligned}$$

(14.544)

$$\begin{aligned} \left( {\mathbf J}_{3,3} \right) _{\mu ,\nu }= & {} \mathsf{E} \left\{ \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf U}_0^* {\mathbf G}^*_{\mu } \tilde{{\mathbf Y}} \right) \right] \left[ \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf U}_0^* {\mathbf G}^*_{\nu } \tilde{{\mathbf Y}} \right) \right] \right\} \nonumber \\= & {} \frac{4}{r^2 \lambda _u^2} \mathsf{E} \left\{ \frac{1}{2} \mathrm{Real} \left( {\mathbf U}_0^* {\mathbf G}_{\mu }^* \tilde{{\mathbf Y}} \tilde{{\mathbf Y}}^* {\mathbf G}_{\nu } {\mathbf U}_0 \right) \right\} \nonumber \\= & {} \frac{2}{r \lambda _u} \mathrm{Real} \left( {\mathbf U}_0^* {\mathbf G}_{\mu }^* {\mathbf G}_{\nu } {\mathbf U}_0 \right) \; . \end{aligned}$$

(14.545)

To evaluate the remaining blocks, one needs also expressions for the fourth-order moments. It holds

$$\begin{aligned} \mathsf{E} \left\{ \left( \tilde{{\mathbf U}}^* \tilde{{\mathbf U}} \right) ^2 \right\}= & {} \mathsf{E} \left\{ \sum _{k=1}^N \sum _{j=1}^N \tilde{U}_k^* \tilde{U}_k \tilde{U}_j^* \tilde{U}_j \right\} \nonumber \\= & {} \sum _{k=1}^N \sum _{j=1}^N \left( \mathsf{E} \left\{ \tilde{U}_k^* \tilde{U}_k \right\} \mathsf{E} \left\{ \tilde{U}_j^* \tilde{U}_j \right\} + \mathsf{E} \left\{ \tilde{U}_k^* \tilde{U}_j \right\} \mathsf{E} \left\{ \tilde{U}_j^* \tilde{U}_k \right\} \right) \nonumber \\= & {} \sum _{k=1}^N \sum _{j=1}^N \left( \lambda _u^2 + \lambda _u^2 \delta _{j, k} \right) \nonumber \\= & {} \left( N^2 + N \right) \lambda _u^2 \; . \end{aligned}$$

(14.546)

Using (14.546) leads quickly to

$$\begin{aligned} {\mathbf J}_{4,4}= & {} \mathsf{E} \left\{ \left( -\frac{2 N}{\lambda _u} + \frac{1}{r \lambda _u^2} \tilde{{\mathbf Y}}^* \tilde{{\mathbf Y}} + \frac{1}{\lambda _u^2} \tilde{{\mathbf U}}^* \tilde{{\mathbf U}} \right) ^2 \right\} \nonumber \\= & {} \frac{4 N^2}{\lambda _u^2} + \frac{1}{r^2 \lambda _u^4} (N^2 + N) r^2 \lambda _u^2 + \frac{1}{\lambda _u^4} (N^2 + N) \lambda _u^2 \nonumber \\&- \frac{4 N}{r \lambda _u^3} N r \lambda _u - \frac{4N }{\lambda _u^3} N \lambda _u + \frac{2}{r \lambda _u^4} N^2 r \lambda _u^2 \nonumber \\= & {} \frac{2 N}{\lambda _u^2} \; , \end{aligned}$$

(14.547)

$$\begin{aligned} {\mathbf J}_{4,5}= & {} \mathsf{E} \left\{ \left( -\frac{2 N}{\lambda _u} + \frac{1}{r \lambda _u^2} \tilde{{\mathbf Y}}^* \tilde{{\mathbf Y}} + \frac{1}{\lambda _u^2} \tilde{{\mathbf U}}^* \tilde{{\mathbf U}} \right) \left( - \frac{N}{r} + \frac{1}{r^2 \lambda _u} \tilde{{\mathbf Y}}^* \tilde{{\mathbf Y}} \right) \right\} \nonumber \\= & {} \frac{2 N^2}{r \lambda _u} - \frac{N}{r^2 \lambda _u^2} N r \lambda _u - \frac{N}{r \lambda _u^2} N \lambda _u - \frac{2N}{r^2 \lambda _u^2} N r \lambda _u \nonumber \\&+ \frac{1}{r^3 \lambda _u^3} (N^2 + N) r^2 \lambda _u^2 + \frac{1}{r^2 \lambda _u^3} N^2 r \lambda _u^2 \nonumber \\= & {} \frac{N}{r \lambda _u} \; , \end{aligned}$$

(14.548)

$$\begin{aligned} {\mathbf J}_{5,5}= & {} \mathsf{E} \left\{ \left( -\frac{N}{r} + \frac{1}{r^2 \lambda _u} \tilde{{\mathbf Y}}^* \tilde{{\mathbf Y}} \right) ^2 \right\} \nonumber \\= & {} \frac{N^2}{r^2} + \frac{1}{r^4 \lambda _u^2} (N^2 + N) r^2 \lambda _u^2 - \frac{2 N}{r^3 \lambda _u} N r \lambda _u \nonumber \\= & {} \frac{N}{r^2} \; . \end{aligned}$$

(14.549)

Noting that ${\mathbf J}_{1,1} = {\mathbf J}_{2,2}$ is diagonal and ${\mathbf J}_{1,2} = {\mathbf 0}$, it is straightforward to evaluate the matrix ${\mathbf X}$ as

$$\begin{aligned} {\mathbf X}_{\mu ,\nu }= & {} \left( {\mathbf J}_{3,1} \right) _{\mu ,:} {\mathbf J}_{1,1}^{-1} \left( {\mathbf J}_{1,3} \right) _{:,\nu } + \left( {\mathbf J}_{3,2} \right) _{\mu ,:} {\mathbf J}_{2,2}^{-1} \left( {\mathbf J}_{2,3} \right) _{:,\nu } \nonumber \\= & {} \sum _k \frac{2}{r \lambda _u} \mathrm{Real} \left( G_k^* G_k^{\mu } U_{0,k} \right) \frac{1}{ \frac{2}{r \lambda _u} |G_k|^2 + \frac{2}{\lambda _u} } \frac{2}{r \lambda _u} \mathrm{Real} \left( G_k^* G_k^{\nu } U_{0,k} \right) \nonumber \\&+ \sum _k \frac{2}{r \lambda _u} \mathrm{Imag} \left( G_k^* G_k^{\mu } U_{0,k} \right) \frac{1}{ \frac{2}{r \lambda _u} |G_k|^2 + \frac{2}{\lambda _u} } \frac{2}{r \lambda _u} \mathrm{Imag} \left( G_k^* G_k^{\nu } U_{0,k} \right) \nonumber \\= & {} \frac{2}{r \lambda _u} \sum _k \frac{1}{|G_k|^2 + r} \left[ \mathrm{Real} \left( G_k^* G_k^{\mu } U_{0,k} \right) \mathrm{Real} \left( G_k^* G_k^{\nu } U_{0,k} \right) \right. \nonumber \\&+ \left. \mathrm{Imag} \left( G_k^* G_k^{\mu } U_{0,k} \right) \mathrm{Imag} \left( G_k^* G_k^{\nu } U_{0,k} \right) \right] \nonumber \\= & {} \frac{2}{r \lambda _u} \sum _k \frac{1}{|G_k|^2 + r} \mathrm{Real} \left( G_k^* G_k^{\mu } U_{0,k} \left( G_k^* G_k^{\nu } U_{0,k} \right) ^* \right) \; . \end{aligned}$$

(14.550)

Using (14.293), (14.294) leads to

$$\begin{aligned} {\mathbf X}_{\mu ,\nu } = \frac{2}{r \lambda _u} \sum _k \frac{ |A_k|^2 }{ |B_k|^2 + r |A_k|^2 } \mathrm{Real} \left( \frac{B_k^*}{A_k^*} \frac{ A_k B_k^{\mu } - A_k^{\mu } B_k}{A_k^2} U_{0,k} \frac{B_k}{A_k} \frac{ A_k^* B_k^{\nu *} - A_k^{\nu *} B_k^*}{ (A_k^2)^* } U_{0,k}^* \right) \; . \end{aligned}$$

(14.551)

Introduce the polynomial $F(q^{-1})$ of degree $\max (n_a, n_b)$, with all zeros inside the unit circle, from the spectral factorization, see also (14.185),

$$\begin{aligned} F F^* = r A A^* + B B^* \; . \end{aligned}$$

(14.552)

Note that the polynomial F is not monic. Paralleling the derivation of (14.297), (14.298) gives

$$\begin{aligned} \lim _{N \rightarrow \infty } \frac{1}{N} {\mathbf X}= & {} \frac{2}{r \lambda _u} {\mathscr {S}}(-B,A) \mathsf{E} \left\{ \varvec{\psi }(t) \varvec{\psi }^T(t) \right\} {\mathscr {S}}^T(-B, A) \; , \end{aligned}$$

(14.553)

$$\begin{aligned} \varvec{\psi }(t)= & {} \frac{B(q^{-1})}{ A^2(q^{-1}) F(q^{-1}) } \left( \begin{array}{c} u_0(t-1) \\ \vdots \\ u_0(t-n_a - n_b) \end{array}\right) \; . \end{aligned}$$

(14.554)

Note that due to (14.552) it holds

$$\begin{aligned} \left| \frac{B({\text {e}}^{{\mathrm{i}}\omega })}{F({\text {e}}^{{\mathrm{i}}\omega })} \right| < 1 \ \ \forall \omega \; , \end{aligned}$$

(14.555)

and

$$\begin{aligned} \varvec{\psi }(t) = \frac{B(q^{-1})}{F(q^{-1})} \varvec{\varphi }(t) \; , \end{aligned}$$

(14.556)

with $\varvec{\varphi }(t)$ introduced in (14.298). Therefore it holds

$$\begin{aligned} \mathsf{E} \left\{ \varvec{\psi }(t) \varvec{\psi }^T (t) \right\} < \mathsf{E} \left\{ \varvec{\varphi }(t) \varvec{\varphi }^T(t) \right\} \; , \end{aligned}$$

(14.557)

which leads to

$$\begin{aligned} {\mathbf J}_{3,3} > {\mathbf X} \end{aligned}$$

(14.558)

as desired.

An explicit calculation shows also, where $\varvec{\varPhi }_{\varvec{\varphi }}(\omega )$ denotes the spectrum of the vector $\varvec{\varphi }(t)$, (14.298),

$$\begin{aligned}\lim _{N \rightarrow \infty } \frac{1}{N} \left( {\mathbf J}_{3,3} - {\mathbf X}\right)= & {} \frac{2}{r \lambda _u} {\mathscr {S}}(-B, A) {\mathbf P}_0 {\mathscr {S}}^T (-B, A) \; , \nonumber \\ {\mathbf P}_0= & {} \mathsf{E} \left\{ \varvec{\varphi }(t) \varvec{\varphi }^T(t) \right\} - \mathsf{E} \left\{ \varvec{\psi }(t) \varvec{\psi }^T(t) \right\} \nonumber \end{aligned}$$

$$\begin{aligned}= & {} \int _{-\pi }^{\pi } \varvec{\varPhi }_{\varvec{\varphi }}(\omega ) {\mathrm{d}}\omega - \int _{-\pi }^{\pi } \left| \frac{ B({\text {e}}^{-{\mathrm{i}}\omega }) }{ F({\text {e}}^{-{\mathrm{i}}\omega })} \right| ^2 \varvec{\varPhi }_{\varvec{\varphi }}(\omega ) {\mathrm{d}}\omega \nonumber \\= & {} \int _{-\pi }^{\pi } r \left| \frac{ A({\text {e}}^{-{\mathrm{i}}\omega }) }{ F({\text {e}}^{-{\mathrm{i}}\omega })} \right| ^2 \varvec{\varPhi }_{\varvec{\varphi }}(\omega ) {\mathrm{d}}\omega \nonumber \\= & {} r \ \mathrm{cov} \left( \frac{1}{A(q^{-1}) F(q^{-1}) } \left( \begin{array}{c} u_0(t-1) \\ \vdots \\ u_0 (t- n_a -n_b) \end{array}\right) \right) \nonumber \\= & {} r \mathsf{E} \left\{ \frac{ A(q^{-1}) }{ F(q^{-1})} \varvec{\varphi }(t) \frac{ A(q^{-1}) }{ F(q^{-1})} \varvec{\varphi }^T(t) \right\} \; , \end{aligned}$$

(14.559)

which proves (14.305).