An iteratively reweighted least-squares approach to adaptive robust adjustment of parameters in linear regression models with autoregressive and t-distributed deviations

Abstract

In this paper, we investigate a linear regression time series model of possibly outlier-afflicted observations and autocorrelated random deviations. This colored noise is represented by a covariance-stationary autoregressive (AR) process, in which the independent error components follow a scaled (Student’s) t-distribution. This error model allows for the stochastic modeling of multiple outliers and for an adaptive robust maximum likelihood (ML) estimation of the unknown regression and AR coefficients, the scale parameter, and the degree of freedom of the t-distribution. This approach is meant to be an extension of known estimators, which tend to focus only on the regression model, or on the AR error model, or on normally distributed errors. For the purpose of ML estimation, we derive an expectation conditional maximization either algorithm, which leads to an easy-to-implement version of iteratively reweighted least squares. The estimation performance of the algorithm is evaluated via Monte Carlo simulations for a Fourier as well as a spline model in connection with AR colored noise models of different orders and with three different sampling distributions generating the white noise components. We apply the algorithm to a vibration dataset recorded by a high-accuracy, single-axis accelerometer, focusing on the evaluation of the estimated AR colored noise model.

Appendices

Appendix

A Used distributions and useful properties

This section reviews the main distributions as well as corresponding densities and expected values used throughout the paper. This compilation is partly based on Koch (1999), but also offers other results that are less well known. Throughout this section, $\mathcal {X}$ denotes any real-valued random variable (r.v.).

1.1 A. 1 The univariate, scaled (Student’s) t-distribution

If $\mathcal {X}$ follows a univariate, scaled Student’s t-distribution with $\nu >0$ degrees of freedom, location parameter $\mu$ and scale parameter $\sigma > 0$, symbolically $\mathcal {X} \sim t_{\nu }(\mu ,\sigma )$, then the density of $\mathcal {X}$ is given by

$$\begin{aligned} f_{\mathcal {X}}(x) = \frac{\mathrm {\Gamma }\left( \frac{\nu +1}{2} \right) }{\sqrt{\nu \pi \sigma ^2} \mathrm {\Gamma }\left( \frac{\nu }{2} \right) } \, \left[ 1 + \left( \frac{x-\mu }{\sigma } \right) ^2 / \nu \right] ^{-\frac{\nu +1}{2}}. \end{aligned}$$

(67)

Here $\mathrm {\Gamma }(.)$ denotes the gamma function (cf.  Abramowitz and Stegun 1972, p. 255). If $\nu > 1$, then the expectation of $\mathcal {X}$ is defined and equals $\mu$; if $\nu > 2$, then the variance of $\mathcal {X}$ is finite and equals $\frac{\nu }{\nu -2} \cdot \sigma ^2$ (cf.  Lange et al. 1989). Setting $\sigma = 1$ leads to the (unscaled) univariate Student’s t-distribution introduced by Student (Gosset WS) (1908).

1.2 A.2 The univariate gamma distribution

If $\mathcal {X}$ follows a univariate gamma distribution with parameters $a>0$ and $b>0$, symbolically $\mathcal {X} \sim G(a,b)$, then the density of $\mathcal {X}$ is given by

$$\begin{aligned} f_{\mathcal {X}}(x) = \left\{ \begin{array}{lll} \frac{b^a}{\mathrm {\Gamma }(a)} \, x^{a-1} \, e^{-bx} &{} \quad \mathrm {for} \; &{} x > 0 \\ 0 &{} \quad \mathrm {for} &{} x \le 0 \end{array} \right. \end{aligned}$$

(68)

The chi-squared distribution (say, with $\nu$ degrees of freedom) is a special case of the gamma distribution in the sense that $\chi ^2_{\nu } = G \left( \frac{1}{2}, \frac{\nu }{2} \right)$.

The moment generating function of G(a, b) is given by

$$\begin{aligned} M_{G(a,b)}(t) = \left( 1 - \frac{t}{b} \right) ^{-a}, \end{aligned}$$

(69)

which allows for the convenient determination of the expectation of $\mathcal {X}$ via the relationship $E_{\mathcal {X}} = M'_{G(a,b)}(0)$, where $M'(.)$ denotes the derivative of the moment generating function. It is easily verified that differentiation of (69) with respect to t and evaluation of that derivative for $t=0$ yields the value $\frac{a}{b}$ for the expectation of $\mathcal {X}$.

Another property we exploit in this paper is the following scaling property. If $s \ne 0$, then

$$\begin{aligned} s \cdot G (a,b) = G \left( \frac{b}{s}, a \right) . \end{aligned}$$

(70)

Using this property and the above-mentioned relationship between the gamma and the chi-squared distribution, we see that

$$\begin{aligned} G \left( \frac{\nu }{2}, \frac{\nu }{2} \right) = \frac{1}{\nu } \cdot G \left( \frac{1}{2}, \frac{\nu }{2} \right) = \frac{\chi ^2_{\nu }}{\nu }. \end{aligned}$$

Next, we show how the expected value of the natural logarithm of a gamma-distributed random variable can be obtained, that is,

$$\begin{aligned} E_{\mathcal {X}} \{ \log \mathcal {X} \}&= \int \limits _0^{\infty } \log x f_{\mathcal {X}}(x) dx. \end{aligned}$$

(71)

Using (68) and the substitution $z = bx$ (with $x > 0$, thus $z > 0$), we obtain

$$\begin{aligned} E_{\mathcal {X}} \{ \log \mathcal {X} \}&= \int \limits _0^{\infty } \log \left( \frac{z}{b} \right) \frac{b^a}{\mathrm {\Gamma }(a)} \, \left( \frac{z}{b} \right) ^{a-1} \, e^{-z} \frac{1}{b} dz \nonumber \\&= \int \limits _0^{\infty } (\log z - \log b) \frac{1}{\mathrm {\Gamma }(a)} z^{a-1} e^{-z} dz \nonumber \\&= \frac{1}{\mathrm {\Gamma }(a)} \int \limits _0^{\infty } \log (z) z^{a-1} e^{-z} dz \nonumber \\&\quad - \frac{\log b}{\mathrm {\Gamma }(a)} \int \limits _0^{\infty } z^{a-1} e^{-z} dz. \end{aligned}$$

(72)

The first integral takes the value $\mathrm {\Gamma }'(a)$ (cf. p. 331, for the derivatives of the gamma function Bronstein and Semendjajew 1991), and the second integral takes the value $\mathrm {\Gamma }(a)$ by definition of the gamma function. Thus, we obtain

$$\begin{aligned} E_{\mathcal {X}} \{ \log \mathcal {X} \} = \frac{\mathrm {\Gamma }'(a)}{\mathrm {\Gamma }(a)} - \log b = \psi (a) - \log b, \end{aligned}$$

(73)

where $\psi (.)$ denotes the digamma (or psi) function (cf. Abramowitz and Stegun 1972, p. 258f.). To evaluate the digamma function we used the standard MATLAB function call psi(0,x); an algorithm for this purpose is given in Bernardo (1976).

B Factorization of the likelihood function

We obtain for the argument of the exponential function in (31) the $N = (p+1)(p+2)/2$ terms

$$\begin{aligned} -\frac{1}{2\sigma ^2} \varvec{\ell }^T \mathbf {W} \varvec{\ell } \,+ \frac{\alpha _1}{\sigma ^2} \varvec{\ell }^T \mathbf {W} (L^1 \varvec{\ell }) +&\cdots + \frac{\alpha _p}{\sigma ^2} \varvec{\ell }^T \mathbf {W} (L^1 \varvec{\ell }) \\ - \frac{\alpha _1^2}{2\sigma ^2} (L^1 \varvec{\ell })^T \mathbf {W} (L^1 \varvec{\ell })\,-&\cdots - \frac{\alpha _1 \alpha _p}{\sigma ^2} (L^1 \varvec{\ell })^T \mathbf {W} (L^p \varvec{\ell })\quad \\&\ddots \\&\quad - \frac{\alpha _p^2}{2\sigma ^2} (L^p \varvec{\ell })^T \mathbf {W} (L^p \varvec{\ell }), \end{aligned}$$

the further N terms

$$\begin{aligned}&-\frac{1}{2\sigma ^2} \varvec{\xi }^T \mathbf {A}^T \mathbf {W} \mathbf {A} \varvec{\xi } + \frac{\alpha _1}{\sigma ^2} \varvec{\xi }^T\mathbf {A}^T \mathbf {W} (L^1 \mathbf {A}) \varvec{\xi } +\cdots \\&\quad + \frac{\alpha _p}{\sigma ^2} \varvec{\xi }^T \mathbf {A}^T \mathbf {W} (L^1 \mathbf {A}) \varvec{\xi } \\&\quad - \frac{\alpha _1^2}{2\sigma ^2} \varvec{\xi }^T (L^1 \mathbf {A})^T \mathbf {W} (L^1 \mathbf {A}) \varvec{\xi } - \cdots \\&\quad - \frac{\alpha _1 \alpha _p}{\sigma ^2} \varvec{\xi }^T (L^1 \mathbf {A})^T \mathbf {W} (L^p \mathbf {A}) \varvec{\xi } \\&\quad \ddots \\&\quad - \frac{\alpha _p^2}{2\sigma ^2} \varvec{\xi }^T (L^p \mathbf {A})^T \mathbf {W} (L^p \mathbf {A}) \varvec{\xi }, \end{aligned}$$

as well as the N terms

$$\begin{aligned}&-\frac{1}{2\sigma ^2} \varvec{\xi }^T \mathbf {A}^T \mathbf {W} \varvec{\ell } + \frac{\alpha _1}{\sigma ^2} \varvec{\xi }^T \mathbf {A}^T \mathbf {W} (L^1 \varvec{\ell }) + \cdots \\&\quad + \frac{\alpha _p}{\sigma ^2} \varvec{\xi }^T \mathbf {A}^T \mathbf {W} (L^1 \varvec{\ell }) \\&\quad - \frac{\alpha _1^2}{2\sigma ^2} \varvec{\xi }^T (L^1 \mathbf {A})^T \mathbf {W} (L^1 \varvec{\ell }) -\cdots \\&\quad - \frac{\alpha _1 \alpha _p}{\sigma ^2} \varvec{\xi }^T (L^1 \mathbf {A})^T \mathbf {W} (L^p \varvec{\ell }) \\&\ddots \\&\quad - \frac{\alpha _p^2}{2\sigma ^2} \varvec{\xi }^T (L^p \mathbf {A})^T \mathbf {W} (L^p \varvec{\ell }), \end{aligned}$$

and finally, the single term $\frac{\nu }{2} \varvec{1}_n^T \bar{\varvec{w}}$. All of the $3N + 1$ occurring scalar products are functions of the augmented data $\varvec{y} = (\varvec{u},\varvec{w})$ and do not involve any parameters. Therefore, they may be viewed as data-dependent functions, as defined by (33)–(36).

C Derivation of the M-step

1.1 C.1 Regarding the CM-step for $\varvec{\xi }$

Concerning the jth functional parameter $\xi _j$ ($j = 1, \ldots , m$), we seek the solution of

$$\begin{aligned} 0= & {} \frac{\partial }{\partial \xi _j}Q(\varvec{\theta } | \varvec{\theta }^{(k)}) \\= & {} - \frac{1}{2\sigma ^2} \sum \limits _{t=1}^n w_t^{(k)} \frac{\partial }{\partial \xi _j} \left[ \alpha (\varvec{L})(\ell _t - \mathbf {A}_t \varvec{\xi }) \right] ^2\\= & {} -\frac{1}{2\sigma ^2} \sum \limits _{t=1}^n w_t^{(k)} \cdot 2\left[ \alpha (\varvec{L})(\ell _t - \mathbf {A}_t \varvec{\xi }) \right] \\&\times \, \frac{\partial }{\partial \xi _j} \left[ -\alpha (\varvec{L})(\mathbf {A}_t \varvec{\xi }) \right] , \end{aligned}$$

or equivalently of

$$\begin{aligned} 0&= \sum \limits _{t=1}^n w_t^{(k)} \left[ \alpha (\varvec{L}) A_{t,j} \right] \left[ \alpha (\varvec{L})(\ell _t - \mathbf {A}_t \varvec{\xi }) \right] \\&= \left[ \alpha (\varvec{L}) A_{1,j} \, \cdots \, \alpha (\varvec{L}) A_{n,j} \right] \mathbf {W}^{(k)} \left[ \begin{array}{c} \alpha (\varvec{L})(\ell _1 - \mathbf {A}_1 \varvec{\xi }) \\ \vdots \\ \alpha (\varvec{L})(\ell _n - \mathbf {A}_n \varvec{\xi }) \end{array} \right] . \end{aligned}$$

Writing all m of these equations jointly, we find (43).

1.2 C.2 CM-step for $\varvec{\alpha }$

Recalling (42) and (2), with respect to the h-th autoregressive coefficient $\alpha _h$ ($h \in \{ 1,\ldots ,p \}$), we obtain for the first-order condition

$$\begin{aligned} 0&= \frac{\partial }{\partial \alpha _h} Q(\varvec{\theta }|\varvec{\theta }^{(k)}) \\&= -\frac{1}{2\sigma ^2} \sum \limits _{t=1}^n w_t^{(k)} \cdot \frac{\partial }{\partial \alpha _h} \left[ \varvec{\alpha }(L)(\ell _t - \mathbf {A}_t \varvec{\xi }) \right] ^2 \\&= -\frac{1}{2\sigma ^2} \sum \limits _{t=1}^n w_t^{(k)} \cdot 2\left[ \varvec{\alpha }(L)(\ell _t - \varvec{A}_t \varvec{\xi }) \right] \\&\quad \times \frac{\partial }{\partial \alpha _h} \left[ \varvec{\alpha }(L)(\ell _t - \varvec{A}_t \varvec{\xi }) \right] \\&= -\frac{1}{\sigma ^2} \sum \limits _{t=1}^n w_t^{(k)} \left[ e_t - \alpha _1 e_{t-1} - \cdots - \alpha _p e_{t-p} \right] \\&\quad \times \frac{\partial }{\partial \alpha _h} \left[ e_t - \alpha _1 e_{t-1} - \cdots - \alpha _p e_{t-p} \right] \\&= -\frac{1}{\sigma ^2} \sum \limits _{t=1}^n e_{t-h} \, w_t^{(k)} (e_t - \alpha _1 e_{t-1} - \ldots - \alpha _p e_{t-p}) \\&= -\frac{1}{\sigma ^2} \left[ e_{1-h} \, \cdots \, e_{n-h} \right] \mathbf {W}^{(k)}\\&\quad \times \left[ \begin{array}{c} e_1 - \alpha _1 e_0 - \ldots - \alpha _p e_{1-p} \\ \vdots \\ e_n - \alpha _1 e_{n-1} - \ldots - \alpha _p e_{n-p} \end{array} \right] . \end{aligned}$$

Writing all p equations in matrix notation, we thus have

$$\begin{aligned} \varvec{0}&= \left[ \begin{array}{ccc} e_0 &{} \cdots &{} e_{n-1} \\ \vdots &{} &{} \vdots \\ e_{1-p} &{} \cdots &{} e_{n-p} \end{array} \right] \mathbf {W}^{(k)}\\&\quad \times \left[ \begin{array}{c} e_1 - \alpha _1 e_0 - \ldots - \alpha _p e_{1-p} \\ \vdots \\ e_n - \alpha _1 e_{n-1} - \ldots - \alpha _p e_{n-p} \end{array} \right] . \end{aligned}$$

Denoting the solution of this equation system by $\varvec{\xi } = \varvec{\xi }^{(k+1)}$ and $\varvec{\alpha } = \varvec{\alpha }^{(k+1)}$, and using furthermore the notations (48)–(49), we find the normal equations

$$\begin{aligned} \varvec{0}&= (\mathbf {E}^{(k+1)})^T \mathbf {W}^{(k)}\\&\quad \times \left[ \begin{array}{c} e_1^{(k+1)} - \alpha _1^{(k+1)} e_0^{(k+1)} - \ldots - \alpha _p^{(k+1)} e_{1-p}^{(k+1)} \\ \vdots \\ e_n^{(k+1)} - \alpha _1^{(k+1)} e_{n-1}^{(k+1)} - \ldots - \alpha _p^{(k+1)} e_{n-p}^{(k+1)} \end{array} \right] \\&= (\mathbf {E}^{(k+1)})^T \mathbf {W}^{(k)} \left( \varvec{e}^{(k+1)} - \mathbf {E}^{(k+1)} \varvec{\alpha }^{(k+1)} \right) , \end{aligned}$$

leading to (50).

1.3 C.3 CM-step for $\sigma ^2$

Solving the first-order condition

$$\begin{aligned} 0&= \frac{\partial }{\partial \sigma ^2} Q(\varvec{\theta }|\varvec{\theta }^{(k)}) \\&= -\frac{n}{2\sigma ^2} + \frac{1}{2\sigma ^4} \sum \limits _{t=1}^n w_t^{(k)} \left[ \alpha (\varvec{L})(\ell _t - \mathbf {A}_t \varvec{\xi }) \right] ^2 \end{aligned}$$

for $\sigma ^2$ by using the already available estimates $\varvec{\xi }^{(k+1)}$, $\varvec{\alpha }^{(k+1)}$, $\varvec{e}^{(k+1)}$ and (51), we obtain (52).

1.4 C.4 CM-step for $\nu$

The first-order condition with respect to $\nu$ reads

$$\begin{aligned} 0&= \frac{\partial }{\partial \nu } Q(\varvec{\theta }|\varvec{\theta }^{(k)}) \\&= \frac{n}{2} \log \nu + \frac{n}{2} - \frac{n}{2} \cdot \frac{\mathrm {\Gamma }' \left( \frac{\nu }{2} \right) }{\mathrm {\Gamma }\left( \frac{\nu }{2} \right) } + \frac{n}{2} \left[ \psi \left( \frac{\nu ^{(k)}+1}{2} \right) \right. \\&\quad - \left. \log (\nu ^{(k)}+1) + \frac{1}{n} \sum \limits _{t=1}^n \left( \log w_t^{(k)} - w_t^{(k)} \right) \right] . \end{aligned}$$

Multiplying with 2 / n and using the definition of the digamma function, a zero $\nu ^{(k+1)}$ of this equation satisfies (53).

Next, we derive the CME-Step (55). Rewriting the log-likelihood function (14) in the form

$$\begin{aligned}&\log L(\varvec{\theta }; \varvec{\ell }) \\&\quad = -\frac{n}{2} \log \pi - \frac{n}{2} \log (\sigma ^2)\\&\qquad + n \log \mathrm {\Gamma }\left( \frac{\nu +1}{2} \right) - n \log \mathrm {\Gamma }\left( \frac{\nu }{2} \right) + \frac{n}{2} \nu \log \nu \\&\qquad - \frac{1}{2} (\nu + 1) \sum \limits _{t=1}^n \log \left[ \nu + \left( \frac{\alpha (\varvec{L}) (\ell _t-\mathbf {A}_t \varvec{\xi })}{\sigma } \right) ^2 \right] , \end{aligned}$$

we obtain for the first-order condition

$$\begin{aligned} 0&= \frac{\partial }{\partial \nu } \log L(\varvec{\xi }, \sigma ^2, \varvec{\alpha }, \nu ; \varvec{\ell }) \\&= \frac{n}{2} \psi \left( \frac{\nu +1}{2} \right) - \frac{n}{2} \psi \left( \frac{\nu }{2} \right) + \frac{n}{2} (\log \nu + 1) \\&\quad -\frac{1}{2} \sum \limits _{t=1}^n \log \left[ \nu + \left( \frac{\alpha (\varvec{L}) (\ell _t-\mathbf {A}_t \varvec{\xi })}{\sigma } \right) ^2 \right] \\&\quad -\frac{1}{2} (\nu +1) \sum \limits _{t=1}^n \left[ \nu + \left( \frac{\alpha (\varvec{L}) (\ell _t-\mathbf {A}_t \varvec{\xi })}{\sigma } \right) ^2 \right] ^{-1}. \end{aligned}$$

Substituting here $\varvec{\xi }^{(k+1)}$, $\varvec{\alpha }^{(k+1)}$, $(\sigma ^2)^{(k+1)}$, $\varvec{e}^{(k+1)}$ from (48), and finally $\varvec{u}^{(k+1)}$ from (51), it follows from the preceding equation that

$$\begin{aligned} 0&= \frac{n}{2} \log \nu \,+\, \frac{n}{2} - \frac{n}{2} \psi \left( \frac{\nu }{2} \right) + \frac{n}{2} \psi \left( \frac{\nu +1}{2} \right) {-} \frac{n}{2} \log (\nu {+}1) \\&\quad + \frac{n}{2} \cdot \frac{1}{n} \sum \limits _{t=1}^n \left( \log \left[ \frac{\nu + 1}{\nu + \left( \frac{\alpha ^{(k+1)}(\varvec{L})(\ell _t - \mathbf {A}_t \varvec{\xi }^{(k+1)})}{\sigma ^{(k+1)}} \right) ^2} \right] \right. \\&\quad \left. - \frac{\nu + 1}{\nu + \left( \frac{\alpha ^{(k+1)}(\varvec{L})(\ell _t - \mathbf {A}_t \varvec{\xi }^{(k+1)})}{\sigma ^{(k+1)}} \right) ^2} \right) . \end{aligned}$$

Using notation (54) and multiplying the preceding equation with 2 / n then yields finally (55).

D Data for the Monte Carlo simulation

See Tables 7 and 8.

Table 7 Values of the true AR(10) and AR(100) coefficients (the true coefficient of the AR(1) model is $\alpha _1 = 0.6828$)

Table 8 Values of the true Fourier coefficients (columns 2 and 3) and of the true spline coefficients (column 4)