Asymptotic properties of $M$-estimators in linear and nonlinear multivariate regression models

Abstract

We consider the (possibly nonlinear) regression model in $\mathbb{R }^q$ with shift parameter $\alpha$ in $\mathbb{R }^q$ and other parameters $\beta$ in $\mathbb{R }^p$. Residuals are assumed to be from an unknown distribution function (d.f.). Let $\widehat{\phi }$ be a smooth $M$-estimator of $\phi = {{\beta }\atopwithdelims (){\alpha }}$ and $T(\phi )$ a smooth function. We obtain the asymptotic normality, covariance, bias and skewness of $T(\widehat{\phi })$ and an estimator of $T(\phi )$ with bias $\sim n^{-2}$ requiring $\sim n$ calculations. (In contrast, the jackknife and bootstrap estimators require $\sim n^2$ calculations.) For a linear regression with random covariates of low skewness, if $T(\phi ) = \nu \beta$, then $T(\widehat{\phi })$ has bias $\sim n^{-2}$ (not $n^{-1}$) and skewness $\sim n^{-3}$ (not $n^{-2}$), and the usual approximate one-sided confidence interval (CI) for $T(\phi )$ has error $\sim n^{-1}$ (not $n^{-1/2}$). These results extend to random covariates.

Appendices

Appendix A

Here, we illustrate how to obtain the derivatives $\delta _{i, j, \ldots } (\theta ) = \partial _i \partial _j \cdots \delta (S_R)$ at $S_R = \theta _R$, where $\theta _R = \mathbb E [ S_R ]$ for $S_R$ of (3.11), where $i, j, \ldots$ range over $1, \ldots ,$ dimension $(S_R)$. The derivatives of order up to $2r$ are required to obtain $C_r$ of

$$\begin{aligned} \mathbb E \left[ \widehat{\phi } \right] \approx \phi + \sum _{r = 1}^\infty n^{-r} C_r, \end{aligned}$$

or more generally $D_r$ of $\mathbb E [ t (\widehat{\phi }) ] \approx t (\phi ) + \sum _{r = 1}^\infty n^{-r} D_r$. The derivatives of order up to $2r$ are also required to obtain an estimator of $\phi$ or $t (\phi )$ with bias $O (n^{-r-1})$. Theorem 9.1 gives formulae for them up to order four, so allowing bias reduction to $O (n^{-3})$ via (3.18).

Theorem 9.1

For $x, y, \ldots \in S_R$, set $(i_1, \ldots , i_r | x, y, \ldots ) = \partial _x \partial _y \cdots (\delta _{i_1} \cdots \delta _{i_r})$, where $\partial x = \partial / \partial x$ and $(\cdot )_0 = (\cdot )$ at $S_R = \theta _R$. Then $\{ (h | x_1, \ldots , x_r)_0 \} = \delta _{h \cdot i_1, \ldots , i_r} (\theta _R )$ for $r \le 4$ are given by

$$\begin{aligned} \left( h | x \right) _0 = -I \left( x = S_h \right) \end{aligned}$$

(9.1)

and

$$\begin{aligned} \left( h | x_1, \ldots , x_s \right) _0&= -\sum ^{s-1}_{r=1} \sum ^s_{x_1, \ldots , x_s} I \left( x_s = S_{h, i_1, \ldots , i_r} \right) \left. \left( i_1, \ldots , i_r \right| x_1, \ldots , x_{s-1} \right) _0 \nonumber \\&-\sum ^s_{r=2} \left( \mathbb E \left[ S_{h, i_1, \ldots , i_r} \right] \right) \left. \left( i_1, \ldots , i_r \right| x_1, \ldots , x_s \right) _0 \end{aligned}$$

(9.2)

for $s \ge 2$, where $I (A) = 1$ if $A$ is true, $I (A) = 0$ if $A$ is false, and

$$\begin{aligned} \displaystyle \left. \left( i_1, \ldots , i_r \right| x_1, \ldots , x_s \right) _0 = \left\{ \begin{array}{l@{\quad }l} 0, &{} \text{ for } s < r, \\ \displaystyle \sum _{x_1, \ldots , x_r}^{r!} \left. \left( i_1 \right| x_1 \right) _0 \cdots \left. \left( i_r \right| x_r \right) _0, \displaystyle &{} \text{ for } s=r. \end{array} \right. \qquad \end{aligned}$$

(9.3)

Also, in an obvious extension of the notation of (1.6),

$$\begin{aligned}&\left. \left( i_1, i_2 \right| x_1, x_2, x_3 \right) _0 = \sum ^6 \left. \left( i_1 \right| x_1, x_2 \right) _0 \left. \left( i_2 \right| x_3 \right) _0\!, \\&\left. \left( i_1, i_2 \right| x_1, \ldots , x_4 \right) _0 = \sum ^8 \left. \left( i_1 \right| x_1, x_2, x_3 \right) _0 \left. \left( i_2 \right| x_4 \right) _0 + \sum ^6 \left. \left( i_1 \right| x_1, x_2 \right) _0 \left. \left( i_2 \right| x_3, x_4 \right) _0\!,\\&\left. \left( i_1, i_2, i_3 \right| x_1, \ldots , x_4 \right) _0 = \sum ^{12} \left. \left( i_1 \right| x_1, x_2 \right) _0 \left. \left( i_2 \right| x_3 \right) _0 \left. \left( i_3 \right| x_4 \right) _0\!. \end{aligned}$$

Similarly, we can obtain $(i_1, \ldots , i_r | x_1, \ldots , x_s)_0$ from their values for $r=1$.

Proof

Differentiating (3.7) gives

$$\begin{aligned} (h|x) = -\sum _{r=0} \left\{ I \left( x = S_{h, i_1, \ldots , i_r} \right) \left( i_1, \ldots , i_r \right) + S_{h, i_1, \ldots , i_r} \left. \left( i_1, \ldots , i_r \right| x \right) \right\} \!, \end{aligned}$$

where $(i_1, \ldots , i_r) = 1$ for $r=0$, and

$$\begin{aligned} \left. \left( h \right| x_1, \ldots , x_s \right)&= -\sum _{r=1} \Bigg \{ \sum ^s_{x_1, \ldots , x_s} I \left( x_s = S_{h, i_1, \ldots , i_r} \right) \left. \left( i_1, \ldots , i_r \right| x_1, \ldots , x_{s-1} \right) \\&\qquad \qquad + S_{h, i_1, \ldots , i_r} \left. \left( i_1, \ldots , i_r \right| x_1, \ldots , x_s \right) \Bigg \} \end{aligned}$$

for $s \ge 2$, where

$$\begin{aligned} \sum ^s_{x_1, \ldots , x_s} a_{x_1, \ldots , x_{s-1}} b_{x_s} = \sum ^s_{j=1} a_{(x)_j} b_{x_j}, \end{aligned}$$

where $(x)_j = x_1, \ldots , x_s$ with $x_j$ deleted. So, (9.1) and (9.2) follow. Note that (9.3) follows since $\delta (\theta _R) = 0$. $\square$

Appendix B

Theorem 10.1

Suppose $Y_1, \ldots , Y_n$ are i.i.d in $\mathbb{R }^r$ with mean $\mu$ and finite covariance $V$.

• (I) Let $\{ a_{N, n} \} \subset \mathbb{R }^r$ satisfy

  $$\begin{aligned} \left( \max _N a'_{N, n} V a_{N, n} \right) / \sigma _n^2 \longrightarrow 0 \end{aligned}$$

  (10.1)

  as $n \rightarrow \infty$, where $\sigma _n^2 = \sum ^n_{N=1} a'_{N, n} V a_{N, n}$. Then $\sum ^n_{N = 1} a'_{N, n} Y_N$ is asymptotically normal with mean $\lim _{n \rightarrow \infty } \sum _{N = 1}^n a'_{N, n} \mu$ and variance $\lim _{n \rightarrow \infty } \sigma ^2_n$.

• (II) Let $\{ A_{N, n} \} \subset \mathbb{R }^{s \times r}$ satisfy

  $$\begin{aligned} \left( \max _N \text{ trace } A_{N, n} V A'_{N, n} \right) / \lambda _n \longrightarrow 0 \end{aligned}$$

  (10.2)

  as $n \rightarrow \infty$, where $\lambda _n$ is the minimum eigenvalue of $C_n = \sum ^n_{N = 1} A_{N, n} V A'_{N, n}$. Then $\sum ^n_{N = 1} A_{N, n} Y_N$ is asymptotically normal with mean $\lim _{n \rightarrow \infty } \sum ^n_{N = 1} A_{N, n} \mu$ and covariance $\lim _{n \rightarrow \infty } C_n$.

Suppose that $V$ is positive-definite. Then (10.1) holds if

$$\begin{aligned} \max _N \left| a_{N, n} \right| ^2 / \sum ^n_{N = 1} \left| a_{N, n} \right| ^2 \longrightarrow 0, \end{aligned}$$

(10.3)

and (10.2) holds if

$$\begin{aligned} \left( \max _N \text{ trace } A_{N, n} A'_{N, n} \right) / \left( \text{ min. } \text{ eigenvalue } \text{ of } \sum A_{N, n} A'_{N, n} \right) \longrightarrow 0. \qquad \end{aligned}$$

(10.4)

Proof

Suppose the minimum eigenvalue of $V$ is positive and (10.3) holds. Then the proof of (I) follows that given on page 153 of Hajek and Sidak (1967) for the case $r=1$. The result in (II) follows under (10.4) by the Cramer-Wold device. That (I), (II) hold under (10.1), (10.2) follows by writing $Y_j = V^{1/2} X_j$, where $\{ X_N \}$ are i.i.d with covariance $I_r$. $\square$