Minimax robust designs for regression models with heteroscedastic errors

Abstract

Minimax robust designs for regression models with heteroscedastic errors are studied and constructed. These designs are robust against possible misspecification of the error variance in the model. We propose a flexible assumption for the error variance and use a minimax approach to define robust designs. As usual it is hard to find robust designs analytically, since the associated design problem is not a convex optimization problem. However, we can show that the objective function of the minimax robust design problem is a difference of two convex functions. An effective algorithm is developed to compute minimax robust designs under the least squares estimator and generalized least squares estimator. The algorithm can be applied to construct minimax robust designs for any linear or nonlinear regression model with heteroscedastic errors. In addition, several theoretical results are obtained for the minimax robust designs.

Appendix: Proofs and derivations

Proof of Theorem 1:

For any $\lambda \in \Lambda (\alpha )$, we have

$$\begin{aligned} \lambda (\mathbf{u}_i) \le g(\mathbf{u}_i) + \alpha , ~~\text{ for } \text{ all } i =1, \ldots , N. \end{aligned}$$

Notice that $\mathbf{B}_i=\lambda (\mathbf{u}_i) \mathbf{A}_i$ and $\mathbf{A}_i$ are positive semidefinite for all $i =1, \ldots , N$. In the following notation $\succeq$ denotes Loewner order for positive semidefinite matrices. From (7) we get

$$\begin{aligned} \mathbf{C}_1(\mathbf{w}, \lambda )= & {} \left( \sum _{i=1}^N w_i \mathbf{A}_i \right) ^{-1} \left( \sum _{i=1}^N w_i \mathbf{B}_i \right) \left( \sum _{i=1}^N w_i \mathbf{A}_i \right) ^{-1} \\\preceq & {} \left( \sum _{i=1}^N w_i \mathbf{A}_i \right) ^{-1} \left( \sum _{i=1}^N w_i (g(\mathbf{u}_i) + \alpha )\mathbf{A}_i \right) \left( \sum _{i=1}^N w_i \mathbf{A}_i \right) ^{-1}, \text{ for } \text{ all } \lambda \in \Lambda (\alpha ), \end{aligned}$$

which leads to the result in (10), i.e.,

$$\begin{aligned} \phi _1(\mathbf{w})= & {} \max _{ \lambda \in \Lambda (\alpha )} \log (\det \left( \mathbf{C}_1(\mathbf{w}, \lambda ) \right) ) \\= & {} -2 \log ( \det \left( \sum _{i=1}^N w_i \mathbf{A}_i \right) ) + \log ( \det \left( \sum _{i=1}^N w_i (g(\mathbf{u}_i) + \alpha )\mathbf{A}_i \right) ). \end{aligned}$$

From Boyd and Vandenberghe (2004, page 387), both $-2 \log ( \det \left( \sum _{i=1}^N w_i \mathbf{A}_i \right) )$ and $-\log ( \det \left( \sum _{i=1}^N w_i (g(\mathbf{u}_i) + \alpha )\mathbf{A}_i \right) )$ are convex functions of $\mathbf{w}$, so $\phi _1(\mathbf{w})$ is a difference of two convex functions of $\mathbf{w}$. The result for $\phi _2(\mathbf{w})$ in (11) can be proved similarly. $\square$

Proof of Theorem 2

Suppose $\mathbf{w}^*$ is a solution to problem (12) with $\phi (\mathbf{w})=\phi _1(\mathbf{w})$. Let $\mathbf{w}$ be any weight vector and let $\mathbf{w}_{\delta } =(1-\delta ) \mathbf{w}^* + \delta \mathbf{w}$ for $\delta \in [0,1]$, so $\mathbf{w}_{\delta }$ is also a weight vector. Since $\phi _1(\mathbf{w})$ is minimized at $\mathbf{w}^*$, we must have

$$\begin{aligned} \frac{\partial \phi _1(\mathbf{w}_{\delta }) }{\partial \delta } \mid _{\delta =0} \ge 0, ~\text{ for } \text{ any } \mathbf{w}. \end{aligned}$$

Straightforward computation of the derivative leads to

$$\begin{aligned} \sum _{i=1}^N w_i \cdot \text{ trace }\left( \left[ 2 \mathbf{A}^{-1}(\mathbf{w}^*) - (g(\mathbf{u}_i) + \alpha ) \mathbf{A}_g^{-1}(\mathbf{w}^*) \right] \mathbf{A}_i \right) \le q, ~~\text{ for } \text{ any } \mathbf{w}, \end{aligned}$$

which gives the necessary condition,

$$\begin{aligned} d_1(\mathbf{u}_i) = \text{ trace }\left( \left[ 2 \mathbf{A}^{-1}(\mathbf{w}^*) - (g(\mathbf{u}_i) + \alpha ) \mathbf{A}_g^{-1}(\mathbf{w}^*) \right] \mathbf{A}_i \right) - q \le 0, ~~\text{ for } \text{ all }~~i=1, \ldots , N. \end{aligned}$$

Similarly we can prove the result when $\phi (\mathbf{w})=\phi _2(\mathbf{w})$. $\square$

The derivative $\frac{\partial v_2(\mathbf{w})}{\partial \mathbf{w}^\top }$ in (15): For $\phi (\mathbf{w}) = \phi _1(\mathbf{w})$, we have

$$\begin{aligned} v_2(\mathbf{w})= - \log ( \det \left( \sum _{i=1}^N w_i (g(\mathbf{u}_i) + \alpha )\mathbf{A}_i \right) ), \end{aligned}$$

and for $i=1, \ldots , N$,

$$\begin{aligned} \frac{\partial v_2(\mathbf{w})}{\partial w_i} = -~\text{ trace }\left( (g(\mathbf{u}_i) + \alpha ) \mathbf{A}_g^{-1}(\mathbf{w}) \mathbf{A}_i \right) . \end{aligned}$$

Similarly, for $\phi (\mathbf{w}) = \phi _2(\mathbf{w})$ we get

$$\begin{aligned} \frac{\partial v_2(\mathbf{w})}{\partial w_i} = -~\text{ trace }\left( \frac{(g(\mathbf{u}_i) + \alpha )}{g(\mathbf{u}_i)} \mathbf{D}_g^{-1}(\mathbf{w}) \mathbf{D}_i \right) , ~~i=1, \ldots , N. \end{aligned}$$

$\square$

Proof of Theorem 3

From (15) and $\lim _{j \rightarrow \infty } \mathbf{w}^{(j)} = \mathbf{w}^*$, we get

$$\begin{aligned} \lim _{j \rightarrow \infty } {\tilde{\phi }}(\mathbf{w}^{(j)}, \mathbf{w}^{(j-1)}) = v_1(\mathbf{w}^*) - v_2(\mathbf{w}^*) = \phi (\mathbf{w}^* ), \end{aligned}$$

and

$$\begin{aligned} \lim _{j \rightarrow \infty } \frac{\partial {\tilde{\phi }}(\mathbf{w}, \mathbf{w}^{(j-1)}) }{\partial \mathbf{w}^\top } \mid _{\mathbf{w}=\mathbf{w}^{(j)}}= \lim _{j \rightarrow \infty } \left( \frac{\partial v_1(\mathbf{w})}{\partial \mathbf{w}^\top } \mid _{\mathbf{w}=\mathbf{w}^{ (j)}} - \frac{\partial v_2(\mathbf{w})}{\partial \mathbf{w}^\top } \mid _{\mathbf{w}=\mathbf{w}^{ (j-1)}} \right) = \frac{\partial {\phi }(\mathbf{w})}{\partial \mathbf{w}^\top } \mid _{\mathbf{w}=\mathbf{w}^{*}}, \end{aligned}$$

which are the results in (18) and (19). To show that $\mathbf{w}^*$ satisfies the necessary condition in Theorem 2, we use the method in the proof of Theorem 2 and let $\mathbf{w}_{\delta } =(1-\delta ) \mathbf{w}^{(j)} + \delta \mathbf{w}$ for $\delta \in [0,1]$. Then we have

$$\begin{aligned} \frac{\partial {\tilde{\phi }}(\mathbf{w}_{\delta }, \mathbf{w}^{(j-1)}) }{\partial \delta } \mid _{\delta =0} \ge 0, ~\text{ for } \text{ any } \mathbf{w}, \end{aligned}$$

which leads to the necessary condition in Theorem 2 as $j \rightarrow \infty$. $\square$

Proof of Theorem 4

First we show that for each j, $\xi (\mathbf{w}^{(j)})$ is symmetric on $S_N$. For any distribution $\xi (\mathbf{w})$ on the $S_N$, we define $\xi (\tilde{\mathbf{w}})$ to be the following distribution,

$$\begin{aligned} \left( \begin{array}{cccc} -\mathbf{u}_1 &{} -\mathbf{u}_2 &{} \cdots &{} -\mathbf{u}_N \\ w_1 &{} w_2 &{} \cdots &{} w_N \end{array} \right) . \end{aligned}$$

If $\xi (\mathbf{w})=\xi (\tilde{\mathbf{w}})$, then $\xi (\mathbf{w})$ is symmetric on the $S_N$. Otherwise, the convex combination $0.5\xi (\mathbf{w})+ 0.5\xi (\tilde{\mathbf{w}})$ is symmetric on the $S_N$. Under the two conditions in Theorem 4, it is easy to verify that ${\tilde{\phi }}(\mathbf{w}, \mathbf{w}^{(j-1)}) = {\tilde{\phi }}(\tilde{\mathbf{w}}, \tilde{\mathbf{w}}^{(j-1)})$, if $\xi (\mathbf{w}^{(0)})$ is symmetric. Note that the choice of $\mathbf{w}^{(0)}$ in the algorithm always makes $\xi (\mathbf{w}^{(0)})$ to be symmetric. Since ${\tilde{\phi }}(\mathbf{w}, \mathbf{w}^{(j-1)})$ is a convex function of $\mathbf{w}$, the solution $\mathbf{w}^{(j)}$ makes $\xi (\mathbf{w}^{(j)})$ symmetric on the $S_N$. Since $\xi (\lim _{j \rightarrow \infty } \mathbf{w}^{(j)}) = \xi (\mathbf{w}^{*})$, $\xi (\mathbf{w}^{*})$ is also symmetric on the $S_N$. $\square$