Using SeDuMi to find various optimal designs for regression models

Abstract

We introduce a powerful and yet seldom used numerical approach in statistics for solving a broad class of optimization problems where the search space is discretized. This optimization tool is widely used in engineering for solving semidefinite programming (SDP) problems and is called self-dual minimization (SeDuMi). We focus on optimal design problems and demonstrate how to formulate A-, A$_s$-, c-, I-, and L-optimal design problems as SDP problems and show how they can be effectively solved by SeDuMi in MATLAB. We also show the numerical approach is flexible by applying it to further find optimal designs based on the weighted least squares estimator or when there are constraints on the weight distribution of the sought optimal design. For approximate designs, the optimality of the SDP-generated designs can be verified using the Kiefer–Wolfowitz equivalence theorem. SDP also finds optimal designs for nonlinear regression models commonly used in social and biomedical research. Several examples are presented for linear and nonlinear models.

Appendix: Proofs and MATLAB program

Proof of Theorem 1:

For the design problem in (6), we first note that there exists a $(q-r) \times q$ matrix $\mathbf{U}$ such that the matrix

$$\begin{aligned} \mathbf{D}=\left( \begin{array}{c} \mathbf{T} \\ \mathbf{U} \end{array} \right) _{q \times q} \end{aligned}$$

(18)

has rank q. This implies that $\mathbf{T}_r =(\mathbf{I}_r, \mathbf{0})$ and it is clear that $\mathbf{T}_r \mathbf{D}=\mathbf{T}$. Thus, problem (8) is the same as problem (6).

Next we claim that problem (9) is a SDP problem. By (5), all the elements of $\mathbf{A}(\mathbf{w})$ are linear functions of weights $w_1, \ldots , w_{N-1}$, so are the elements of $\mathbf{B}(\mathbf{w})$. From (10) and $\mathbf{W}_N$, all the elements of $\mathbf{B}_1, \ldots , \mathbf{B}_r$ and $\mathbf{W}_N$ are linear functions of $\mathbf{v}=(w_1, \ldots , w_{N-1}, v_N, \ldots , v_{N+r-1})^\top$, so the constraint in (9) is a linear matrix constraint. It is obvious that the objective function in (9) is a linear function of $\mathbf{v}$ and our claim holds.

Now we show that a solution to problem (9) provides a solution to problem (8). Since $\mathbf{B}(\mathbf{w})=\mathbf{D}^{-\top } \mathbf{A}(\mathbf{w}) \mathbf{D}^{-1}$, it is easy to verify that $\mathbf{T}_r \mathbf{D} (\mathbf{A}(\mathbf{w}))^{-1} \mathbf{D}^\top \mathbf{T}_r^\top = \mathbf{T}_r (\mathbf{B}(\mathbf{w}))^{-1} \mathbf{T}_r^\top$. Let $b_{ii}$ ($i=1, \ldots , q$) be the diagonal elements of $(\mathbf{B}(\mathbf{w}))^{-1}$. Then we have

$$\begin{aligned} \text{ trace } \left( \mathbf{T}_r \mathbf{D} (\mathbf{A}(\mathbf{w}))^{-1} \mathbf{D}^\top \mathbf{T}_r^\top \right) = \text{ trace } \left( \mathbf{T}_r (\mathbf{B}(\mathbf{w}))^{-1} \mathbf{T}_r^\top \right) = \sum _{i=1}^r b_{ii}. \end{aligned}$$

(19)

The constraints in (8) is equivalent to have $\mathbf{W}_N \succeq 0$. Thus, problem (8) is to minimize $\sum _{i=1}^r b_{ii}$ over the design weights subject to $\mathbf{W}_N \succeq 0$. By (10) and $\mathbf{B}_i \succeq 0$, we have

$$\begin{aligned} v_{N+i-1} \ge \mathbf{e}_i^\top \mathbf{B}(\mathbf{w}))^{-1} \mathbf{e}_i=b_{ii}, ~~i=1, \ldots , r. \end{aligned}$$

(20)

Since we minimize $\sum _{i=1}^r v_{N+i-1}$ in (9), a solution to (9) must have $v_{N+i-1}^* = b_{ii}$ from (20) and $\sum _{i=1}^r b_{ii}$ is minimized. By (9), the solution satisfies $\mathbf{W}_N \succeq 0$. It follows that if $\mathbf{v}^*=(w_1^*, \ldots , w_{N-1}^*, v_N^*, \ldots , v_{N+r-1}^*)^\top$ is a solution to problem (9), $\mathbf{w}^*=(w_1^*, \ldots , w_{N-1}^*, 1-\sum _{i=1}^{N-1} w_i^*)^\top$ is a solution to problem (8). $\square$

Proof of Lemma 1:

Let $\mathbf{w}_0$ and $\mathbf{w}_1$ be two weight vectors and $\alpha \in [0, 1]$, and define $\mathbf{w}_\alpha =(1-\alpha ) \mathbf{w}_0 + \alpha \mathbf{w}_1$. Assume $\mathbf{A}(\mathbf{w}_0)$ and $\mathbf{A}(\mathbf{w}_1)$ are nonsingular. We need to show that $\phi (\mathbf{w}_\alpha )$ is a convex function of $\alpha$. It is easy to get

$$\begin{aligned} \frac{\partial \phi (\mathbf{w}_\alpha )}{\partial \alpha }= & {} - \text{ trace } \left( \mathbf{T} \mathbf{A}^{-1}(\mathbf{w}_\alpha ) ( \mathbf{A}(\mathbf{w}_1) - \mathbf{A}(\mathbf{w}_0) )\mathbf{A}^{-1}(\mathbf{w}_\alpha ) \mathbf{T}^\top \right) , \\ \frac{\partial ^2 \phi (\mathbf{w}_\alpha )}{\partial \alpha ^2}= & {} 2 ~ \text{ trace } \left( \mathbf{T} \mathbf{A}^{-1}(\mathbf{w}_\alpha ) ( \mathbf{A}(\mathbf{w}_1) - \mathbf{A}(\mathbf{w}_0) ) \mathbf{A}^{-1}(\mathbf{w}_\alpha ) ( \mathbf{A}(\mathbf{w}_1)\right. \nonumber \\&\left. - \mathbf{A}(\mathbf{w}_0) ) \mathbf{A}^{-1}(\mathbf{w}_\alpha ) \mathbf{T}^\top \right) . \end{aligned}$$

Since the information matrices $\mathbf{A}(\mathbf{w}_0)$ and $\mathbf{A}(\mathbf{w}_1)$ are positive definite, $\mathbf{A}(\mathbf{w}_\alpha )$ is also positive definite. Then it is clear that $\frac{\partial ^2 \phi (\mathbf{w}_\alpha )}{\partial \alpha ^2} \ge 0$, which implies that $\phi (\mathbf{w}_\alpha )$ is a convex function of $\alpha$. $\square$

Proof of Lemma 2:

For any $\mathbf{w}$, define $\mathbf{w}_\alpha =(1-\alpha ) \hat{\mathbf{w}} + \alpha \mathbf{w}$. If $\hat{\mathbf{w}}$ is an optimal design, then we must have $\frac{\partial \phi (\mathbf{w}_\alpha )}{\partial \alpha } |_{\alpha =0} \ge 0$ for any $\mathbf{w}$. Similar to the proof of Lemma 1, we have

$$\begin{aligned} \frac{\partial \phi (\mathbf{w}_\alpha )}{\partial \alpha } \mid _{\alpha =0}= & {} - \text{ trace } \left( \mathbf{T} \mathbf{A}^{-1}(\hat{\mathbf{w}}) ( \mathbf{A}(\mathbf{w}) - \mathbf{A}(\hat{\mathbf{w}}) )\mathbf{A}^{-1}(\hat{\mathbf{w}}) \mathbf{T}^\top \right) , \\= & {} - \text{ trace } \left( \mathbf{T} \mathbf{A}^{-1}(\hat{\mathbf{w}}) \mathbf{A}(\mathbf{w})\mathbf{A}^{-1}(\hat{\mathbf{w}}) \mathbf{T}^\top \right) + \phi (\hat{\mathbf{w}}) \\= & {} - \text{ trace } \left( \mathbf{T} \mathbf{A}^{-1}(\hat{\mathbf{w}}) \sum _{i=1}^N w_i \mathbf{f}(\mathbf{u}_i) (\mathbf{f}(\mathbf{u}_i))^\top \mathbf{A}^{-1}(\hat{\mathbf{w}}) \mathbf{T}^\top \right) + \sum _{i=1}^N w_i \phi (\hat{\mathbf{w}}), \\& ~\text{ by } \text{(5) } \text{ and } \text{(11), } \\= & {} - \sum _{i=1}^N w_i \left( \phi _{Ai}(\hat{\mathbf{w}})- \phi (\hat{\mathbf{w}}) \right) , ~~~~~\text{ by } \text{(14) }, \\\ge & {} 0, ~~\text{ for } \text{ any } ~~\mathbf{w}, \end{aligned}$$

which leads to $\phi _{Ai}(\hat{\mathbf{w}})- \phi (\hat{\mathbf{w}}) \le 0$, for all $i=1, \ldots , N$. $\square$