Conditional selective inference for robust regression and outlier detection using piecewise-linear homotopy continuation

Abstract

In this paper, we consider conditional selective inference (SI) for a linear model estimated after outliers are removed from the data. To apply the conditional SI framework, it is necessary to characterize the events of how the robust method identifies outliers. Unfortunately, the existing conditional SIs cannot be directly applied to our problem because they are applicable to the case where the selection events can be represented by linear or quadratic constraints. We propose a conditional SI method for popular robust regressions such as least-absolute-deviation regression and Huber regression by introducing a new computational method using a convex optimization technique called homotopy method. We show that the proposed conditional SI method is applicable to a wide class of robust regression and outlier detection methods and has good empirical performance on both synthetic data and real data experiments.

Appendix A: Proofs

Proof of Lemma 1

Let $\rho _i(z) = y_i(z)-\varvec{x}_i^\top \varvec{\beta }$ be a parametric residual for each i. Then, (21) can be written as

$$\begin{aligned} \hat{\varvec{\beta }}^{R}(z) = \mathop \mathrm{arg~min}\limits _{\varvec{\beta }} \sum _{i=1}^n |\rho _i(z)|. \end{aligned}$$

Furthermore, for each i, let $\rho _i^+(z)$ and $\rho _i^-(z)$ be non-negative numbers with $\rho _i(z) =\rho _i^+(z)-\rho _i^-(z)$. Since at least one of $\rho _i(z)^+$ and $\rho _i(z)^-$ can be set to zero, the following equation holds:

$$\begin{aligned} |\rho _i(z)|=|\rho _i^+(z) -\rho _i^-(z)| = \rho _i^+(z) +\rho _i^-(z),\ i \in [n]. \end{aligned}$$

Therefore, $\hat{\varvec{\beta }}^{R}(z)$ in the parametric approach can be given by solving the following parametric linear programming problem as

$$\begin{aligned}&\mathrm{min} \sum _{i=1}^n\left( \rho _i^+(z) +\rho _i^-(z)\right) \nonumber \\&\mathrm{s.t.}\;\; \rho _i^+(z) - \rho _i^-(z)=y_i(z)-\varvec{x}_i^{\top }\varvec{\beta }(z), \rho _i^+(z) \ge 0, \rho _i^-(z) \ge 0,\ i \in [n]. \end{aligned}$$

(26)

Next, we show that (26) can be expressed as (22). The $\varvec{\beta }(z)$ in (26) can be re-written as follows:

$$\begin{aligned} \varvec{\beta }(z)=\varvec{\beta }^+(z) - \varvec{\beta }^-(z) , \;\;\; \varvec{\beta }^+(z)\ge \varvec{0},\;\; \varvec{\beta }^-(z)\ge \varvec{0}. \end{aligned}$$

Thus, since ${\varvec{Y}} = {\varvec{a}} + {\varvec{b}} z$, by letting

$$\begin{aligned} \varvec{r}&= \left( \begin{array}{cccccccccc} \rho _1^+ (z)&\rho _1^- (z)&\cdots&\rho _n^+ (z)&\rho _n^- (z)&\beta _{1}^{+} (z)&\beta _{1}^{-} (z)&\cdots&\beta _{p}^{+} (z)&\beta _{p}^{-} (z) \end{array} \right) ^\top \in {\mathbb {R}}^{2n+2p}, \end{aligned}$$

$$\begin{aligned} S&= \left( \begin{array}{cccccccccccccc} 1 &{} -1 &{} 0 &{}0&{}\cdots &{} 0 &{}0 &{}x_{11} &{}-x_{11}&{}x_{12} &{}-x_{12}&{}\cdots &{}x_{1p}&{}-x_{1p}\\ 0 &{} 0 &{} 1 &{} -1 &{}\cdots &{} 0 &{}0 &{}x_{21} &{}-x_{21}&{}x_{22} &{}-x_{22}&{}\cdots &{}x_{2p}&{}-x_{2p}\\ &{}{\vdots }&{}&{}&{}{\ddots }&{} &{}{\vdots }&{}&{}{\vdots }&{}&{}{\vdots }&{} &{}&{}{\vdots } \\ 0&{} 0 &{} 0 &{}0 &{}\cdots &{}1 &{}-1 &{}x_{n1} &{}-x_{n1}&{}x_{n2} &{}-x_{n2}&{}\cdots &{}x_{np}&{}-x_{np}\\ \end{array} \right) \in {\mathbb {R}}^{n \times (2n+2p)},\\ \varvec{q}&= \left( \begin{array}{cccccccccc} 1&{}1&{}\cdots &{}1&{}1&{}0&{}0&{}\cdots &{}0&{}0 \\ \end{array} \right) ^\top \in {\mathbb {R}}^{2n+2p}, \varvec{u}_0=\varvec{a}, \varvec{u}_1=\varvec{b}, \end{aligned}$$

we have (22). Finally, we show piecewise linearity. It is known that the optimal value of the parametric linear programming problem given in the form (22) is a (convex) piecewise-linear function with respect to the parameter z (see, e.g., Section 8.6 in Murty (1983)). Hence, the solution path of $\varvec{\beta }^{R}(z)$ can be represented as a piecewise-linear function with respect to z. $\square$

Proof of Lemma 2

Let $\varvec{e}(z) = \varvec{y}(z)-X^\top \varvec{\beta }$ be a parametric residual vector. Suppose that $\varvec{u}(z)$ and $\varvec{v}(z)$ are vectors, and the i-th element $u(z)_i$ and $v(z)_i$ of $\varvec{u}(z)$ and $\varvec{v}(z)$ are respectively given by

$$\begin{aligned} u(z)_i = \min \{ |e (z)_i |, \delta \}, \ v(z)_i = \max \{|e (z)_i | -\delta ,0 \}, \end{aligned}$$

where $e(z)_i$ is the i-th element of ${\varvec{\rho }} (z)$. Then, the following inequality holds:

$$\begin{aligned} -\varvec{u}(z)-\varvec{v}(z) \le \varvec{y}(z)-X^\top \varvec{\beta } \le \varvec{u}(z)+\varvec{v}(z). \end{aligned}$$

In addition, the objective function can be expressed as follows:

$$\begin{aligned} \frac{1}{2} \varvec{u}(z)^\top \varvec{u}(z) + \delta \cdot \varvec{1}^\top \varvec{v}(z), \end{aligned}$$

where $\varvec{1}$ is a vector with all elements 1. Thus, $\hat{\varvec{\beta }}^{R}(z)$ in the parametric approach can be given by solving the following parametric quadratic programming problem as

$$\begin{aligned} \min&\frac{1}{2} \varvec{u}(z)^\top \varvec{u}(z) + \delta \cdot \varvec{1}^\top \varvec{v}(z)\nonumber \\ \mathrm {s.t.}&-\varvec{u}(z)-\varvec{v}(z) \le \varvec{y}(z)-X\varvec{\beta }(z) \le \varvec{u}(z) + \varvec{v}(z), \nonumber \\&\varvec{0} \le \varvec{u}(z) \le \delta \cdot \varvec{1},\;\; \varvec{v}(z) \ge {\varvec{0}}. \end{aligned}$$

(27)

Therefore, noting that ${\varvec{y}} (z) = {\varvec{a}} + {\varvec{b}} z$, letting

$$\begin{aligned} \varvec{r}&= \left( \begin{array}{ccc} \varvec{\beta } (z) ,\varvec{u} (z) ,\varvec{v} (z) \end{array} \right) ^\top \in {\mathbb {R}}^{p+2n}, \\ P&= \left( \begin{array}{ccc} \varvec{0} &{}\varvec{0} &{}\varvec{0}\\ \varvec{0} &{} I_n &{} \varvec{0}\\ \varvec{0} &{} \varvec{0} &{} \varvec{0}\\ \end{array} \right) \in {\mathbb {R}}^{(p+2n) \times (p+2n)}, \\ \varvec{q}&=(\varvec{0} \;\; \varvec{0}\;\; \delta \cdot \varvec{1}_n)^\top \in {\mathbb {R}}^{(p+2n)}\\ S&= \left( \begin{array}{ccc} X &{}-I_n &{}-I_n\\ -X &{} -I_n &{} -I_n\\ \varvec{0} &{} -I_n &{} \varvec{0}\\ \varvec{0} &{} I_n &{} \varvec{0}\\ \varvec{0} &{} \varvec{0} &{} -I_n \end{array} \right) \in {\mathbb {R}}^{5n \times (p+2n)},\\ {\varvec{u}}_0&= ({\varvec{a}} \;\; -{\varvec{a}} \;\; \varvec{0} \;\; \delta \cdot \varvec{1_n} \;\; \varvec{0} ) ^\top \in {\mathbb {R}}^{5n}, \\ {\varvec{u}}_1&= ({\varvec{b}} \;\; -{\varvec{b}} \;\; \varvec{0} \;\; \varvec{0} \;\; \varvec{0} ) ^\top \in {\mathbb {R}}^{5n}, \end{aligned}$$

it can be shown that (27) is the same as (23). $\square$

Next, we consider the KKT optimality conditions of the reformulated optimization problem (21), and show that the optimal solutions are represented as piecewise-linear functions of z by showing that, when the active variables (nonzero variables at the optimal solution) do not change, the optimal solutions linearly change with z (Proposition 1). Let $\hat{\varvec{u}}(z)$ and $\hat{\varvec{r}} (z)$ be the vectors of optimal Lagrange multipliers. Then, the KKT conditions of (23) are given by

$$\begin{aligned} P \varvec{\hat{r}}(z)+\varvec{q}+S^\top \hat{\varvec{u}}(z)&={\varvec{0}},\\ S\hat{\varvec{r}}(z)-\varvec{h}(z)&\le {\varvec{0}}, \\ \hat{u}_i (z) (S \hat{\varvec{r}}(z)-{\varvec{h}} (z))_i&= 0, \;\;\; \forall i \in [m], \\ \hat{u} _i (z)&\ge 0, \;\;\; \forall i \in [m], \end{aligned}$$

where $\varvec{h}(z)=(\varvec{y}(z)\;\;-\varvec{y}(z) \;\; \varvec{0} \;\; \delta \cdot \varvec{1_n} \;\; \varvec{0})^\top \in {\mathbb {R}}^{5n}$. Furthermore, letting ${\mathcal {A}}_z=\{i \in [m] : \hat{u}_i(z)> 0\}$ and ${\mathcal {A}}_z^c=[m] \; \backslash \; {\mathcal {A}}_z$, we have

$$\begin{aligned} P \varvec{\hat{r}}(z)+\varvec{q}+S^\top \hat{\varvec{u}}(z)&={\varvec{0}}, \nonumber \\ (S\hat{\varvec{r}}(z)-\varvec{h}(z))_i&= 0, \;\;\; \forall i \in {\mathcal {A}}_z \nonumber \\ (S\hat{\varvec{r}}(z)-\varvec{h}(z))_i&\le 0, \;\;\; \forall i \in {\mathcal {A}}_z^c. \end{aligned}$$

(28)

Then, the following proposition holds:

Proposition 1

Let $S_{{\mathcal {A}}_z}$ be the rows of matrix S in a set ${\mathcal {A}}_z$. Consider two real values z and $z'(z < z')$. If ${\mathcal {A}}_z = {\mathcal {A}}_{z'}$, then we have

$$\begin{aligned} \hat{\varvec{r}}(z')-\hat{\varvec{r}}(z)&= \psi (z) \times (z'-z), \end{aligned}$$

(29)

$$\begin{aligned} \hat{\varvec{u}}_{{\mathcal {A}}_z}(z')-\hat{\varvec{u}}_{{\mathcal {A}}_z}(z)&= \gamma (z) \times (z'-z), \end{aligned}$$

(30)

where $\psi (z) \in {\mathbb {R}}^n, \gamma (z) \in {\mathcal {R}}^{|{\mathcal {A}}_z|}$, $\begin{bmatrix} \psi (z)\\ \gamma (z) \end{bmatrix} = \left[ \begin{array}{cc} P &{} S_{{\mathcal {A}}_z}^\top \\ S_{{\mathcal {A}}_z} &{} 0 \end{array} \right] ^{-1} \left[ \begin{array}{c} \varvec{0}\\ \varvec{u}_{1,{\mathcal {A}}_z} \end{array} \right] .$

Proposition 1 means that $\hat{\varvec{\beta }}^{R}(z)$ is a piecewise-linear function with respect to z on the interval $[z,z^\prime ]$. Finally, we show Proposition 1.

Proof

According to (28), we have the following linear system

$$\begin{aligned} \left[ \begin{array}{cc} P &{} S_{{\mathcal {A}}_z}^\top \\ S_{{\mathcal {A}}_z} &{} \varvec{0} \end{array} \right] \left[ \begin{array}{c} \hat{\varvec{r}}(z)\\ \hat{\varvec{u}}_{{\mathcal {A}}_z}(z)\\ \end{array} \right] = \left[ \begin{array}{c} \varvec{q}\\ \varvec{h}_{{\mathcal {A}}_z}(z) \end{array} \right] . \end{aligned}$$

(31)

By decomposing $\varvec{h}(z)=\varvec{u}_0+\varvec{u}_1 z$, (31) can be re-written as

$$\begin{aligned}&\left[ \begin{array}{cc} P &{} S_{{\mathcal {A}}_z}^\top \\ S_{{\mathcal {A}}_z} &{} \varvec{0} \end{array} \right] \left[ \begin{array}{c} \hat{\varvec{r}}(z)\\ \hat{\varvec{u}}_{{\mathcal {A}}_z}(z)\\ \end{array} \right] = \left[ \begin{array}{c} \varvec{q}\\ \varvec{u}_{0,{\mathcal {A}}}(z) \end{array} \right] + \left[ \begin{array}{c} \varvec{0} \\ \varvec{u}_{1,{\mathcal {A}}_z} \end{array} \right] z, \\ \left[ \begin{array}{c} \hat{\varvec{r}}(z)\\ \hat{\varvec{u}}_{{\mathcal {A}}_z}(z)\\ \end{array} \right] =&\left[ \begin{array}{cc} P &{} S_{{\mathcal {A}}_z}^\top \\ S_{{\mathcal {A}}_z} &{} \varvec{0} \end{array} \right] ^{-1} \left[ \begin{array}{c} \varvec{q}\\ \varvec{u}_{0,{\mathcal {A}}}(z) \end{array} \right] + \left[ \begin{array}{cc} P &{} S_{{\mathcal {A}}_z}^\top \\ S_{{\mathcal {A}}_z} &{} \varvec{0} \end{array} \right] ^{-1} \left[ \begin{array}{c} \varvec{0} \\ \varvec{u}_{1,{\mathcal {A}}_z} \end{array} \right] z. \end{aligned}$$

From now, let us denote $\begin{bmatrix} \psi (z)\\ \gamma (z) \end{bmatrix} = \left[ \begin{array}{cc} P &{} S_{{\mathcal {A}}_z}^\top \\ S_{{\mathcal {A}}_z} &{} \varvec{0} \end{array} \right] ^{-1} \left[ \begin{array}{c} \varvec{0} \\ \varvec{u}_{1,{\mathcal {A}}_z} \end{array} \right]$ with $\psi (z)\in {\mathbb {R}}^n$ and $\gamma (z) \in {\mathbb {R}}^{|{\mathcal {A}}_z|}$, the results in proposition 1 is straightforward. $\square$

Proof of Lemma 3

We derive the truncation region ${\mathcal {Z}}$ in the interval $[z_{t-1},z_t].$ For any $t \in [T]$ and $i \in [n]$, the residual $r_i^R(z)$ in the interval $[z_{t-1},z_t]$ is given by

$$\begin{aligned} r_i^R(z)=f_{t,i}+g_{t,i}z \;\; z \in [z_{t-1}, z_t]. \end{aligned}$$

The condition for the truncation region is the following inequality:

$$\begin{aligned} |r_i^R(z)|&\ge \xi . \end{aligned}$$

This implies that

$$\begin{aligned}&f_{t,i}+g_{t,i}z \ge \xi , \end{aligned}$$

(32)

$$\begin{aligned} \text { or }&f_{t,i}+g_{t,i}z \le -\xi . \end{aligned}$$

(33)

From (32) and (33), z is classified into the following four cases based on the value of $g_{t,i}$:

$$\begin{aligned}&\left\{ \begin{array}{c} z\ge \frac{\xi -f_{t,i}}{g_{t,i}} \text { if } \;g_{t,i}> 0,\\ z\le \frac{\xi -f_{t,i}}{g_{t,i}} \text { if } \;g_{t,i}< 0,\\ \end{array} \right. \\&\left\{ \begin{array}{c} z\le \frac{-\xi -f_{t,i}}{g_{t,i}} \text { if } \;g_{t,i} > 0,\\ z\ge \frac{-\xi -f_{t,i}}{g_{t,i}} \text { if }\;g_{t,i} < 0. \end{array} \right. \end{aligned}$$

If $g_{t,i} = 0$, the region that satisfies $|f_{t,i}|\ge \xi$ is the truncation region. From the above, $\mathcal {V}_{t, i}$ is derived as

$$\begin{aligned} \mathcal {V}_{t, i} = \left\{ \begin{array}{ll} \left[ z_{t-1}, \min \left\{ z_{t}, \frac{- \xi - f_{t,i}}{g_{t,i}}\right\} \right] \cup \left[ \max \left\{ z_{t-1}, \frac{\xi - f_{t,i}}{g_{t,i}}\right\} , z_t \right] &{} \text { if } g_{t,i} > 0,\\ \left[ z_{t-1}, \min \left\{ z_{t}, \frac{\xi - f_{t,i}}{g_{t,i}}\right\} \right] \cup \left[ \max \left\{ z_{t-1}, \frac{-\xi - f_{t,i}}{g_{t,i}}\right\} , z_t \right] &{} \text { if } g_{t,i}< 0,\\ \left[ z_{t-1}, z_{t} \right] &{} \text { if } g_{t,i} = 0 \text { and } |f_{t,i}| \ge \xi ,\\ \emptyset &{} \text { if } g_{t,i} = 0 \text { and } |f_{t,i}| < \xi , \end{array} \right. \end{aligned}$$

where we define $[\ell , u] = \emptyset$ if $\ell > u$. $\square$

Proof of Lemma 4

We derive the truncation region ${\mathcal {Z}}$ in the interval $[z_{t-1},z_t]$. For any $t \in [T]$ and $(i, i^\prime ) \in \mathcal {O}(y^{\mathrm{observed}}) \times \left( [n] \setminus \mathcal {O}(y^\mathrm{observed})\right)$, the residual $r_i^R(z)$ in the interval $[z_{t-1},z_t]$ is given by

$$\begin{aligned} r_i^R(z)=f_{t,i}+g_{t,i}z,\;\;\; r_{i'}^R(z)=f_{t,i'}+g_{t,i'}z \;\; z \in [z_{t-1}, z_t]. \end{aligned}$$

The condition of the truncation region is the following:

$$\begin{aligned} |f_{t,i}+g_{t,i} z| \ge |f_{t,i'} +g_{t,i'}z|. \end{aligned}$$

(34)

Since the case classification by absolute value is complicated, we take the method of squaring both sides:

$$\begin{aligned}&(f_{t,i}+g_{t,i} z)^2 \ge (f_{t,i'} +g_{t,i'}z)^2,\nonumber \\&(g_{t,i}^2-g_{t,i'}^2)z^2 + (2f_{t,i}g_{t,i}-2f_{t,i'}g_{t,i'})z + (f_{t,i'}^2-f_{t,i'}^2) \ge 0,\nonumber \\&\alpha z^2 + \beta z + \gamma \ge 0, \end{aligned}$$

(35)

where $\alpha =g_{t,i}^2-g_{t,i'}^2, \beta =2f_{t,i}g_{t,i}-2f_{t,i'}g_{t,i'}$ and $\gamma =f_{t,i}^2-f_{t,i'}^2$. The truncation region that satisfies (35) is equivalent to the condition that satisfies equation (34). Next, we consider the three cases of $\alpha$.

When $\alpha =0$, from $\beta z + \gamma \ge 0$, z is classified into

$$\begin{aligned} \left\{ \begin{array}{c} z\ge \frac{-\gamma }{\beta } \text { if } \; \beta > 0,\\ z\le \frac{-\gamma }{\beta } \text { if }\; \beta < 0. \end{array} \right. \end{aligned}$$

If $\beta =0$, the region that satisfies $\gamma \ge 0$ is the truncation region. Therefore, when $\alpha = 0$, the truncation region can be expressed as

$$\begin{aligned} \left\{ \begin{array}{ll} \left[ \max \left\{ z_{t-1}, \frac{-\gamma }{\beta }\right\} , z_t \right] &{} \text { if } \alpha =0 \text { and } \beta >0,\\ \left[ z_{t-1}, \min \left\{ z_{t},\frac{-\gamma }{\beta }\right\} \right] &{} \text { if } \alpha =0 \text { and } \beta<0,\\ \left[ z_{t-1}, z_{t} \right] &{} \text { if } \alpha =0 \text { and } \beta =0 \text { and } \gamma \ge 0,\\ \emptyset &{} \text { if } \alpha =0 \text { and } \beta =0 \text { and } \gamma <0. \end{array} \right. \end{aligned}$$

When $\alpha >0$, using the quadratic formula, inequality (35) is

$$\begin{aligned}&z>\frac{-\beta +\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha },\\ \text { or }&z<\frac{-\beta -\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }. \end{aligned}$$

Therefore, the truncation region can be derived using the union as

$$\begin{aligned} \left[ \max \left\{ z_{t-1}, \frac{-\beta +\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha } \right\} , z_t \right] \cup \left[ z_{t-1}, \min \left\{ z_{t}, \frac{-\beta -\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }\right\} \right] \text { if } \alpha >0. \end{aligned}$$

When $\alpha <0$, using the quadratic formula, inequality (35) is

$$\begin{aligned}&z<\frac{-\beta -\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }, \\ \text { and }&z>\frac{-\beta +\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }. \end{aligned}$$

Therefore, the truncation region can be derived using the intersection as

$$\begin{aligned} \left[ z_{t-1}, \min \left\{ z_{t}, \frac{-\beta -\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha } \right\} \right] \cap \left[ \max \left\{ z_{t-1}, \frac{-\beta +\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }\right\} , z_{t} \right] \text { if } \alpha <0. \end{aligned}$$

From the above, $\mathcal {W}_{t, i}$ is derived as

$$\begin{aligned} \mathcal {W}_{t, (i, i^\prime )} = \left\{ \begin{array}{ll} \left[ \max \left\{ z_{t-1}, \frac{-\gamma }{\beta }\right\} , z_t \right] &{} \text { if } \alpha =0 \text { and } \beta>0,\\ \left[ z_{t-1}, \min \left\{ z_{t},\frac{-\gamma }{\beta }\right\} \right] &{} \text { if } \alpha =0 \text { and } \beta<0,\\ \left[ z_{t-1}, z_{t} \right] &{} \text { if } \alpha =0 \text { and } \beta =0 \text { and } \gamma \ge 0,\\ \emptyset &{} \text { if } \alpha =0 \text { and } \beta =0 \text { and } \gamma<0,\\ \left[ \max \left\{ z_{t-1}, \frac{-\beta +\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha } \right\} , z_t \right] \cup \left[ z_{t-1}, \min \left\{ z_{t}, \frac{-\beta -\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }\right\} \right] &{} \text { if } \alpha >0,\\ \left[ z_{t-1}, \min \left\{ z_{t}, \frac{-\beta -\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha } \right\} \right] \cap \left[ \max \left\{ z_{t-1}, \frac{-\beta +\sqrt{\beta ^2-4 \alpha \gamma }}{2 \alpha }\right\} , z_{t} \right] &{} \text { if } \alpha <0. \end{array} \right. \end{aligned}$$

$\square$