Piecewise Linear L1 Modeling

Abstract

Lester Taylor (Taylor and Houthakker 2010) instilled a deep respect for estimating parameters in statistical models by minimizing the sum of absolute errors (the L1 criterion) as an important alternative to minimizing the sum of the squared errors (the ordinary least squares or OLS criterion).

Appendices

Appendix 1. Standard L1 or QR Multiple Regression

The estimation of a single multiple regression with L1 or QR is the following LP problem:

$$ {\text{min}}:\sum\limits_{i = 1}^{n} {\left( {\theta e_{i + } + (1 - \theta )e_{i - } } \right)}. $$

Such that:

$$ y_{i} - \beta^{\prime } x_{i} = e_{i + } - e_{i - } ;\;\forall i $$

$$ e_{i + } \ge 0;\;\forall i $$

$$ e_{i - } \ge 0;\;\forall i, $$

β:

unrestricted.

In this primal LP problem, the $x_{i}$ are known p-vectors and the $y_{i}$ are known scalar values. $\beta$ is a p-vector of decision variables. For L1, choose θ = 0.5; for QR choose any $\theta \in \left[ {0,\,1} \right]$. This well-known LP formulation has 2n + p decision variables and n linear equality constraints. For this primal LP formulation, duality theory applies and the dual LP problem is:

$$ {\text{max}}:\sum\limits_{i = 1}^{n} {\lambda_{i} } y_{i}. $$

Such that:

$$ X^{\prime } \lambda = 0 $$

$$ \theta - 1 \le \lambda_{i} \le \theta :\;\forall i. $$

This LP problem has n decision variables, p linear equality constraints, and n bounded variables, so it is usually a bit faster to solve for large n. Importantly, the optimal values in $\lambda$ may be associated with important test statistics developed by Koenker and Bassett.

Appendix 2. L1 or QR Piecewise Multiple Regression with Known Hinges

With one known hinge, Eq. (2.2) describes the predictor and Eq. (2.3) defines the hinge. Let x be a p-vector of known values of independent variables. Typically, the first element of x is unity for a constant term in the multiple regression. The hinge given by Eq. (2.3) will then be a p-vector, H, which is here assumed known. Define the p-vector $z = \left\{ {\begin{array}{*{20}l} {x;} & {x \le H} \\ {x - H;} & {x > H} \\ \end{array} } \right\}$ with individual calculations for each element of x and H. Since x and H are known, z has known values. This results in the LP problem:

$$ {\text{min}}:\sum\limits_{i = 1}^{n} {\left( {\theta e_{i + } + (1 - \theta )e_{i - } } \right)} $$

Such that:

$$ y_{i} - \beta_{1}^{\prime } x_{i} - \beta_{2}^{\prime } z_{i} = e_{i + } - e_{i - } ;\;\forall i $$

$$ e_{i + } \ge 0;\;\forall i $$

$$ e_{i - } \ge 0;\;\forall i $$

β₁, β₂ :

unrestricted.

For more than one known hinge, this LP can be easily extended; simply add additional $\beta$ vectors and additional z vectors for each additional hinge to the formulation.

Appendix 3. L1 or QR Piecewise Multiple Regression with Unknown Hinges

The solution for H = 1 hinge and two pieces is clearly found with the MILP formulation in the second section. Let this solution be denoted by $\hat{y}_{i} = \hat{y}(1)_{i} ;\;\forall i$ [with notation changes to Eq. (2.7)] which chooses one of the linear pieces $\left( {\hat{y}_{1i} ,\;\hat{y}_{2i} } \right)$ as the regression for each i.

For H = 2 hinges, there are three possible pieces $\left( {\hat{y}_{1i} ,\;\hat{y}_{2i} ,\;\hat{y}_{3i} } \right).$ This reduces to a choice between one of two linear pieces $\left( {\hat{y}_{3i} ,\;\hat{y}(1)_{i} } \right)$ and a second set of binary variables and constraints such as Eq. (2.7) (with notation changes) enforces this choice to solve the problem for H = 2. This solution can be denoted by $\hat{y}(2)_{i} ;\;\forall i.$

This inductive argument can be continued for H = 3, 4, etc. For any number of hinges, an MILP formulation can be created with H(n + 1) binary variables, the main determinant of computing time.