A generalized Newton algorithm for quantile regression models

Abstract

This paper formulates the quadratic penalty function for the dual problem of the linear programming associated with the $L_1$ constrained linear quantile regression model. We prove that the solution of the original linear programming can be obtained by minimizing the quadratic penalty function, with the formulas derived. The obtained quadratic penalty function has no constraint, thus could be minimized efficiently by a generalized Newton algorithm with Armijo step size. The resulting algorithm is easy to implement, without requiring any sophisticated optimization package other than a linear equation solver. The proposed approach can be generalized to the quantile regression model in reproducing kernel Hilbert space with slight modification. Extensive experiments on simulated data and real-world data show that, the proposed Newton quantile regression algorithms can achieve performance comparable to state-of-the-art.

Appendix: The proof of Eq. (25)

If $a>\lambda >0$, then

$$\begin{aligned} a-\lambda >0, \quad -a-\lambda <0,\quad \text {and}\quad |a|-\lambda >0, \end{aligned}$$

thus, from the definition of $(\cdot )_*$,

$$\begin{aligned} (a-\lambda )_*+(-a-\lambda )_* = 1 = (|a|-\lambda )_*. \end{aligned}$$

If $a<-\lambda <0$, we have

$$\begin{aligned} a-\lambda <0, \quad -a-\lambda >0,\quad \text {and}\quad |a|-\lambda >0, \end{aligned}$$

thus,

$$\begin{aligned} (a-\lambda )_*+(-a-\lambda )_* = 1 = (|a|-\lambda )_*. \end{aligned}$$

If $-\lambda <a<\lambda$, it implies that

$$\begin{aligned} a-\lambda <0, \quad -a-\lambda <0,\quad \text {and}\quad |a|-\lambda <0, \end{aligned}$$

thus,

$$\begin{aligned} (a-\lambda )_*+(-a-\lambda )_* = 0 = (|a|-\lambda )_*. \end{aligned}$$

In summary, it is checked for all the cases that $(a-\lambda )_*+(-a-\lambda )_* = (|a|-\lambda )_*$.