Skip to main content

Gradient descent algorithms for quantile regression with smooth approximation

International Journal of Machine Learning and Cybernetics volume 2, pages 191–207 (2011)Cite this article

Abstract

Gradient based optimization methods often converge quickly to a local optimum. However, the check loss function used by quantile regression model is not everywhere differentiable, which prevents the gradient based optimization methods from being applicable. As such, this paper introduces a smooth function to approximate the check loss function so that the gradient based optimization methods could be employed for fitting quantile regression model. The properties of the smooth approximation are discussed. Two algorithms are proposed for minimizing the smoothed objective function. The first method directly applies gradient descent, resulting the gradient descent smooth quantile regression model; the second approach minimizes the smoothed objective function in the framework of functional gradient descent by changing the fitted model along the negative gradient direction in each iteration, which yields boosted smooth quantile regression algorithm. Extensive experiments on simulated data and real-world data show that, compared to alternative quantile regression models, the proposed smooth quantile regression algorithms can achieve higher prediction accuracy and are more efficient in removing noninformative predictors.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (Canada)

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. For example, |x| is not smooth at x = 0, and its second order derivative at x = 0 could be understood as \(\infty\) in some sense.

  2. A random variable X follows a double exponential distribution if its probability density function is \(f(x)=\frac{1}{2} e^{-|x|}. \)

  3. Quantile regression forest was implemented based on the R package “quantregForest”, while all other algorithms in this paper were implemented using MATLAB, thus the computing time of QReg forest is not comparable to other algorithms. As such, we choose not to provide the training time of QReg forest.

  4. IP-QReg and MM-QReg were implemented based on the MATLAB code downloaded from http://www.stat.psu.edu/∼dhunter/code/qrmatlab/. When the dimensionality is greater than the sample size, the software package gives error message. Thus, the performances of IP-QReg and MM-QReg are not provided.

References

  1. Belloni A, Chernozhukov V (2011) ℓ1-Penalized quantile regression in high-dimensional sparse models. Ann Stat 39(1):82–130

    Article  MathSciNet  MATH  Google Scholar 

  2. Bühlmann P, Hothorn T (2007) Boosting algorithms: regularization, prediction and model fitting. Stat Sci 22:477–505

    Article  Google Scholar 

  3. Bühlmann P, Yu B (2003) Boosting with the L 2 Loss: regression and classification. J Am Stat Assoc 98:324–340

    Article  MATH  Google Scholar 

  4. Cade BS, Noon BR (2003) A gentle introduction to quantile regression for ecologists. Front Ecol Environ 1(8):412–420

    Article  Google Scholar 

  5. Chen C, Mangasarian OL (1996) A class of smoothing functions for nonlinear and mixed complementarity problems. Comput Optim Appl 5:97–138

    Article  MathSciNet  MATH  Google Scholar 

  6. Duffy N, Helmbold D (2002) Boosting methods for regression. Mach Learn 47(2–3):153–200

    Article  MATH  Google Scholar 

  7. Freund Y, Schapire R (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139

    Article  MathSciNet  MATH  Google Scholar 

  8. Friedman J (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29:1189–1232

    Article  MATH  Google Scholar 

  9. Friedman J, Hastie T, Tibshirani R (2000) Additive logistic regression: a statistical view of boosting. Ann Stat 28(2):337–407

    Article  MathSciNet  MATH  Google Scholar 

  10. Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction. 2nd edn. Springer, New York

    MATH  Google Scholar 

  11. Hendricks W, Koenker R (1992) Hierarchical spline models for conditional quantiles and the demand for electricity. J Am Stat Assoc 93:58–68

    Article  Google Scholar 

  12. Hunter DR, Lange K (2000) Quantile regression via an MM algorithm. J Comput Graph Stat 19(1):60–77

    Google Scholar 

  13. Hwang C, Shim J (2005) A simple quantile regression via support vector machine. In: Lecture notes in computer science, vol 3610/2005, pp 512–520

  14. Koenker R (2005) Quantile regression. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  15. Koenker R, Bassett G (1978) Regression quantiles. Econometrica 46:33–50

    Article  MathSciNet  MATH  Google Scholar 

  16. Koenker R, Geling R (2001) Reappraising medfly longevity: a quantile regression survival analysis. J Am Stat Assoc 96:458–468

    Article  MathSciNet  MATH  Google Scholar 

  17. Koenker R, Hallock K (2001) Quantile regression. J Econ Perspect 15:143–156

    Article  Google Scholar 

  18. Koenker R, Park BJ (1996) An interior point algorithm for nonlinear quantile regression. J Econ 71:265–283

    MathSciNet  MATH  Google Scholar 

  19. Kriegler B, Berk R (2007) Boosting the quantile distribution: a cost-sensitive statistical learning procedure. Technical report, Department of Statitics. University of California, Los Angeles

  20. Langford J, Oliveira R, Zadrozny B (2006) Predicting conditional quantiles via reduction to classification. In: Proceedings of uncertainty in artificical intelligence, pp 257–264

  21. Lee Y-J, Mangasarian OL (2001) SSVM: a smooth support vector machine for classification. Comput Optim Appl 20(1):5–22

    Article  MathSciNet  MATH  Google Scholar 

  22. Li C, Wei Y, Chappell R, He X (2011) Bent line quantile regression with application to an allometric study of land mammals’ speed and mass. Biometrics 67(1):242–249

    Article  MATH  Google Scholar 

  23. Li SZ, Zhang Z (2004) FloatBoost learning and statistical face detection. IEEE Trans Pattern Anal Mach Intell 26(9):1112–1123

    Article  Google Scholar 

  24. Li Y, Liu Y, Zhu J (2007) Quantile regression in reproducing Kernel Hilbert spaces. J Am Stat Assoc 102:255–268

    Article  MathSciNet  MATH  Google Scholar 

  25. Li Y, Zhu J (2008) L 1-norm quantile regression. J Comput Graph Stat 17(1):163–185

    Article  Google Scholar 

  26. Mason L, Baxter J, Bartlett PL, Frean M (2000) Boosting algorithms as gradient descent. Adv Neural Inform Process Syst 12:512–518

    Google Scholar 

  27. Meinshausen N (2006) Quantile regression forests. J Mach Learn Res 7:983–999

    MathSciNet  Google Scholar 

  28. Sohn I, Kim S, Hwang C, Lee JW (2008) New normalization methods using support vector machine quantile regression approach in microarray analysis. Comput Stat Data Anal 52(8):4104–4115

    Article  MathSciNet  MATH  Google Scholar 

  29. Sohn I, Kim S, Hwang C, Lee JW, Shim J (2008) Support vector machine quantile regression for detecting differentially expressed genes in microarray analysis. Methods Inf Med 47(5):459–467

    Google Scholar 

  30. Takeuchi I, Le QV, Sears TD, Smola AJ (2006) Nonparametric quantile estimation. J Mach Learn Res 7:1231–1264

    MathSciNet  Google Scholar 

  31. Torralba A, Murphy KP, Freeman WT (2004) Sharing features: efficient boosting procedures for multiclass object detection. In: Proceeding of IEEE conference on computer vision and pattern recognition (CVPR), pp 762–769

  32. Walsh GR (1975) Methods of optimization. Wiley, New York

    MATH  Google Scholar 

  33. Weisberg S (2005) Applied linear regression, 3rd edn. Wiley, New York

    Book  MATH  Google Scholar 

  34. Wu Y, Liu Y (2009) Variable selection in quantile regression. Stat Sin 19:801–817

    MATH  Google Scholar 

  35. Zemel R, Pitassi T (2001) A gradient-based boosting algorithm for regression problems. In: Proceedings of advances in neural information processing systems

  36. Zhao P, Yu B (2007) Stagewise Lasso. J Mach Learn Res 8:2701–2726

    MathSciNet  Google Scholar 

  37. Zheng S (2010) Boosting based conditional quantile estimation for regression and binary classification. In: The 9th Mexican international conference on artificial intelligence, Pachuca, LNAI 6438, pp 67–79

Download references

Acknowledgments

The short version of this paper [37] was published on the 9th Mexican International Conference on Artificial Intelligence (MICAI). The author gratefully acknowledges the anonymous reviewers of MICAI and International Journal of Machine Learning and Cybernetics for their constructive comments and would like to extend his gratitude to Prof. Grigori Sidorov for his excellent work in coordinating the preparation and the reviewing of the manuscript. This work was partially supported by a 2011 Summer Faculty Fellowship of Missouri State University.

Author information

Authors and Affiliations

  1. Department of Mathematics, Missouri State University, Springfield, MO, 65897, USA

    Songfeng Zheng

Authors
  1. Songfeng Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Songfeng Zheng.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Zheng, S. Gradient descent algorithms for quantile regression with smooth approximation. Int. J. Mach. Learn. & Cyber. 2, 191–207 (2011). https://doi.org/10.1007/s13042-011-0031-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-011-0031-2

Keywords

  • Quantile regression
  • Gradient descent
  • Boosting
  • Variable selection