Advertisement

A Tutorial on Open image in new window: R Package for the Linearized Bregman Algorithm in High-Dimensional Statistics

  • Jiechao Xiong
  • Feng Ruan
  • Yuan Yao
Chapter
Part of the Springer Handbooks of Computational Statistics book series (SHCS)

Abstract

The R package, Open image in new window , stands for the LInearized BRegman Algorithm in high-dimensional statistics. The Linearized Bregman Algorithm is a simple iterative procedure which generates sparse regularization paths of model estimation. This algorithm was firstly proposed in applied mathematics for image restoration, and is particularly suitable for parallel implementation in large-scale problems. The limit of such an algorithm is a sparsity-restricted gradient descent flow, called the Inverse Scale Space, evolving along a parsimonious path of sparse models from the null model to overfitting ones. In sparse linear regression, the dynamics with early stopping regularization can provably meet the unbiased oracle estimator under nearly the same condition as LASSO, while the latter is biased. Despite its successful applications, proving the consistency of such dynamical algorithms remains largely open except for some recent progress on linear regression. In this tutorial, algorithmic implementations in the package are discussed for several widely used sparse models in statistics, including linear regression, logistic regression, and several graphical models (Gaussian, Ising, and Potts). Besides the simulation examples, various applications are demonstrated, with real-world datasets such as diabetes, publications of COPSS award winners, as well as social networks of two Chinese classic novels, Journey to the West and Dream of the Red Chamber.

Keywords

Linearized Bregman iteration LASSO Variable selection Regularization path 

Notes

Acknowledgements

The authors would like to thank Chendi Huang, Stanley J. Osher, Ming Yan, and Wotao Yin for helpful discussions. The research of Jiechao Xiong and Yuan Yao was supported in part by National Basic Research Program of China: 2015CB85600 and 2012CB825501, National Natural Science Foundation of China: 61370004 and 11421110001 (A3 project), as well as grants from Baidu and Microsoft Research Asia. The research of Feng Ruan was partially supported by the E.K. Potter Stanford Graduate Fellowship.

References

  1. Burger M, Osher S, Xu J, Gilboa G (2005) Nonlinear inverse scale space methods for image restoration. In: Variational, geometric, and level set methods in computer vision. Springer, Berlin, pp 25–36Google Scholar
  2. Burger M, Möller M, Benning M, Osher S (2013) An adaptive inverse scale space method for compressed sensing. Math Comput 82(281):269–299MathSciNetCrossRefGoogle Scholar
  3. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499Google Scholar
  4. Fan J, Li R (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96:1348–1360MathSciNetCrossRefGoogle Scholar
  5. Friedman J, Hastie T, Tibshirani R (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3):432–441CrossRefGoogle Scholar
  6. Hassner M, Sklansky J (1980) The use of Markov random fields as models of texture. Comput Graph Image Process 12(4):357–370CrossRefGoogle Scholar
  7. Hastie TJ, Tibshirani RJ, Friedman JH (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New York. http://opac.inria.fr/record=b1127878. Autres impressions: 2011 (corr.), 2013 (7e corr.)
  8. Ising E (1925) Beitrag zur theorie des ferromagnetismus. Z Phys A Hadrons Nucl 31(1):253–258CrossRefGoogle Scholar
  9. Ji P, Jin J (2016) Coauthorship and citation networks for statisticians. Ann Appl Stat 10(4):1779–1812. http://dx.doi.org/10.1214/15-AOAS896 MathSciNetCrossRefGoogle Scholar
  10. Manning CD, Schütze H (1999) Foundations of statistical natural language processing, vol 999. MIT, CambridgeGoogle Scholar
  11. Osher S, Ruan F, Xiong J, Yao Y, Yin W (2016) Sparse recovery via differential inclusions. Appl Comput Harmon Anal. https://doi.org/10.1016/j.acha.2016.01.002 MathSciNetCrossRefGoogle Scholar
  12. Ravikumar P, Wainwright MJ, Lafferty JD et al (2010) High-dimensional Ising model selection using l 1-regularized logistic regression. Ann Stat 38(3):1287–1319MathSciNetCrossRefGoogle Scholar
  13. Shi JV, Yin W, Osher SJ (2013) Linearized Bregman for l 1-regularized logistic regression. In: Proceedings of the 30th international conference on machine learning (ICML)Google Scholar
  14. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B 58:267–288MathSciNetzbMATHGoogle Scholar
  15. Xue L, Zou H, Cai T (2012) Nonconcave penalized composite conditional likelihood estimation of sparse Ising models. Ann Stat 40(3):1403–1429. https://doi.org/10.1214/12-AOS1017 MathSciNetCrossRefGoogle Scholar
  16. Yin W, Osher S, Darbon J, Goldfarb D (2008) Bregman iterative algorithms for compressed sensing and related problems. SIAM J Imag Sci 1(1):143–168CrossRefGoogle Scholar
  17. Zhao T, Liu H (2012) The huge package for high-dimensional undirected graph estimation in R. J Mach Learn Res 13:1059–1062MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Peking UniversitySchool of Mathematical SciencesBeijingChina
  2. 2.Stanford UniversityDepartment of StatisticsStanfordUSA
  3. 3.Hong Kong University of Science & TechnologyClear Water BayHong Kong
  4. 4.Peking UniversityBeijingChina

Personalised recommendations