Acta Mathematica Sinica, English Series

, Volume 34, Issue 12, pp 1892–1906 | Cite as

An Alternating Direction Method of Multipliers for MCP-penalized Regression with High-dimensional Data

  • Yue Yong Shi
  • Yu Ling Jiao
  • Yong Xiu Cao
  • Yan Yan LiuEmail author


The minimax concave penalty (MCP) has been demonstrated theoretically and practically to be effective in nonconvex penalization for variable selection and parameter estimation. In this paper, we develop an efficient alternating direction method of multipliers (ADMM) with continuation algorithm for solving the MCP-penalized least squares problem in high dimensions. Under some mild conditions, we study the convergence properties and the Karush–Kuhn–Tucker (KKT) optimality conditions of the proposed method. A high-dimensional BIC is developed to select the optimal tuning parameters. Simulations and a real data example are presented to illustrate the efficiency and accuracy of the proposed method.


Alternating direction method of multipliers coordinate descent continuation high-dimensional BIC minimax concave penalty penalized least squares 

MR(2010) Subject Classification

62J05 62J07 62J99 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The authors sincerely thank the associate editor and the referees for their valuable comments and suggestions that have led to significant improvement of this article.


  1. [1]
    Becker, S., Bobin, J., Candès, E. J.: NESTA: A fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci., 4, 1–39 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  2. [2]
    Boyd, S., Parikh, N., Chu, E., et al.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3, 1–122 (2011)CrossRefzbMATHGoogle Scholar
  3. [3]
    Breheny, P., Huang, J.: Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann. Appl. Stat., 5, 232–253 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  4. [4]
    Bühlmann, P., Van De Geer, S.: Statistics for High-dimensional Data: Methods, Theory and Applications, Springer-Verlag, Berlin, 2011CrossRefzbMATHGoogle Scholar
  5. [5]
    Fan, Q., Jiao, Y., Lu, X.: A primal dual active set algorithm with continuation for compressed sensing. IEEE Trans. Signal Process., 62, 6276–6285 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  6. [6]
    Friedman, J., Hastie, T., Tibshirani, R.: Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw., 33, 1–22 (2010)CrossRefGoogle Scholar
  7. [7]
    Gao, H. Y., Bruce, A. G.: WaveShrink with firm shrinkage. Statist. Sinica, 7, 855–874 (1997)MathSciNetzbMATHGoogle Scholar
  8. [8]
    Golub, G. H., Van Loan, C. F.: Matrix Computations (4th Edition), John Hopkins University Press, Baltimore, 2013zbMATHGoogle Scholar
  9. [9]
    Hong, M., Luo, Z. Q., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM J. Optim., 26, 337–364 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  10. [10]
    Huang, J., Breheny, P., Lee, S., et al.: The Mnet method for variable selection. Statist. Sinica, 26, 903–923 (2016)MathSciNetzbMATHGoogle Scholar
  11. [11]
    Huang, J., Jiao, Y., Liu, Y., et al.: A constructive approach to sparse linear regression in high-dimensions, arXiv preprint arXiv:1701.05128v1, 2017Google Scholar
  12. [12]
    Huang, J., Ma, S., Zhang, C. H.: Adaptive Lasso for sparse high-dimensional regression models. Statist. Sinica, 18, 1603–1618 (2008)MathSciNetzbMATHGoogle Scholar
  13. [13]
    Jiao, Y., Jin, B., Lu, X.: A primal dual active set with continuation algorithm for the 0-regularized optimization problem. Appl. Comput. Harmon. Anal., 39, 400–426 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  14. [14]
    Jiao, Y., Jin, B., Lu, X., et al.: A primal dual active set algorithm for a class of nonconvex sparsity optimization, arXiv preprint arXiv:1310.1147v3, 2016Google Scholar
  15. [15]
    Jiao, Y., Jin, Q., Lu, X., et al.: Alternating direction method of multipliers for linear inverse problems. SIAM J. Numer. Anal., 54, 2114–2137 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  16. [16]
    Jin, Z. F., Wan, Z., Jiao, Y., et al.: An alternating direction method with continuation for nonconvex low rank minimization. J. Sci. Comput., 66, 849–869 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  17. [17]
    Li, X., Zhao, T., Yuan, X., et al.: The flare package for high dimensional linear regression and precision matrix estimation in R. J. Mach. Learn. Res., 16, 553–557 (2015)MathSciNetzbMATHGoogle Scholar
  18. [18]
    Lu, Z., Pong, T. K., Zhang, Y.: An alternating direction method for finding Dantzig selectors. Comput. Statist. Data Anal., 56, 4037–4046 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  19. [19]
    Mazumder, R., Friedman, J. H., Hastie, T.: Sparsenet: Coordinate descent with nonconvex penalties. J. Amer. Statist. Assoc., 106, 1125–1138 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  20. [20]
    Peng, B., Wang, L.: An iterative coordinate descent algorithm for high-dimensional nonconvex penalized quantile regression. J. Comput. Graph. Statist., 24, 676–694 (2015)MathSciNetCrossRefGoogle Scholar
  21. [21]
    Scheetz, T., Kim, K., Swiderski, R., et al.: Regulation of gene expression in the mammalian eye and its relevance to eye disease. Proc. Natl. Acad. Sci. USA, 103, 14429–14434 (2006)CrossRefGoogle Scholar
  22. [22]
    Simon, N., Friedman, J., Hastie, T., et al.: Regularization paths for Cox’s proportional hazards model via coordinate descent. J. Stat. Softw., 39, 1–13 (2011)CrossRefGoogle Scholar
  23. [23]
    Song, C., Yoon, S., Pavlovic, V.: Fast ADMM algorithm for distributed optimization with adaptive penalty. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence and the Twenty-Eighth Innovative Applications of Artificial Intelligence Conference, 2016Google Scholar
  24. [24]
    Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl., 109, 475–494 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  25. [25]
    Wang, H., Li, B., Leng, C.: Shrinkage tuning parameter selection with a diverging number of parameters. J. R. Stat. Soc. Ser. B Stat. Methodol., 71, 671–683 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  26. [26]
    Wang, H., Li, R., Tsai, C. L.: Tuning parameter selectors for the smoothly clipped absolute deviation method. Biometrika, 94, 553–568 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  27. [27]
    Wang, L., Kim, Y., Li, R.: Calibrating nonconvex penalized regression in ultra-high dimension. Ann. Statist., 41, 2505–2536 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  28. [28]
    Wang, Y., Yin, W., Zeng, J.: Global convergence of ADMM in nonconvex nonsmooth optimization. arXiv preprint, arXiv:1511.06324v5, 2017Google Scholar
  29. [29]
    Xu, Z., Figueiredo, M. A. T., Goldstein, T.: Adaptive ADMM with spectral penalty parameter selection. arXiv preprint, arXiv:1605.07246v5, 2017Google Scholar
  30. [30]
    Yang, J., Zhang, Y.: Alternating direction algorithms for 1-problems in compressive sensing. SIAM J. Sci. Comput., 33, 250–278 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  31. [31]
    Yu, L., Lin, N., Wang, L.: A parallel algorithm for large-scale nonconvex penalized quantile regression. J. Comput. Graph. Statist., DOI:10.1080/10618600.2017.1328366, 2017 (just-accepted)Google Scholar
  32. [32]
    Yu, Y., Feng, Y.: APPLE: Approximate path for penalized likelihood estimators. Stat. Comput., 24, 803–819 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  33. [33]
    Yuan, X.: Alternating direction method for covariance selection models. J. Sci. Comput., 51, 261–273 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  34. [34]
    Zhang, C. H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Statist., 38, 894–942 (2010)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Institute of Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Chinese Mathematical Society and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Yue Yong Shi
    • 1
    • 2
  • Yu Ling Jiao
    • 3
  • Yong Xiu Cao
    • 3
  • Yan Yan Liu
    • 4
    Email author
  1. 1.School of Economics and ManagementChina University of GeosciencesWuhanP. R. China
  2. 2.Center for Resources and Environmental Economic ResearchChina University of GeosciencesWuhanP. R. China
  3. 3.School of Statistics and MathematicsZhongnan University of Economics and LawWuhanP. R. China
  4. 4.School of Mathematics and StatisticsWuhan UniversityWuhanP. R. China

Personalised recommendations