Journal of Scientific Computing

, Volume 51, Issue 2, pp 261–273 | Cite as

Alternating Direction Method for Covariance Selection Models

  • Xiaoming YuanEmail author


The covariance selection problem captures many applications in various fields, and it has been well studied in the literature. Recently, an l 1-norm penalized log-likelihood model has been developed for the covariance selection problem, and this novel model is capable of completing the model selection and parameter estimation simultaneously. With the rapidly increasing magnitude of data, it is urged to consider efficient numerical algorithms for large-scale cases of the l 1-norm penalized log-likelihood model. For this purpose, this paper develops the alternating direction method (ADM). Some preliminary numerical results show that the ADM approach is very efficient for large-scale cases of the l 1-norm penalized log-likelihood model.


Covariance selection problem l1-norm Log-likelihood Alternating direction method 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Banerjee, O., El Ghaoui, L., d’Aspremont, A.: Model selection through sparse maximum likelihood estimation. J. Mach. Learn. Res. 9, 485–516 (2008) MathSciNetzbMATHGoogle Scholar
  2. 2.
    Bauschke, H.H., Combettes, P.L.: A weak-to-strong convergence principle for Fejér-monotone methods in Hilbert spaces. Math. Oper. Res. 26(2), 248–264 (2001) MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Cox, D.R., Wermuth, N.: Multivariate Dependencies: Models, Analysis and Interpretation. Chapman and Hall, London (1996) zbMATHGoogle Scholar
  5. 5.
    d’Aspremont, A., Banerjee, O., El Ghaoui, L.: First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30(1), 56–66 (2008) MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Dempster, A.: Covariance selection. Biometrics 28, 157–175 (1972) CrossRefGoogle Scholar
  7. 7.
    Eckstein, J., Fukushima, M.: Some reformulation and applications of the alternating directions method of multipliers. In: Hager, W.W., et al. (eds.) Large Scale Optimization: State of the Art, pp. 115–134. Kluwer Academic, Norwell (1994) CrossRefGoogle Scholar
  8. 8.
    Fazel, M., Hindi, H., Boyd, S.: A rank minimization heuristic with application to minimum order system approximation. Proc. Am. Control Conf. 6, 4734–4739 (2001) Google Scholar
  9. 9.
    Fukushima, M.: Application of the alternating direction method of multipliers to separable convex programming problems. Comput. Optim. Appl. 1, 93–111 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Gabay, D.: Application of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian methods: Application to the Numerical Solution of Boundary-Value Problem, pp. 299–331. North-Holland, Amsterdam (1983) CrossRefGoogle Scholar
  11. 11.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 17–40 (1976) zbMATHCrossRefGoogle Scholar
  12. 12.
    Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator Splitting Methods in Nonlinear Mechanics. SIAM Studies in Applied Mathematics, (1989) zbMATHCrossRefGoogle Scholar
  13. 13.
    Golub, G.H., van Loan, C.F.: Matrix Computation, 3rd edn. The Johns Hopkins University Press, Baltimore (1996) Google Scholar
  14. 14.
    He, B.S., Liao, L.Z., Han, D., Yang, H.: A new inexact alternating directions method for monotone variational inequalities. Math. Program. 92, 103–118 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    He, B.S., Yang, H.: Some convergence properties of a method of multipliers for linearly constrained monotone variational inequalities. Oper. Res. Lett. 23, 151–161 (1998) MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Hey, T., Tansley, S., Tolle, K.: The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research (2009) Google Scholar
  17. 17.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969) MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Hughes, T.R., Marton, M.J., Jones, A.R., Roberts, C.J., Stoughton, R., Armour, C.D., Bennett, H.A., Coffey, E., Dai, H., He, Y.D., Kidd, M.J., King, A.M., Meyer, M.R., Slade, D., Lum, P.Y., Stepaniants, S.B., Shoemaker, D.D., Gachotte, D., Chakraburtty, K., Simon, J., Bard, M., Friend, S.H.: Functional discovery via a compendium of expression profiles. Cell 102(1), 109–126 (2000) CrossRefGoogle Scholar
  19. 19.
    Kontogiorgis, S., Meyer, R.R.: A variable-penalty alternating directions method for convex optimization. Math. Program. 83, 29–53 (1998) MathSciNetzbMATHGoogle Scholar
  20. 20.
    Li, L., Toh, K.C.: An inexact interior point method for l 1-regularized sparse covariance selection. Math. Program. Comput. 2, 291–315 (2010) MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Lu, Z.: Smooth optimization approach for sparse covariance selection. SIAM J. Control Optim. 19(4), 1807–1827 (2009) zbMATHGoogle Scholar
  22. 22.
    Natsoulis, G., Pearson, C.I., Gollub, J., Eynon, B.P., Ferng, J., Nair, R., Idury, R., Lee, M.D., Fielden, M.R., Brennan, R.J., Roter, A.H., Jarnagin, K.: The liver pharmacological and xenobiotic gene response repertoire. Mol. Syst. Biol. 175(4), 1–12 (2008) Google Scholar
  23. 23.
    Ng, M., Weiss, P.A., Yuan, X.M.: Solving constrained total-variation problems via alternating direction methods. SIAM J. Sci. Comput. 32(5), 2710–2736 (2010) MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Nesterov, Y.E.: A method for unconstrained convex minimization problem with the rate of convergence O(1/k 2). Dokl. Akad. Nauk SSSR 269, 543–547 (1983) MathSciNetGoogle Scholar
  25. 25.
    Nesterov, Y.E.: Smooth minimization of nonsmooth functions. Math. Program. 103, 127–152 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Nesterov, Y.E., Nemirovski, A.S.: Interior point Polynomial algorithms in Convex Programming: Theory and Applications. Society for Industrial and Applied Mathematics, Philadelphia (1994) zbMATHCrossRefGoogle Scholar
  27. 27.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969) Google Scholar
  28. 28.
    Sturm, J.F.: Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optim. Methods Softw. 11–12, 625–653 (1999) MathSciNetCrossRefGoogle Scholar
  29. 29.
    Tseng, P.: Alternating projection-proximal methods for convex programming and variational inequalities. SIAM J. Control Optim. 7, 951–965 (1997) MathSciNetzbMATHGoogle Scholar
  30. 30.
    Tütüncü, R.H., Toh, K.C., Todd, M.J.: Solving semidefinite-quadratic-linear programs using SDPT3. Math. Program. 95, 189–217 (2003) MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Vandenberghe, L., Boyd, S., Wu, S.-P.: Determinant maximization with linear matrix inequality constraints. SIAM J. Matrix Anal. Appl. 19, 499–533 (1998) MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Wang, C.J., Sun, D.F., Toh, K.C.: Solving log-determinant optimization problems by a Newton-CG primal proximal point algorithm. SIAM J. Control Optim. 20, 2994–3013 (2010) MathSciNetzbMATHGoogle Scholar
  33. 33.
    Yang, J.-F., Zhang, Y.: Alternating direction method for L1 problems in compressive sensing. SIAM J. Sci. Comput., 33(1), 250–278 (2011) MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Ye, C.H., Yuan, X.M.: A descent method for structured monotone variational inequalities. Optim. Methods Softw. 22, 329–338 (2007) MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Yuan, M., Lin, Y.: Model selection and estimation in the Gaussian graphical model. Biometrika 94(1), 19–35 (2007) MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Yuan, Y.-X., Sun, W.Y.: Optimization Theory and Methods. Springer, Berlin (2006) zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Department of MathematicsHong Kong Baptist UniversityHong KongChina

Personalised recommendations