Mathematical Programming Computation

, Volume 2, Issue 3–4, pp 291–315 | Cite as

An inexact interior point method for L 1-regularized sparse covariance selection

  • Lu Li
  • Kim-Chuan TohEmail author
Full Length Paper


Sparse covariance selection problems can be formulated as log-determinant (log-det) semidefinite programming (SDP) problems with large numbers of linear constraints. Standard primal–dual interior-point methods that are based on solving the Schur complement equation would encounter severe computational bottlenecks if they are applied to solve these SDPs. In this paper, we consider a customized inexact primal–dual path-following interior-point algorithm for solving large scale log-det SDP problems arising from sparse covariance selection problems. Our inexact algorithm solves the large and ill-conditioned linear system of equations in each iteration by a preconditioned iterative solver. By exploiting the structures in sparse covariance selection problems, we are able to design highly effective preconditioners to efficiently solve the large and ill-conditioned linear systems. Numerical experiments on both synthetic and real covariance selection problems show that our algorithm is highly efficient and outperforms other existing algorithms.


Log-determinant semidefinite programming Sparse inverse covariance selection Inexact interior point method Inexact search direction Iterative solver 

Mathematics Subject Classification (2000)

90C06 90C22 90C25 65F10 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Banerjee O., El Ghaoui L., d’Aspremont A.: Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data. J. Mach. Learn. Res. 9, 485–516 (2008)MathSciNetGoogle Scholar
  2. 2.
    Bilmes, J.A.: Natural statistical models for automatic speech recognition. PhD thesis, University of California, Berkeley (1999)Google Scholar
  3. 3.
    Burer S., Monteiro R.D.C., Zhang Y.: A computational study of a gradient-based log-barrier algorithm for a class of large-scale SDPs. Math. Program. 95, 359–379 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Chen, S.S., Gopinath, R.A.: Model selection in acoustic modeling. In: Proc. EUROSPEECH’99, pp. 1087–1090, Budapest, Hungary (1999)Google Scholar
  5. 5.
    Dahl J., Vandenberghe L., Roychowdhury V.: Covariance selection for nonchordal graphs via chordal embedding. Optim. Methods Softw. 23, 501–520 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    d’Aspremont, A.: Identifying small mean reverting portfolios. Quant. Finance (2010, to appear)Google Scholar
  7. 7.
    d’Aspremont A., Banerjee O., El Ghaoui L.: First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30, 56–66 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Dempster A.P.: Covariance selection. Biometrics 28, 157–175 (1972)CrossRefGoogle Scholar
  9. 9.
    Dobra A.: Variable selection and dependency networks for genomewide data. Biostatistics 10, 621–639 (2009)CrossRefGoogle Scholar
  10. 10.
    Edwards D.: Introduction to graphical modelling, 2nd edn. Springer, New York (2000)zbMATHGoogle Scholar
  11. 11.
    Fan J., Feng Y., Wu Y.: Network exploration via the adaptive LASSO and SCAD penalties. Ann. Appl. Stat. 3, 521–541 (2009)zbMATHCrossRefGoogle Scholar
  12. 12.
    Freund, R., Nachtigal, N.: A new Krylov-subspace method for symmetric indefinite linear system. In: Proceedings of the 14th IMACS World Congress on Computational and Applied Mathematics, Atlanta, USA, pp. 1253–1256 (1994)Google Scholar
  13. 13.
    Friedman J., Hastie T., Tibshirani R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9, 432–441 (2008)zbMATHCrossRefGoogle Scholar
  14. 14.
    Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999)CrossRefGoogle Scholar
  15. 15.
    Hedenfalk I., Duggan D., Chen Y., Radmacher M., Bittner M., Simon R., Meltzer P., Gusterson B., Esteller M., Raffeld M., Yakhini Z., Ben-Dor A., Dougherty E., Kononen J., Bubendorf L., Fehrle W., Pittaluga S., Gruvberger S., Loman N., Johannsson O., Olsson H., Wilfond B., Sauter G., Kallioniemi O.-P., Borg A., Trent J.: Gene-expression profiles in hereditary breast cancer. N. Engl. J. Med. 344, 539–548 (2001)CrossRefGoogle Scholar
  16. 16.
    Jarre F., Rendl F.: An augmented primal-dual method for linear conic programs. SIAM J. Optim. 19, 808–823 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Krishnamurthy, V., d’Aspremont, A.: A pathwise algorithm for covariance selection. Preprint (2009)Google Scholar
  18. 18.
    Lan, G., Lu, Z., Monterio, R.D.: Primal-dual first-order methods with \({\mathcal{O}(1/\epsilon)}\) iteration-complexity for cone programming. Math. Program. (2010, to appear)Google Scholar
  19. 19.
    Lauritzen, S.L.: Graphical models. In: Oxford Statistical Science Series, vol. 17. The Clarendon Press/Oxford University Press/Oxford Science Publications, New York (1996)Google Scholar
  20. 20.
    Lu Z.: Smooth optimization approach for sparse covariance selection. SIAM J. Optim. 19, 1807–1827 (2008)CrossRefMathSciNetGoogle Scholar
  21. 21.
    Lu Z.: Adaptive first-order methods for general sparse inverse covariance selection. SIAM J. Matrix Anal. Appl. 31, 2000–2016 (2010)zbMATHCrossRefGoogle Scholar
  22. 22.
    Meinshausen N., Bühlmann P.: High-dimensional graphs and variable selection with the lasso. Ann. Stat. 34, 1436–1462 (2006)zbMATHCrossRefGoogle Scholar
  23. 23.
    Nesterov Y.: Smooth minimization of non-smooth functions. Math. Program. 103, 127–152 (2005)zbMATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Nesterov Y., Todd M.J.: Primal-dual interior-point methods for self-scaled cones. SIAM J. Optim. 8, 324–364 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Pittman J., Huang E., Dressman H., Horng C.-F., Cheng S.H., Tsou M.-H., Chen C.-M., Bild A., Iversen E.S., Huang A.T., Nevins J.R., West M.: Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomes. Proc. Natl. Acad. Sci. USA 101(22), 8431–8436 (2004)CrossRefGoogle Scholar
  26. 26.
    Rockafellar R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)zbMATHCrossRefMathSciNetGoogle Scholar
  27. 27.
    Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2003)Google Scholar
  28. 28.
    Sachs K., Perez O., Pe’er D., Lauffenburger D.A., Nolan G.P.: Causal protein-signaling networks derived from multiparameter single-cell data. Science 308, 523–529 (2005)CrossRefGoogle Scholar
  29. 29.
    Scheinberg, K., Rish, I.: Learning sparse Gaussian Markov networks using a greedy coordinate ascent approach. In: Balcázar, J., Bonchi, F., Gionis, A., Sebag, M. (eds.) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science 6323. pp. 196–212 (2010)Google Scholar
  30. 30.
    Storey J.D., Tibshirani R.: Statistical significance for genome-wide studies. Proc. Natl. Acad. Sci. USA 100(16), 9440–9445 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  31. 31.
    Sturm J.F.: Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones. Optim. Methods Softw. 11/12, 625–653 (1999)CrossRefMathSciNetGoogle Scholar
  32. 32.
    Toh K.-C.: Solving large scale semidefinite programs via an iterative solver on the augmented systems. SIAM J. Optim. 14, 670–698 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Toh K.-C.: An inexact primal-dual path following algorithm for convex quadratic SDP. Math. Program. 112, 221–254 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    Toh K.-C., Todd M.J., Tütüncü R.H.: SDPT3—a MATLAB software package for semidefinite programming, version 1.3. Optim. Methods Softw. 11/12, 545–581 (1999)CrossRefGoogle Scholar
  35. 35.
    Tsuchiya T., Xia Y.: An extension of the standard polynomial-time primal-dual path-following algorithm to the weighted determinant maximization problem with semidefinite constraints. Pac. J. Optim. 3, 165–182 (2007)zbMATHMathSciNetGoogle Scholar
  36. 36.
    Ueno U., Tsuchiya T.: Covariance regularization in inverse space. Q. J. R. Meteorol. Soc. 135, 1133–1156 (2009)CrossRefGoogle Scholar
  37. 37.
    Vandenberghe L., Boyd S., Wu S.-P.: Determinant maximization with linear matrix inequality constraints. SIAM J. Matrix Anal. Appl. 19, 499–533 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  38. 38.
    Wang C., Sun D., Toh K.-C.: Solving log-determinant optimization problems by a newton-cg proximal point algorithm. SIAM J. Optim. 20, 2994–3013 (2010)CrossRefGoogle Scholar
  39. 39.
    Whittaker, J.: Graphical models in applied multivariate statistics. In: Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, Chichester (1990)Google Scholar
  40. 40.
    Wille, A., Zimmermann, P., Vranová, E., Fürholz, A., Laule, O., Bleuler, S., Hennig, L., Prelić, A., von Rohr, P., Thiele, L., Zitzler, E., Gruissem, W., Bühlmann, P.: Sparse graphical gaussian modeling of the isoprenoid gene network in Arabidopsis thaliana. Genome Biol. 5, R92 (2004)Google Scholar
  41. 41.
    Wong F., Carter C.K., Kohn R.: Efficient estimation of covariance selection models. Biometrika 90, 809–830 (2003)CrossRefMathSciNetGoogle Scholar
  42. 42.
    Wu W.B., Pourahmadi M.: Nonparameteric estimation of large covariance matrices of longitudinal data. Biometrika 90, 831–844 (2003)CrossRefMathSciNetGoogle Scholar
  43. 43.
    Yeung K.Y., Bumgarner R.E., Raftery A.E.: Bayesian model averaging: development of an improved multi-class, gene selection and classification tool for microarray data. Bioinformatics 21, 2394–2402 (2005)CrossRefGoogle Scholar
  44. 44.
    Yuan M., Lin Y.: Model selection and estimation in the Gaussian graphical model. Biometrika 94, 19–35 (2007)zbMATHCrossRefMathSciNetGoogle Scholar
  45. 45.
    Yuan, X.: Alternating direction methods for sparse covariance selection. Preprint (2009)Google Scholar
  46. 46.
    Zhang Y.: On extending some primal-dual interior-point algorithms from linear programming to semidefinite programming. SIAM J. Optim. 8, 365–386 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  47. 47.
    Zhao X.Y., Sun D., Toh K.-C.: A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. 20, 1737–1765 (2010)CrossRefMathSciNetGoogle Scholar
  48. 48.
    Zhou G., Toh K.-C.: Polynomiality of an inexact infeasible interior point algorithm for semidefinite programming. Math. Program. 99, 261–282 (2004)zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer and Mathematical Optimization Society 2010

Authors and Affiliations

  1. 1.Department of MathematicsNational University of SingaporeSingaporeSingapore
  2. 2.Singapore-MIT AllianceSingaporeSingapore

Personalised recommendations