Abstract
We suggest a novel approach to the sparse covariance matrix estimation (SCME) problem using the ℓ1-norm. The resulting optimization problem is nonconvex and very hard to solve. Fortunately, it can be reformulated as DC (Difference of Convex functions) programs to which DC programming and DC Algorithms can be investigated. The main contribution of this paper is to propose a more suitable DC decomposition for solving the SCME problem. The experimental results on both simulated datasets and two real datasets in classification problem illustrate the efficiency of the proposed algorithms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bickel, P.J., Levina, E.: Some theory for Fisher’s linear discriminant function, naive Bayes, and some alternatives when there are many more variables than observations. Bernoulli 10(6), 989–1010 (2004)
Bickel, P.J., Levina, E.: Regularized estimation of large covariance matrices. The Annals of Statistichs 36, 199–227 (2008)
Bien, J., Tibshirani, R.: Sparse estimation of a covariance matrix. Biometrika 98(4), 807–820 (2011)
Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundat. Trends Mach. Learn. 3(1), 1–122 (2011)
Cai, T., Zhang, C., Zou, H.: Optimal rates of convergence for covariance matrix estimation. The Annals of Statistic 38, 2118–2144 (2010)
Chaudhuri, S., Drton, M., Richardson, T.S.: Estimation of a covariance matrix with zeros. Biometrika 94, 199–216 (2007)
Fawzi, A., Davies, M., Frossard, P.: Dictionary learning for fast classification based on soft-thresholding. International Journal of Computer Vision (2014), http://arxiv.org/abs/1402.1973
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer, New York (2009)
Huang, J.Z., Liu, N., Pourahmadi, M., Liu, L.: Covariance matrix selection and estimation via penalised normal likelihood. Biometrika 93, 85–98 (2006)
Khare, K., Rajaratnam, B.: Wishart distributions for decomposable covariance graph models. Ann. Statist. 39, 514–555 (2011)
Lam, C., Fan, J.: Sparsistency and rates of convergence in large covariance matrix estimation. The Annals of Statistics 37, 4254–4278 (2009)
Le Thi, H.A., Huynh Van, N., Pham Dinh, T.: Exact penalty and error bounds in DC programming. Journal of Global Optimization 52(3), 509–535 (2012)
Le Thi, H.A., Le Hoai, M., Nguyen, V.V., Pham Dinh, T.: A DC Programming approach for feature selection in support vector machines learning. Journal of Advances in Data Analysis and Classification 2(3), 259–278 (2008)
Le Thi, H.A., Le Hoai, M., Pham Dinh, T.: Optimization based DC programming and DCA for hierarchical clustering. European Journal of Operational Research 183, 1067–1085 (2007)
Le Thi, H.A., Le Hoai, M., Pham Dinh, T.: Feature selection in machine learning: An exact penalty approachusing a difference of convex function algorithm. Machine Learning (2014), (published online July 04, 2014), doi:10.1007/s10994-014-5455-y
Le Thi, H.A., Nguyen, M.C.: Self-organizing maps by difference of convex functions optimization. Data Mining and Knowledge Discovery 28, 1336–1365 (2014)
Le Thi, H.A., Pham Dinh, T.: Solving a class of linearly constrained indefinite quadratic problems by D.C. algorithms. Journal of Global Optimization 11, 253–285 (1997)
Le Thi, H.A., Pham Dinh, T.: The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Annals of Operations Research 133, 23–46 (2005)
Le Thi, H.A., Pham Dinh, T., Le Hoai, M., Vo Xuan, T.: DC approximation approaches for sparse optimization. To appear in European Journal of Operational Research (2014)
Le Thi, H.A., Vo Xuan, T., Pham Dinh, T.: Feature selection for linear SVMs under uncertain data: robust optimization based on difference of convex functions Algorithms. Neural Networks 59, 36–50 (2014)
Levina, E., Rothman, A., Zhu, J.: Sparse estimation of large covariance matrices via a nested lasso penalty. Ann. Appl. Stat. 2(1), 245–263 (2008)
Liu, H., Wang, L., Zhao, T.: Sparse covariance matrix estimation with eigenvalue contraints. Journal of Computational and Graphical Statistics 23(2), 439–459 (2014)
Liu, Y., Shen, X.: Multicategory ψ-Learning. Journal of the American Statistical Association 101, 500–509 (2006)
Liu, Y., Shen, X., Doss, H.: Multicategory ψ-Learning and Support Vector Machine: Computational Tools. Journal of Computational and Graphical Statistics 14, 219–236 (2005)
Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Academic (1979)
Pham Dinh, T., Le Thi, H.A.: Convex analysis approach to D.C. programming: Theory, algorithms and applications. Acta Mathematica Vietnamica 22(1), 289–355 (1997)
Pham Dinh, T., Le Thi, H.A.: DC optimization algorithm for solving the trust-region subproblem. SIAM Journal of Optimization 8(1), 476–505 (1998)
Rothman, A.J., Levina, E., Zhu, J.: Generalized thresholding of large covariance matrices. J. Am. Statist. Assoc. 104, 177–186 (2009)
Rothman, A.J., Levina, E., Zhu, J.: A new approach to Cholesky-based covariance regularization in high dimensions. Biometrika 97, 539–550 (2010)
Thiao, M., Pham Dinh, T., Le Thi, H.A.: DC Programming. In: Le Thi, H.A., Bouvry, P., Pham Dinh, T. (eds.) MCO 2008. CCIS, vol. 14, pp. 348–357. Springer, Heidelberg (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Phan, D.N., Le Thi, H.A., Dinh, T.P. (2015). A DC Programming Approach for Sparse Estimation of a Covariance Matrix. In: Le Thi, H., Pham Dinh, T., Nguyen, N. (eds) Modelling, Computation and Optimization in Information Systems and Management Sciences. Advances in Intelligent Systems and Computing, vol 359. Springer, Cham. https://doi.org/10.1007/978-3-319-18161-5_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-18161-5_12
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-18160-8
Online ISBN: 978-3-319-18161-5
eBook Packages: EngineeringEngineering (R0)