Advertisement

Mathematical Programming

, Volume 167, Issue 1, pp 129–154 | Cite as

Interpreting latent variables in factor models via convex optimization

  • Armeen Taeb
  • Venkat ChandrasekaranEmail author
Full Length Paper Series B
  • 377 Downloads

Abstract

Latent or unobserved phenomena pose a significant difficulty in data analysis as they induce complicated and confounding dependencies among a collection of observed variables. Factor analysis is a prominent multivariate statistical modeling approach that addresses this challenge by identifying the effects of (a small number of) latent variables on a set of observed variables. However, the latent variables in a factor model are purely mathematical objects that are derived from the observed phenomena, and they do not have any interpretation associated to them. A natural approach for attributing semantic information to the latent variables in a factor model is to obtain measurements of some additional plausibly useful covariates that may be related to the original set of observed variables, and to associate these auxiliary covariates to the latent variables. In this paper, we describe a systematic approach for identifying such associations. Our method is based on solving computationally tractable convex optimization problems, and it can be viewed as a generalization of the minimum-trace factor analysis procedure for fitting factor models via convex optimization. We analyze the theoretical consistency of our approach in a high-dimensional setting as well as its utility in practice via experimental demonstrations with real data.

Keywords

Canonical correlations analysis Factor analysis Log-determinant optimization Minimum-trace factor analysis Nuclear-norm relaxation Semidefinite programming 

Mathematics Subject Classification

90C34 90C47 90C90 90C25 62-07 62F12 62H25 

Notes

Acknowledgements

The authors were supported in part by NSF Career Award CCF-1350590, by Air Force Office of Scientific Research Grant Nos. FA9550-14-1-0098 and FA9550-16-1-0210, by a Sloan research fellowship, and the Resnick Sustainability Institute at Caltech. Armeen Taeb would like to thank Yong Sheng Soh for many helpful discussions.

Supplementary material

10107_2017_1187_MOESM1_ESM.pdf (285 kb)
Supplementary material 1 (pdf 285 KB)

References

  1. 1.
    Bickel, P., Doksum, K.: Mathematical Statistics, Basic Ideas and Selected Topics. Prentice-Hall, Prentice (2007)zbMATHGoogle Scholar
  2. 2.
    Bühlmann, P., van de Geer, S.: Statistics for High-Dimensional Data. Springer, New York (2011)CrossRefzbMATHGoogle Scholar
  3. 3.
    Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9, 717–772 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Cavalho, C., Chang, J., Lucas, J., Nevins, J., Wang, Q., West, M.: High-dimensional sparse factor modeling: applications in gene expression genomics. J. Am. Stat. Assoc. 103, 1438–1456 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chandrasekaran, V., Sanghavi, S., Parrilo, P.A., Willsky, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21, 572–596 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Chandrasekaran, V., Parrilo, P.A., Willsky, A.S.: Latent variable graphical model selection via convex optimization. Ann. Stat. 40, 1935–1967 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Davidson, K.R., Szarek, S.J.: Local operator theory, random matrices and Banach spaces. Handb. Geom. Banach Spaces 1, 317–366 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Fazel, M.: Matrix rank minimization with applications. Ph.D. thesis, Department of Electrical Engineering, Stanford University (2002)Google Scholar
  9. 9.
    Hotelling, H.: Relations between two sets of variants. Biometrika 28, 321–377 (2002)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kay, S.M.: Modern Spectral Estimation, Theory and Application. Prentice-Hall, Prentice (1988)zbMATHGoogle Scholar
  11. 11.
    Ledermann, W.: On a problem concerning matrices with variable diagonal elements. Proc. R. Soc. Edinb. 60, 1–17 (1940)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Meinhausen, N., Bülmann, A.P.: High dimensional graphs and variable selection with the Lasso. Ann. Stat. 34, 1436–1462 (2006)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Mesbahi, M., Papavassilopoulos, G.: On the rank minimization problem over a positive semidefinite linear matrix inequality. IEEE Trans. Autom. Control 42, 239–243 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Ravikumar, P., Wainwright, M.J., Raskutti, G., Yu, B.: High-dimensional covariance estimation by minimizing \(\ell _1\)-penalized log-determinant divergence. Electron. J. Stat. 4, 935–980 (2011)CrossRefzbMATHGoogle Scholar
  16. 16.
    Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum rank solutions to linear matrix equations via nuclear norm minimization. SIAM Rev. 52, 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Saunderson, J., Chandrasekaran, V., Parrillo, P., Willsky, S.: Diagonal and low-rank matrix decompositions, correlation matrices, and ellipsoid fitting. SIAM J. Matrix Anal. 33, 1395–1416 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Shapiro, A.: Rank-reducibility of a symmetric matrix and sampling theory of minimum trace factor analysis. Psychometrika 47, 187–199 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Shapiro, A.: Weighted minimum trace factor analysis. Psychometrika 47, 243–264 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Shapiro, A.: Identifiability of factor analysis: some results and open problems. Linear Algebra Appl. 15, 201–292 (1904)Google Scholar
  21. 21.
    Spearman, C.: ‘General intelligence’, objectively determined and measured. Am. J. Psychol. 15, 201–292 (1904)CrossRefGoogle Scholar
  22. 22.
    Toh, K.C., Todd, M.J., Tutuncu, R.H.: SDPT3—a MATLAB software package for semidefinite-quadratic-linear programming. http://www.math.nus.edu.sg/~mattohkc/sdpt3.html
  23. 23.
    Wainwright, M.J.: Sharp thresholds for noisy and high-dimensional recovery of sparsity using \(\ell _1\)-constrained quadratic programming (Lasso). IEEE Trans. Inf. Theory 55, 2183–2202 (2009)CrossRefzbMATHGoogle Scholar
  24. 24.
    Wainwright, M.J.: Structured regularizers for high-dimensional problems: statistical and computational issues. Annu. Rev. Stat. Appl. 1, 233–253 (2014)CrossRefGoogle Scholar
  25. 25.
    Zhao, P., Yu, B.: On model selection consistency of the lasso. J. Mach. Learn. Res. 7, 2541–2567 (2006)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany and Mathematical Optimization Society 2017

Authors and Affiliations

  1. 1.Department of Electrical EngineeringCalifornia Institute of TechnologyPasadenaUSA
  2. 2.Departments of Computing and Mathematical Sciences and of Electrical EngineeringCalifornia Institute of TechnologyPasadenaUSA

Personalised recommendations