Skip to main content
SpringerLink
Log in

Sparse PCA on fixed-rank matrices

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

Sparse PCA is the optimization problem obtained from PCA by adding a sparsity constraint on the principal components. Sparse PCA is NP-hard and hard to approximate even in the single-component case. In this paper we settle the computational complexity of sparse PCA with respect to the rank of the covariance matrix. We show that, if the rank of the covariance matrix is a fixed value, then there is an algorithm that solves sparse PCA to global optimality, whose running time is polynomial in the number of features. We also prove a similar result for the version of sparse PCA which requires the principal components to have disjoint supports.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Asteris, M., Papailiopoulos, D., Karystinos, G.: Sparse principal component of a rank-deficient matrix. In: Proceedings of ISIT (2011)

  2. Asteris, M., Papailiopoulos, D., Karystinos, G.: The sparse principal component of a constant-rank matrix. IEEE Trans. Inf. Theor. 60(4), 2281–2290 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  3. Asteris, M., Papailiopoulos, D., Kyrillidis, A., Dimakis, A.: Sparse PCA via bipartite matchings. In: Proceedings of NIPS (2015)

  4. Berk, L., Bertsimas, D.: Certifiably optimal sparse principal component analysis. Math. Progr. Comput. 11, 381–420 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bertsimas, D., Tsitsiklis, J.: Introduction to Linear Optimization. Athena Scientific, Belmont (1997)

    Google Scholar 

  6. Boutsidis, C., Drineas, P., Magdon-Ismail, M.: Sparse features for PCA-like linear regression. In: Proceedings of NIPS, pp. 2285–2293 (2011)

  7. Cadima, J., Jolliffe, I.: Loading and correlations in the interpretation of principle compenents. J. Appl. Stat. 22(2), 203–214 (1995)

    Article  Google Scholar 

  8. Chan, S., Papailiopoulos, D., Rubinstein, A.: On the worst-case approximability of sparse PCA. Proceedings of COLT (2016)

  9. d’Aspremont, A., Bach, F., Ghaoui, L.: Optimal solutions for sparse principal component analysis. J. Mach. Learning Res. 9, 1269–1294 (2008)

    MathSciNet  MATH  Google Scholar 

  10. d’Aspremont, A., Bach, F., Ghaoui, L.: Approximation bounds for sparse principal component analysis. Mathematical Programming Series . 148(1), pp. 89–110

  11. d’Aspremont, A., El Ghaoui, L., Jordan, M., Lanckriet, G.: A direct formulation for sparse PCA using semidefinite programming. SIAM Rev. 49(3), 434–448 (2007)

  12. Dey, S., Mazumder, R., Wang, G.: A convex integer programming approach for optimal sparse PCA. arXiv preprint arXiv:1810.09062 (2018)

  13. Edelsbrunner, H., O’Rourke, J., Seidel, R.: Constructing arrangements of lines and hyperplanes with applications. SIAM J. Comput. 15(2), 341–363 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  14. Goldberg, A., Tarjan, R.: Finding minimum-cost circulations by canceling negative cycles. In: Proceedings of STOC, pp. 388–397 (1988)

  15. Goldberg, A., Tarjan, R.: Finding minimum-cost circulations by canceling negative cycles. J. Assoc. Comput. Mach. 36, 873–886 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  16. Hastie, T., Tibshirani, R., Wainwright, M.: Stat. learning Sparsity. CRC Press, Boca Raton (2015)

    Book  MATH  Google Scholar 

  17. He, Y., Monteiro, R., Park, H.: An efficient algorithm for rank-1 sparse PCA. In: working paper (2010)

  18. Jolliffe, I., Trendafilov, N., Uddin, M.: A modified principal component technique based on the lasso. J. Comput. Graph. Stat. 12(3), 531–547 (2003)

    Article  MathSciNet  Google Scholar 

  19. Journée, M., Nesterov, Y., Richtárik, P., Sepulchre, R.: Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11, 517–553 (2010)

    MathSciNet  MATH  Google Scholar 

  20. Karystinos, G., Liavas, A.: Efficient computation of the binary vector that maximizes a rank-deficient quadratic form. IEEE Trans. Inf. Theor. 56(7), 3581–3593 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  21. Karystinos, G., Pados, D.: Rank-2-optimal adaptive design of binary spreading codes. IEEE Trans. Inf. Theor. 53(9), 3075–3080 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  22. Mackenthun, K.: A fast algorithm for multiple-symbol differential detection of MPSK. IEEE Trans. Commun. 42(234), 1471–1474 (1994)

    Article  Google Scholar 

  23. Mackey, L.: Deflation methods for sparse PCA. In: Proceedings of NIPS, vol. 21, pp. 1017–1024 (2009)

  24. Magdon-Ismail, M.: NP-hardness and inapproximability of sparse PCA. Inf. Process. Lett. 126, 35–38 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  25. Moghaddam, B., Weiss, Y., Avidan, S.: Spectral bounds for sparse PCA: exact and greedy algorithms. In: Proceedings of NIPS, vol. 18, p. 915 (2006)

  26. Motedayen, I., Krishnamoorthy, A., Anastasopoulos, A.: Optimal joint detection/estimation in fading channels with polynomial complexity. IEEE Trans. Inf. Theor. 53(1), 209–223 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  27. Papailiopoulos, D., Dimakis, A., Korokythakis, S.: Sparse PCA through low-rank approximations. In: Proceedings of ICML (2013)

  28. Schrijver, A.: Combinatorial Optimization . Polyhedra and Efficiency. Springer-Verlag, Berlin (2003)

    MATH  Google Scholar 

  29. Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning. Cambridge University Press, Cambridge (2014)

    Book  MATH  Google Scholar 

  30. Sigg, C., Buhmann, J.: Expectation-maximization for sparse and non-negative PCA. In: Proceedings of ICML, pp. 960–967 (2008)

  31. Sweldens, W.: Fast block noncoherent decoding. IEEE Commun. Lett. 5(4), 132–134 (2001)

    Article  MathSciNet  Google Scholar 

  32. Vu, V., Lei, J.: Minimax rates of estimation for sparse PCA in high dimensions. In: Proceedings of AIStats, pp. 1278–1286 (2012)

  33. Yuan, X., Zhang, T.: Truncated power method for sparse eigenvalue problems. J. Mach. Learn. Res. 14, 899–925 (2013)

    MathSciNet  MATH  Google Scholar 

  34. Zhang, Y., d’Aspremont, A., Ghaoui, L.E.: Sparse PCA: convex relaxations, algorithms and applications. In: Anjos, M., Lasserre, J. (eds.) Handbook on Semidefinite, Conic and Polynomial Optimization. International Series in Operations Research & Management Science, vol. 166, pp. 915–940. Springer, Boston, MA (2011). https://doi.org/10.1007/978-1-4614-0769-0_31

    Chapter  MATH  Google Scholar 

  35. Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. J. Comput. Graph. Stat. 15(2), 265–286 (2006)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial and Systems Engineering & Wisconsin Institute for Discovery, University of Wisconsin-Madison, Madison, WI, USA

    Alberto Del Pia

Authors
  1. Alberto Del Pia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alberto Del Pia.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is supported by ONR grant N00014-19-1-2322. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Office of Naval Research.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Del Pia, A. Sparse PCA on fixed-rank matrices. Math. Program. 198, 139–157 (2023). https://doi.org/10.1007/s10107-022-01769-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-022-01769-9

Keywords

Mathematics Subject Classification

Search

Navigation