Skip to main content

Nonsmooth rank-one matrix factorization landscape

Optimization Letters volume 16, pages 1611–1631 (2022)Cite this article

Abstract

We provide the first positive result on the nonsmooth optimization landscape of robust principal component analysis, to the best of our knowledge. It is the object of several conjectures and remains mostly uncharted territory. We identify a necessary and sufficient condition for the absence of spurious local minima in the rank-one case. Our proof exploits the subdifferential regularity of the objective function in order to eliminate the existence quantifier from the first-order optimality condition known as Fermat’s rule.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Anderson, B. G., Sojoudi, S.: Global optimality guarantees for nonconvex unsupervised video segmentation. Annual Allerton Conference on Communication, Control, and Computing, (2019)

  2. Baldi, P., Hornik, K.: neural networks and principal component analysis: learning from examples without local minima. Neural Netw. 2(1), 53–58 (1989)

    Article  Google Scholar 

  3. Beck, A.: First-order methods in optimization. SIAM, New Delhi (2017)

    Book  Google Scholar 

  4. Bertsimas, D., Cory-Wright, R., Pauphilet, J.: Solving large-scale sparse pca to certifiable (near) optimality. arXiv preprint arXiv:2005.05195, (2020)

  5. Bouwmans, T., Sobral, A., Javed, S., Jung, S.K., Zahzah, E.-H.: Decomposition into low-rank plus additive matrices for background/foreground separation: A review for a comparative evaluation with a large-scale dataset. Comput. Sci. Rev. 23, 1–71 (2017)

    Article  Google Scholar 

  6. Burke, J.V., Ferris, M.C.: Weak sharp minima in mathematical programming. SIAM J. Control Optim. 31(5), 1340–1359 (1993)

    MathSciNet  Article  Google Scholar 

  7. Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM (JACM) 58(3), 1–37 (2011)

    MathSciNet  Article  Google Scholar 

  8. Candès, E.J., Tao, T.: The power of convex relaxation: Near-optimal matrix completion. IEEE Trans. Inf. Theory 56(5), 2053–2080 (2010)

    MathSciNet  Article  Google Scholar 

  9. Chandrasekaran, V., Sanghavi, S., Parrilo, P. A., Willsky, A. S.: Rank-sparsity incoherence for matrix decomposition. SIOPT, (2011)

  10. Charisopoulos, V., Chen, Y., Davis, D., Díaz, M., Ding, L., Drusvyatskiy, D.: Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence. Found. Comput. Math., pp. 1–89, 2021

  11. Chen, Y., Chi, Y.: Harnessing structures in big data via guaranteed low-rank matrix estimation: Recent theory and fast algorithms via convex and nonconvex optimization. IEEE Signal Process. Mag. 35(4), 14–31 (2018)

    Article  Google Scholar 

  12. Chi, Y., Lu, Y.M., Chen, Y.: Nonconvex optimization meets low-rank matrix factorization: an overview. IEEE Trans. Signal Process. 67(20), 5239–5269 (2019)

    MathSciNet  Article  Google Scholar 

  13. Clarke, F. H.: Optimization and nonsmooth analysis. SIAM Classics Appl. Math., 1990

  14. Du, Simon S., Hu, Wei, Lee, Jason D.: Algorithmic regularization in learning deep homogeneous models: layers are automatically balanced. NeurIPS 2018, (2018)

  15. Fattahi, S., Sojoudi, S.: Exact guarantees on the absence of spurious local minima for non-negative rank-1 robust principal component analysis. JMLR, 2020

  16. Garcia-Garcia, B., Bouwmans, T., Silva, A.J.R.: Background subtraction in real applications: challenges, current models and future directions. Comput. Sci. Rev. 35, 100–204 (2020)

    MathSciNet  Article  Google Scholar 

  17. Gillis, Nicolas, Vavasis, Stephen A.: On the complexity of robust PCA and l1-norm low-rank matrix approximation. Math. Op. Res. 43(4), 1072–1084 (2018)

    Article  Google Scholar 

  18. Gross, D.: Recovering low-rank matrices from few coefficients in any basis. IEEE Trans. Inf. Theory, (2011)

  19. Ioffe, A.D.: An invitation to tame optimization. SIAM J. Optim. 19(4), 1894–1917 (2009)

    MathSciNet  Article  Google Scholar 

  20. Jain, P., Kar, P.: Non-convex optimization for machine learning. Found Trends Mach. Learn., (2017)

  21. Josz, C., Ouyang, Y., Zhang, R. Y., Lavaei, J., Sojoudi, S.: A theory on the absence of spurious solutions for nonconvex and nonsmooth optimization. NeurIPS, (2018)

  22. Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Comput. (2009)

  23. Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. NeurIPS, (2014)

  24. Li, X., Zhu, Z., Man-Cho So, A., Vidal, R.: Nonconvex robust low-rank matrix recovery. SIAM J. Optim. (2019)

  25. Li, Xiao, Zhu, Zhihui, Man-Cho So, Anthony, Lee, Jason D.: Incremental methods for weakly convex optimization. 12th Annual Workshop on Optimization for Machine Learning, (2020)

  26. Moitra, A.: Algorithmic aspects of machine learning. Cambridge University Press, Cambridge (2018)

    Book  Google Scholar 

  27. Nesterov, Yu.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    MathSciNet  Article  Google Scholar 

  28. Nguyen, L.T., Kim, J., Shim, B.: Low-rank matrix completion: a contemporary survey. IEEE Access 7, 94215–94237 (2019)

    Article  Google Scholar 

  29. Recht, B.: A simpler approach to matrix completion. JMLR, (2011)

  30. Seidenberg, A.: A new decision method for elementary algebra. Annals Math., 365–374, (1954)

  31. Tarski, A.: A decision method for elementary algebra and geometry: prepared for publication with the assistance of JCC McKinsey. (1951)

  32. Valavi, H., Liu, S., Ramadge, P. J.: The landscape of matrix factorization revisited. AISTATS, (2020)

Download references

Acknowledgements

The authors acknowledge support from DTRA grant 13-1-0021, DARPA grant Lagrange, NSF grant 2023032, and ONR grant N00014-21-1-2282. Data sharing not applicable to this article as no datasets were generated or analysed during the current study

Author information

Authors and Affiliations

  1. Department of Industrial Engineering and Operations Research, Columbia University, Columbia, USA

    Cédric Josz & Lexiao Lai

Authors
  1. Cédric Josz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lexiao Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cédric Josz.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Josz, C., Lai, L. Nonsmooth rank-one matrix factorization landscape. Optim Lett 16, 1611–1631 (2022). https://doi.org/10.1007/s11590-021-01819-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-021-01819-9