Mathematical Programming

, Volume 162, Issue 1–2, pp 325–361 | Cite as

Finding a low-rank basis in a matrix subspace

  • Yuji Nakatsukasa
  • Tasuku SomaEmail author
  • André Uschmajew
Full Length Paper Series A


For a given matrix subspace, how can we find a basis that consists of low-rank matrices? This is a generalization of the sparse vector problem. It turns out that when the subspace is spanned by rank-1 matrices, the matrices can be obtained by the tensor CP decomposition. For the higher rank case, the situation is not as straightforward. In this work we present an algorithm based on a greedy process applicable to higher rank problems. Our algorithm first estimates the minimum rank by applying soft singular value thresholding to a nuclear norm relaxation, and then computes a matrix with that rank using the method of alternating projections. We provide local convergence results, and compare our algorithm with several alternative approaches. Applications include data compression beyond the classical truncated SVD, computing accurate eigenvectors of a near-multiple eigenvalue, image separation and graph Laplacian eigenproblems.


Low-rank matrix subspace \(\ell ^1\) relaxation Alternating projections Singular value thresholding Matrix compression 

Mathematics Subject Classification

90C26 Nonconvex programming, global optimization 


  1. 1.
    Abolghasemi, V., Ferdowsi, S., Sanei, S.: Blind separation of image sources via adaptive dictionary learning. IEEE Trans. Image Process. 21(6), 2921–2930 (2012)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Ames, B.P.W., Vavasis, S.A.: Nuclear norm minimization for the planted clique and biclique problems. Math. Program. 129(1 Ser. B), 69–89 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Andersson, F., Carlsson, M.: Alternating projections on nontangential manifolds. Constr. Approx. 38(3), 489–525 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bai, Z., Demmel, J., Dongarra, J., Ruhe, A., van der Vorst, H. (eds.): Templates for the solution of algebraic eigenvalue problems. A practical guide. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (2000)Google Scholar
  5. 5.
    Barak, B., Kelner, J.A., Steurer, D.: Rounding sum-of-squares relaxations. In: Proceedings of the 46th Annual ACM Symposium on Theory of Computing, pp. 31–40 (2014)Google Scholar
  6. 6.
    Bell, A.J., Sejnowski, T.J.: An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7(6), 1129–1159 (1995)CrossRefGoogle Scholar
  7. 7.
    Bühlmann, P., van de Geer, S.: Statistics for High-Dimensional Data. Methods, Theory and Applications. Springer, Heidelberg (2011)CrossRefzbMATHGoogle Scholar
  8. 8.
    Cai, J.-F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20(4), 1956–1982 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Candès, E.J.: The restricted isometry property and its implications for compressed sensing. C. R. Math. Acad. Sci. Paris 346(9–10), 589–592 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Candes, E.J., Tao, T.: Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inform. Theory 52(12), 5406–5425 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Candès, E.J., Tao, T.: The power of convex relaxation: near-optimal matrix completion. IEEE Trans. Inform. Theory 56(5), 2053–2080 (2010)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Carroll, J.D., Chang, J.-J.: Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition. Psychometrika 35(3), 283–319 (1970)CrossRefzbMATHGoogle Scholar
  14. 14.
    Cichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., Phan, H.A.: Tensor decompositions for signal processing applications: from two-way to multiway component analysis. IEEE Signal Proc. Mag. 32(2), 145–163 (2015)CrossRefGoogle Scholar
  15. 15.
    Coleman, T.F., Pothen, A.: The null space problem. I. Complexity. SIAM J. Algebraic Discrete Methods 7(4), 527–537 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    De Lathauwer, L.: A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl. 28(3), 642–666 (2006). (electronic)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    De Lathauwer, L.: Decompositions of a higher-order tensor in block terms. II. Definitions and uniqueness. SIAM J. Matrix Anal. Appl. 30(3), 1033–1066 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000). (electronic)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition. SIAM J. Matrix Anal. Appl. 26(2), 295–327 (electronic) (2004/2015)Google Scholar
  20. 20.
    Demanet, L., Hand, P.: Scaling law for recovering the sparsest element in a subspace. Inf. Inference 3(4), 295–309 (2014)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Demmel, J.W.: Applied Numerical Linear Algebra. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (1997)CrossRefzbMATHGoogle Scholar
  22. 22.
    Domanov, I., De Lathauwer, L.: Canonical polyadic decomposition of third-order tensors: reduction to generalized eigenvalue decomposition. SIAM J. Matrix Anal. Appl. 35(2), 636–660 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Drusvyatskiy, D., Ioffe, A.D., Lewis, A.S.: Transversality and alternating projections for nonconvex sets. Found. Comput. Math. 15(6), 1637–1651 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Edmonds, J.: Systems of distinct representatives and linear algebra. J. Res. Nat. Bur. Stand. Sect. B 71B, 241–245 (1967)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Fazel, M.: Matrix rank minimization with applications. Ph.D. thesis, Electrical Engineering Deptartment Stanford University (2002)Google Scholar
  26. 26.
    Fazel, M., Hindi, H., Boyd, S.P.: A rank minimization heuristic with application to minimum order system approximation. In: Proceedings of the 2001 American Control Conference, pp. 4734–4739 (2001)Google Scholar
  27. 27.
    Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins University Press, Baltimore, MD (2013)zbMATHGoogle Scholar
  28. 28.
    Grant, M., Boyd, S.: CVX: Matlab Software for Disciplined Convex Programming, version 2.1, March 2014.
  29. 29.
    Gurvits, L.: Classical complexity and quantum entanglement. J. Comput. Syst. Sci. 69(3), 448–484 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Harshman, R.A.: Foundations of the PARAFAC procedure: models and conditions for an “explanatory” multi-modal factor analysis. UCLA Working Papers in Phonetics 16, 1–84 (1970)Google Scholar
  31. 31.
    Harvey, N.J.A., Karger, D.R., Murota, K.: Deterministic network coding by matrix completion. In: Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 489–498 (2005)Google Scholar
  32. 32.
    Harvey, N. J. A., Karger, D. R., Yekhanin, S.: The complexity of matrix completion. In: Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithm, pp. 1103–1111 (2006)Google Scholar
  33. 33.
    Håstad, J.: Tensor rank is NP-complete. J. Algorithms 11(4), 644–654 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Helmke, U., Shayman, M.A.: Critical points of matrix least squares distance functions. Linear Algebra Appl. 215, 1–19 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Hillar, C.J., Lim, L.-H.: Most tensor problems are NP-hard. J. ACM 60(6), Art. 45, 39 (2013)Google Scholar
  36. 36.
    Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys. 6, 164–189 (1927)CrossRefzbMATHGoogle Scholar
  37. 37.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07-49, University of Massachusetts, Amherst (2007)Google Scholar
  38. 38.
    Ivanyos, G., Karpinski, M., Qiao, Y., Santha, M.: Generalized Wong sequences and their applications to Edmonds’ problems. In: Proceedings of the 31st International Symposium on Theoretical Aspects of Computer Science, vol. 117543, pp. 397–408 (2014)Google Scholar
  39. 39.
    Kindermann, S., Navasca, C.: News algorithms for tensor decomposition based on a reduced functional. Numer. Linear Algebra Appl. 21(3), 340–374 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Leurgans, S.E., Ross, R.T., Abel, R.B.: A decomposition for three-way arrays. SIAM J. Matrix Anal. Appl. 14(4), 1064–1083 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Lewis, A.S., Luke, D.R., Malick, J.: Local linear convergence for alternating and averaged nonconvex projections. Found. Comput. Math. 9(4), 485–513 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Lewis, A.S., Malick, J.: Alternating projections on manifolds. Math. Oper. Res. 33(1), 216–234 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Li, N., Kindermann, S., Navasca, C.: Some convergence results on the regularized alternating least-squares method for tensor decomposition. Linear Algebra Appl. 438(2), 796–812 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Liu, Y.-J., Sun, D., Toh, K.-C.: An implementable proximal point algorithmic framework for nuclear norm minimization. Math. Program. 133(1—-2, Ser. A), 399–436 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Liu, Z., Vandenberghe, L.: Interior-point method for nuclear norm approximation with application to system identification. SIAM J. Matrix Anal. Appl. 31(3), 1235–1256 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Lovász, L.: Singular spaces of matrices and their application in combinatorics. Bol. Soc. Brasil. Math. 20(1), 87–99 (1989)Google Scholar
  48. 48.
    Mohlenkamp, M.J.: Musings on multilinear fitting. Linear Algebra Appl. 438(2), 834–852 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Motwani, R., Raghavan, P.: Randomized Algorithms. Chapman and Hall/CRC, Routledge (2010)zbMATHGoogle Scholar
  50. 50.
    Noll, D., Rondepierre, A.: On local convergence of the method of alternating projections. Found. Comput. Math. 16(2), 425–455 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  51. 51.
    Oxley, J.: Infinite matroids. In: White, N. (ed.) Matroid Applications, pp. 73–90. Cambridge University Press, Cambridge (1992)CrossRefGoogle Scholar
  52. 52.
    Qu, Q., Sun, J., Wright, J.: Finding a sparse vector in a subspace: linear sparsity using alternating directions. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, pp. 3401–3409. Curran Associates, Inc, Red Hook (2014)Google Scholar
  53. 53.
    Qu, Q., Sun, J., Wright, J.: Finding a sparse vector in a subspace: linear sparsity using alternating directions. arXiv:1412.4659 (2014)
  54. 54.
    Recht, B.: A simpler approach to matrix completion. J. Mach. Learn. Res. 12, 3413–3430 (2011)MathSciNetzbMATHGoogle Scholar
  55. 55.
    Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  56. 56.
    Sorber, L., Van Barel, M., De Lathauwer, L.: Tensorlab v2.0.
  57. 57.
    Spielman, D.A., Wang, H., Wright, J.: Exact recovery of sparsely-used dictionaries. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI ’13, pp. 3087–3090. AAAI Press (2013)Google Scholar
  58. 58.
    Stewart, G.W.: Matrix Algorithms. Vol. II. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (2001)CrossRefzbMATHGoogle Scholar
  59. 59.
    Stewart, G.W., Sun, J.G.: Matrix Perturbation Theory. Computer Science and Scientific Computing. Academic Press, Inc., Boston, MA (1990)Google Scholar
  60. 60.
    Sun, J., Qu, Q., Wright, J.: Complete dictionary recovery over the sphere I: Overview and the geometric picture. arXiv:1511.03607 (2015)
  61. 61.
    Sun, J., Qu, Q., Wright, J.: Complete dictionary recovery over the sphere II: Recovery by Riemannian trust-region method. arXiv:1511.04777 (2015)
  62. 62.
    Uschmajew, A.: Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM J. Matrix Anal. Appl. 33(2), 639–652 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  63. 63.
    Uschmajew, A.: A new convergence proof for the higher-order power method and generalizations. Pac. J. Optim. 11(2), 309–321 (2015)MathSciNetzbMATHGoogle Scholar
  64. 64.
    Wang, L., Chu, M.T.: On the global convergence of the alternating least squares method for rank-one approximation to generic tensors. SIAM J. Matrix Anal. Appl. 35(3), 1058–1072 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  65. 65.
    Wedin, P.-Å.: Perturbation bounds in connection with singular value decomposition. Nordisk Tidskr. Informationsbehandling (BIT) 12, 99–111 (1972)Google Scholar
  66. 66.
    Xu, Y., Yin, W.: A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci. 6(3), 1758–1789 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  67. 67.
    Zhao, X., Zhou, G., Dai, W., Xu, T., Wang, W.: Joint image separation and dictionary learning. In: 18th International Conference on Digital Signal Processing (DSP), pp. 1–6. IEEE (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2016

Authors and Affiliations

  1. 1.Mathematical InstituteUniversity of OxfordOxfordUK
  2. 2.Graduate School of Information Science and TechnologyThe University of TokyoTokyoJapan
  3. 3.Hausdorff Center for Mathematics & Institute for Numerical SimulationUniversity of BonnBonnGermany

Personalised recommendations