# Finding a low-rank basis in a matrix subspace

## Abstract

For a given matrix subspace, how can we find a basis that consists of low-rank matrices? This is a generalization of the sparse vector problem. It turns out that when the subspace is spanned by rank-1 matrices, the matrices can be obtained by the tensor CP decomposition. For the higher rank case, the situation is not as straightforward. In this work we present an algorithm based on a greedy process applicable to higher rank problems. Our algorithm first estimates the minimum rank by applying soft singular value thresholding to a nuclear norm relaxation, and then computes a matrix with that rank using the method of alternating projections. We provide local convergence results, and compare our algorithm with several alternative approaches. Applications include data compression beyond the classical truncated SVD, computing accurate eigenvectors of a near-multiple eigenvalue, image separation and graph Laplacian eigenproblems.

## Keywords

Low-rank matrix subspace \(\ell ^1\) relaxation Alternating projections Singular value thresholding Matrix compression## Mathematics Subject Classification

90C26 Nonconvex programming, global optimization## References

- 1.Abolghasemi, V., Ferdowsi, S., Sanei, S.: Blind separation of image sources via adaptive dictionary learning. IEEE Trans. Image Process.
**21**(6), 2921–2930 (2012)MathSciNetCrossRefGoogle Scholar - 2.Ames, B.P.W., Vavasis, S.A.: Nuclear norm minimization for the planted clique and biclique problems. Math. Program.
**129**(1 Ser. B), 69–89 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 3.Andersson, F., Carlsson, M.: Alternating projections on nontangential manifolds. Constr. Approx.
**38**(3), 489–525 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 4.Bai, Z., Demmel, J., Dongarra, J., Ruhe, A., van der Vorst, H. (eds.): Templates for the solution of algebraic eigenvalue problems. A practical guide. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (2000)Google Scholar
- 5.Barak, B., Kelner, J.A., Steurer, D.: Rounding sum-of-squares relaxations. In: Proceedings of the 46th Annual ACM Symposium on Theory of Computing, pp. 31–40 (2014)Google Scholar
- 6.Bell, A.J., Sejnowski, T.J.: An information-maximization approach to blind separation and blind deconvolution. Neural Comput.
**7**(6), 1129–1159 (1995)CrossRefGoogle Scholar - 7.Bühlmann, P., van de Geer, S.: Statistics for High-Dimensional Data. Methods, Theory and Applications. Springer, Heidelberg (2011)CrossRefzbMATHGoogle Scholar
- 8.Cai, J.-F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim.
**20**(4), 1956–1982 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 9.Candès, E.J.: The restricted isometry property and its implications for compressed sensing. C. R. Math. Acad. Sci. Paris
**346**(9–10), 589–592 (2008)MathSciNetCrossRefzbMATHGoogle Scholar - 10.Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math.
**9**(6), 717–772 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 11.Candes, E.J., Tao, T.: Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inform. Theory
**52**(12), 5406–5425 (2006)MathSciNetCrossRefzbMATHGoogle Scholar - 12.Candès, E.J., Tao, T.: The power of convex relaxation: near-optimal matrix completion. IEEE Trans. Inform. Theory
**56**(5), 2053–2080 (2010)MathSciNetCrossRefGoogle Scholar - 13.Carroll, J.D., Chang, J.-J.: Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition. Psychometrika
**35**(3), 283–319 (1970)CrossRefzbMATHGoogle Scholar - 14.Cichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., Phan, H.A.: Tensor decompositions for signal processing applications: from two-way to multiway component analysis. IEEE Signal Proc. Mag.
**32**(2), 145–163 (2015)CrossRefGoogle Scholar - 15.Coleman, T.F., Pothen, A.: The null space problem. I. Complexity. SIAM J. Algebraic Discrete Methods
**7**(4), 527–537 (1986)MathSciNetCrossRefzbMATHGoogle Scholar - 16.De Lathauwer, L.: A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl.
**28**(3), 642–666 (2006). (electronic)MathSciNetCrossRefzbMATHGoogle Scholar - 17.De Lathauwer, L.: Decompositions of a higher-order tensor in block terms. II. Definitions and uniqueness. SIAM J. Matrix Anal. Appl.
**30**(3), 1033–1066 (2008)MathSciNetCrossRefzbMATHGoogle Scholar - 18.De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl.
**21**(4), 1253–1278 (2000). (electronic)MathSciNetCrossRefzbMATHGoogle Scholar - 19.De Lathauwer, L., De Moor, B., Vandewalle, J.: Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition. SIAM J. Matrix Anal. Appl.
**26**(2), 295–327 (electronic) (2004/2015)Google Scholar - 20.Demanet, L., Hand, P.: Scaling law for recovering the sparsest element in a subspace. Inf. Inference
**3**(4), 295–309 (2014)MathSciNetCrossRefGoogle Scholar - 21.Demmel, J.W.: Applied Numerical Linear Algebra. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (1997)CrossRefzbMATHGoogle Scholar
- 22.Domanov, I., De Lathauwer, L.: Canonical polyadic decomposition of third-order tensors: reduction to generalized eigenvalue decomposition. SIAM J. Matrix Anal. Appl.
**35**(2), 636–660 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 23.Drusvyatskiy, D., Ioffe, A.D., Lewis, A.S.: Transversality and alternating projections for nonconvex sets. Found. Comput. Math.
**15**(6), 1637–1651 (2015)MathSciNetCrossRefzbMATHGoogle Scholar - 24.Edmonds, J.: Systems of distinct representatives and linear algebra. J. Res. Nat. Bur. Stand. Sect. B
**71B**, 241–245 (1967)MathSciNetCrossRefzbMATHGoogle Scholar - 25.Fazel, M.: Matrix rank minimization with applications. Ph.D. thesis, Electrical Engineering Deptartment Stanford University (2002)Google Scholar
- 26.Fazel, M., Hindi, H., Boyd, S.P.: A rank minimization heuristic with application to minimum order system approximation. In: Proceedings of the 2001 American Control Conference, pp. 4734–4739 (2001)Google Scholar
- 27.Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins University Press, Baltimore, MD (2013)zbMATHGoogle Scholar
- 28.Grant, M., Boyd, S.: CVX: Matlab Software for Disciplined Convex Programming, version 2.1, March 2014. http://cvxr.com/cvx
- 29.Gurvits, L.: Classical complexity and quantum entanglement. J. Comput. Syst. Sci.
**69**(3), 448–484 (2004)MathSciNetCrossRefzbMATHGoogle Scholar - 30.Harshman, R.A.: Foundations of the PARAFAC procedure: models and conditions for an “explanatory” multi-modal factor analysis. UCLA Working Papers in Phonetics
**16**, 1–84 (1970)Google Scholar - 31.Harvey, N.J.A., Karger, D.R., Murota, K.: Deterministic network coding by matrix completion. In: Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 489–498 (2005)Google Scholar
- 32.Harvey, N. J. A., Karger, D. R., Yekhanin, S.: The complexity of matrix completion. In: Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithm, pp. 1103–1111 (2006)Google Scholar
- 33.Håstad, J.: Tensor rank is NP-complete. J. Algorithms
**11**(4), 644–654 (1990)MathSciNetCrossRefzbMATHGoogle Scholar - 34.Helmke, U., Shayman, M.A.: Critical points of matrix least squares distance functions. Linear Algebra Appl.
**215**, 1–19 (1995)MathSciNetCrossRefzbMATHGoogle Scholar - 35.Hillar, C.J., Lim, L.-H.: Most tensor problems are NP-hard. J. ACM
**60**(6), Art. 45, 39 (2013)Google Scholar - 36.Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys.
**6**, 164–189 (1927)CrossRefzbMATHGoogle Scholar - 37.Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07-49, University of Massachusetts, Amherst (2007)Google Scholar
- 38.Ivanyos, G., Karpinski, M., Qiao, Y., Santha, M.: Generalized Wong sequences and their applications to Edmonds’ problems. In: Proceedings of the 31st International Symposium on Theoretical Aspects of Computer Science, vol. 117543, pp. 397–408 (2014)Google Scholar
- 39.Kindermann, S., Navasca, C.: News algorithms for tensor decomposition based on a reduced functional. Numer. Linear Algebra Appl.
**21**(3), 340–374 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 40.Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev.
**51**(3), 455–500 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 41.Leurgans, S.E., Ross, R.T., Abel, R.B.: A decomposition for three-way arrays. SIAM J. Matrix Anal. Appl.
**14**(4), 1064–1083 (1993)MathSciNetCrossRefzbMATHGoogle Scholar - 42.Lewis, A.S., Luke, D.R., Malick, J.: Local linear convergence for alternating and averaged nonconvex projections. Found. Comput. Math.
**9**(4), 485–513 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 43.Lewis, A.S., Malick, J.: Alternating projections on manifolds. Math. Oper. Res.
**33**(1), 216–234 (2008)MathSciNetCrossRefzbMATHGoogle Scholar - 44.Li, N., Kindermann, S., Navasca, C.: Some convergence results on the regularized alternating least-squares method for tensor decomposition. Linear Algebra Appl.
**438**(2), 796–812 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 45.Liu, Y.-J., Sun, D., Toh, K.-C.: An implementable proximal point algorithmic framework for nuclear norm minimization. Math. Program.
**133**(1—-2, Ser. A), 399–436 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 46.Liu, Z., Vandenberghe, L.: Interior-point method for nuclear norm approximation with application to system identification. SIAM J. Matrix Anal. Appl.
**31**(3), 1235–1256 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 47.Lovász, L.: Singular spaces of matrices and their application in combinatorics. Bol. Soc. Brasil. Math.
**20**(1), 87–99 (1989)Google Scholar - 48.Mohlenkamp, M.J.: Musings on multilinear fitting. Linear Algebra Appl.
**438**(2), 834–852 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 49.Motwani, R., Raghavan, P.: Randomized Algorithms. Chapman and Hall/CRC, Routledge (2010)zbMATHGoogle Scholar
- 50.Noll, D., Rondepierre, A.: On local convergence of the method of alternating projections. Found. Comput. Math.
**16**(2), 425–455 (2016)MathSciNetCrossRefzbMATHGoogle Scholar - 51.Oxley, J.: Infinite matroids. In: White, N. (ed.) Matroid Applications, pp. 73–90. Cambridge University Press, Cambridge (1992)CrossRefGoogle Scholar
- 52.Qu, Q., Sun, J., Wright, J.: Finding a sparse vector in a subspace: linear sparsity using alternating directions. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, pp. 3401–3409. Curran Associates, Inc, Red Hook (2014)Google Scholar
- 53.Qu, Q., Sun, J., Wright, J.: Finding a sparse vector in a subspace: linear sparsity using alternating directions. arXiv:1412.4659 (2014)
- 54.Recht, B.: A simpler approach to matrix completion. J. Mach. Learn. Res.
**12**, 3413–3430 (2011)MathSciNetzbMATHGoogle Scholar - 55.Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev.
**52**(3), 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 56.Sorber, L., Van Barel, M., De Lathauwer, L.: Tensorlab v2.0. http://www.tensorlab.net/
- 57.Spielman, D.A., Wang, H., Wright, J.: Exact recovery of sparsely-used dictionaries. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI ’13, pp. 3087–3090. AAAI Press (2013)Google Scholar
- 58.Stewart, G.W.: Matrix Algorithms. Vol. II. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (2001)CrossRefzbMATHGoogle Scholar
- 59.Stewart, G.W., Sun, J.G.: Matrix Perturbation Theory. Computer Science and Scientific Computing. Academic Press, Inc., Boston, MA (1990)Google Scholar
- 60.Sun, J., Qu, Q., Wright, J.: Complete dictionary recovery over the sphere I: Overview and the geometric picture. arXiv:1511.03607 (2015)
- 61.Sun, J., Qu, Q., Wright, J.: Complete dictionary recovery over the sphere II: Recovery by Riemannian trust-region method. arXiv:1511.04777 (2015)
- 62.Uschmajew, A.: Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM J. Matrix Anal. Appl.
**33**(2), 639–652 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 63.Uschmajew, A.: A new convergence proof for the higher-order power method and generalizations. Pac. J. Optim.
**11**(2), 309–321 (2015)MathSciNetzbMATHGoogle Scholar - 64.Wang, L., Chu, M.T.: On the global convergence of the alternating least squares method for rank-one approximation to generic tensors. SIAM J. Matrix Anal. Appl.
**35**(3), 1058–1072 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 65.Wedin, P.-Å.: Perturbation bounds in connection with singular value decomposition. Nordisk Tidskr. Informationsbehandling (BIT)
**12**, 99–111 (1972)Google Scholar - 66.Xu, Y., Yin, W.: A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci.
**6**(3), 1758–1789 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 67.Zhao, X., Zhou, G., Dai, W., Xu, T., Wang, W.: Joint image separation and dictionary learning. In: 18th International Conference on Digital Signal Processing (DSP), pp. 1–6. IEEE (2013)Google Scholar