Finding a low-rank basis in a matrix subspace

Abstract

For a given matrix subspace, how can we find a basis that consists of low-rank matrices? This is a generalization of the sparse vector problem. It turns out that when the subspace is spanned by rank-1 matrices, the matrices can be obtained by the tensor CP decomposition. For the higher rank case, the situation is not as straightforward. In this work we present an algorithm based on a greedy process applicable to higher rank problems. Our algorithm first estimates the minimum rank by applying soft singular value thresholding to a nuclear norm relaxation, and then computes a matrix with that rank using the method of alternating projections. We provide local convergence results, and compare our algorithm with several alternative approaches. Applications include data compression beyond the classical truncated SVD, computing accurate eigenvectors of a near-multiple eigenvalue, image separation and graph Laplacian eigenproblems.

Appendix: Finding rank-one bases via tensor decomposition

In this appendix, we describe the rank-one basis problem as a tensor decomposition problem. Recall that in this problem, we are promised that the given subspace ${\mathcal {M}}$ is spanned by rank-one matrices. Thus we can apply Algorithm 3 (Phase II) with the precise rank guess directly. Alternatively, we can also stop after Algorithm 2 (Phase I), which in practice performs well (see Sect. 5.1). The following tensor decomposition viewpoint leads to further algorithms.

Let $M_1,\ldots ,M_d$ be an arbitrary basis of ${\mathcal {M}}$, and let ${\mathcal {T}}$ be the $m\times n\times d$ tensor whose 3-slices are $M_1,\ldots ,M_d$. The fact that ${\mathcal {M}}$ possesses a rank-one basis is equivalent to the existence of d (and not less) triplets of vectors $({\mathbf {a}}_\ell ^{}, {\mathbf {b}}_\ell ^{}, {\mathbf {c}}_\ell ^{})$ where ${\mathbf {a}}_\ell ^{}\in {\mathbb {R}}^{m}, {\mathbf {b}}_\ell ^{}\in {\mathbb {R}}^{n}, {\mathbf {c}}_\ell ^{}\in {\mathbb {R}}^{d},$ such that

$$\begin{aligned} M_k = \sum _{\ell = 1}^d c_{k,\ell }^{} {\mathbf {a}}_\ell ^{} {\mathbf {b}}_\ell ^T, \quad k = 1,\ldots ,d \end{aligned}$$

(24)

(here $c_{k,\ell }$ denotes the kth entry of ${\mathbf {c}}_\ell$). Namely, if such triplets $({\mathbf {a}}_\ell ^{}, {\mathbf {b}}_\ell ^{}, {\mathbf {c}}_\ell ^{})$ exist, then the assumed linear independence of the $M_k$ automatically implies that rank-one matrices ${\mathbf {a}}_\ell ^{} {\mathbf {b}}_\ell ^T$ belong to ${\mathcal {M}}$. Using the outer product of vectors (denoted by $\circ$), we may express this relation in terms of the tensor ${\mathcal {T}}$ as

$$\begin{aligned} {\mathcal {T}} = \sum _{\ell =1}^d {\mathbf {a}}_\ell \circ {\mathbf {b}}_\ell \circ {\mathbf {c}}_\ell . \end{aligned}$$

(25)

This type of tensor decomposition into a sum of outer products is called the CP decomposition, and is due to Hitchcock [36] (although the term CP decomposition appeared later). In general, the smallest d required for a representation of the form (25) is called the (canonical) rank of the tensor ${\mathcal {T}}$. We refer to [40] and references therein for more details. In summary, we have the following trivial conclusion.

Proposition 8

The d-dimensional matrix space ${\mathcal {M}} = {{\mathrm{{span}}}}(M_1,\ldots ,M_d)$ possesses a rank-one basis if and only if the tensor ${\mathcal {T}}$ whose 3-slices are the $M_1,\ldots ,M_d$ has (canonical) rank d. Any CP decomposition (25) of ${\mathcal {T}}$ provides a rank-one basis ${\mathbf {a}}_1^{} {\mathbf {b}}_1^T, \ldots , {\mathbf {a}}_d^{} {\mathbf {b}}_d^T$ of ${\mathcal {M}}$.

We remark that computing the rank of a general third-order tensor is known to be NP-hard [33, 35]. Therefore, it is NP-hard to check whether a matrix space ${\mathcal {M}}$ admits a rank-one basis. Nevertheless, we might try to find a rank-one basis by trying to calculate a CP decomposition (25) from linearly independent $M_1,\ldots ,M_d$. We outline two common algorithms.

1.1 Simultaneous diagonalization

If the tensor ${\mathcal {T}} \in {\mathbb {R}}^{m \times n \times r}$ is known to have rank d and $d\le \min (m,n)$, it is “generically” possible to find a CP decomposition (25) in polynomial time using simultaneous diagonalization [16, 19, 41].

Let us introduce the factor matrices ${\mathbf {A}} = [{\mathbf {a}}_1,\ldots ,{\mathbf {a}}_d] \in {\mathbb {R}}^{m\times d}$, ${\mathbf {B}} = [{\mathbf {b}}_d,\ldots ,{\mathbf {b}}_d] \in {\mathbb {R}}^{n\times d}$, and ${\mathbf {C}} = [{\mathbf {c}}_1,\ldots ,{\mathbf {c}}_d] \in {\mathbb {R}}^{d\times d}$. Then (24) reads

$$\begin{aligned} M_k = {\mathbf {A}} D_k {\mathbf {B}}^T, \quad k=1,\ldots ,d, \end{aligned}$$

where $D_k = {{\mathrm{{diag}}}}({\mathbf {c}}_k^T)$, in which ${\mathbf {c}}_k^T$ denotes the kth row of ${\mathbf {C}}$. In other words, a rank-one basis exists, if the $M_1,\ldots ,M_d$ can be simultaneously diagonalized. The basic idea of the algorithm of Leurgans, Ross, and Abel in [41] is as follows. One assumes ${{\mathrm{rank}}}({\mathbf {A}}) = {{\mathrm{rank}}}({\mathbf {B}}) = d$. Pick a pair $(k,\ell )$, and assume that $D_k$ and $D_\ell$ are invertible, and that $D_k^{} D_\ell ^{-1}$ has d distinct diagonal entries. Then it holds

$$\begin{aligned} M_k^{} M_\ell ^+ {\mathbf {A}} = {\mathbf {A}} D_k^{} {\mathbf {B}}^T ({\mathbf {B}}^T)^+ D_\ell ^{-1} {\mathbf {A}}^+ {\mathbf {A}} = {\mathbf {A}} D_k^{} D_\ell ^{-1}, \end{aligned}$$

where superscript $^+$ denotes the Moore-Penrose inverse. In other words, ${\mathbf {A}}$ contains the eigenvectors of $M_k^{} M_\ell ^+$ to distinct eigenvalues, and is essentially uniquely determined (up to scaling and permutation of the columns). Alternatively, for more numerical reliability, one can compute an eigenvalue decompositions of a linear combination of all $M_k^{} M_\ell ^+$ instead, assuming that the corresponding linear combination of $D_k^{} D_\ell ^{-1}$ has distinct diagonal entries. Similarly, ${\mathbf {B}}$ can be obtained from an eigendecomposition, e.g. of $M_k^T (M_\ell ^T)^+$ or linear combinations. Finally,

$$\begin{aligned} D_k = {\mathbf {A}}^+ M_k ({\mathbf {B}}^T)^+, \quad k = 1,\ldots ,d, \end{aligned}$$

which gives ${\mathbf {C}}$. The algorithm requires the construction of Moore-Penrose inverses of matrices whose larger dimension is at most $\max (m,n)$. Hence, the complexity is $O(mn^2)$.

The condition that the $D_k^{} D_\ell ^{-1}$ or a linear combination of them should have distinct diagonal entries is not very critical, since it holds generically, if the matrices $M_1,\ldots ,M_d$ are randomly drawn from ${\mathcal {M}}$, or, when this is not possible, are replaced by random linear combination of themselves. The condition ${{\mathrm{rank}}}({\mathbf {A}}) = {{\mathrm{rank}}}({\mathbf {B}}) = d$ on the other hand, is a rather strong assumption on the rank-one basis ${\mathbf {a}}^{}_1{\mathbf {b}}_1^T,\ldots ,{\mathbf {a}}_d^{} {\mathbf {b}}_d^T$. It implies uniqueness of the basis, and restricts the applicability of the outlined algorithm a priori to dimension $d \le \min (m,n)$ of ${\mathcal {M}}$. There is an interesting implication on the condition (14) that we used for the local convergence proof of our algorithms. Theorem 4 and Corollary 5 therefore apply at every basis element ${\mathbf {a}}_\ell ^{} {\mathbf {b}}_\ell ^T$ in this setting.

Proposition 9

Let ${\mathbf {a}}^{}_1{\mathbf {b}}_1^T,\ldots ,{\mathbf {a}}_d^{} {\mathbf {b}}_d^T$ be a rank-one basis such that ${{\mathrm{rank}}}({\mathbf {A}}) = {{\mathrm{rank}}}({\mathbf {B}}) = d$. Then (14) holds at any basis element $X_* = X_\ell = {\mathbf {a}}_\ell ^{} {\mathbf {b}}_\ell ^T$.

Proof

This follows immediately from Lemma 7 by taking $\tilde{{\mathcal {M}}} = {{\mathrm{span}}}\{ {\mathbf {a}}_k^{} {\mathbf {b}}_k^T :k \ne \ell \}$. $\square$

De Lathauwer [16] developed the idea of simultaneous diagonalization further. His algorithm requires the matrix ${\mathbf {C}}$ to have full column rank, which in our case is always true as ${\mathbf {C}}$ must contain the basis coefficients for d linearly independent elements $M_1,\ldots ,M_d$. The conditions on the full column rank of ${\mathbf {A}}$ and ${\mathbf {B}}$ can then be replaced by some weaker conditions, but, simply speaking, too many linear dependencies in ${\mathbf {A}}$ and ${\mathbf {B}}$ will still lead to a failure. A naive implementation of De Lathauwer’s algorithm in [16] seems to require $O(n^6)$ operations (assuming $m=n$).

Further progress on finding the CP decomposition algebraically under even milder assumptions has been made recently in [22]. It is partially based on the following observation: denoting by $m_\ell =\text{ vec }(M_\ell )$ the $n^2\times 1$ vectorization of $M_\ell$ (which stacks the column on top of each other), and defining $\text{ Matr }({\mathcal {T}}) = [m_1,\ldots , m_r]\in {\mathbb {R}}^{mn\times d}$, we have

$$\begin{aligned} \text{ Matr }({\mathcal {T}}) = ({\mathbf {B}} \odot {\mathbf {A}} ) {\mathbf {C}}^T, \end{aligned}$$

(26)

where ${\mathbf {B}} \odot {\mathbf {A}} = [{\mathbf {a}}_1 \otimes {\mathbf {b}}_1,\ldots ,{\mathbf {a}}_d \otimes {\mathbf {b}}_d] \in {\mathbb {R}}^{mn \times d}$ is the so called Khatri-Rao product of ${\mathbf {B}}$ and ${\mathbf {A}}$ (here $\otimes$ is the ordinary Kronecker product). If ${\mathbf {C}}$ (which is of full rank in our scenario) would be known, then ${\mathbf {A}}$ and ${\mathbf {B}}$ can be retrieved from the fact that the matricizations of the columns of $\text{ Matr }({\mathcal {T}}){\mathbf {C}}^{-T} = {\mathbf {B}} \odot {\mathbf {A}}$ must be rank-one matrices. In [22] algebraic procedures are proposed that find the matrix ${\mathbf {C}}$ from ${\mathcal {T}}$.

Either way, by computing the CP decomposition for ${\mathcal {T}}$ we can, at least in practice, recover the rank one basis $\{{\mathbf {a}}_\ell ^{}{\mathbf {b}}_\ell ^T\}$ in polynomial time if we know it exists. This is verified in our MATLAB experiments using Tensorlab’s cpd in Sect. 5.1.1.

1.2 Alternating least squares

An alternative and cheap workaround are optimization algorithms to calculate an approximate CP decomposition of a given third-order tensor, a notable example being alternating least squares (ALS), which was developed in statistics along with the CP model for data analysis [13, 30]. In practice, they often work astonishingly well when the exact rank is provided.

Assuming the existence of a rank-one basis, that is, ${{\mathrm{rank}}}({\mathcal {T}}) = d$, the basic ALS algorithm is equivalent to a block coordinate descent method applied to the function

$$\begin{aligned} f({\mathbf {A}},{\mathbf {B}},{\mathbf {C}}) = \frac{1}{2} \left\| {\mathcal {T}} - \sum _{\ell =1}^d {\mathbf {a}}_\ell \circ {\mathbf {b}}_\ell \circ {\mathbf {c}}_\ell \right\| ^2_F. \end{aligned}$$

The name of the algorithm comes from the fact that a block update consists in solving a least squares problem for one of the matrices ${\mathbf {A}}$, ${\mathbf {B}}$ or ${\mathbf {C}}$, since f is quadratic with respect to each of them. It is easy to derive the explicit formulas. For instance, fixing ${\mathbf {A}}$ and ${\mathbf {B}}$, an optimal ${\mathbf {C}}$ with minimal Frobenius norm is found from (26) as ${\mathbf {C}} = \text{ Matr }({\mathcal {T}})^T ({\mathbf {B}}^T \odot {\mathbf {A}}^T)^+.$ The updates for the other blocks look similar when using appropriate reshapes of ${\mathcal {T}}$ into a matrix; the formulas can be found in [40].

The question of convergence of ALS is very delicate, and has been subject to many studies. As it is typical for these block coordinate type optimization methods for nonconvex functions, convergence can, if at all, ensured only to critical points, but regularization might be necessary, see [44, 48, 62–64, 66] for some recent studies, and [40] in general. Practical implementations are typically a bit more sophisticated than the simple version outlined above, for instance the columns of every factor matrix should be rescaled during the process for more numerical stability. Also a good initialization of ${\mathbf {A}}$, ${\mathbf {B}}$, and ${\mathbf {C}}$ can be crucial for the performance. For instance one may take the leading HOSVD vectors [18] or the result of other methods [39] as a starting guess for ALS.