Deterministic Sparse Column Based Matrix Reconstruction via Greedy Approximation of SVD

Abstract

Given a matrix A ∈ ℝ^{m ×n} of rank r, and an integer k < r, the top k singular vectors provide the best rank-k approximation to A. When the columns of A have specific meaning, it is desirable to find (provably) “good” approximations to A _k which use only a small number of columns in A. Proposed solutions to this problem have thus far focused on randomized algorithms. Our main result is a simple greedy deterministic algorithm with guarantees on the performance and the number of columns chosen. Specifically, our greedy algorithm chooses c columns from A with $c=O \left({{k^2\log k} \over {\epsilon^2}} \mu^2(A)\ln\left({\sqrt{k}\|{A_k}\|_F} \over {\epsilon}\|{A-A_k}\|_F\right)\right)$ such that

$${\|A-C_{gr}C_{gr}^+A\|}_F \leq \left(1+\epsilon \right)\|{A-A_k}_F,$$

where C _gr is the matrix composed of the c columns, $C_{gr}^+$ is the pseudo-inverse of C _gr ($C_{gr}C_{gr}^+A$ is the best reconstruction of A from C _gr ), and μ(A) is a measure of the coherence in the normalized columns of A. The running time of the algorithm is O(SVD(A _k ) + mnc) where SVD(A _k ) is the running time complexity of computing the first k singular vectors of A. To the best of our knowledge, this is the first deterministic algorithm with performance guarantees on the number of columns and a (1 + ε) approximation ratio in Frobenius norm. The algorithm is quite simple and intuitive and is obtained by combining a generalization of the well known sparse approximation problem from information theory with an existence result on the possibility of sparse approximation. Tightening the analysis along either of these two dimensions would yield improved results.