Robust Principal Component Pursuit via Inexact Alternating Minimization on Matrix Manifolds

Michael Hintermüller; Tao Wu

doi:10.1007/s10851-014-0527-y

Journal of Mathematical Imaging and Vision

March 2015, Volume 51, Issue 3, pp 361–377 | Cite as

Robust Principal Component Pursuit via Inexact Alternating Minimization on Matrix Manifolds

Authors
Authors and affiliations

Michael HintermüllerEmail author
Tao Wu

Article

First Online: 29 August 2014

Received: 07 January 2014

Accepted: 31 July 2014

623 Downloads
4 Citations

Abstract

Robust principal component pursuit (RPCP) refers to a decomposition of a data matrix into a low-rank component and a sparse component. In this work, instead of invoking a convex-relaxation model based on the nuclear norm and the $\ell ^1$-norm as is typically done in this context, RPCP is solved by considering a least-squares problem subject to rank and cardinality constraints. An inexact alternating minimization scheme, with guaranteed global convergence, is employed to solve the resulting constrained minimization problem. In particular, the low-rank matrix subproblem is resolved inexactly by a tailored Riemannian optimization technique, which favorably avoids singular value decompositions in full dimension. For the overall method, a corresponding $q$-linear convergence theory is established. The numerical experiments show that the newly proposed method compares competitively with a popular convex-relaxation based approach.

Keywords

Matrix decomposition Low-rank matrix Sparse matrix Image processing Alternating minimization Riemannian manifold Optimization on manifolds

Mathematics subject classification

15A83 53B21 65K10 90C30 94A08

This is a preview of subscription content, to check access.

Notes

Acknowledgments

This research was supported by the Austrian Science Fund (FWF) through START project Y305 “Interfaces and Free Boundaries” and through SFB project F3204 “Mathematical Optimization and Applications in Biomedical Sciences”.

Appendix: Local Convergence of an Alternating Projection Method.

Here we consider a heuristic alternating projection method for the RPCP problem. This method, which can be interpreted as an exact alternating minimizer scheme for the optimization problem (1) with $\mu =0$, can be shortly described as follows. Given $A^k\in \mathcal {M}$, one generates

$$\begin{aligned} \left\{ \begin{array}{l} B^{k+1}:=P_\mathcal {N}(Z-A^k), \\ A^{k+1}:=P_\mathcal {M}(Z-B^{k+1}). \end{array}\right. \end{aligned}$$

(24)

The name “alternating projection method” is termed, since the iterative procedure (on $\{A^k\}$) can be expressed as

$$\begin{aligned} A^{k+1}=\psi (A^k):=(P_\mathcal {M}\circ \iota \circ P_\mathcal {N}\circ \iota )(A^k), \end{aligned}$$

(25)

with $\iota :A\mapsto Z-A$, and thus generalizes the classical alternating projection (where $\iota $ is the identity map) in, e.g, [21]. The following theorem asserts the local convergence of the alternating projection method. However, we note that the global convergence for this method is not guaranteed in general.

Theorem 4.1

Given $A^0\in \mathcal {M}$, let the sequence $\{(A^k,B^k)\}$ be iteratively generated by formula (24). Assume that $(A^k,B^k)$ is sufficiently close to some $(A^*,B^*)$ such that $\mathrm{rank}(A^*)=r$, $\Vert B^*\Vert =s$, $T_\mathcal {M}(A^*)\cap T_\mathcal {N}(B^*)=\{0\}$, and moreover

$$\begin{aligned} \left\{ \begin{array}{l} B^*:=P_\mathcal {N}(Z-A^*), \\ A^*:=P_\mathcal {M}(Z-B^*). \end{array}\right. \end{aligned}$$

(26)

Then $\{(A^k,B^k)\}$ converges to $(A^*,B^*)$ $q$-linearly at rate $\kappa _p$; i.e.

$$\begin{aligned} \limsup _{k\rightarrow \infty }\frac{\Vert (A^{k+1},B^{k+1})-(A^*,B^*)\Vert }{\Vert (A^k,B^k)-(A^*,B^*)\Vert }\le \kappa _p, \end{aligned}$$

where $\kappa _p\in [0,1)$ is a constant (same as in Lemma 3.10) such that

$$\begin{aligned} \Vert (P_{T_\mathcal {M}(A^*)}\circ P_{T_\mathcal {N}(B^*)})(\Delta )\Vert \le \kappa _p\Vert \Delta \Vert , \end{aligned}$$

for all $\Delta \in \mathbb {R}^{m\times n}$.

Proof

We only prove the $q$-linear convergence on $\{A^k\}$, as the proof for $\{B^k\}$ is almost identical. Note that $\mathcal {M}$ and $\mathcal {N}$ are two smooth manifolds near $A^*$ and $B^*$, respectively. For the existence of a qualified constant $\kappa _p$, we refer to Lemma 3.10(iii).

In the following, we perturb both equations in (26) with respect to $A^*$ by an arbitrarily fixed $\Delta \in \mathbb {R}^{m\times n}$. The perturbation of the first equation gives

$$\begin{aligned} P_\mathcal {N}(Z-A^*-\Delta )&= B^*+P_\mathcal {N}(Z-A^*-\Delta )\nonumber \\&-P_\mathcal {N}(Z-A^*)\\&= B^*+P_{T_\mathcal {N}(B^*)}(-\Delta )+O(\Vert \Delta \Vert ^2). \end{aligned}$$

Since $A^*$ is a fixed point of the map $\psi $ in (25), the second equation in (26) can be written as $\psi (A^*)=P_\mathcal {M}(Z-B^*)$. Then we have

$$\begin{aligned} \psi (A^*+\Delta )&=P_\mathcal {M}(Z-P_\mathcal {N}(Z-A^*-\Delta ))\\&=P_\mathcal {M}(Z-B^*-P_{T_\mathcal {N}(B^*)}(-\Delta )+O(\Vert \Delta \Vert ^2)) \\&=A^*+P_\mathcal {M}(Z-B^*-P_{T_\mathcal {N}(B^*)}(-\Delta )\\&\quad +O(\Vert \Delta \Vert ^2))-P_\mathcal {M}(Z-B^*) \\&=A^*+(P_{T_\mathcal {M}(A^*)}\circ P_{T_\mathcal {N}(B^*)})(\Delta )+O(\Vert \Delta \Vert ^2)). \end{aligned}$$

Thus, by considering $\Delta =A^k-A^*$ and passing $\Delta \rightarrow 0$, we conclude that

$$\begin{aligned} \limsup _{k\rightarrow \infty }\frac{\Vert A^{k+1}-A^*\Vert }{\Vert A^k-A^*\Vert }\le \kappa _p. \end{aligned}$$

$\square $

References

1.
LMaFit: Low-rank matrix fitting. http://lmafit.blogs.rice.edu
2.
Low-rank matrix recovery and completion via convex optimization. http://perception.csl.illinois.edu/matrix-rank/sample_code.html
3.
PROPACK–Software for large and sparse SVD calculations. http://sun.stanford.edu/rmunk/PROPACK/
4.
Statistical modeling of complex background for foreground object detection. http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html
5.
Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, New Jersey (2008)CrossRefMATHGoogle Scholar
6.
Absil, P.-A., Malick, J.: Projection-like retractions on matrix manifolds. SIAM J. Optim. 22, 135–158 (2012)CrossRefMATHMathSciNetGoogle Scholar
7.
Boumal, N., Absil, P.-A., RTRMC: A Riemannian trust-region method for low-rank matrix completion. In: Advances in Neural Information Processing Systems, pp. 406–414 (2011)Google Scholar
8.
Boumal, N., Mishra, B., Absil, P.-A., Sepulchre, R.: Manopt, a Matlab toolbox for optimization on manifolds. J. Mach. Learn. Res. 15, 1455–1459 (2014). http://www.manopt.org
9.
Cai, J.F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20, 1956–1982 (2010)CrossRefMATHMathSciNetGoogle Scholar
10.
Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58, 1–37 (2011)CrossRefGoogle Scholar
11.
Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9, 717–772 (2009)CrossRefMATHMathSciNetGoogle Scholar
12.
Chandrasekaran, V., Sanghavi, S., Parrilo, P.A., Willsky, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21, 572–596 (2011)CrossRefMATHMathSciNetGoogle Scholar
13.
Eckart, C., Young, G.: The approximation of one matrix by another of lower rank. Psychometrika 1, 211–218 (1936)CrossRefMATHGoogle Scholar
14.
Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)CrossRefMATHMathSciNetGoogle Scholar
15.
Huang, P.-S., Chen, S. D., Smaragdis, P., Hasegawa-Johnson, M.: Singing-voice separation from monaural recordings using robust principal component analysis. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 57–60 (2012)Google Scholar
16.
Ji, H., Huang, S., Shen, Z., Xu, Y.: Robust video restoration by joint sparse and low rank matrix approximation. SIAM J. Imaging Sci. 4, 1122–1142 (2011)CrossRefMATHMathSciNetGoogle Scholar
17.
Jia, K., Chan, T.-H., Ma, Y.: Robust and practical face recognition via structured sparsity. In: Proceedings of the 12th European Conference on Computer Vision, vol. 4, pp. 331–344 (2012)Google Scholar
18.
Keshavan, R.H., Montanari, A., Oh, S.: Matrix completion from a few entries. IEEE Trans. Inf. Theory 56, 2980–2998 (2010)CrossRefMathSciNetGoogle Scholar
19.
Keshavan, R.H., Montanari, A., Oh, S.: Matrix completion from noisy entries. J. Mach. Learn. Res. 11, 2057–2078 (2010)MATHMathSciNetGoogle Scholar
20.
Knuth, D.: Sorting and Searching. The art of computer programming, vol. 3. Addison-Wesley, Reading (1997)MATHGoogle Scholar
21.
Lewis, A.S., Malick, J.: Alternating projections on manifolds. Math. Oper. Res. 33, 216–234 (2008)CrossRefMATHMathSciNetGoogle Scholar
22.
Li, L., Huang, W., Gu, I.Y.-H., Tian, Q.: Statistical modeling of complex backgrounds for foreground object detection. IEEE Trans. Image Process. 13, 1459–1472 (2004)CrossRefGoogle Scholar
23.
Lin, Z., Chen, M., Wu, L., Ma, Y.: The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices, Technical Report UILU-ENG-09-2215, UIUC, (2009)Google Scholar
24.
Meyer, G., Bonnabel, S., Sepulchre, R.: Regression on fixed-rank positive semidefinite matrices: a Riemannian approach. J. Mach. Learn. Res. 12, 593–625 (2011)MATHMathSciNetGoogle Scholar
25.
Min, K., Zhang, Z., Wright, J., Ma, Y.: Decomposing background topics from keywords by principal component pursuit. In: Proceedings of the 19th ACM international conference on Information and knowledge management, pp. 269–278 (2010)Google Scholar
26.
Ngo, T.T., Saad, Y.: Scaled gradients on Grassmann manifolds for matrix completion. Adv. Neural Inf. Process. Syst. 25, 1421–1429 (2012)Google Scholar
27.
Nocedal, J., Wright, S.: Numerical Optimization, 2nd edn. Springer, New York (2006)MATHGoogle Scholar
28.
Seber, G.A.F.: Multivariate Observations. Wiley, Hoboken, NJ (1984)CrossRefMATHGoogle Scholar
29.
Simonsson, L., Eldén, L.: Grassmann algorithms for low rank approximation of matrices with missing values. BIT Numer. Math. 50, 173–191 (2010)CrossRefMATHGoogle Scholar
30.
Smith, S.T.: Geometric Optimization Methods for Adaptive Filtering, PhD thesis, Harvard University (1993)Google Scholar
31.
Tao, M., Yuan, X.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)CrossRefMATHMathSciNetGoogle Scholar
32.
Trefethen, L.N., Bau III, D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)CrossRefMATHGoogle Scholar
33.
Vandereycken, B.: Low-rank matrix completion by Riemannian optimization. SIAM J. Optim. 23, 1214–1236 (2013)CrossRefMATHMathSciNetGoogle Scholar
34.
Wen, Z., Yin, W., Zhang, Y.: Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm. Math. Prog. Comput. 4, 333–361 (2012)CrossRefMATHMathSciNetGoogle Scholar
35.
Zhang, Z., Ganesh, A., Liang, X., Ma, Y.: TILT: transform invariant low-rank textures. Int. J. Comput. Vis. 99, 1–24 (2012)CrossRefMATHMathSciNetGoogle Scholar

Copyright information

Authors and Affiliations

Michael Hintermüller
- 1
Email author
Tao Wu
- 2

1.Department of MathematicsHumboldt-Universität zu BerlinBerlinGermany
2.Institute for Mathematics and Scientific ComputingKarl-Franzens-University of GrazGrazAustria

Robust Principal Component Pursuit via Inexact Alternating Minimization on Matrix Manifolds

Abstract

Keywords

Mathematics subject classification

Notes

Acknowledgments

References

Copyright information

Authors and Affiliations

Personalised recommendations

Cite article