Mathematical Programming Computation

, Volume 4, Issue 4, pp 333–361 | Cite as

Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm

  • Zaiwen Wen
  • Wotao Yin
  • Yin Zhang
Full Length Paper


The matrix completion problem is to recover a low-rank matrix from a subset of its entries. The main solution strategy for this problem has been based on nuclear-norm minimization which requires computing singular value decompositions—a task that is increasingly costly as matrix sizes and ranks increase. To improve the capacity of solving large-scale problems, we propose a low-rank factorization model and construct a nonlinear successive over-relaxation (SOR) algorithm that only requires solving a linear least squares problem per iteration. Extensive numerical experiments show that the algorithm can reliably solve a wide range of problems at a speed at least several times faster than many nuclear-norm minimization algorithms. In addition, convergence of this nonlinear SOR algorithm to a stationary point is analyzed.


Matrix completion Alternating minimization Nonlinear GS method Nonlinear SOR method 

Mathematics Subject Classification (2000)

65K05 90C06 93C41 68Q32 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beck A., Teboulle M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Candès E., Plan Y.: Matrix completion with noise. Proc. IEEE 98, 925–936 (2010)CrossRefGoogle Scholar
  3. 3.
    Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. (2009)Google Scholar
  4. 4.
    Candès E.J., Tao T.: The power of convex relaxation near-optimal matrix completion. IEEE Trans. Inf. Theory 56, 2053–2080 (2010)CrossRefGoogle Scholar
  5. 5.
    Chan T.F.: Rank revealing QR factorizations. Linear Algebra Appl. 88(89), 67–82 (1987)MathSciNetGoogle Scholar
  6. 6.
    Chan T.F., Hansen P.C.: Low-rank revealing QR factorizations. Numer. Linear Algebra Appl. 1, 33–44 (1994)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Dai, W., Milenkovic, O.: Set an algorithm for consistent matrix completion CoRR. abs/0909.2705 (2009)Google Scholar
  8. 8.
    Eldén L.: Matrix Methods in Data Mining and Pattern Recognition (Fundamentals of Algorithms). Society for Industrial and Applied Mathematics, Philadelphia (2007)CrossRefGoogle Scholar
  9. 9.
    Goldberg K., Roeder T., Gupta D., Perkins C.: Eigentaste A constant time collaborative filtering algorithm. Inf. Retr. 4, 133–151 (2001)zbMATHCrossRefGoogle Scholar
  10. 10.
    Goldfarb, D., Ma, S., Wen, Z.: Solving low-rank matrix completion problems efficiently. In: Proceedings of the 47th Annual Allerton Conference on Communication, Control, and Computing, Allerton’09, pp. 1013–1020 (2009)Google Scholar
  11. 11.
    Golub G.H., Van Loan C.F.: Matrix computations. In: Johns Hopkins Studies in the Mathematical Sciences, 3rd edn. Johns Hopkins University Press, Baltimore (1996)Google Scholar
  12. 12.
    Grippo L., Sciandrone M.: On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper. Res. Lett. 26, 127–136 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Gross, D.: Recovering low-rank matrices from few coefficients in any basis, tech. rep. Leibniz University (2009)Google Scholar
  14. 14.
    Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: SIGIR ’99: Proceedings of the 22nd annual international ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 230–237. ACM, New York (1999)Google Scholar
  15. 15.
    Jian-Feng C., Candes E.J., Zuowei S.: A singular value thresholding algorithm for matrix completion export. SIAM J. Optim. 20, 1956–1982 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Keshavan R.H., Montanari A., Oh S.: Matrix completion from a few entries. IEEE Trans. Inform. Theory 56, 2980–2998 (2010)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Keshavan R.H., Montanari A., Oh S.: Matrix completion from noisy entries. J. Mach. Learn. Res. 99, 2057–2078 (2010)MathSciNetGoogle Scholar
  18. 18.
    Keshavan, R.H., Oh, S.: A gradient descent algorithm on the grassman manifold for matrix completion, tech. rep., Dept. of Electrical Engineering, Stanford University (2009)Google Scholar
  19. 19.
    Larsen, R.M.: PROPACK Software for large and sparse svd calculations.
  20. 20.
    Lee K., Bresler Y.: Admira atomic decomposition for minimum rank approximation. IEEE Trans. Inf. Theor. 56, 4402–4416 (2010)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Liu Y.-J., Sun D., Toh K.-C.: An implementable proximal point algorithmic framework for nuclear norm minimization. Math. Program. 133(1–2), 399–436 (2011)MathSciNetGoogle Scholar
  22. 22.
    Liu Z., Vandenberghe L.: Interior-point method for nuclear norm approximation with application to system identification. SIAM J. Matrix Anal. Appl. 31, 1235–1256 (2009)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Luo Q.Z., Tseng P.: On the convergence of the coordinate descent method for convex differentiable minimization. J. Optim. Theory Appl. 72, 7–35 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Ma S., Goldfarb D., Chen L.: Fixed point and bregman iterative methods for matrix rank minimization. Math. Program. 128(1–2), 321–353 (2009)MathSciNetGoogle Scholar
  25. 25.
    Mazumder R., Hastie T., Tibshirani R.: Regularization methods for learning incomplete matrices. Stanford University, tech. rep. (2009)Google Scholar
  26. 26.
    Meka, R., Jain, P., Dhillon, I.S.: Guaranteed rank minimization via singular value projection. CoRR, abs/0909.5457 (2009)Google Scholar
  27. 27.
    Morita T., Kanade T.: A sequential factorization method for recovering shape and motion from image streams. IEEE Trans. Pattern Anal. Mach. Intell. 19, 858–867 (1997)CrossRefGoogle Scholar
  28. 28.
    Nocedal J., Wright S.J.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, Berlin (2006)Google Scholar
  29. 29.
    Recht, B.: A simpler approach to matrix completion. J. Mach. Learn. Res. (2010)Google Scholar
  30. 30.
    Recht B., Fazel M., Parrilo P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52, 471–501 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Shen, Y., Wen, Z., Zhang, Y.: Augmented lagrangian alternating direction method for matrix separation based on low-rank factorization, tech. rep. Rice University (2010)Google Scholar
  32. 32.
    Stewart G.W.: On the continuity of the generalized inverse. SIAM J. Appl. Math. 17, 33–45 (1969)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Toh K.-C., Yun S.: An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems. Pacific J. Optim. 6, 615–640 (2010)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Tomasi C., Kanade T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9, 137–154 (1992)CrossRefGoogle Scholar
  35. 35.
    Tseng P.: Dual ascent methods for problems with strictly convex costs and linear constraints: a unified approach. SIAM J. Control Optim. 28, 214–242 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Tseng P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109, 475–494 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Xu, Y., Yin, W., Wen, Z., Zhang, Y.: An alternating direction algorithm for matrix completion with nonnegative factors. tech. rep., Shanghai Jiaotong University (2011)Google Scholar
  38. 38.
    Yang, J., Yuan, X.: An inexact alternating direction method for trace norm regularized least squares problem. tech. rep., Dept. of Mathematics. Nanjing University (2010)Google Scholar
  39. 39.
    Yuan, X., Yang, J.: Sparse and low-rank matrix decomposition via alternating direction methods. tech. rep. Dept. of Mathematics, Hong Kong Baptist University (2009)Google Scholar
  40. 40.
    Zhang, Y.: LMaFit Low-rank matrix fitting (2009).
  41. 41.
    Zhang, Y.: An alternating direction algorithm for nonnegative matrix factorization. tech. rep. Rice University (2010)Google Scholar
  42. 42.
    Zhu, Z., So, A.M.-C., Ye, Y.: Fast and near-optimal matrix completion via randomized basis pursuit, tech. rep. Stanford University (2009)Google Scholar

Copyright information

© Springer and Mathematical Optimization Society 2012

Authors and Affiliations

  1. 1.Department of Mathematics and Institute of Natural SciencesShanghai Jiaotong UniversityShanghaiChina
  2. 2.Department of Computational and Applied MathematicsRice UniversityHoustonUSA

Personalised recommendations