Skip to main content
SpringerLink
Log in

Scalable algorithms for locally low-rank matrix modeling

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

We consider the problem of modeling data matrices with locally low-rank (LLR) structure, a generalization of the popular low-rank structure widely used in a variety of real-world application domains ranging from medical imaging to recommendation systems. While LLR modeling has been found to be promising in real-world application domains, limited progress has been made on the design of scalable algorithms for such structures. In this paper, we consider a convex relaxation of LLR structure and propose an efficient algorithm based on dual projected gradient descent (D-PGD) for computing the proximal operator. While the original problem is non-smooth, so that primal (sub)gradient algorithms will be slow, we show that the proposed D-PGD algorithm has geometrical convergence rate. We present several practical ways to further speed up the computations, including acceleration and approximate SVD computations. With experiments on both synthetic and real data from MRI (magnetic resonance imaging) denoising, we illustrate the superior performance of the proposed D-PGD algorithm compared to several baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://stanford.edu/~boyd/papers/prox_algs/matrix_decomp.html.

References

  1. Banerjee A, Chen S, Fazayeli F, Sivakumar V (2014) Estimation with norm regularization. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems 27. Curran Associates, Inc., pp 1556–1564

  2. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202

    Article  MathSciNet  MATH  Google Scholar 

  3. Bertsekas DP (1999) Nonlinear programming. Athena Scientific, Belmont

    MATH  Google Scholar 

  4. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2010) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122

    Article  MATH  Google Scholar 

  5. Cai J-F, Candès EJ, Shen Z (2010) A singular value thresholding algorithm for matrix completion. SIAM J Optim 20(4):1956–1982

    Article  MathSciNet  MATH  Google Scholar 

  6. Candès EJ, Recht B (2009) Exact matrix completion via convex optimization. Found Comput Math 9(6):717–772

    Article  MathSciNet  MATH  Google Scholar 

  7. Candès EJ, Sing-long CA, Trzasko JD (2013) Unbiased risk estimates for singular value thresholding and spectral estimators. IEEE Trans Signal Process 61(19):4643–4657

    Article  MathSciNet  MATH  Google Scholar 

  8. Chen S, Banerjee A (2016) Structured matrix recovery via the generalized dantzig selector. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R (eds) Advances in neural information processing systems 29. Curran Associates Inc., New York, pp 3252–3260

    Google Scholar 

  9. Chen X, Tseng P (2003) Non-interior continuation methods for solving semidefinite complementarity problems. Math Program 95(3):431–474

    Article  MathSciNet  MATH  Google Scholar 

  10. Donoho D, Gavish M (2014) Minimax risk of matrix denoising by singular value thresholding. Ann Stat 42(6):2413–2440

    Article  MathSciNet  MATH  Google Scholar 

  11. Goud S, Hu Y, Jacob M (2010) Real-time cardiac MRI using low-rank and sparsity penalties. In: ISBI, pp. 988–991

  12. Gunasekar S, Banerjee A, Ghosh J (2015) Unified view of matrix completion under general structural constraints. Adv Neural Inf Process Syst 28:1180–1188

    Google Scholar 

  13. Haldar JP, Liang ZP (2010) Spatiotemporal imaging with partially separable functions: a matrix recovery approach. In: ISBI, pp. 716–719

  14. Hoffman AJ (1952) On approximate solutions of systems of linear inequalities. J Res Natl Bur Stand 49(4):263–265

    Article  MathSciNet  Google Scholar 

  15. Horn RA, Johnson CR (1985) Matrix analysis, vol 169. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  16. Koren Y (2008) Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 426–434

  17. Lee J, Kim S, Lebanon G, Singer Y, Bengio S (2016) Llorma: local low-rank matrix approximation. J Mach Learn Res 17(15):1–24

    MathSciNet  MATH  Google Scholar 

  18. Lin Z, Liu R, Su Z (2011) Linearized alternating direction method with adaptive penalty for low-rank representation. Adv Neural Inf Process Syst 24(1):1–9

    Google Scholar 

  19. Luo ZQ, Tseng P (1993) Error bounds and convergence analysis of feasible descent methods: a general approach. Ann Oper Res 46–47(1):157–178

    Article  MathSciNet  MATH  Google Scholar 

  20. Ma S, Goldfarb D, Chen L (2011) Fixed point and bregman iterative methods for matrix rank minimization. Math Program 128(1):321–353

    Article  MathSciNet  MATH  Google Scholar 

  21. Moreau JJ (1962) Decomposition orthogonale d’un espace hilbertien selon deux cones mutuellement polaires. Comptes Rendus de l’Académie des Sci 255:238–240

    MATH  Google Scholar 

  22. Negahban S, Wainwright MJ (2011) Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann Stat 39(2):1069–1097

    Article  MathSciNet  MATH  Google Scholar 

  23. Nesterov Y (2004) Introductory lectures on convex optimization : a basic course, applied optimization. Kluwer Academic Publ, Boston

    Book  MATH  Google Scholar 

  24. O’Donoghue B, Candés E (2015) Adaptive restart for accelerated gradient schemes. Found Comput Math 15(3):715–732

    Article  MathSciNet  MATH  Google Scholar 

  25. Parikh N, Boyd S (2014) Proximal algorithms. Found Trends Optim 1(3):127–239

    Article  Google Scholar 

  26. Peng Z, Yan M, Yin W (2013) Parallel and distributed sparse optimization. In: 2013 Asilomar conference on signals, systems and computers, pp. 659–646

  27. Recht B, Fazel M, Parrilo PA (2010) Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev 52(3):471–501

    Article  MathSciNet  MATH  Google Scholar 

  28. Rockafellar RT (1970) Convex analysis. Princeton University Press, Princeton

    Book  MATH  Google Scholar 

  29. Toh KC, Yun S (2010) An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pac J Optim 6:615–640

    MathSciNet  MATH  Google Scholar 

  30. Trzasko JD (2013) Exploiting local low-rank structure in higher-dimensional mri applications. In: Proceedings of SPIE, vol 8858, pp 885821–885828

  31. Trzasko JD, Mostardi PM, Riederer SJ, Manduca A (2013) Estimating t1 from multichannel variable flip angle SPGR sequences. Magn Reson Med 69(6):1787–1794

    Article  Google Scholar 

  32. Trzasko J, Manduca A (2011) Local versus global low-rank promotion in dynamic MRI series reconstruction. In: ISMRM, vol 24, p 4371

  33. Watson GA (1992) Characterization of the subdifferential of some matrix norms. Linear Algebra Appl 170(C):33–45

    Article  MathSciNet  MATH  Google Scholar 

  34. Yao Q, Kwok JT (2015) Colorization by patch-based local low-rank matrix completion. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence, AAAI’15. AAAI Press, pp. 1959–1965

  35. Zhou Z, So AM-C (2015) A unified approach to error bounds for structured convex optimization problems, pp. 1–32. arXiv:1512.03518

Download references

Acknowledgements

The research was supported by NSF grants IIS-1563950, IIS-1447566, IIS-1447574, IIS-1422557, CCF-1451986, CNS- 1314560, IIS-0953274, IIS-1029711, CCF:CIF:Small:1318347, NASA grant NNX12AQ39A, “Mayo Clinic Discovery Translation Program”, and gifts from Adobe, IBM, and Yahoo.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Minnesota, Twin Cities, USA

    Qilong Gu & Arindam Banerjee

  2. Department of Radiology, Mayo Clinic, Rochester, USA

    Joshua D. Trzasko

Authors
  1. Qilong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joshua D. Trzasko
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arindam Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qilong Gu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, Q., Trzasko, J.D. & Banerjee, A. Scalable algorithms for locally low-rank matrix modeling. Knowl Inf Syst 61, 1457–1484 (2019). https://doi.org/10.1007/s10115-018-1320-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1320-9

Keywords

Search

Navigation