Data Mining and Knowledge Discovery

, Volume 29, Issue 6, pp 1665–1684 | Cite as

Quadratic regularization projected Barzilai–Borwein method for nonnegative matrix factorization

  • Yakui HuangEmail author
  • Hongwei Liu
  • Shuisheng Zhou


In this paper, based on the alternating nonnegative least squares framework, we present a new efficient method for nonnegative matrix factorization that uses a quadratic regularization projected Barzilai–Borwein (QRPBB) method to solve the subproblems. At each iteration, the QRPBB method first generates a point by solving a strongly convex quadratic minimization problem, which has a simple closed-form solution that is inexpensive to calculate, and then applies a projected Barzilai–Borwein method to update the solution of NMF. Global convergence result is established under mild conditions. Numerical comparisons of methods on both synthetic and real-world datasets show that the proposed method is efficient.


Nonnegative matrix factorization Alternating nonnegative least squares Quadratic regularization Projected Barzilai–Borwein method 



The authors would like to thank the handling associate editor Professor Kristian Kersting and the anonymous referees for their constructive comments and useful suggestions. This work is supported by the National Natural Science Foundation of China (NNSFC) under Grant No. 61072144 and No. 61179040 and the Fundamental Research Funds for the Central Universities No. K50513100007.


  1. Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8(1):141–148MathSciNetCrossRefzbMATHGoogle Scholar
  2. Berry MW, Browne M, Langville AN, Pauca VP, Plemmons RJ (2007) Algorithms and applications for approximate nonnegative matrix factorization. Computat Statist Data Anal 52(1):155–173MathSciNetCrossRefzbMATHGoogle Scholar
  3. Bertsekas DP (1999) Nonlinear programming, 2nd edn. Athena Scientific, BelmontzbMATHGoogle Scholar
  4. Birgin EG, Martínez JM, Raydan M (2000) Nonmonotone spectral projected gradient methods on convex sets. SIAM J Optim 10(4):1196–1211MathSciNetCrossRefzbMATHGoogle Scholar
  5. Birgin EG, Martínez JM, Raydan M (2012) Spectral projected gradient methods: Review and perspectives.
  6. Bonettini S, Zanella R, Zanni L (2009) A scaled gradient projection method for constrained image deblurring. Inverse Probl 25(1):015002MathSciNetCrossRefGoogle Scholar
  7. Cai D, He X, Han J (2005) Document clustering using locality preserving indexing. IEEE Trans Knowl Data Eng 17(12):1624–1637CrossRefGoogle Scholar
  8. Cai D, He X, Han J, Huang TS (2011) Graph regularized nonnegative matrix factorization for data representation. IEEE Trans Pattern Anal Mach Intell 33(8):1548–1560CrossRefGoogle Scholar
  9. Cichocki A, Zdunek R, Amari SI (2006) New algorithms for non-negative matrix factorization in applications to blind source separation. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 621–624Google Scholar
  10. Cichocki A, Zdunek R, Phan AH, Amari SI (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, ChichesterCrossRefGoogle Scholar
  11. Cieri C, Graff D, Liberman M, Martey N, Strassel S (1999) The TDT-2 text and speech corpus. In: Proceedings of the DARPA Broadcast News Workshop, pp 57–60Google Scholar
  12. Cores D, Escalante R, González-Lima M, Jimenez O (2009) On the use of the spectral projected gradient method for support vector machines. Comp Optim Appl 28(3):327–364zbMATHGoogle Scholar
  13. Dai YH, Zhang H (2001) Adaptive two-point stepsize gradient algorithm. Numer Algor 27(4):377–385MathSciNetCrossRefzbMATHGoogle Scholar
  14. Dai YH, Liao LZ (2002) \(R\)-linear convergence of the Barzilai and Borwein gradient method. IMA J Numer Anal 22(1):1–10MathSciNetCrossRefzbMATHGoogle Scholar
  15. Dai YH, Fletcher R (2005) Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer Math 100(1):21–47MathSciNetCrossRefzbMATHGoogle Scholar
  16. Dai YH, Hager WW, Schittkowski K, Zhang H (2006) The cyclic Barzilai–Borwein method for unconstrained optimization. IMA J Numer Anal 26(3):604–627MathSciNetCrossRefzbMATHGoogle Scholar
  17. Figueiredo MA, Nowak RD, Wright SJ (2007) Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J Sel Top Signal Process 1(4):586–597CrossRefGoogle Scholar
  18. Fletcher R (2005) On the Barzilai–Borwein method. Optimization and control with applications, applied optimization. Springer, New YorkGoogle Scholar
  19. Gong P, Zhang C (2012) Efficient nonnegative matrix factorization via projected Newton method. Pattern Recognit 45(9):3557–3565CrossRefzbMATHGoogle Scholar
  20. Grippo L, Sciandrone M (2000) On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper Res Lett 26(3):127–136MathSciNetCrossRefzbMATHGoogle Scholar
  21. Grippo L, Sciandrone M (2002) Nonmonotone globalization techniques for the Barzilai–Borwein gradient method. Comput Optim Appl 23(2):143–169MathSciNetCrossRefzbMATHGoogle Scholar
  22. Guan N, Tao D, Luo Z, Yuan B (2012) NeNMF: an optimal gradient method for nonnegative matrix factorization. IEEE Trans Signal Process 60(6):2882–2898MathSciNetCrossRefGoogle Scholar
  23. Han L, Neumann M, Prasad U (2009) Alternating projected Barzilai-Borwein methods for nonnegative matrix factorization. Electron Trans Numer Anal 36(6):54–82MathSciNetzbMATHGoogle Scholar
  24. Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5:1457–1469MathSciNetzbMATHGoogle Scholar
  25. Huang Y, Liu H, Zhou S (2013) A Barzilai–Borwein type method for stochastic linear complementarity problems. Numer Algor. doi: 10.1007/s11075-013-9803-y
  26. Kim H, Park H (2008) Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM J Matrix Anal Appl 30(2):713–730MathSciNetCrossRefzbMATHGoogle Scholar
  27. Kim D, Sra S, Dhillon IS (2007) Fast Newton-type methods for the least squares nonnegative matrix approximation problem. In: Proceedings of the 2007 SIAM International Conference on Data Mining, pp 343–354Google Scholar
  28. Lee DD, Seung HS (2001) Algorithms for nonnegative matrix factorization. In: Advances in Neural Information Processing Systems, pp 556–562Google Scholar
  29. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791CrossRefGoogle Scholar
  30. Lewis DD, Yang Y, Rose TG, Li F (2004) RCV1: a new benchmark collection for text categorization research. J Mach Learn Res 5:361–397Google Scholar
  31. Li SZ, Hou XW, Zhang HJ, Cheng QS (2001) Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 207–212Google Scholar
  32. Lin CJ (2007) Projected gradient methods for nonnegative matrix factorization. Neural Comput 19(10):2756–2779MathSciNetCrossRefzbMATHGoogle Scholar
  33. Nesterov Y (1983) A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). In: Soviet Mathematics Doklady, pp 372–376Google Scholar
  34. Paatero P, Tapper U (1994) Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5(2):111–126CrossRefGoogle Scholar
  35. Pascual-Montano A, Carazo JM, Kochi K, Lehmann D, Pascual-Marqui R (2006) Nonsmooth nonnegative matrix factorization (nsNMF). IEEE Trans Pattern Anal Mach Intell 28(3):403–415CrossRefGoogle Scholar
  36. Pauca VP (2004) Text mining using nonnegative matrix factorizations. In: Proceedings of the 2004 SIAM International Conference on Data Mining, pp 22–24Google Scholar
  37. Raydan M (1997) The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J Optim 7(1):26–33MathSciNetCrossRefzbMATHGoogle Scholar
  38. Shahnaz F, Berry MW, Pauca VP, Plemmons RJ (2006) Document clustering using nonnegative matrix factorization. J Inf Process Manage 42(2):373–386CrossRefzbMATHGoogle Scholar
  39. Vavasis S (2009) On the complexity of nonnegative matrix factorization. SIAM J Optim 20(3):1364–1377MathSciNetCrossRefzbMATHGoogle Scholar
  40. Xu W, Liu X, Gong Y (2003) Document clustering based on non-negative matrix factorization. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 267–273Google Scholar
  41. Zdunek R, Cichocki A (2006) Non-negative matrix factorization with quasi-Newton optimization. In: Proceedings of the 8th Artificial Intelligence and Soft Computing, pp 870–879Google Scholar
  42. Zhang H, Hager WW (2004) A nonmonotone line search technique and its application to unconstrained optimization. SIAM J Optim 14(4):1043–1056MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© The Author(s) 2014

Authors and Affiliations

  1. 1.School of Mathematics and StatisticsXidian UniversityXi’anPeople’s Republic of China

Personalised recommendations