Modified subspace Barzilai-Borwein gradient method for non-negative matrix factorization

  • Hongwei Liu
  • Xiangli LiEmail author


Non-negative matrix factorization (NMF) is a problem to obtain a representation of data using non-negativity constraints. Since the NMF was first proposed by Lee, NMF has attracted much attention for over a decade and has been successfully applied to numerous data analysis problems. Recent years, many variants of NMF have been proposed. Common methods are: iterative multiplicative update algorithms, gradient descent methods, alternating least squares (ANLS). Since alternating least squares has nice optimization properties, various optimization methods can be used to solve ANLS’s subproblems. In this paper, we propose a modified subspace Barzilai-Borwein for subproblems of ANLS. Moreover, we propose a modified strategy for ANLS. Global convergence results of our algorithm are established. The results of numerical experiments are reported to show the effectiveness of the proposed algorithm.


Non-negative matrix factorization Alternating least squares Active sets Non-monotone technique 


  1. 1.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999) CrossRefGoogle Scholar
  2. 2.
    Pauca, V.P., Shahnaz, F., Berry, M.W., Plemmons, R.J.: Text mining using non-negative matrix factorization. In: Proceedings of the Fourth SIAM International Conference on Data Mining (2004) Google Scholar
  3. 3.
    Kim, P.M., Tidor, B.: Subsystem identification through dimensionality reduction of large-scale gene expression data. Genome Res. 13, 1706–1718 (2003) CrossRefGoogle Scholar
  4. 4.
    Brunet, J.-P., Tamayo, P., Golub, T.R., Mesirov, J.P.: Metagenes and molecular pattern discovery using matrix factorization. In: Proceedings of the National Academy of Sciences of the United States of America, vol. 101, pp. 4164–4169 (2004) Google Scholar
  5. 5.
    Gao, Y., Church, G.: Improving molecular cancer class discovery through sparse non-negative matrix factorization. Bioinformatics 21, 3970–3975 (2005) CrossRefGoogle Scholar
  6. 6.
    Kim, H., Park, H.: Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23, 1495–1502 (2007) CrossRefGoogle Scholar
  7. 7.
    Richardson, W.H.: Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 (1972) CrossRefGoogle Scholar
  8. 8.
    Cho, Y.-C., Choi, S.: Nonnegative features of spectro-temporal sounds for classification. Pattern Recognit. Lett. 26, 1327–1336 (2005) CrossRefGoogle Scholar
  9. 9.
    Rao, N., Shepherd, S.J.: Extracting characteristic patterns from genome—wide expression data by non-negative matrix factorization. In: Computational Systems Bioinformatics Conference (2004) Google Scholar
  10. 10.
    Guillamet, D., Vitria, J., Schiele, B.: Introducing a weighted non-negative matrix factorization for image classification. Pattern Recognit. Lett. 24, 2447–2454 (2003) zbMATHCrossRefGoogle Scholar
  11. 11.
    Guillamet, D., Schiele, B., Vitrià, J.: Analyzing non-negative matrix factorization for image classification. In: 16th International Conference on Pattern Recognition, vol. 2, pp. 116–119 (2002) Google Scholar
  12. 12.
    Guillamet, D., Vitria, J.: Classifying faces with nonnegative matrix factorization. In: Procedure of the Fifth Catalan Conference for Artificial Intelligence (2002) Google Scholar
  13. 13.
    Ahn, J.-H., Kim, S.-K., Oh, J.-H., Choi, S.: Multiple nonnegativematrix factorization of dynamic PET images. In: ACCV (2004) Google Scholar
  14. 14.
    Lee, J.S., Lee, D.D., Choi, S., Lee, D.S.: Application of nonnegative matrix factorization to dynamic positron emission tomography. In: 3rd International Conference on Independent Component Analysis and Blind Signal Separation, pp. 556–562 (2002) Google Scholar
  15. 15.
    Li, H., Adali, T., Wang, W., Emge, D.: Non-negative matrix factorization with orthogonality constraints for chemical agent detection in Raman spectra. In: IEEE Workshop on Machine Learning for Signal Processing, pp. 253–258 (2005) Google Scholar
  16. 16.
    Kochi, K., Lehmann, D., Pascual Marqui, R.D.: Nonsmooth nonnegative matrix factorization (nsNMF). IEEE Trans. Pattern Anal. Mach. Intell. 28, 403–415 (2006) CrossRefGoogle Scholar
  17. 17.
    Okun, O., Priisalu, H.: Fast nonnegative matrix factorization and its application for protein fold recognition. EURASIP J. Appl. Signal Process. 2006, 62 (2009) Google Scholar
  18. 18.
    Wang, Y., Jia, Y., Hu, C., Matthew, T.: Non-negative matrix factorization framework for face recognition. Int. J. Pattern Recognit. Artif. Intell. 19, 495–511 (2005) CrossRefGoogle Scholar
  19. 19.
    Liu, W., Zheng, N.: Non-negative matrix factorization based methods for object recognition. Pattern Recognit. Lett. 25, 893–897 (2004) CrossRefGoogle Scholar
  20. 20.
    Spratling, M.W.: Learning image components for object recognition. J. Mach. Learn. Res. 7, 793–815 (2006) MathSciNetzbMATHGoogle Scholar
  21. 21.
    Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Proceedings of Neural Information Processing Systems, vol. 13, pp. 556–562 (2001) Google Scholar
  22. 22.
    Chu, M., Diele, F., Plemmons, R., Ragni, S.: Optimality, computation, and interpretations of nonnegative matrix factorizations. SIAM J. Matrix Anal. 4–8030 (2004) Google Scholar
  23. 23.
    Paatero, P., Tapper, U.: Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5, 111–126 (1994) CrossRefGoogle Scholar
  24. 24.
    Lin, C.-J.: Projected gradient methods for nonnegative matrix factorization. Neural Comput. 19, 2756–2779 (2007) MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Piper, J., Pauca, P., Plemmons, R., Giffin, M.: Object characterization from spectral data using nonnegative factorization and information theory. In: Proceedings of AMOS Technical Conference (2004) Google Scholar
  26. 26.
    Lin, C.-J.: On the convergence of multiplicative update algorithms for non-negative matrix factorization. IEEE Trans. Neural Netw. 18, 1589–1596 (2007) CrossRefGoogle Scholar
  27. 27.
    Shahnaz, F., Berry, M.W., Pauca, V.P., Plemmons, R.J.: Document clustering using nonnegative matrix factorization. Inf. Process. Manag. 42, 373–386 (2006) zbMATHCrossRefGoogle Scholar
  28. 28.
    Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. Mach. Learn. Res. 5, 1457–1469 (2004) MathSciNetzbMATHGoogle Scholar
  29. 29.
    Grippo, L., Sciandrone, M.: On the convergence of the block nonlinear Gauss-Seidel method under convex constraints. Oper. Res. Lett. 26, 127–136 (2000) MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Zdunek, R., Cichocki, A.: Non-negative matrix factorization with quasi-newton optimization. In: International Conference on Artificial Intelligence and Soft Computing (ICAISC 2006), pp. 870–879 (2006) CrossRefGoogle Scholar
  31. 31.
    Park, H., Kim, H.: One-sided non-negative matrix factorization and non-negative centroid dimension reduction for text classification. In: Proceedings of the Workshop on Text Mining at the 6th SIAM International Conference on Data Mining (SDM06) (2006) Google Scholar
  32. 32.
    Xiao, Y., Hu, Q.: Subspace Barzilai-Borwein gradient method for large-scale bound constrained optimization. Appl. Math. Optim. 58, 275–290 (2008) MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988) MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Dai, Y.-H., Fletcher, R.: On the asymptotic behaviour of some new gradient methods. Math. Program. Ser. A, B 103, 541–559 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Dai, Y.-H., Liao, L.-Z.: R-linear convergence of the Barzilai and Borwein gradient method. IMA J. Numer. Anal. 22, 1–10 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Dai, Y., Yuan, J., Yuan, Y.-X.: Modified two-point stepsize gradient methods for unconstrained optimization. Comput. Optim. Appl. 22, 103–109 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Birgin, E.G., Martínez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Control Optim. 10, 1196–1211 (2000) zbMATHCrossRefGoogle Scholar
  38. 38.
    Frassoldati, G., Zanni, L., Zanghirati, G.: New adaptive stepsize selections in gradient methods. J. Ind. Manag. Optim. 4, 299–312 (2008) MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Dai, Y.-H., Fletcher, R.: Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100, 21–47 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986) MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Control Optim. 14, 1043–1056 (2004) MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Toint, P.L.: An assessment of nonmonotone linesearch techniques for unconstrained optimization. SIAM J. Sci. Comput. 17, 725–739 (1996) MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Gafni, E.M., Bertsekas, D.P.: Two-metric projection methods for constrained optimization. SIAM J. Control Optim. 22, 936–964 (1984) MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Zhang, Y.: An alternating direction algorithm for nonnegative matrix factorization. Technical report, Rice University (2010) Google Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  1. 1.Department of MathematicsXidian UniversityXi’anP.R. China
  2. 2.College of Mathematics and Computing ScienceGuilin University of Electronic TechnologyGuilinP.R. China

Personalised recommendations