Non-negative Matrix Factorization with Quasi-Newton Optimization

  • Rafal Zdunek
  • Andrzej Cichocki
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4029)


Non-negative matrix factorization (NMF) is an emerging method with wide spectrum of potential applications in data analysis, feature extraction and blind source separation. Currently, most applications use relative simple multiplicative NMF learning algorithms which were proposed by Lee and Seung, and are based on minimization of the Kullback-Leibler divergence and Frobenius norm. Unfortunately, these algorithms are relatively slow and often need a few thousands of iterations to achieve a local minimum. In order to increase a convergence rate and to improve performance of NMF, we proposed to use a more general cost function: so-called Amari alpha divergence. Taking into account a special structure of the Hessian of this cost function, we derived a relatively simple second-order quasi-Newton method for NMF. The validity and performance of the proposed algorithm has been extensively tested for blind source separation problems, both for signals and images. The performance of the developed NMF algorithm is illustrated for separation of statistically dependent signals and images from their linear mixtures.


Nonnegative Matrix Factorization Positive Matrix Factorization Nonlinear Projection Blind Source Separation Problem Negative Matrix Factorization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Paatero, P., Tapper, U.: Positive matrix factorization: A nonnegative factor model with optimal utilization of error estimates of data values. Environmetrics 5, 111–126 (1994)CrossRefGoogle Scholar
  2. 2.
    Lee, D.D., Seung, H.S.: Learning of the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  3. 3.
    Dhillon, I., Sra, S.: Generalized nonnegative matrix approximations with Bregman divergences. In: Neural Information Proc. Systems, Vancouver, Canada (2005)Google Scholar
  4. 4.
    Chu, M., Plemmons, R.J.: Nonnegative matrix factorization and applications. Bulletin of the International Linear Algebra Society 34, 2–7 (2005)Google Scholar
  5. 5.
    Pauca, V.P., Shahnaz, F., Berry, M.W., Plemmons, R.J.: Text mining using non-negative matrix factorizations. In: Proc. SIAM Inter. Conf. on Data Mining, Orlando, FL (2004)Google Scholar
  6. 6.
    Cichocki, A., Amari, S.: Adaptive Blind Signal And Image Processing (New revised and improved edition). John Wiley, New York (2003)Google Scholar
  7. 7.
    Amari, S.: Information geometry of the EM and em algorithms for neural networks. Neural Networks 8(9), 1379–1408 (1995)CrossRefGoogle Scholar
  8. 8.
    Byrne, C.: Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative (RBI) methods. IEEE Transactions on Image Processing 7, 100–109 (1998)MATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Amari, S.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)MATHGoogle Scholar
  10. 10.
    Cressie, N.A., Read, T.: Goodness-of-Fit Statistics for Discrete Multivariate Data. Springer, New York (1988)MATHGoogle Scholar
  11. 11.
    Cichocki, A., Zdunek, R., Amari, S.: Csiszar’s divergences for non-negative matrix factorization: Family of new algorithms. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 32–39. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  12. 12.
    Cullum, J.: The effective choice of the smoothing norm in regularization. Math. Comp. 3, 149–170 (1979)CrossRefMathSciNetGoogle Scholar
  13. 13.
    Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)Google Scholar
  14. 14.
    Hansen, P.C.: Rank-Deficient and Discrete Ill-Posed Problems. SIAM, Philadelphia (1998)Google Scholar
  15. 15.
    Cichocki, A., Zdunek, R.: NMFLAB for Signal Processing. Technical report, Laboratory for Advanced Brain Signal Processing, BSI RIKEN, Saitama, Japan (2006),

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Rafal Zdunek
    • 1
  • Andrzej Cichocki
    • 1
  1. 1.Laboratory for Advanced Brain Signal ProcessingBSI, RIKENWako-shiJapan

Personalised recommendations