Csiszár’s Divergences for Non-negative Matrix Factorization: Family of New Algorithms

  • Andrzej Cichocki
  • Rafal Zdunek
  • Shun-ichi Amari
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3889)


In this paper we discus a wide class of loss (cost) functions for non-negative matrix factorization (NMF) and derive several novel algorithms with improved efficiency and robustness to noise and outliers. We review several approaches which allow us to obtain generalized forms of multiplicative NMF algorithms and unify some existing algorithms. We give also the flexible and relaxed form of the NMF algorithms to increase convergence speed and impose some desired constraints such as sparsity and smoothness of components. Moreover, the effects of various regularization terms and constraints are clearly shown. The scope of these results is vast since the proposed generalized divergence functions include quite large number of useful loss functions such as the squared Euclidean distance,Kulback-Leibler divergence, Itakura-Saito, Hellinger, Pearson’s chi-square, and Neyman’s chi-square distances, etc. We have applied successfully the developed algorithms to blind (or semi blind) source separation (BSS) where sources can be generally statistically dependent, however they satisfy some other conditions or additional constraints such as nonnegativity, sparsity and/or smoothness.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amari, S.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)MATHGoogle Scholar
  2. 2.
    Amari, S.: Information geometry of the EM and em algorithms for neural networks. Neural Networks 8, 1379–1408 (1995)CrossRefGoogle Scholar
  3. 3.
    Lee, D.D., Seung, H.S.: Learning of the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  4. 4.
    Hoyer, P.: Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research 5, 1457–1469 (2004)MathSciNetGoogle Scholar
  5. 5.
    Sajda, P., Du, S., Parra, L.: Recovery of constituent spectra using non-negative matrix factorization. In: Proceedings of SPIE. Wavelets: Applications in Signal and Image Processing, vol. 5207, pp. 321–331 (2003)Google Scholar
  6. 6.
    Cho, Y.C., Choi, S.: Nonnegative features of spectro-temporal sounds for classification. Pattern Recognition Letters 26, 1327–1336 (2005)CrossRefGoogle Scholar
  7. 7.
    Lee, D.D., Seung, H.S.: Algorithms for nonnegative matrix factorization. NIPS, vol. 13. MIT Press, Cambridge (2001)Google Scholar
  8. 8.
    Cichocki, A., Amari, S.: Adaptive Blind Signal and Image Processing (New revised and improved edition). John Wiley, New York (2003)Google Scholar
  9. 9.
    Byrne, C.: Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative (RBI) methods. IEEE Transactions on Image Processing 7, 100–109 (1998)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Csiszár, I.: Information measures: A critical survey. In: Prague Conference on Information Theory, Academia Prague, vol. A, pp. 73–86 (1974)Google Scholar
  11. 11.
    Cressie, N.A., Read, T.: Goodness-of-Fit Statistics for Discrete Multivariate Data. Springer, New York (1988)MATHGoogle Scholar
  12. 12.
    Kompass, R.: A generalized divergence measure for nonnegative matrix factorization. In: Neuroinfomatics Workshop, Torun, Poland (September 2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Andrzej Cichocki
    • 1
  • Rafal Zdunek
    • 1
  • Shun-ichi Amari
    • 2
  1. 1.Laboratory for Advanced Brain Signal Processing 
  2. 2.Laboratory for Mathematical NeuroscienceRIKEN BSIWako-shiJapan

Personalised recommendations