Extended SMART Algorithms for Non-negative Matrix Factorization

Invited Paper
  • Andrzej Cichocki
  • Shun-ichi Amari
  • Rafal Zdunek
  • Raul Kompass
  • Gen Hori
  • Zhaohui He
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4029)


In this paper we derive a family of new extended SMART (Simultaneous Multiplicative Algebraic Reconstruction Technique) algorithms for Non-negative Matrix Factorization (NMF). The proposed algorithms are characterized by improved efficiency and convergence rate and can be applied for various distributions of data and additive noise. Information theory and information geometry play key roles in the derivation of new algorithms. We discuss several loss functions used in information theory which allow us to obtain generalized forms of multiplicative NMF learning adaptive algorithms. We also provide flexible and relaxed forms of the NMF algorithms to increase convergence speed and impose an additional constraint of sparsity. The scope of these results is vast since discussed generalized divergence functions include a large number of useful loss functions such as the Amari α– divergence, Relative entropy, Bose-Einstein divergence, Jensen-Shannon divergence, J-divergence, Arithmetic-Geometric (AG) Taneja divergence, etc. We applied the developed algorithms successfully to Blind (or semi blind) Source Separation (BSS) where sources may be generally statistically dependent, however are subject to additional constraints such as nonnegativity and sparsity. Moreover, we applied a novel multilayer NMF strategy which improves performance of the most proposed algorithms.


Loss Function Blind Source Separation Nonnegative Matrix Factorization Blind Signal Positive Matrix Factorization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lee, D.D., Seung, H.S.: Learning of the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  2. 2.
    Cho, Y.-C., Choi, S.: Nonnegative features of spectro-temporal sounds for classification. Pattern Recognition Letters 26(9), 1327–1336 (2005)CrossRefGoogle Scholar
  3. 3.
    Sajda, P., Du, S., Parra, L.: Recovery of constituent spectra using non-negative matrix factorization. In: Proceedings of SPIE. Wavelets: Applications in Signal and Image Processing, vol. 5207, pp. 321–331 (2003)Google Scholar
  4. 4.
    Guillamet, D., Vitri‘a, J., Schiele, B.: Introducing a weighted nonnegative matrix factorization for image classification. Pattern Recognition Letters 24, 2447 (2004)CrossRefGoogle Scholar
  5. 5.
    Li, D., Adali, T., Wang, D.E.W.: Non-negative matrix factorization with orthogonality constraints for chemical agent detection in Raman spectra. In: IEEE Workshop on Machine Learning for Signal Processing, Mystic USA (2005)Google Scholar
  6. 6.
    Cichocki, A., Zdunek, R., Amari, S.: Csiszar’s divergences for non-negative matrix factorization: Family of new algorithms. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 32–39. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Paatero, P., Tapper, U.: Positive matrix factorization: A nonnegative factor model with optimal utilization of error estimates of data values. Environmetrics 5, 111–126 (1994)CrossRefGoogle Scholar
  8. 8.
    Oja, E., Plumbley, M.: Blind separation of positive sources using nonnegative PCA. In: 4th International Symposium on Independent Component Analysis and Blind Signal Separation, Nara, Japan (2003)Google Scholar
  9. 9.
    Hoyer, P.: Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research 5, 1457–1469 (2004)MathSciNetGoogle Scholar
  10. 10.
    Kompass, R.: A generalized divergence measure for nonnegative matrix factorization, Neuroinfomatics Workshop, Torun, Poland (2005)Google Scholar
  11. 11.
    Dhillon, I., Sra, S.: Generalized nonnegative matrix approximations with Bregman divergences. In: NIPS -Neural Information Proc. Systems, Vancouver Canada (2005)Google Scholar
  12. 12.
    Berry, M., Browne, M., Langville, A., Pauca, P., Plemmons, R.: Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics and Data Analysis (2006),
  13. 13.
    Lee, D.D., Seung, H.S.: Algorithms for nonnegative matrix factorization, vol. 13, pp. 556–562. NIPS, MIT Press, Cambridge (2001)Google Scholar
  14. 14.
    Novak, M., Mammone, R.: Use of nonnegative matrix factorization for language model adaptation in a lecture transcription task. In: Proceedings of the 2001 IEEE Conference on Acoustics, Speech and Signal Processing, Salt Lake City, UT, vol. 1, pp. 541–544 (2001)Google Scholar
  15. 15.
    Feng, T., Li, S.Z., Shum, H.Y., Zhang, H.: Local nonnegative matrix factorization as a visual representation. In: Proceedings of the 2nd International Conference on Development and Learning, Cambridge, MA, pp. 178–193 (2002)Google Scholar
  16. 16.
    Chen, Z., Cichocki, A., Rutkowski, T.: Constrained non-negative matrix factorization method for EEG analysis in early detection of Alzheimer’s disease. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2006, Toulouse, France (2006)Google Scholar
  17. 17.
    Cichocki, A., Amari, S.: Adaptive Blind Signal And Image Processing (New revised and improved edition). John Wiley, New York (2003)Google Scholar
  18. 18.
    Cichocki, A., Zdunek, R.: NMFLAB Toolboxes for Signal and Image Processing, Japan (2006),
  19. 19.
    Merritt, M., Zhang, Y.: An interior-point gradient method for large-scale totally nonnegative least squares problems. Technical report, Department of Computational and Applied Mathematics, Rice University, Houston, Texas, USA (2004)Google Scholar
  20. 20.
    Minami, M., Eguchi, S.: Robust blind source separation by beta-divergence. Neural Computation 14, 1859–1886 (2002)MATHCrossRefGoogle Scholar
  21. 21.
    Jorgensen, B.: The Theory of Dispersion Models. Chapman and Hall, Boca Raton (1997)Google Scholar
  22. 22.
    Csiszár, I.: Information measures: A critical survey. Prague Conference on Information Theory, Academia Prague A, 73–86 (1974)Google Scholar
  23. 23.
    Amari, S., Nagaoka, H.: Methods of Information Geometry. Oxford University Press, Oxford (2000)MATHGoogle Scholar
  24. 24.
    Zhang, J.: Divergence function, duality and convex analysis. Neural Computation 16, 159–195 (2004)MATHCrossRefGoogle Scholar
  25. 25.
    Schraudolf, N.: Gradient-based manipulation of non-parametric entropy estimates. IEEE Trans. on Neural Networks 16, 159–195 (2004)Google Scholar
  26. 26.
    Byrne, C.: Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative (RBI) methods. IEEE Transactions on Image Processing 7, 100–109 (1998)MATHCrossRefMathSciNetGoogle Scholar
  27. 27.
    Byrne, C.: Choosing parameters in block-iterative or ordered subset reconstruction algorithms. IEEE Transactions on Image Progressing 14, 321–327 (2005)CrossRefMathSciNetGoogle Scholar
  28. 28.
    Amari, S.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)MATHGoogle Scholar
  29. 29.
    Amari, S.: Information geometry of the EM and em algorithms for neural networks. neural networks 8(9), 1379–1408 (1995)CrossRefGoogle Scholar
  30. 30.
    Cressie, N.A., Read, T.C.R.: Goodness-of-Fit Statistics for Discrete Multivariate Data. Springer, Heidelberg (1988)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Andrzej Cichocki
    • 1
  • Shun-ichi Amari
    • 2
  • Rafal Zdunek
    • 1
  • Raul Kompass
    • 1
  • Gen Hori
    • 1
  • Zhaohui He
    • 1
  1. 1.Laboratory for Advanced Brain Signal ProcessingJapan
  2. 2.Amari Research Unit for Mathematical Neuroscience, BSI, RIKENWako-shiJapan

Personalised recommendations