Novel Multi-layer Non-negative Tensor Factorization with Sparsity Constraints

  • Andrzej Cichocki
  • Rafal Zdunek
  • Seungjin Choi
  • Robert Plemmons
  • Shun-ichi Amari
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4432)


In this paper we present a new method of 3D non-negative tensor factorization (NTF) that is robust in the presence of noise and has many potential applications, including multi-way blind source separation (BSS), multi-sensory or multi-dimensional data analysis, and sparse image coding. We consider alpha- and beta-divergences as error (cost) functions and derive three different algorithms: (1) multiplicative updating; (2) fixed point alternating least squares (FPALS); (3) alternating interior-point gradient (AIPG) algorithm. We also incorporate these algorithms into multilayer networks. Experimental results confirm the very useful behavior of our multilayer 3D NTF algorithms with multi-start initializations.


Monte Carlo Blind Source Separation Nonnegative Matrix Factorization Alternate Little Square Parallel Factor Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Workshop on tensor decompositions and applications. CIRM, Marseille, France (2005)Google Scholar
  2. 2.
    Hazan, T., Polak, S., Shashua, A.: Sparse image coding using a 3D non-negative tensor factorization. In: International Conference of Computer Vision (ICCV), pp. 50–57 (2005),
  3. 3.
    Heiler, M., Schnörr, C.: Controlling sparseness in non-negative tensor factorization. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 56–67. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Smilde, A., Bro, R., Geladi, P.: Multi-way Analysis: Applications in the Chemical Sciences. John Wiley and Sons, New York (2004)Google Scholar
  5. 5.
    Berry, M., Browne, M., Langville, A., Pauca, P., Plemmons, R.: Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics and Data Analysis, submitted (2006)Google Scholar
  6. 6.
    Cichocki, A., Zdunek, R., Amari, S.-i.: Csiszár’s divergences for non-negative matrix factorization: Family of new algorithms. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 32–39. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Cichocki, A., Amari, S.-i., Zdunek, R., Kompass, R., Hori, G., He, Z.: Extended SMART algorithms for non-negative matrix factorization. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. LNCS (LNAI), vol. 4029, pp. 548–562. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  8. 8.
    Cichocki, A., Zdunek, R.: NMFLAB for Signal and Image Processing. Technical report, Laboratory for Advanced Brain Signal Processing, BSI, RIKEN, Saitama, Japan (2006)Google Scholar
  9. 9.
    Cichocki, A., Zdunek, R.: NTFLAB for Signal Processing. Technical report, Laboratory for Advanced Brain Signal Processing, BSI, RIKEN, Saitama, Japan (2006)Google Scholar
  10. 10.
    Dhillon, I., Sra, S.: Generalized nonnegative matrix approximations with Bregman divergences. In: Neural Information Proc. Systems, Vancouver, Canada, December 2005, pp. 283–290 (2005)Google Scholar
  11. 11.
    Hoyer, P.: Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research 5, 1457–1469 (2004)MathSciNetGoogle Scholar
  12. 12.
    Kim, M., Choi, S.: Monaural music source separation: Nonnegativity, sparseness, and shift-invariance. In: Rosca, J., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 617–624. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  13. 13.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by nonnegative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  14. 14.
    Vasilescu, M.A.O., Terzopoulos, D.: Multilinear analysis of image ensembles: TensorFaces. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 447–460. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. 15.
    Morup, M., Hansen, L.K., Herrmann, C.S., Parnas, J., Arnfred, S.M.: Parallel factor analysis as an exploratory tool for wavelet transformed event-related EEG. NeuroImage 29(3), 938–947 (2006)CrossRefGoogle Scholar
  16. 16.
    Miwakeichi, F., Martínez-Montes, E., Valdés-Sosa, P.A., Nishiyama, N., Mizuhara, H., Yamaguchi, Y.: Decomposing EEG data into space-time-frequency components using Parallel Factor Analysis. NeuroImage 22, 1035–1045 (2004)CrossRefGoogle Scholar
  17. 17.
    Amari, S.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)zbMATHGoogle Scholar
  18. 18.
    Merritt, M., Zhang, Y.: An interior-point gradient method for large-scale totally nonnegative least squares problems. J. Optimization Theory and Applications 126(1), 191–202 (2005)zbMATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Cichocki, A., Zdunek, R.: Multilayer nonnegative matrix factorization. Electronics Letters 42(16), 947–948 (2006)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Andrzej Cichocki
    • 1
  • Rafal Zdunek
    • 1
  • Seungjin Choi
    • 2
  • Robert Plemmons
    • 3
  • Shun-ichi Amari
    • 1
  1. 1.RIKEN Brain Science Institute, Wako-shiJapan
  2. 2.POSTECHKorea
  3. 3.Dept. of Mathematics and Computer Science, Wake Forest UniversityUSA

Personalised recommendations