Controlling Sparseness in Non-negative Tensor Factorization

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3951)


Non-negative tensor factorization (NTF) has recently been proposed as sparse and efficient image representation (Welling and Weber, Patt. Rec. Let., 2001). Until now, sparsity of the tensor factorization has been empirically observed in many cases, but there was no systematic way to control it. In this work, we show that a sparsity measure recently proposed for non-negative matrix factorization (Hoyer, J. Mach. Learn. Res., 2004) applies to NTF and allows precise control over sparseness of the resulting factorization. We devise an algorithm based on sequential conic programming and show improved performance over classical NTF codes on artificial and on real-world data sets.


Nonnegative Matrix Factorization Positive Matrix Factorization Tensor Factorization Sparsity Constraint Alternate Minimization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Welling, M., Weber, M.: Positive tensor factorization. Pattern Recog. Letters 22(12), 1255–1261 (2001)CrossRefzbMATHGoogle Scholar
  2. 2.
    Shashua, A., Hazan, T.: Non-negative tensor factorization with applications to statistics and computer vision. In: Proc. of ICML (2005)Google Scholar
  3. 3.
    Hazan, T., Polak, S., Shashua, A.: Sparse image coding using a 3D non-negative tensor factorization. In: Proc. of ICCV (2005)Google Scholar
  4. 4.
    Shen, J., Israël, G.W.: A receptor model using a specific non-negative transformation technique for ambient aerosol. Atmospheric Environment 23(10), 2289–2298 (1989)CrossRefGoogle Scholar
  5. 5.
    Paatero, P., Tapper, U.: Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5, 111–126 (1994)CrossRefGoogle Scholar
  6. 6.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)CrossRefzbMATHGoogle Scholar
  7. 7.
    Shashua, A., Levin, A.: Linear image coding for regression and classification using the tensor-rank principle. In: Proc. of CVPR (2001)Google Scholar
  8. 8.
    Donoho, D., Stodden, V.: When does non-negative matrix factorization give a correct decomposition into parts? In: Adv. in NIPS, vol. 17 (2004)Google Scholar
  9. 9.
    Kruskal, J.B.: Three-way arrays: Rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra and its Applications 18, 95–138 (1977)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Sidiropoulos, N.D., Bro, R.: On the uniqueness of multilinear decompositions of N-way arrays. J. of Chemometrics 14, 229–239 (2000)CrossRefGoogle Scholar
  11. 11.
    Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. of Mach. Learning Res. 5, 1457–1469 (2004)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Adv. in NIPS (2000)Google Scholar
  13. 13.
    Chu, M., Diele, F., Plemmons, R., Ragni, S.: Optimality, computations, and interpretation of nonnegative matrix factorizations. SIAM J. Mat. Anal. Appl. (submitted, 2004),
  14. 14.
    Horst, R., Tuy, H.: Global Optimization. Springer, Berlin (1996)CrossRefzbMATHGoogle Scholar
  15. 15.
    Heiler, M., Schnörr, C.: Learning non-negative sparse image codes by convex programming. In: Proc. of ICCV (2005)Google Scholar
  16. 16.
    Littlestone, N., Warmuth, M.: Relating data compression, learnability, and the Vapnik-Chervonenkis dimension. Tech. Rep., Univ. of Calif. Santa Cruz (1986)Google Scholar
  17. 17.
    Herbrich, R., Williamson, R.C.: Algorithmic luckiness. J. of Mach. Learning Res. 3, 175–212 (2002)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research 37, 3311–3325 (1997)CrossRefGoogle Scholar
  19. 19.
    Lobo, M.S., Vandenberghe, L., Boyd, S., Lebret, H.: Applications of second-order cone programming. In: Linear Algebra and its Applications (1998)Google Scholar
  20. 20.
    Sturm, J.F.: Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones (updated version 1.05). Department of Econometrics, Tilburg University, Tilburg, The Netherlands (2001)Google Scholar
  21. 21.
    Mittelmann, H.: An independent benchmarking of SDP and SOCP solvers. Math. Programming, Series B 95(2), 407–430 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    MOSEK ApS, Denmark, The MOSEK optimization tools version 3.2 (Revision 8) User’s manual and reference (2005)Google Scholar
  23. 23.
    Buchanan, A.M., Fitzgibbon, A.W.: Damped Newton algorithms for matrix factorization with missing data. In: CVPR 2005, vol. 2, pp. 316–322 (2005)Google Scholar
  24. 24.
    Tuy, H.: Convex programs with an additional reverse convex constraint. J. of Optim. Theory and Applic. 52, 463–486 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Rockafellar, R., Wets, R.-B.: Variational Analysis. Grundlehren der math. Wissenschaften, vol. 317. Springer, Heidelberg (1998)zbMATHGoogle Scholar
  26. 26.
    CBCL: CBCL face database #1. MIT Center For Biological and Computational Learning (2000),

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  1. 1.Computer Vision, Graphics, and Pattern Recognition Group, Department of Mathematics and Computer ScienceUniversity of MannheimMannheimGermany

Personalised recommendations