Sparse Non-negative Matrix Factorization with Generalized Kullback-Leibler Divergence

  • Jingwei Chen
  • Yong Feng
  • Yang LiuEmail author
  • Bing Tang
  • Wenyuan Wu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9937)


Non-negative Matrix Factorization (NMF), especially with sparseness constraints, plays a critically important role in data engineering and machine learning. Hoyer (2004) presented an algorithm to compute NMF with exact sparseness constraints. The exact sparseness constraints depends on a projection operator. In the present work, we first give a very simple counterexample, for which the projection operator of the Hoyer (2004) algorithm fails. After analysing the reason geometrically, we fix this bug by adding some random terms and show that the fixed one works correctly. Based on the fixed projection operator, we propose another sparse NMF algorithm aiming at optimizing the generalized Kullback-Leibler divergence, hence named SNMF-GKLD. Experimental results show that SNMF-GKLD not only has similar effects with Hoyer (2004) on the same data sets, but is also efficient.


Non-negative Matrix Factorization Projection operator Generalized Kullback-Leibler divergence 



This work was partially supported by NSFC (11471307, 11501540, 61572024), CAS “Light of West China” Program (2014), NSF of Hunan Province (2015JJ3071) and Chongqing Research Program (cstc2015jcyjys40001).


  1. 1.
    Berry, M.W., Browne, M., Langville, A.N., Pauca, V.P., Plemmons, R.J.: Algorithms and applications for approximate nonnegative matrix factorization. Comput. Stat. Data Anal. 52(1), 155–173 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Boutsidis, C., Gallopoulos, E.: SVD based initialization: a head start for nonnegative matrix factorization. Pattern Recogn. 41(4), 1350–1362 (2008)CrossRefzbMATHGoogle Scholar
  3. 3.
    Cichocki, A., Amari, S.I., Zdunek, R., Kompass, R., Hori, G., He, Z.: Extended SMART algorithms for non-negative matrix factorization. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. Lecture Notes in Artificial Intelligence (LNAI), vol. 4029, pp. 548–562. Springer, Heidelberg (2006). doi: 10.1007/11785231_58 CrossRefGoogle Scholar
  4. 4.
    Cichocki, A., Zdunek, R., Phan, A.H., Amari, S.I.: Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. John Wiley & Sons Ltd., Chichester (2009)CrossRefGoogle Scholar
  5. 5.
    Hoyer, P.O.: Non-negative sparse coding. In: Bourlard, H., Adali, T., Bengio, S., Larsen, J., Douglas, S. (eds.) NNSP 2012, pp. 557–565. IEEE, New York (2002)Google Scholar
  6. 6.
    Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. Mach. Learn. Res. 5, 1457–1469 (2004)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Huang, Z., Zhou, A., Zhang, G.: Non-negative matrix factorization: a short survey on methods and applications. In: Li, K., Li, J., Liu, Y., Castiglione, A. (eds.) ISICA 2015. CCIS, vol. 575, pp. 331–340. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-34289-9_37 Google Scholar
  8. 8.
    Hyvärinen, A., Karhunen, J., Oja, E.: Independent Component Analysis. John Wiley & Sons, New York (2001)CrossRefGoogle Scholar
  9. 9.
    Jolliffe, I.T.: Principal Component Analysis, 2nd edn. Springer, New York (2002)zbMATHGoogle Scholar
  10. 10.
    Kim, H., Park, H.: Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23(12), 1495–1502 (2007)CrossRefGoogle Scholar
  11. 11.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401(6755), 788–791 (1999)CrossRefGoogle Scholar
  12. 12.
    Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) NIPS*2000, pp. 556–562. MIT Press, Cambridge (2001)Google Scholar
  13. 13.
    Li, S.Z., Hou, X., Zhang, H., Cheng, Q.: Learning spatially localized, parts-based representation. In: CVPR 2001, vol. 1, pp. 207–212. IEEE, Los Alamitos (2001)Google Scholar
  14. 14.
    Liu, W., Zheng, N.: Learning sparse features for classification by mixture models. Pattern Recogn. Lett. 25(2), 155–161 (2004)CrossRefGoogle Scholar
  15. 15.
    Liu, W., Zheng, N., Lu, X.: Non-negative matrix factorization for visual coding. In: ICASSP 2003, vol. 3, pp. 293–296. IEEE, Piscataway (2003)Google Scholar
  16. 16.
    Pascual-Montano, A., Carazo, J.M., Kochi, K., Lehmann, D., Pascual-Marqui, R.D.: Nonsmooth nonnegative matrix factorization (nsNMF). IEEE Trans. Pattern Anal. Mach. Intell. 28(3), 403–415 (2006)CrossRefGoogle Scholar
  17. 17.
    Stadlthanner, K., Theis, F.J., Puntonet, C.G., Lang, E.W.: Extended sparse nonnegative matrix factorization. In: Cabestany, J., Prieto, A., Sandoval, F. (eds.) IWANN 2005. LNCS, vol. 3512, pp. 249–256. Springer, Heidelberg (2005). doi: 10.1007/11494669_31 CrossRefGoogle Scholar
  18. 18.
    Tong, M., Guo, J., Tao, S., Wu, Y.: Independent detection and self-recovery video authentication mechanism using extended NMF with different sparseness constraints. Multimedia Tools Appl. 75(13), 8045–8069 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Jingwei Chen
    • 1
  • Yong Feng
    • 1
  • Yang Liu
    • 2
    Email author
  • Bing Tang
    • 3
  • Wenyuan Wu
    • 1
  1. 1.Chongqing Key Laboratory of Automated Reasoning and CognitionChongqing Institute of Green and Intelligent Technology, CASChongqingChina
  2. 2.College of Information Science and EngineeringChongqing Jiaotong UniversityChongqingChina
  3. 3.School of Computer Science and EngineeringHunan University of Science and TechnologyXiangtanChina

Personalised recommendations