Skip to main content
SpringerLink
Log in

Initialization for non-negative matrix factorization: a comprehensive review

  • Review
  • Published:
International Journal of Data Science and Analytics Aims and scope Submit manuscript

Abstract

Non-negative matrix factorization (NMF) has become a popular method for representing meaningful data by extracting a non-negative basis feature from an observed non-negative data matrix. Some of the unique features of this method in identifying hidden data place this method among the powerful methods in the machine learning area. The NMF is a known non-convex optimization problem, and the initial point has a significant effect on finding an efficient local solution. In this paper, we investigate the most popular initialization procedures proposed for NMF so far. We describe each method and present some of their advantages and disadvantages. Finally, some numerical results to illustrate the performance of each algorithm are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The dataset used in this paper is available from the following link https://paperswithcode.com/dataset/orl

Code availability

In this paper, the NMF library was employed. https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html

Notes

  1. https://nimfa.biolab.si/nimfa.examples.orl_-images.html.

Abbreviations

NMF:

Non-negative matrix factorization

LR:

Low-rank

SVD:

Singular value decomposition

PCA:

Principal component analysis

KL:

Kullback-Leibler

MU:

Multiplicative update

NPCA:

Non-negative PCA

ICA:

Independent component analysis

NICA:

Non-negative ICA

NNDSVD:

Non-negative double SVD

NNSVD-LRC:

Non-negative SVD LR correction

FCM:

Fuzzy C-means

DE:

Differential evolution

PSO:

Particle swarm optimization

References

  1. Mehrmann, V., Miedlar, A., Nkengla, M., Friedland, S.: Fast low rank approximations of matrices and tensors. Electronic J. Linear Algebr. 22, 1031–1048 (2011)

    MathSciNet  MATH  Google Scholar 

  2. Eldén, Lars.: Numerical linear algebra and applications in data mining and it. (2003)

  3. Skillicorn, D.: Understanding complex datasets: data mining with matrix decompositions. Chapman and Hall, London (2007)

    Book  MATH  Google Scholar 

  4. Murphy, K.P.: Machine learning: a probabilistic perspective. MIT press, Cambridge (2012)

    MATH  Google Scholar 

  5. Lee, J., Kim, S., Lebanon, G., Singer, Y.: Matrix approximation under local low-rank assumption. http://arxiv.org/abs/1301.3192, (2013)

  6. Ling, Y.T., Sani, N.F.M., Abdullah, M.A., Hamid, N.A.W.A.: Structural features with nonnegative matrix factorization for metamorphic malware detection. Computers Secur. 104, 102216 (2021)

    Article  Google Scholar 

  7. Ling, Y.T., Sani, N.F.M., Abdullah, M.T., Hamid, N.A.W.A.: Nonnegative matrix factorization and metamorphic malware detection. J. Computer Virol. Hack. Tech. 15(3), 195–208 (2019)

    Article  Google Scholar 

  8. Ling, Y.T., Sani, N.F.M., Abdullah, M.T., Hamid, N.A.W.A.: Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model. J. Computer Virol. Hack. Tech. 25, 1–21 (2021)

    Google Scholar 

  9. Golub, G., Loan, C.F.V.: Matrix computations, 3rd edn. The John Hopkins University, Baltimore (1996)

    MATH  Google Scholar 

  10. Alter, O.B., Patrick, O., Botstein, D.: Singular value decomposition for genome-wide expression data processing and modeling. Proc. Nat. Acad. Sci. 97(18), 10101–10106 (2000)

    Article  Google Scholar 

  11. Wall, Michael E., Rechtsteiner, A., Rocha, Luis M.: Singular value decomposition and principal component analysis in a practical approach to microarray data analysis. Springer, Cham (2003)

    Google Scholar 

  12. Datta, BN.: Numerical linear algebra and applications, volume 116. Siam, (2010)

  13. Sundarapandian, V.: Numerical linear algebra. PHI Learning Pvt. Ltd., NY (2008)

    MATH  Google Scholar 

  14. Trefethen, Lloyd N., Bau, David III.: Numerical linear algebra, volume 50. Siam, (1997)

  15. Paruolo, Paolo.: Multivariate reduced rank regression, theory and applications, (2000)

  16. Miller, D.N., Callafon, Raymond A.: Identification of linear time-invariant systems via constrained step-based realization. IFAC Proc 45(16), 1155–1160 (2012)

    Article  Google Scholar 

  17. Chu, M.T., Funderlic, R.E.: Structured low rank approximation. Linear Algebra Appl. 366, 157–172 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  18. Drineas, P., Kannan, R., Mahoney, M.: Fast monte carlo algorithms for matrices i:approximating matrix multiplication. SIAM J. Comput. 36(1), 132–157 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  19. Kolda, T.G., Oleary, D.P.: A semidiscrete matrix decomposition for latent semantic indexing information retrieval. ACM Trans. Inf. Syst. (TOIS) 16(4), 322–346 (1998)

    Article  Google Scholar 

  20. Abdi, H., Williams, L.: Principal component analysis. Wiley Interdisciplinary Rev.: Comput. Stat. 2(4), 433–459 (2010)

    Article  Google Scholar 

  21. Kwak, K.-C., Pedrycz, W.: Face recognition using an enhanced independent component analysis approach. IEEE Trans. Neural Netw. 18(2), 530–541 (2007)

    Article  Google Scholar 

  22. Recht, B., Fazel, M., Parrilo, P.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  23. Gander, W.: Algorithms for the qr decomposition. Res. Rep 80(02), 1251–1268 (1980)

    Google Scholar 

  24. Esposito, F., Del, B., Nicoletta, S.L.: Nonnegative matrix factorization models for knowledge extraction from biomedical and other real world data. PAMM 20(1), e202000032 (2021)

    Article  Google Scholar 

  25. Peng, C., Zhang, Z., Chen, C., Kang, Z., Cheng, Q.: Two-dimensional semi-nonnegative matrix factorization for clustering. Inf. Sci. 590, 106–141 (2022)

    Article  Google Scholar 

  26. Kang, Z., Lin, Z., Zhu, X., Xu, W.: Structured graph learning for scalable subspace clustering: From single view to multiview. IEEE Trans. Cybern 26, 71 (2021)

    Google Scholar 

  27. Peng, C., Zhang, Z., Kang, Z., Chen, C., Cheng, Q.: Nonnegative matrix factorization with local similarity learning. Inf. Sci. 562, 325–346 (2021)

    Article  MathSciNet  Google Scholar 

  28. Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401(6755), 788–791 (1999)

    Article  MATH  Google Scholar 

  29. Aggarwal, Charu C., Reddy, Chandan K.: Data clustering. Algorithms and applications. Chapman &Hall/CRC Data mining and Knowledge Discovery series, Londra, (2014)

  30. Boutsidis, C., Gallopoulos, E.: Svd based initialization: a head start for nonnegative matrix factorization. Pattern Recogn. 41(4), 1350–1362 (2008)

    Article  MATH  Google Scholar 

  31. Wild, S., Wild, W-S., Curry, J., Dougherty, A., Betterton, M.: Seeding non-negative matrix factorizations with the spherical k-means clustering. PhD thesis, Citeseer, (2003)

  32. Gong, P., Zhang, C.: Efficient nonnegative matrix factorization via projected newton method. Pattern Recognit. 45(9), 3557–3565 (2012)

    Article  MATH  Google Scholar 

  33. Hafshejani, S-F., Gaur, D., Hossain, S., Benkoczi, R.: Barzilai and borwein conjugate gradient method equipped with a non-monotone line search technique and its application on non-negative matrix factorization. http://arxiv.org/abs/2109.05685, (2021)

  34. Li, X., Zhang, W., Dong, X.: A class of modified fr conjugate gradient method and applications to non-negative matrix factorization. Computers Math. Appl. 73(2), 270–276 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  35. Bonettini, S., Zanella, R., Zanni, L.: A scaled gradient projection method for constrained image deblurring. Inverse Probl. 25(1), 015002 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  36. Guan, N., Tao, D., Luo, Z., Yuan, B.: Nenmf: an optimal gradient method for nonnegative matrix factorization. IEEE Trans. Signal Process. 60(6), 2882–2898 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  37. Fathi-Hafshejani, S., Moaberfard, Z.: An interior-point algorithm for linearly constrained convex optimization based on kernel function and application in non-negative matrix factorization. Optim. Eng. 21(3), 1019–1051 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  38. Zdunek, R., Cichocki, A.: Non-negative matrix factorization with quasi-newton optimization. In International conference on artificial intelligence and soft computing, pages 870–879. Springer, (2006)

  39. Casalino, G., Del, B., Nicoletta, M.C.: Subtractive clustering for seeding non-negative matrix factorizations. Inf. Sci. 257, 369–387 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  40. Albright, R., Cox, J., Duling, D., Langville, A N., Meyer, C.: Algorithms, initializations, and convergence for the nonnegative matrix factorization. Technical report, Tech. rep. 919. NCSU Technical Report Math 81706. http://meyer.math.ncsu (2006)

  41. Zheng, Z., Yang, J., Zhu, Y.: Initialization enhancer for non-negative matrix factorization. Eng. Appl. Artif. Intell. 20(1), 101–110 (2007)

    Article  Google Scholar 

  42. Yun, X., Chong Sze, T., Ying, C., Wen-Sheng, C.: Clustering-based initialization for non-negative matrix factorization. Appl. Math. Comput. 205(2), 525–536 (2008)

    MathSciNet  MATH  Google Scholar 

  43. Ossama SA, ME Ghoneim, WA Awad, and Aboul Ella H.: Underdetermined blind source separation based on fuzzy c-means and semi-nonnegative matrix factorization. In 2012 Federated Conference on Computer Science and Information Systems (FedCSIS), pages 695–700. IEEE, 2012

  44. Rezaei, M., Boostani, R., Rezaei, M.: An efficient initialization method for nonnegative matrix factorization. J. Appl. Sci. 11(2), 354–359 (2011)

    Article  Google Scholar 

  45. Yong-Deok K and Seungjin C.: A method of initialization for nonnegative matrix factorization. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07, volume 2, pages II–537. IEEE, 2007

  46. Kurt S, Dominik L, Fabian JT, Elmar Wolfgang L, Ana Maria T, Petia G, and Carlos García P.: Sparse nonnegative matrix factorization with genetic algorithms for microarray analysis. In 2007 International Joint Conference on Neural Networks, pages 294–299. IEEE, 2007

  47. Václav S, Jan P, and Pavel K.: Developing genetic algorithms for boolean matrix factorization. Databases, Texts, 61, 2008a

  48. Price, K., Storn, R.M., Lampinen, J.A.: Differential evolution: a practical approach to global optimization. Springer Science & Business Media, Cham (2006)

    MATH  Google Scholar 

  49. Andreas J and Ying T.: Using population based algorithms for initializing nonnegative matrix factorization. In International Conference in Swarm Intelligence, pages 307–316. Springer, 2011

  50. Syed, M.A., Sameer, Q., Nicolas, G.: Improved svd-based initialization for nonnegative matrix factorization using low-rank correction. Pattern Recognit. Lett. 122, 53–59 (2019)

    Article  Google Scholar 

  51. Lihong Z, Guibin Z, and Xinhe X.: Facial expression recognition based on pca and nmf. In 2008 7th World Congress on Intelligent Control and Automation, pages 6826–6829. IEEE, 2008

  52. Daichi K and Nobutaka O.: Efficient initialization for nonnegative matrix factorization based on nonnegative independent component analysis. In 2016 IEEE International Workshop on Acoustic Signal Enhancement (IWAENC), pages 1–5. IEEE, 2016

  53. Oja, E., Plumbley, M.: Blind separation of positive sources by globally convergent gradient search. Neural Comput. 16(9), 1811–1825 (2004)

    Article  MATH  Google Scholar 

  54. Djaouad B., Shahram H., Yannick D., Moussa Sofiane K., and Abdelkader H.: Modified independent component analysis for initializing non-negative matrix factorization: An approach to hyperspectral image unmixing. In 2013 IEEE 11th International Workshop of Electronics, Control, Measurement, Signals and their application to Mechatronics, pages 1–6. IEEE, 2013

  55. Sandler, M.: On the use of linear programming for unsupervised text classification. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 256–264, (2005)

  56. Liu, Zhaoqiang., Tan, Vincent YF.: Rank-one nmf-based initialization for nmf and relative error bounds under a geometric assumption. In 2018 Information Theory and Applications Workshop (ITA), pages 1–15. IEEE, (2018)

  57. Zdunek, R.: Initialization of nonnegative matrix factorization with vertices of convex polytope. In International Conference on Artificial Intelligence and Soft Computing, pages 448–455. Springer, (2012)

  58. Sauwen, N., Acou, M., Bharath, Halandur N., Sima, Diana M., Veraart, J., Maes, F., Himmelreich, U., Achten, E., Van Huffel, Sabine: The successive projection algorithm as an initialization method for brain tumor segmentation using non-negative matrix factorization. Plos one 12(8), e0180268 (2017)

    Article  Google Scholar 

  59. Mahoney, Michael W.: Randomized algorithms for matrices and data. http://arxiv.org/abs/1104.5557, (2011)

  60. Drineas, P., Mahoney, M.: Randnla: randomized numerical linear algebra. Commun. ACM 59(6), 80–90 (2016)

    Article  Google Scholar 

  61. Wang, F., Li, P.: Efficient nonnegative matrix factorization with random projections. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 281–292. SIAM, (2010)

  62. Tepper, M., Sapiro, G.: Compressed nonnegative matrix factorization is fast and accurate. IEEE Trans. Signal Process. 64(9), 2269–2283 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  63. Langville Amy N., Meyer, Carl D., Albright, Russell., Cox, James., Duling, David.: Alternating least squares algorithms for the nonnegative matrix factorization, (2006)

  64. Bassiou, Nikoletta K., Kotropoulos, C.: Online plsa: Batch updating techniques including out-of-vocabulary words. IEEE Trans. Neural Netw. Learn. Syst. 25(11), 1953–1966 (2014)

    Article  Google Scholar 

  65. Zhang, C., Wang, H., Cao, L., Wang, W., Fanjiang, X.: A hybrid term-term relations analysis approach for topic detection. Knowledge-Based Syst. 93, 109–120 (2016)

    Article  Google Scholar 

  66. Berthold, Michael R., Borgelt, C., Höppner, F., Klawonn, F.: Guide to intelligent data analysis: how to intelligently make sense of real data. Springer Science & Business Media, Cham (2010)

    Book  MATH  Google Scholar 

  67. Xu, W., Liu, X., Gong, Y.: Document clustering based on non-negative matrix factorization. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 267–273, (2003)

  68. Chrétien, S., Guyeux, C., Conesa, B., Delage-Mouroux, R., Jouvenot, M., Huetz, P., Descôtes, F.: A bregman-proximal point algorithm for robust non-negative matrix factorization with possible missing values and outliers-application to gene expression analysis. BMC Bioinf. 17(8), 623–631 (2016)

    Google Scholar 

  69. Cichocki, A., Amari, S.-I.: Adaptive blind signal and image processing: learning algorithms and applications. Wiley, NY (2002)

    Book  Google Scholar 

  70. Perronnin, F., Bouchard, G.: Clustering using non-negative matrix factorization on sparse graphs, August 8 (2017). US Patent 9,727,532

  71. Wild, S., Curry, J., Dougherty, A.: Improving non-negative matrix factorizations through structured initialization. Pattern Recognit. 37(11), 2217–2232 (2004)

    Article  Google Scholar 

  72. Forgey, Edward: Cluster analysis of multivariate data: Efficiency vs. interpretability of classification. Biometrics 21(3), 768–769 (1965)

    Google Scholar 

  73. Hartigan, John A.: Clustering algorithms. Wiley, NY (1975)

    MATH  Google Scholar 

  74. Ding, C., He, X., Simon, H D.: On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM international conference on data mining, pages 606–610. SIAM, (2005)

  75. Anderberg, Michael R.: Cluster analysis for applications: probability and mathematical statistics: a series of monographs and textbooks. Academic press, Cambridge (2014)

    Google Scholar 

  76. Wild, S., Curry, J., Dougherty, A.: Improving non-negative matrix factorizations through structured initialization. Pattern Recognit. 37(11), 2217–2232 (2004)

    Article  Google Scholar 

  77. Zadeh, Lotfi Asker., Klir, George J., Yuan, Bo.: Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers, volume 6. World Scientific, (1996)

  78. Dong, B., Lin, Matthew M., Chu, Moody T.: Rank factorization-a heuristic approach via rank reduction. Numer. Algorith. 65(2), 251–274 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  79. Goldberg, David E., Holland, J H.: Genetic algorithms and machine learning. (1988)

  80. Snášel, Václav., Platoš, Jan., Krömer, Pavel.: Developing genetic algorithms for boolean matrix factorization. Databases, Texts, 61, (2008b)

  81. Eberhart, R., Kennedy, J.: Particle swarm optimization. In Proceedings of the IEEE international conference on neural networks, volume 4, pages 1942–1948. Citeseer, (1995)

  82. Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. 41(6), 391–407 (1990)

    Article  Google Scholar 

  83. Eckart, Carl, Young, Gale: The approximation of one matrix by another of lower rank. Psychometrika 1(3), 211–218 (1936)

    Article  MATH  Google Scholar 

  84. Cao, L. Singular value decomposition applied to digital image processing, pp. 1–15. Division of Computing Studies, Arizona State University Polytechnic Campus, Mesa, Arizona State University polytechnic Campus (2006)

  85. Cichocki, A., Zdunek, R., Phan, A.H., Amari, S.-I.: Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, NY (2009)

  86. Guillamet, David, Vitria, Jordi: Evaluation of distance metrics for recognition based on non-negative matrix factorization. Pattern Recogniti. Lett. 24(9–10), 1599–1605 (2003)

  87. Turk, Matthew, Pentland, Alex: Eigenfaces for recognition. Journal of cognitive neuroscience 3(1), 71–86 (1991)

    Article  Google Scholar 

  88. Geng, Xiu-rui, Ji, Lu.-yan, Sun, Kang: Non-negative matrix factorization based unmixing for principal component transformed hyperspectral data. Front. Inf. Technol. Electronic Eng. 17(5), 403–412 (2016)

    Article  Google Scholar 

  89. Shen, H., Huang, J.: Sparse principal component analysis via regularized low rank matrix approximation. J. Multiv. Anal. 99(6), 1015–1034 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  90. Zass, R., Shashua, A.: Nonnegative sparse pca. In Advances in neural information processing systems, pages 1561–1568. Citeseer, (2007)

  91. Bauckhage, C.: A purely geometric approach to non-negative matrix factorization. In LWA, pages 125–136. Citeseer, (2014)

  92. Smaragdis, P., Raj, B., Shashanka, M.: A probabilistic latent variable model for acoustic modeling. Adv. Models Acoust. Process., NIPS 148, 1–8 (2006)

    Google Scholar 

  93. Yao, F, Coquery, J, Cao, K-AL.: Independent principal component analysis for biologically meaningful dimension reduction of large biological data sets. BMC bioinformatics 13(1), 1–15 (2012)

  94. Gong, L., Nandi, AK.: Clustering by non-negative matrix factorization with independent principal component initialization. In 21st European Signal Processing Conference (EUSIPCO 2013), pages 1–5. IEEE, (2013)

Download references

Acknowledgements

The authors would like to thank the editors and anonymous reviewers for their constructive comments.

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, Shiraz University of Technology, Shiraz, Iran

    Sajad Fathi Hafshejani & Zahra Moaberfard

Authors
  1. Sajad Fathi Hafshejani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zahra Moaberfard
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sajad Fathi Hafshejani.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fathi Hafshejani, S., Moaberfard, Z. Initialization for non-negative matrix factorization: a comprehensive review. Int J Data Sci Anal 16, 119–134 (2023). https://doi.org/10.1007/s41060-022-00370-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41060-022-00370-9

Keywords

Search

Navigation