Advertisement

Journal of Global Optimization

, Volume 65, Issue 2, pp 369–400 | Cite as

Heuristics for exact nonnegative matrix factorization

  • Arnaud Vandaele
  • Nicolas Gillis
  • François Glineur
  • Daniel Tuyttens
Article

Abstract

The exact nonnegative matrix factorization (exact NMF) problem is the following: given an m-by-n nonnegative matrix X and a factorization rank r, find, if possible, an m-by-r nonnegative matrix W and an r-by-n nonnegative matrix H such that \(X = WH\). In this paper, we propose two heuristics for exact NMF, one inspired from simulated annealing and the other from the greedy randomized adaptive search procedure. We show empirically that these two heuristics are able to compute exact nonnegative factorizations for several classes of nonnegative matrices (namely, linear Euclidean distance matrices, slack matrices, unique-disjointness matrices, and randomly generated matrices) and as such demonstrate their superiority over standard multi-start strategies. We also consider a hybridization between these two heuristics that allows us to combine the advantages of both methods. Finally, we discuss the use of these heuristics to gain insight on the behavior of the nonnegative rank, i.e., the minimum factorization rank such that an exact NMF exists. In particular, we disprove a conjecture on the nonnegative rank of a Kronecker product, propose a new upper bound on the extension complexity of generic n-gons and conjecture the exact value of (i) the extension complexity of regular n-gons and (ii) the nonnegative rank of a submatrix of the slack matrix of the correlation polytope.

Keywords

Nonnegative matrix factorization Exact nonnegative matrix factorization Heuristics Simulated annealing GRASP Hybridization Nonnegative rank Linear Euclidean distance matrices Slack matrices Extension complexity 

Notes

Acknowledgments

The authors would like to thank the reviewers and the editor for their insightful comments which helped improve the paper.

References

  1. 1.
    Arora, S., Ge, R., Kannan, R., Moitra, A.: Computing a nonnegative matrix factorization—provably. in Proceedings of the 44th Symposium on Theory of Computing, STOC ’12, pp. 145–162, (2012)Google Scholar
  2. 2.
    Beasley, L., Laffey, T.: Real rank versus nonnegative rank. Linear Algebra Appl. 431(12), 2330–2335 (2009)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Beasley, L., Lee, T., Klauck, H., Theis, D.: Dagstuhl report 13082: communication complexity, linear optimization, and lower bounds for the nonnegative rank of matrices (2013). arXiv:1305.4147
  4. 4.
    Ben-Tal, A., Nemirovski, A.: On polyhedral approximations of the second-order cone. Math. Oper. Res. 26(2), 193–205 (2001)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Bocci, C., Carlini, E., Rapallo, F.: Perturbation of matrices and nonnegative rank with a view toward statistical models. SIAM J. Matrix Anal. Appl. 32(4), 1500–1512 (2011)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Boutsidis, C., Gallopoulos, E.: SVD based initialization: a head start for nonnegative matrix factorization. Pattern Recognit. 41(4), 1350–1362 (2008)CrossRefMATHGoogle Scholar
  7. 7.
    Brown, C.W.: Qepcad b: a program for computing with semi-algebraic sets using cads. ACM SIGSAM Bull. 37(4), 97–108 (2003)CrossRefMATHGoogle Scholar
  8. 8.
    Carlini, E., Rapallo, F.: Probability matrices, non-negative rank, and parameterization of mixture models. Linear Algebra Appl. 433, 424–432 (2010)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Cichocki, A., Amari, S.-I., Zdunek, R., Phan, A.: Non-negative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. Wiley, London (2009)CrossRefGoogle Scholar
  10. 10.
    Cichocki, A., Phan, A.H.: Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Trans. Fundam. Electron. E92–A(3), 708–721 (2009)CrossRefGoogle Scholar
  11. 11.
    Cichocki, A., Zdunek, R., Amari, S.-i.: Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization. Lecture notes in computer science (Springer, 2007), pp. 169–176Google Scholar
  12. 12.
    Cohen, J., Rothblum, U.: Nonnegative ranks, decompositions and factorization of nonnegative matrices. Linear Algebra Appl. 190, 149–168 (1993)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Conforti, M., Cornuéjols, G., Zambelli, G.: Extended formulations in combinatorial optimization. 4OR A Q.J. Oper. Res. 10(1), 1–48 (2010)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    de Caen, D., Gregory, D.A., Pullman, N.J.: The boolean rank of zero-one matrices. in Proceedings of Third Caribbean Conference on Combinatorics and Computing (Barbados), pp. 169–173 (1981)Google Scholar
  15. 15.
    Fawzi, H., Gouveia, J., Parrilo, P., Robinson, R., Thomas, R.: Positive Semidefinite Rank (2014). arXiv:1407.4095
  16. 16.
    Fiorini, S., Kaibel, V., Pashkovich, K., Theis, D.: Combinatorial bounds on nonnegative rank and extended formulations. Discret. Math. 313(1), 67–83 (2013)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Fiorini, S., Massar, S., Pokutta, S., Tiwary, H., de Wolf, R.: Linear Versus Semidefinite Extended Formulations: Exponential Separation and Strong Lower Bounds. in Proceedings of the Forty-Fourth Annual ACM Symposium on Theory of Computing, ACM, pp. 95–106, (2012)Google Scholar
  18. 18.
    Fiorini, S., Rothvoss, T., Tiwary, H.: Extended formulations for polygons. Discret. Comput. Geom. 48(3), 658–668 (2012)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Gillis, N.: Sparse and unique nonnegative matrix factorization through data preprocessing. J. Mach. Learn. Res. 13(Nov), 3349–3386 (2012)MathSciNetMATHGoogle Scholar
  20. 20.
    Gillis, N.: The why and how of nonnegative matrix factorization. In: Suykens, J., Signoretto, M., Argyriou, A. (eds.) Regularization, Optimization, Kernels, and Support Vector Machines. Machine Learning and Pattern Recognition Series. Chapman & Hall/CRC, London (2014)Google Scholar
  21. 21.
    Gillis, N., Glineur, F.: Using underapproximations for sparse nonnegative matrix factorization. Pattern Recognit. 43(4), 1676–1687 (2010)CrossRefMATHGoogle Scholar
  22. 22.
    Gillis, N., Glineur, F.: Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization. Neural Comput. 24(4), 1085–1105 (2012)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Gillis, N., Glineur, F.: On the geometric interpretation of the nonnegative rank. Linear Algebra Appl. 437(11), 2685–2712 (2012)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Gillis, N., Vavasis, S.: Semidefinite programming based preconditioning for more robust near-separable nonnegative matrix factorization. SIAM J.Optim. 25, 677–698 (2015)MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Goemans, M.: Smallest Compact Formulation for the Permutahedron (2009). http://math.mit.edu/~goemans/PAPERS/permutahedron
  26. 26.
    Gouveia, J.: Personnal Comunication (2014)Google Scholar
  27. 27.
    Gouveia, J., Fawzi, H., Robinson, R.: Rational and Real Positive Srank can be Different (2014). arXiv:1404.4864
  28. 28.
    Gouveia, J., Parrilo, P., Thomas, R.: Lifts of convex sets and cone factorizations. Math. Oper. Res. 38(2), 248–264 (2013)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Gouveia, J., Robinson, R., Thomas, R.: Worst-case Results for Positive Semidefinite Rank (2013). arXiv:1305.4600
  30. 30.
    Gregory, D.A., Pullman, N.J.: Semiring rank: boolean rank and nonnegative rank factorizations. J. Combin. Inform. Syst. Sci. 8(3), 223–233 (1983)MathSciNetMATHGoogle Scholar
  31. 31.
    Hrubeš, P.: On the nonnegative rank of distance matrices. Inf. Process. Lett. 112(11), 457–461 (2012)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Janecek, A., Tan, Y.: Iterative improvement of the multiplicative update NMF algorithm using nature-inspired optimization. in Seventh International Conference on Natural Computation vol. 3 (2011), pp. 1668–1672Google Scholar
  33. 33.
    Janecek, A., Tan, Y.: Swarm intelligence for non-negative matrix factorization. Int. J. Swarm Intell. Res. 2(4), 12–34 (2011)CrossRefGoogle Scholar
  34. 34.
    Janecek, A., Tan, Y.: Using population based algorithms for initializing nonnegative matrix factorization. Adv. Swarm Intell. 6729, 307–316 (2011)Google Scholar
  35. 35.
    Kaibel, V.: Extended formulations in combinatorial optimization. Optima 85, 2–7 (2011)Google Scholar
  36. 36.
    Kaibel, V., Weltge, S.: A Short Proof that the Extension Complexity of the Correlation Polytope Grows Exponentially (2013). arXiv:1307.3543
  37. 37.
    Kim, J., He, Y., Park, H.: Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework. J. Global Optim. 58(2), 285–319 (2014)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Kim, J., Park, H.: Fast nonnegative matrix factorization: an active-set-like method and comparisons. SIAM J. Sci. Comput. 33(6), 3261–3281 (2011)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Lee, D., Seung, H.: Learning the parts of objects by nonnegative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  40. 40.
    Lee, D., Seung, H.: Algorithms for non-negative matrix factorization. In: Advances in Neural Information Processing Systems, vol. 13, pp. 556–562 (2001)Google Scholar
  41. 41.
    Lee, T., Shraibman, A.: Lower Bounds in Communication Complexity. Found. Trends Theor. Comput. Sci. 3(4), 263–399 (2007)Google Scholar
  42. 42.
    Moitra, A.: An Almost Optimal Algorithm for Computing Nonnegative Rank. in Proceedings of the 24th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA ’13), pp. 1454–1464 (2013)Google Scholar
  43. 43.
    Oelze, M., Vandaele, A., Weltge, S.: Computing the Extension Complexities of all 4-Dimensional 0/1-polytopes (2014). arXiv:1406.4895
  44. 44.
    Padrol, A., Pfeifle, J.: Polygons as Slices of Higher-Dimensional Polytopes (2014). arXiv:1404.2443
  45. 45.
    Pirlot, M.: General local search methods. Eur. J. Oper. Res. 92(3), 493–511 (1996)CrossRefMATHGoogle Scholar
  46. 46.
    Rothvoss, T.: The Matching Polytope has Exponential Extension Complexity (2013). arXiv:1311.2369
  47. 47.
    Shitov, Y.: Sublinear Extensions of Polygons (2014). arXiv:1412.0728
  48. 48.
    Shitov, Y.: An upper bound for nonnegative rank. J. Combin. Theory Ser. A 122, 126–132 (2014)MathSciNetCrossRefMATHGoogle Scholar
  49. 49.
    Shitov, Y.: Nonnegative Rank Depends on the Field (2015). arXiv:1505.01893
  50. 50.
    Takahashi, N., Hibi, R.: Global convergence of modified multiplicative updates for nonnegative matrix factorization. Comput. Optim. Appl. 57(2), 417–440 (2014)MathSciNetCrossRefMATHGoogle Scholar
  51. 51.
    Thomas, L.: Rank factorization of nonnegative matrices. SIAM Rev. 16(3), 393–394 (1974)CrossRefGoogle Scholar
  52. 52.
    Vandaele, A., Gillis, N., Glineur, F.: On the Linear Extension Complexity of Regular n-gons (2015). arXiv:1505.08031
  53. 53.
    Vavasis, S.: On the complexity of nonnegative matrix factorization. SIAM J. Optim. 20(3), 1364–1377 (2010)MathSciNetCrossRefMATHGoogle Scholar
  54. 54.
    Watson, T.: Sampling Versus Unambiguous Nondeterminism in Communication Complexity (2014). http://www.cs.toronto.edu/~thomasw/papers/nnr
  55. 55.
    Yannakakis, M.: Expressing combinatorial optimization problems by linear programs. J. Comput. Syst. Sci. 43(3), 441–466 (1991)MathSciNetCrossRefMATHGoogle Scholar
  56. 56.
    Zdunek, R.: Initialization of nonnegative matrix factorization with vertices of convex polytope. In: Artificial Intelligence and Soft Computing, vol. 7267, pp. 448–455. Lecture Notes in Computer Science (2012)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Mathematics and Operational Research, Faculté PolytechniqueUniversité de MonsMonsBelgium
  2. 2.Center for Operations Research and EconometricsUniversité catholique de LouvainLouvain-La-NeuveBelgium
  3. 3.ICTEAM InstituteUniversité catholique de LouvainLouvain-La-NeuveBelgium

Personalised recommendations