, Volume 76, Issue 3, pp 597–629 | Cite as

Common Information and Unique Disjointness



We provide an information-theoretic framework for establishing strong lower bounds on the nonnegative rank of matrices by means of common information, a notion previously introduced in Wyner (IEEE Trans Inf Theory 21(2):163–179, 1975). The framework is a generalization of the one in Braverman and Moitra (Proceedings of the forty-fifth annual ACM symposium on theory of computing, pp 161–170, 2013) for the shifted uniqe disjointness (UDISJ) matrix to arbitrary nonnegative matrices. Common information is a natural lower bound for the nonnegative rank of a matrix and by combining it with Hellinger distance estimations we compute the (almost) exact common information of UDISJ partial matrix. The bounds are obtained very naturally. We also establish robustness of this estimation under random or adversarial removal of rows and columns of the UDISJ partial matrix. This robustness translates, via a variant of Yannakakis’s factorization theorem, to lower bounds on the average case and adversarial approximate extension complexity of removals. We present the first family of polytopes, the hard pair introduced in Braun et al. (Math Oper Res 40(3):756–772, 2015) related to the CLIQUE problem, with high average case and adversarial approximate extension complexity of removals. The framework relies on a strengthened version of the link between information theory and Hellinger distance from Bar-Yossef (J Comput Syst Sci 68(4):702–732, 2004). We also provide an information theoretic variant of the fooling set method that allows us to extend fooling set lower bounds from extension complexity to approximate extension complexity.


Unique disjointness Common information Information theory Extended formulations Correlation polytope Extension complexity Nonnegative rank 


  1. 1.
    Arora, S., Ge, R., Kannan, R., Moitra, A.: Computing a nonnegative matrix factorization-provably. In: Proceedings of the 44th Symposium on Theory of Computing, pp. 145–162. ACM (2012)Google Scholar
  2. 2.
    Avis, D., Tiwary, H.R.: On the extension complexity of combinatorial polytopes. Math. Program. Ser. B 153(1), 95–115 (2015). doi:10.1007/s10107-014-0764-2
  3. 3.
    Bar-Yossef, Z., Jayram, T., Kumar, R., Sivakumar, D.: An information statistics approach to data stream and communication complexity. J. Comput. Syst. Sci. 68(4), 702–732 (2004). doi:10.1016/j.jcss.2003.11.006
  4. 4.
    Ben-Tal, A., Nemirovski, A.: On polyhedral approximations of the second-order cone. Math. Oper. Res. 26, 193–205 (2001). doi:10.1287/moor. MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Braun, G., Pokutta, S.: The matching polytope does not admit fully-polynomial size relaxation schemes. In: Proceedings of the Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 837–846 (2015)Google Scholar
  6. 6.
    Braun, G., Fiorini, S., Pokutta, S., Steurer, D.: Approximation limits of linear programs (beyond hierarchies). Math. Oper. Res. 40(3), 756–772 (2015). doi:10.1287/moor.2014.0694
  7. 7.
    Braun, G., Fiorini, S., Pokutta, S.: Average case polyhedral complexity of the maximum stable set problem. Math. Program. (2013). doi:10.1007/s10107-016-0989-3
  8. 8.
    Braun, G., Jain, R., Lee, T., Pokutta, S.: Information-theoretic approximations of the nonnegative rank. Electron. Colloq. Comput. Complex. (ECCC) 13(158) (2013).
  9. 9.
    Braverman, M., Moitra, A.: An information complexity approach to extended formulations. In: Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing, pp. 161–170 (2013). doi:10.1145/2488608.2488629
  10. 10.
    Braverman, M., Garg, A., Pankratov, D., Weinstein, O.: Information lower bounds via self-reducibility. In: 8th International Computer Science Symposium in Russia, CSR 2013, pp. 183–194 (2013). doi:10.1007/978-3-642-38536-0_16
  11. 11.
    Braverman, M., Garg, A., Pankratov, D., Weinstein, O.: From information to exact communication. In: Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing, pp. 151–160 (2013). doi:10.1145/2488608.2488628
  12. 12.
    Chan, S., Lee, J., Raghavendra, P., Steurer, D.: Approximate constraint satisfaction requires large LP relaxations. In: IEEE 54th Annual Symposium on Foundations of Computer Science (2013). doi:10.1109/FOCS.2013.45
  13. 13.
    Conforti, M., Cornuéjols, G., Zambelli, G.: Extended formulations in combinatorial optimization. 4OR 8, 1–48 (2010). doi:10.1007/s10288-010-0122-z MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Cover, T., Thomas, J.: Elements of Information Theory. Wiley-interscience, Hoboken, NJ (2006)Google Scholar
  15. 15.
    Faenza, Y., Fiorini, S., Grappe, R., Tiwary, H.R.: Extended formulations, non-negative factorizations and randomized communication protocols. In: Proceedings of the Second International Conference on Combinatorial Optimization (ISCO 2012), pp. 129–140 (2012)Google Scholar
  16. 16.
    Fiorini, S., Massar, S., Pokutta, S., Tiwary, H.R., de Wolf, R.: Linear vs. semidefinite extended formulations: exponential separation and strong lower bounds. In: Proceedings of STOC 2012 (2012)Google Scholar
  17. 17.
    Fiorini, S., Rothvoß, T., Tiwary, H.: Extended formulations for polygons. J. Discrete Comput. Geom. 48(3), 658–668 (2012)Google Scholar
  18. 18.
    Fiorini, S., Kaibel, V., Pashkovich, K., Theis, D.O.: Combinatorial bounds on nonnegative rank and extended formulations. Discrete Math. 313, 67–83 (2013)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Gillis, N.: Sparse and unique nonnegative matrix factorization through data preprocessing. J. Mach. Learn. Res. 13, 3349–3386 (2012)MathSciNetMATHGoogle Scholar
  20. 20.
    Gillis, N., Glineur, F., Theis, D.O.: Personal communication. 2 (2013)Google Scholar
  21. 21.
    Håstad, J.: Clique is hard to approximate within \(1-\varepsilon \). Acta Math. 182(1), 105–142 (1999)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Jain, R., Shi, Y., Wei, Z., Zhang, S.: Efficient protocols for generating bipartite classical distributions and quantum states. In: Proceedings of SODA 2013 (2013)Google Scholar
  23. 23.
    Kaibel, V.: Extended formulations in combinatorial optimization. Optima 85, 2–7 (2011)Google Scholar
  24. 24.
    Lin, J.: Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory 37(I), 145–151 (1991). doi:10.1109/18.61115 MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Moitra, A.: An almost optimal algorithm for computing nonnegative rank. In: Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2013), pp. 1454–1464 (2013)Google Scholar
  26. 26.
    Pashkovich, K.: Extended formulations for combinatorial polytopes. PhD thesis, Magdeburg Universität (2012)Google Scholar
  27. 27.
    Pokutta, S., Van Vyve, M.: A note on the extension complexity of the knapsack polytope. Oper. Res. Lett. 41(4), 347–350 (2013)Google Scholar
  28. 28.
    Razborov, A.A.: On the distributional complexity of disjointness. Theor. Comput. Sci. 106(2), 385–390 (1992)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Razborov, A.A.: Personal communication (2012)Google Scholar
  30. 30.
    Rothvoß, T.: Some 0/1 polytopes need exponential size extended formulations, (2011). arXiv:1105.0036
  31. 31.
    Rothvoß, T.: The matching polytope has exponential extension complexity. In: Proceedings of STOC, pp. 263–272 (2014). doi:10.1145/2591796.2591834
  32. 32.
    Shitov, Y.: An upper bound for nonnegative rank. J. Comb. Theory 122, 126–132 (2014)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Witsenhausen, H.S.: Values and bounds for the common information of two discrete random variables. SIAM J. Appl. Math. 31(2), 313–333 (1976)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    de Wolf, R.: Nondeterministic quantum query and communication complexities. SIAM J. Comput. 32(3), 681–699 (2003)Google Scholar
  35. 35.
    Wyner, A.: The common information of two dependent random variables. IEEE Trans. Inf. Theory 21(2), 163–179 (1975)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Yannakakis, M.: Expressing combinatorial optimization problems by linear programs (extended abstract). In: Proceedings of STOC 1988, pp. 223–228 (1988)Google Scholar
  37. 37.
    Yannakakis, M.: Expressing combinatorial optimization problems by linear programs. J. Comput. Syst. Sci. 43(3), 441–466 (1991). doi:10.1016/0022-0000(91)90024-Y MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Zhang, S.: Quantum strategic game theory. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 39–59. ACM (2012)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.ISyEGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations