, Volume 76, Issue 3, pp 597–629 | Cite as

Common Information and Unique Disjointness

  • Gábor Braun
  • Sebastian Pokutta


We provide an information-theoretic framework for establishing strong lower bounds on the nonnegative rank of matrices by means of common information, a notion previously introduced in Wyner (IEEE Trans Inf Theory 21(2):163–179, 1975). The framework is a generalization of the one in Braverman and Moitra (Proceedings of the forty-fifth annual ACM symposium on theory of computing, pp 161–170, 2013) for the shifted uniqe disjointness (UDISJ) matrix to arbitrary nonnegative matrices. Common information is a natural lower bound for the nonnegative rank of a matrix and by combining it with Hellinger distance estimations we compute the (almost) exact common information of UDISJ partial matrix. The bounds are obtained very naturally. We also establish robustness of this estimation under random or adversarial removal of rows and columns of the UDISJ partial matrix. This robustness translates, via a variant of Yannakakis’s factorization theorem, to lower bounds on the average case and adversarial approximate extension complexity of removals. We present the first family of polytopes, the hard pair introduced in Braun et al. (Math Oper Res 40(3):756–772, 2015) related to the CLIQUE problem, with high average case and adversarial approximate extension complexity of removals. The framework relies on a strengthened version of the link between information theory and Hellinger distance from Bar-Yossef (J Comput Syst Sci 68(4):702–732, 2004). We also provide an information theoretic variant of the fooling set method that allows us to extend fooling set lower bounds from extension complexity to approximate extension complexity.


Unique disjointness Common information Information theory Extended formulations Correlation polytope Extension complexity Nonnegative rank 



We are indebted to Samuel Fiorini for providing extensive feedback through the various stages of this work and who helped to significantly improve the presentation. We would also like to thank Daniel Dadush, Kostya Pashkovich, Santosh Vempala, and Ronald de Wolf for extremely valuable feedback as well as Jérémie Roland for pointing us to [22]. Part of this work was conducted at the Dagstuhl Seminar 13082 on Communication Complexity, Linear Optimization, and lower bounds for the nonnegative rank of matrices ( The authors would like to thank the organizers for providing such a stimulating environment. Research reported in this paper was partially supported by NSF Grant CMMI-1300144.


  1. 1.
    Arora, S., Ge, R., Kannan, R., Moitra, A.: Computing a nonnegative matrix factorization-provably. In: Proceedings of the 44th Symposium on Theory of Computing, pp. 145–162. ACM (2012)Google Scholar
  2. 2.
    Avis, D., Tiwary, H.R.: On the extension complexity of combinatorial polytopes. Math. Program. Ser. B 153(1), 95–115 (2015). doi: 10.1007/s10107-014-0764-2
  3. 3.
    Bar-Yossef, Z., Jayram, T., Kumar, R., Sivakumar, D.: An information statistics approach to data stream and communication complexity. J. Comput. Syst. Sci. 68(4), 702–732 (2004). doi: 10.1016/j.jcss.2003.11.006
  4. 4.
    Ben-Tal, A., Nemirovski, A.: On polyhedral approximations of the second-order cone. Math. Oper. Res. 26, 193–205 (2001). doi: 10.1287/moor. MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Braun, G., Pokutta, S.: The matching polytope does not admit fully-polynomial size relaxation schemes. In: Proceedings of the Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 837–846 (2015)Google Scholar
  6. 6.
    Braun, G., Fiorini, S., Pokutta, S., Steurer, D.: Approximation limits of linear programs (beyond hierarchies). Math. Oper. Res. 40(3), 756–772 (2015). doi: 10.1287/moor.2014.0694
  7. 7.
    Braun, G., Fiorini, S., Pokutta, S.: Average case polyhedral complexity of the maximum stable set problem. Math. Program. (2013). doi: 10.1007/s10107-016-0989-3
  8. 8.
    Braun, G., Jain, R., Lee, T., Pokutta, S.: Information-theoretic approximations of the nonnegative rank. Electron. Colloq. Comput. Complex. (ECCC) 13(158) (2013).
  9. 9.
    Braverman, M., Moitra, A.: An information complexity approach to extended formulations. In: Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing, pp. 161–170 (2013). doi: 10.1145/2488608.2488629
  10. 10.
    Braverman, M., Garg, A., Pankratov, D., Weinstein, O.: Information lower bounds via self-reducibility. In: 8th International Computer Science Symposium in Russia, CSR 2013, pp. 183–194 (2013). doi: 10.1007/978-3-642-38536-0_16
  11. 11.
    Braverman, M., Garg, A., Pankratov, D., Weinstein, O.: From information to exact communication. In: Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing, pp. 151–160 (2013). doi: 10.1145/2488608.2488628
  12. 12.
    Chan, S., Lee, J., Raghavendra, P., Steurer, D.: Approximate constraint satisfaction requires large LP relaxations. In: IEEE 54th Annual Symposium on Foundations of Computer Science (2013). doi: 10.1109/FOCS.2013.45
  13. 13.
    Conforti, M., Cornuéjols, G., Zambelli, G.: Extended formulations in combinatorial optimization. 4OR 8, 1–48 (2010). doi: 10.1007/s10288-010-0122-z MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Cover, T., Thomas, J.: Elements of Information Theory. Wiley-interscience, Hoboken, NJ (2006)Google Scholar
  15. 15.
    Faenza, Y., Fiorini, S., Grappe, R., Tiwary, H.R.: Extended formulations, non-negative factorizations and randomized communication protocols. In: Proceedings of the Second International Conference on Combinatorial Optimization (ISCO 2012), pp. 129–140 (2012)Google Scholar
  16. 16.
    Fiorini, S., Massar, S., Pokutta, S., Tiwary, H.R., de Wolf, R.: Linear vs. semidefinite extended formulations: exponential separation and strong lower bounds. In: Proceedings of STOC 2012 (2012)Google Scholar
  17. 17.
    Fiorini, S., Rothvoß, T., Tiwary, H.: Extended formulations for polygons. J. Discrete Comput. Geom. 48(3), 658–668 (2012)Google Scholar
  18. 18.
    Fiorini, S., Kaibel, V., Pashkovich, K., Theis, D.O.: Combinatorial bounds on nonnegative rank and extended formulations. Discrete Math. 313, 67–83 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Gillis, N.: Sparse and unique nonnegative matrix factorization through data preprocessing. J. Mach. Learn. Res. 13, 3349–3386 (2012)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Gillis, N., Glineur, F., Theis, D.O.: Personal communication. 2 (2013)Google Scholar
  21. 21.
    Håstad, J.: Clique is hard to approximate within \(1-\varepsilon \). Acta Math. 182(1), 105–142 (1999)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Jain, R., Shi, Y., Wei, Z., Zhang, S.: Efficient protocols for generating bipartite classical distributions and quantum states. In: Proceedings of SODA 2013 (2013)Google Scholar
  23. 23.
    Kaibel, V.: Extended formulations in combinatorial optimization. Optima 85, 2–7 (2011)Google Scholar
  24. 24.
    Lin, J.: Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory 37(I), 145–151 (1991). doi: 10.1109/18.61115 MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Moitra, A.: An almost optimal algorithm for computing nonnegative rank. In: Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2013), pp. 1454–1464 (2013)Google Scholar
  26. 26.
    Pashkovich, K.: Extended formulations for combinatorial polytopes. PhD thesis, Magdeburg Universität (2012)Google Scholar
  27. 27.
    Pokutta, S., Van Vyve, M.: A note on the extension complexity of the knapsack polytope. Oper. Res. Lett. 41(4), 347–350 (2013)Google Scholar
  28. 28.
    Razborov, A.A.: On the distributional complexity of disjointness. Theor. Comput. Sci. 106(2), 385–390 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Razborov, A.A.: Personal communication (2012)Google Scholar
  30. 30.
    Rothvoß, T.: Some 0/1 polytopes need exponential size extended formulations, (2011). arXiv:1105.0036
  31. 31.
    Rothvoß, T.: The matching polytope has exponential extension complexity. In: Proceedings of STOC, pp. 263–272 (2014). doi: 10.1145/2591796.2591834
  32. 32.
    Shitov, Y.: An upper bound for nonnegative rank. J. Comb. Theory 122, 126–132 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Witsenhausen, H.S.: Values and bounds for the common information of two discrete random variables. SIAM J. Appl. Math. 31(2), 313–333 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    de Wolf, R.: Nondeterministic quantum query and communication complexities. SIAM J. Comput. 32(3), 681–699 (2003)Google Scholar
  35. 35.
    Wyner, A.: The common information of two dependent random variables. IEEE Trans. Inf. Theory 21(2), 163–179 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Yannakakis, M.: Expressing combinatorial optimization problems by linear programs (extended abstract). In: Proceedings of STOC 1988, pp. 223–228 (1988)Google Scholar
  37. 37.
    Yannakakis, M.: Expressing combinatorial optimization problems by linear programs. J. Comput. Syst. Sci. 43(3), 441–466 (1991). doi: 10.1016/0022-0000(91)90024-Y MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Zhang, S.: Quantum strategic game theory. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 39–59. ACM (2012)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.ISyEGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations