Abstract
Common information was introduced by Wyner (IEEE Trans Inf Theory 21(2):163–179, 1975) as a measure of dependence of two random variables. This measure has been recently resurrected as a lower bound on the logarithm of the nonnegative rank of a nonnegative matrix in Braun and Pokutta (Proceedings of FOCS, 2013) and Jain et al. (Proceedings of SODA, 2013). Lower bounds on nonnegative rank have important applications to several areas such as communication complexity and combinatorial optimization. We begin a systematic study of common information extending the dual characterization of Witsenhausen (SIAM J Appl Math 31(2):313–333, 1976). Our main results are: (i) Common information is additive under tensoring of matrices. (ii) It characterizes the (logarithm of the) amortized nonnegative rank of a matrix, i.e., the minimal nonnegative rank under tensoring and small \({\ell_1}\) perturbations. We also provide quantitative bounds generalizing previous asymptotic results by Wyner (IEEE Trans Inf Theory 21(2):163–179, 1975). (iii) We deliver explicit witnesses from the dual problem for several matrices leading to explicit lower bounds on common information, which are robust under \({\ell_1}\) perturbations. This includes improved lower bounds for perturbations of the all important unique disjointness partial matrix, as well as new insights into its information-theoretic structure.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Z. Bar-Yossef, T.S. Jayram, R. Kumar & D. Sivakumar (2004). An information statistics approach to data stream and communication complexity. Journal of Computer and System Sciences 68(4), 702–732. http://www.sciencedirect.com/science/article/pii/S0022000003001855.
B. Barak, M. Braverman, X. Chen & A. Rao (2010). How to compress interactive communication. In 42nd ACM Symposium on Theory of Computing, 67–76.
G. Braun & S. Pokutta (2013). Common information and unique disjointness. In Proceedings of the 54th IEEE Symposium on Foundations of Computer Science, 688–697.
G. Braun, S. Fiorini, S. Pokutta & D. Steurer (2012). Approximation limits of linear programs (beyond hierarchies). In 53rd IEEE Symposium on Foundations of Computer Science, 480–489. ISBN 978-1-4673-4383-1.
G. Braun, S. Pokutta & D. Zink (2015). Inapproximability of combinatorial problems via small LPs and SDPs. In Proceedings of the 47th Annual Symposium on the Theory of Computing, 107–116.
M. Braverman & A. Moitra (2012). An information complexity approach to extended formulations. IEEE Trans. Inf. Theory 60(10), 6058–6069.
M. Braverman & A. Rao (2011). Information equals amortized communication. In Foundations of Computer Science (FOCS), 2011 IEEE 52nd Annual Symposium on, 748–757. IEEE.
A. Chakrabarti, Y. Shi, A. Wirth & A. Yao (2001). Informational complexity and the direct sum problem for simultaneous message complexity. In Proceedings of the 42nd IEEE Symposium on Foundations of Computer Science, 270–278.
S.O. Chan, J. Lee, P. Raghavendra & D. Steurer (2013). Approximate constraint satisfaction requires large LP relaxations. In Proceedings of the 54th IEEE Symposium on Foundations of Computer Science, 350–359.
T. Cover & J. Thomas (2006). Elements of information theory. Wiley-interscience.
H. Fawzi & P. Parrilo (2015). Lower bounds on nonnegative rank via nonnegative nuclear norms. Mathematical Programming Series B 153(1), 41–66.
S. Fiorini, S. Massar, S. Pokutta, H.R. Tiwary & R. de Wolf (2015). Exponential lower bounds for polytopes in combinatorial optimization. Journal of the ACM 62(2).
R. Jain, Y. Shi, Z. Wei & S. Zhang (2013). Efficient protocols for generating bipartite classical distributions and quantum states. Proceedings of SODA 2013.
V. Kaibel & S. Weltge (2015). A short proof that the extension complexity of the correlation polytope grows exponentially. Discrete & Computational Geometry 2, 397–401.
Karchmer M., Kushilevitz E., Nisan N. (1995) Fractional covers and communication complexity. SIAM J. Discrete Math. 8: 76–92
L. Lovász (1975). On the ratio of optimal integral and fractional covers. Discrete Mathematics 13(4), 383–390. ISSN 0012-365X.
L. Lovász (1990). Communication complexity: A survey. In B. Korte, L. Lovász, H. Prömel & A. Schrijver, editors, Paths, flows, and VLSI-layout, 235–265. Springer.
A. Moitra (2016). An almost optimal algorithm for computing non-negative rank. SIAM J. Comput. 45(1), 156–173.
Razborov A. A. (1992) On the Distributional Complexity of Disjointness. Theoret. Comput. Sci. 106(2): 385–390
T. Rothvoß (2014). The matching polytope has exponential extension complexity. In Proceedings of the 46th Annual Symposium on the Theory of Computing, 263–272.
Vavasis S.A. (2009) On the complexity of nonnegative matrix factorization. SIAM J. Optim. 20(3): 1364–1377
H.S. Witsenhausen (1976) Values and bounds for the common information of two discrete random variables. SIAM J. App. Math. 31(2): 313–333
de Wolf R. (2003) Nondeterministic quantum query and communication complexities. SIAM J. Comput. 32(3): 681–699
A. Wyner (1975) The common information of two dependent random variables. IEEE Trans. Inf. Theory 21(2): 163–179
M. Yannakakis (1988). Expressing combinatorial optimization problems by linear programs (extended abstract). In Proceedings of the 20th Annual Symposium on the Theory of Computing, 223–228.
Yannakakis M. (1991) Expressing combinatorial optimization problems by linear programs. J. Comput. System Sci. 43(3): 441–466
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Braun, G., Jain, R., Lee, T. et al. Information-theoretic approximations of the nonnegative rank. comput. complex. 26, 147–197 (2017). https://doi.org/10.1007/s00037-016-0125-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00037-016-0125-z