Conditional Random Fields, Planted Constraint Satisfaction and Entropy Concentration

  • Emmanuel Abbe
  • Andrea Montanari
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8096)

Abstract

This paper studies a class of probabilistic models on graphs, where edge variables depend on incident node variables through a fixed probability kernel. The class includes planted constraint satisfaction problems (CSPs), as well as more general structures motivated by coding and community clustering problems. It is shown that under mild assumptions on the kernel and for sparse random graphs, the conditional entropy of the node variables given the edge variables concentrates around a deterministic threshold. This implies in particular the concentration of the number of solutions in a broad class of planted CSPs, the existence of a threshold function for the disassortative stochastic block model, and the proof of a conjecture on parity check codes. It also establishes new connections among coding, clustering and satisfiability.

Keywords

Planted models constraint satisfaction problems graphical models community clustering parity-check codes entropy concentration interpolation method 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abbe, E., Montanari, A.: On the concentration of the number of solutions of random satisfiability formulas. Random Structures & Algorithms (2013) ISSN: 1098-2418, http://dx.doi.org/10.1002/rsa.20501, doi:10.1002/rsa.20501
  2. 2.
    Abbe, E., Montanari, A.: Conditional Random Fields, Planted Constraint Satisfaction, and Entropy Concentration. arXiv:1305.4274 [math.PR] (2013)Google Scholar
  3. 3.
    Achlioptas, D., Coja-Oghlan, A.: Algorithmic barriers from phase transitions. In: Proceedings of the 2008 49th Annual IEEE Symposium on Foundations of Computer Science, FOCS 2008, pp. 793–802. IEEE Computer Society, Washington, DC (2008)CrossRefGoogle Scholar
  4. 4.
    Krivelevich, M., Coja-Oghlan, A., Vilenchik, D.: Why almost all satisfiable k-cnf formulas are easy. In: Proceedings of the 13th International Conference on Analysis of Algorithms, pp. 89–102 (2007)Google Scholar
  5. 5.
    Achlioptas, D., Jia, H., Moore, C.: Hiding satisfying assignments: two are better than one. In: Proceedings of AAAI 2004, pp. 131–136 (2004)Google Scholar
  6. 6.
    Achlioptas, D., Kautz, H., Gomes, C.: Generating satisfiable problem instancesGoogle Scholar
  7. 7.
    Achlioptas, D., Han Kim, J., Krivelevich, M., Tetali, P.: Two-coloring random hypergraphs. Random Structures and Algorithms 20(2), 249–259 (2002)MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Altarelli, F., Monasson, R., Zamponi, F.: Can rare SAT formulas be easily recognized? On the efficiency of message passing algorithms for K-SAT at large clause-to-variable ratios. Computing Research Repository abs/cs/060 (2006)Google Scholar
  9. 9.
    Achlioptas, D., Naor, A., Peres, Y.: Rigorous Location of Phase Transitions in Hard Optimization Problems. Nature 435, 759–764 (2005)CrossRefGoogle Scholar
  10. 10.
    Bayati, M., Gamarnik, D., Tetali, P.: Combinatorial approach to the interpolation method and scaling limits in sparse random graphs. In: 42nd Annual ACM Symposium on Theory of Computing, Cambridge, MA, pp. 105–114 (June 2010)Google Scholar
  11. 11.
    Barthel, W., Hartmann, A.K., Leone, M., Ricci-Tersenghi, F., Weigt, M., Zecchina, R.: Hiding solutions in random satisfiability problems: A statistical mechanics approach. Phys. Rev. Lett. 88, 188701 (2002)CrossRefGoogle Scholar
  12. 12.
    Berlekamp, E., McEliece, R.J., Van Tilborg, H.C.A.: On the inherent intractability of certain coding problems (corresp.). IEEE Transactions on Information Theory 24(3), 384–386 (1978)MATHCrossRefGoogle Scholar
  13. 13.
    Coja-oghlan, A.: Graph partitioning via adaptive spectral techniques. Comb. Probab. Comput. 19(2), 227–284 (2010)MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Dyer, M.E., Frieze, A.M.: The solution of some random np-hard problems in polynomial expected time. Journal of Algorithms 10(4), 451–489 (1989)MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Dietzfelbinger, M., Goerdt, A., Mitzenmacher, M., Montanari, A., Pagh, R., Rink, M.: Tight thresholds for cuckoo hashing via XORSAT. In: Abramsky, S., Gavoille, C., Kirchner, C., Meyer auf der Heide, F., Spirakis, P.G. (eds.) ICALP 2010, Part I. LNCS, vol. 6198, pp. 213–225. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  16. 16.
    Decelle, A., Krzakala, F., Moore, C., Zdeborová, L.: Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Phys. Rev. E 84, 066106 (2011)Google Scholar
  17. 17.
    Dubois, O., Mandler, J.: The 3-XORSAT threshold. In: Proceedings of the 43rd Symposium on Foundations of Computer Science, FOCS 2002, pp. 769–778. IEEE Computer Society, Washington, DC (2002)Google Scholar
  18. 18.
    Daudé, H., Ravelomanana, V.: Random 2-XORSAT at the satisfiability threshold. In: Laber, E.S., Bornstein, C., Nogueira, L.T., Faria, L. (eds.) LATIN 2008. LNCS, vol. 4957, pp. 12–23. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  19. 19.
    Elias, P.: Coding for noisy channels. IRE Convention Record 4, 37–46 (1955)Google Scholar
  20. 20.
    Franz, S., Leone, M.: Replica bounds for optimization problems and diluted spin systems. J. Stat. Phys. 111, 535 (2003)MathSciNetMATHCrossRefGoogle Scholar
  21. 21.
    Franz, S., Leone, M., Toninelli, F.L.: Replica bounds for diluted non-Poissonian spin systems. J. Phys. A 36, 10967 (2003)MathSciNetMATHCrossRefGoogle Scholar
  22. 22.
    Feige, U., Mossel, E., Vilenchik, D.: Complete convergence of message passing algorithms for some satisfiability problems. In: Díaz, J., Jansen, K., Rolim, J.D.P., Zwick, U. (eds.) APPROX and RANDOM 2006. LNCS, vol. 4110, pp. 339–350. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  23. 23.
    Fortunato, S.: Community detection in graphs. Physics Reports 486(3-5), 75–174 (2010)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Friedgut, E.: Sharp thresholds of graph properties, and the k-sat problem. J. Amer. Math. Soc. 12, 1017–1054 (1999); appendix by Bourgain, J.Google Scholar
  25. 25.
    Gallager, R.G.: Low-density parity-check codes. MIT Press, Cambridge (1963)Google Scholar
  26. 26.
    Guerra, F., Toninelli, F.L.: The thermodynamic limit in mean field spin glasses. Commun. Math. Phys. 230, 71–79 (2002)MathSciNetMATHCrossRefGoogle Scholar
  27. 27.
    Goldenberg, A., Zheng, A.X., Fienberg, S.E., Airoldi, E.M.: A survey of statistical network models. Foundations and Trends® in Machine Learning 2(2), 129–233 (2010)CrossRefGoogle Scholar
  28. 28.
    Haanpää, H., Järvisalo, M., Kaski, P., Niemelä, I.: Hard satisfiable clause sets for benchmarking equivalence reasoning techniques (2005)Google Scholar
  29. 29.
    Jia, H., Moore, C., Strain, D.: Generating hard satisfiable formulas by hiding solutions deceptively. In: AAAI, pp. 384–389. AAAI Press (2005)Google Scholar
  30. 30.
    Kudekar, S., Macris, N.: Sharp bounds for optimal decoding of Low-Density Parity-Check codes. IEEE Trans. on Inform. Theory 55, 4635–4650 (2009)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Karrer, B., Newman, M.E.J.: Stochastic blockmodels and community structure in networks. Phys. Rev. E 83, 016107 (2011)Google Scholar
  32. 32.
    Krzakala, F., Zdeborová, L.: Hiding quiet solutions in random constraint satisfaction problems. Phys. Rev. Lett. 102, 238701 (2009)CrossRefGoogle Scholar
  33. 33.
    Lafferty, J.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data, pp. 282–289. Morgan Kaufmann (2001)Google Scholar
  34. 34.
    Luby, M., Mitzenmacher, M., Shokrollahi, A., Spielman, D.A., Stemann, V.: Practical loss-resilient codes. In: 29th Annual ACM Symposium on Theory of Computing, pp. 150–159 (1997)Google Scholar
  35. 35.
    Luby, M., Mitzenmacher, M., Shokrollahi, A., Spielman, D.A.: Efficient erasure correcting codes. IEEE Trans. on Inform. Theory 47(2), 569–584 (2001)MathSciNetMATHCrossRefGoogle Scholar
  36. 36.
    Mossel, E., Neeman, J., Sly, A.: Stochastic Block Models and Reconstruction. arXiv:1202.1499 [math.PR]Google Scholar
  37. 37.
    Montanari, A.: Tight bounds for LDPC and LDGM codes under MAP decoding. IEEE Trans. on Inform. Theory 51, 3221–3246 (2005)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Montanari, A.: Estimating random variables from random sparse observations. European Transactions on Telecommunications 19(4), 385–403 (2008)CrossRefGoogle Scholar
  39. 39.
    Montanari, A., Restrepo, R., Tetali, P.: Reconstruction and Clustering in Random Constraint Satisfaction Problems. CoRR abs/0904.2751 (2009)Google Scholar
  40. 40.
    Newman, M.E.J.: Communities, modules and large-scale structure in networks. Nature Physics 8(1), 25–31 (2011)CrossRefGoogle Scholar
  41. 41.
    Pittel, B., Sorkin, G.B.: The Satisfiability Threshold for k-XORSAT. arXiv:1212.1905 (2012)Google Scholar
  42. 42.
    Panchenko, D., Talagrand, M.: Bounds for diluted mean-field spin glass models. Prob. Theor. Rel. Fields 130, 319–336 (2004)MathSciNetMATHCrossRefGoogle Scholar
  43. 43.
    Raj Kumar, K., Pakzad, P., Salavati, A.H., Shokrollahi, A.: Phase transitions for mutual information. In: 2010 6th International Symposium on Turbo Codes and Iterative Information Processing (ISTC), pp. 137–141 (2010)Google Scholar
  44. 44.
    Richardson, T., Urbanke, R.: Modern Coding Theory. Cambridge University Press, Cambridge (2008)MATHCrossRefGoogle Scholar
  45. 45.
    Zdeborová, L., Krzakala, F.: Quiet planting in the locked constraint satisfaction problems. SIAM Journal on Discrete Mathematics 25(2), 750–770 (2011)MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Emmanuel Abbe
    • 1
  • Andrea Montanari
    • 2
  1. 1.Department of Electrical Engineering and Program in Applied and Computational MathematicsPrinceton UniversityUSA
  2. 2.Departments of Electrical Engineering and StatisticsStanford UniversityUSA

Personalised recommendations