Journal of Automated Reasoning

, Volume 24, Issue 1–2, pp 67–100 | Cite as

Heavy-Tailed Phenomena in Satisfiability and Constraint Satisfaction Problems

  • Carla P. Gomes
  • Bart Selman
  • Nuno Crato
  • Henry Kautz

Abstract

We study the runtime distributions of backtrack procedures for propositional satisfiability and constraint satisfaction. Such procedures often exhibit a large variability in performance. Our study reveals some intriguing properties of such distributions: They are often characterized by very long tails or “heavy tails”. We will show that these distributions are best characterized by a general class of distributions that can have infinite moments (i.e., an infinite mean, variance, etc.). Such nonstandard distributions have recently been observed in areas as diverse as economics, statistical physics, and geophysics. They are closely related to fractal phenomena, whose study was introduced by Mandelbrot. We also show how random restarts can effectively eliminate heavy-tailed behavior. Furthermore, for harder problem instances, we observe long tails on the left-hand side of the distribution, which is indicative of a non-negligible fraction of relatively short, successful runs. A rapid restart strategy eliminates heavy-tailed behavior and takes advantage of short runs, significantly reducing expected solution time. We demonstrate speedups of up to two orders of magnitude on SAT and CSP encodings of hard problems in planning, scheduling, and circuit synthesis.

satisfiability constraint satisfaction heavy tails backtracking 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Adler, R., Feldman, R. and Taggu, M. (eds.): A Practical Guide to Heavy Tails, Birkhäuser, 1998Google Scholar
  2. 2.
    Alt, H., Guibas, L., Mehlhorn, K., Karp, R. and Wigderson, A.: A method for obtaining randomized algorithms with small tail probabilities, Algorithmica 16(4–5) (1996), 543–547.Google Scholar
  3. 3.
    Anderson, L.: Completing partial latin squares, Math. Fys. Meddelelser 41 (1985), 23–69.Google Scholar
  4. 4.
    Bayardo, R.: Personal communications, 1999.Google Scholar
  5. 5.
    Bayardo, R. and Miranker, D.: A complexity analysis of space-bounded learning algorithms for the constraint satisfaction problem, in Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), Portland, OR, 1996, pp. 558–562.Google Scholar
  6. 6.
    Bayardo, R. and Schrag, R.: Using CSP look-back techniques to solve real-world SAT instances, in Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97), New Providence, RI, 1997, pp. 203–208.Google Scholar
  7. 7.
    Brelaz, D.: New methods to color the vertices of a graph, Comm. ACM 22(4) (1979), 251–256.Google Scholar
  8. 8.
    Colbourn, C.: Embedding partial Steiner triple systems is NP-complete, J. Combin. Theory A 35 (1983), 100–105.Google Scholar
  9. 9.
    Cook, S. and Mitchell, D.: Finding hard instances of the satisfiability problem: A survey, in Du, Gu, and Pardalos (eds), Satisfiability Problem: Theory and Applications, Dimacs Series in Discrete Math. and Theoret. Comput. Sci. 35, 1998.Google Scholar
  10. 10.
    Crato, N., Gomes, C. and Selman, B.: On estimating the index of stability with truncated data, Manuscript in preparation.Google Scholar
  11. 11.
    Crawford, J. and Baker, A.: Experimental results on the application of satisfiability algorithms to scheduling problems, in Proceedings of The Tenth National Conference on Artificial Intelligence (AAAI-94), Austin, TX, 1994, pp. 1092–1098.Google Scholar
  12. 12.
    Davis, M., Logemann, G. and Loveland, D.: A machine program for theorem proving, Comm. ACM 5 (1979), 394–397.Google Scholar
  13. 13.
    Davis, M. and Putman, H.: A computing procedure for quantification theory, J. ACM 7 (1960), 201–215.Google Scholar
  14. 14.
    Denes, J. and Keedwell, A.: Latin Squares and Their Applications, Akademiai Kiado, Budapest, and English Universities Press, 1974.Google Scholar
  15. 15.
    Ertel, W. and Luby, M.: Optimal parallelization of Las Vegas algorithms, in Symp. on Theoretical Aspects of Computer Science, Lecture Notes in Comput. Sci. 775, 1994, pp. 463–475.Google Scholar
  16. 16.
    Feller,W.: An Introduction to Probability Theory and Its Applications, Vol. I, Wiley, New York, 1968.Google Scholar
  17. 17.
    Feller, W.: An Introduction to Probability Theory and Its Applications, Vol. II, Wiley, New York, 1971.Google Scholar
  18. 18.
    Frost, D., Rish, I. and Vila, L.: Summarizing CSP hardness with continuous probability distributions, in Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97), New Providence, RI, 1997, pp. 327–334.Google Scholar
  19. 19.
    Fujita, M., Slaney, J. and Bennett, F.: Automatic generation of some results in finite algebra, in Proceedings of the International Joint Conference on Artificial Intelligence, France, 1993, pp. 52–57.Google Scholar
  20. 20.
    Gent, I. and Walsh, T.: Easy problems are sometimes hard, Artif. Intell. 70 (1993), 335–345.Google Scholar
  21. 21.
    Gomes, C. P. and Selman, B.: Algorit portfolio design: Theory vs. practice, in Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence (UAI-97), Providence, RI, 1997, pp. 190–197.Google Scholar
  22. 22.
    Gomes, C. P. and Selman, B.: Problem structure in the presence of perturbations, in Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97), New Providence, RI, 1997, pp. 221–227.Google Scholar
  23. 23.
    Gomes, C. P., Selman, B. and Crato, N.: Heavy-tailed distributions in combinatorial search, in G. Smolka (ed.), Principles and Practice of Constraint Programming (CP97), Lecture Notes in Comput. Sci., Linz, Austria, 1997, pp. 121–135.Google Scholar
  24. 24.
    Hall, P.: On some simple estimates of an exponent of regular variation, J. Roy. Statist. Soc. 44 (1982), 37–42.Google Scholar
  25. 25.
    Hayes, B.: Computer science: Can't get no satisfaction, American Scientist 85(2) (1996), 108–112.Google Scholar
  26. 26.
    Hill, B.: A simple general approach to inference about the tail of a distribution, Ann. Statist. 3 (1975), 1163–1174.Google Scholar
  27. 27.
    Hogg, T., Huberman, B. and Williams, C. (eds.): Phase transitions and complexity (Special Issue), Artif. Intell. 81(1–2) (1996).Google Scholar
  28. 28.
    Hogg, T. and Williams, C.: Expected gains from parallelizing constraint solving for hard problems, in Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94), Seattle, WA, 1994, pp. 1310–1315.Google Scholar
  29. 29.
    Hoos, H.: Stochastic local search-methods, models, applications, Ph.D. Thesis, TU Darmstadt, 1998.Google Scholar
  30. 30.
    Huberman, B., Lukose, R. and Hogg, T.: An economics approach to hard computational problems, Science 275 (1993), 51–54.Google Scholar
  31. 31.
    Johnson, D.: A theoretician's guide to the experimental analysis of algorithms, Preliminary manuscript. Invited talk presented at AAAI-96, Portland, OR, 1996.Google Scholar
  32. 32.
    Johnson, D. and Trick,M.: Cliques, coloring, and satisfiability: Second dimacs implementation challenge, in Dimacs Series in Discrete Math. and Theoret. Comput. Sci. 36, 1996.Google Scholar
  33. 33.
    Kamath, A., Karmarkar, N., Ramakrishnan, K. and Resende, M.: Computational experience with an interior point algorithm on the satisfiability problem, in Proceedings of Integer Programming and Combinatorial Optimization, Waterloo, Canada, 1990, pp. 333–349.Google Scholar
  34. 34.
    Kautz, H. and Selman, B.: Pushing the envelope: Planning, propositional logic, and stochastic search, in Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), Portland, OR, 1996, pp. 1188–1194.Google Scholar
  35. 35.
    Kirkpatrick, S. and Selman, B.: Critical behavior in the satisfiability of random Boolean expressions, Science 264 (1994), 1297–1301.Google Scholar
  36. 36.
    Kwan, A.: Validity of normality assumption in CSP research, in Proceedings of the Pacific Rim International Conference on Artificial Intelligence, 1995, pp. 459–465.Google Scholar
  37. 37.
    Lam, C., Thiel, L. and Swiercz, S.: The non-existence of finite projective planes of order 10, Canad. J. Math. XLI(6) (1994), 1117–1123.Google Scholar
  38. 38.
    Li, C. M.: Personal communications, 1999.Google Scholar
  39. 39.
    Li, C. M. and Anbulagan: Heuristics based on unit propagation for satisfiability problems, in Proceedings of the International Joint Conference on Artificial Intelligence, 1997, pp. 366–371.Google Scholar
  40. 40.
    Luby, M., Sinclair, A. and Zuckerman, D.: Optimal speedup of Las Vegas algorithms, Inform. Process. Lett., 1993, pp. 173–180.Google Scholar
  41. 41.
    Macready, W., Siapas, A. and Kauffman, S.: Criticality and parallelism in combinatorial optimization, Science 271 (1996), 56–59.Google Scholar
  42. 42.
    Mandelbrot, B.: The Pareto-Lévy law and the distribution of income, Internat. Econom. Rev. 1 (1960), 79–106.Google Scholar
  43. 43.
    Mandelbrot, B.: The variation of certain speculative prices, J. Business 36 (1963), 394–419.Google Scholar
  44. 44.
    Mandelbrot, B.: The Fractal Geometry of Nature, Freeman, New York, 1983Google Scholar
  45. 45.
    McAloon, K., Tretkoff, C. and Wetzel, G.: Sports league scheduling, in Proceedings of Third Ilog International Users Meeting, 1997.Google Scholar
  46. 46.
    Mitchell, D. and Levesque, H.: Some pitfalls for experimenters with random SAT, Artif. Intell. 81(1–2) (1996), 111–125.Google Scholar
  47. 47.
    Monasson, R., Zecchina, R., Kirkpatrick, S., Selman, B. and Troyansky, L.: Determining computational complexity from characteristic 'phase transitions', Nature 400(8) (1999), 133–137.Google Scholar
  48. 48.
    Nemhauser, G. and Trick, M.: Scheduling a major college basketball conference, Oper. Res. 46(1) (1998), 1–8.Google Scholar
  49. 49.
    Niemela, I.: Logic programs with stable model semantics as a constraint programming paradigm. Ext. version of paper presented at the Workshop on Computational Aspects of Nonmonotonic Reasoning, 1998.Google Scholar
  50. 50.
    Niemela, I.: Personal communications, 1999.Google Scholar
  51. 51.
    Nolan, J.: Stable distributions, Manuscript, in preparation, 1999.Google Scholar
  52. 52.
    Prosser, P.: Hybrid algorithms for the constraint satisfaction problem, Comput. Intell. 9(3) (1993), 268–299.Google Scholar
  53. 53.
    Puget, J. F. and Leconte, M.: Beyond the black box: Constraints as objects, in Proceedings of ILPS'95, Lisbon, Portugal, 1995, pp. 513–527.Google Scholar
  54. 54.
    Regin, J. C.: A filtering algorithm for constraints of difference in CSP, in Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94), Seattle, WA, 1994, pp. 362–367.Google Scholar
  55. 55.
    Rish, I. and Frost, D.: Statistical analysis of backtracking on inconsistent CSPs, in Proceedings of Third International Conference on Principles and Practices of Constraint Programming, 1997, pp. 150–162.Google Scholar
  56. 56.
    Samorodnitsky, G. and Taqqu, M.: Stable Non-Gaussian Random Processes: Stochastic Models with Infinite Variance, Chapman and Hall, 1994.Google Scholar
  57. 57.
    Selman, B., Kautz, H. and Cohen, B.: Local search strategies for satisfiability testing, in D. Johnson and M. Trick (eds), Dimacs Series in Discrete Math. and Theoret. Comput. Sci. 26, 1993, pp. 521–532.Google Scholar
  58. 58.
    Selman, B. and Kirkpatrick, S.: Finite-size scaling of the computational cost of systematic search, Artif. Intell. 81(1–2) (1996), 273–295.Google Scholar
  59. 59.
    Shaw, P., Stergiou, K. and Walsh, T.: Arc consistency and quasigroup completion, in Proceedings of ECAI-98, Workshop on Binary Constraints, 1998.Google Scholar
  60. 60.
    Shonkwiler, R., E. and van Vleck, E.: Parallel speedup of Monte Carlo methods for global optimization, J. Complexity 10 (1994), 64–95.Google Scholar
  61. 61.
    Slaney, J., Fujita, M. and Stickel, M.: Automated reasoning and exhaustive search: Quasigroup existence problems, Comput. Math. Appl. 29 (1995), 115–132.Google Scholar
  62. 62.
    Smith, B. and Grant, S.: Sparse constraint graphs and exceptionally hard problems, in Proceedings of the International Joint Conference on Artificial Intelligence, 1995, pp. 646–651.Google Scholar
  63. 63.
    Stickel, M.: Personal communications, 1998.Google Scholar
  64. 64.
    Vandengriend and Culberson: The Gnm phase transition is not hard for the Hamiltonian cycle problem, J. Artif. Intell. Res. 9 (1999), 219–245.Google Scholar
  65. 65.
    Walsh, T.: Search in a small world, in Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 1999Google Scholar
  66. 66.
    Zolotarev, V.: One-Dimensional Stable Distributions, Transl. Math. Monographs 65, Amer. Math. Soc., 1986. Translation from the original 1983 Russian ed.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Carla P. Gomes
  • Bart Selman
  • Nuno Crato
  • Henry Kautz

There are no affiliations available

Personalised recommendations