# Heavy-Tailed Phenomena in Satisfiability and Constraint Satisfaction Problems

## Abstract

We study the runtime distributions of backtrack procedures for propositional satisfiability and constraint satisfaction. Such procedures often exhibit a large variability in performance. Our study reveals some intriguing properties of such distributions: They are often characterized by very long tails or “heavy tails”. We will show that these distributions are best characterized by a general class of distributions that can have infinite moments (i.e., an infinite mean, variance, etc.). Such nonstandard distributions have recently been observed in areas as diverse as economics, statistical physics, and geophysics. They are closely related to fractal phenomena, whose study was introduced by Mandelbrot. We also show how random restarts can effectively eliminate heavy-tailed behavior. Furthermore, for harder problem instances, we observe long tails on the left-hand side of the distribution, which is indicative of a non-negligible fraction of relatively short, successful runs. A rapid restart strategy eliminates heavy-tailed behavior and takes advantage of short runs, significantly reducing expected solution time. We demonstrate speedups of up to two orders of magnitude on SAT and CSP encodings of hard problems in planning, scheduling, and circuit synthesis.

## Preview

Unable to display preview. Download preview PDF.

### References

- 1.Adler, R., Feldman, R. and Taggu, M. (eds.):
*A Practical Guide to Heavy Tails*, Birkhäuser, 1998Google Scholar - 2.Alt, H., Guibas, L., Mehlhorn, K., Karp, R. and Wigderson, A.: A method for obtaining randomized algorithms with small tail probabilities,
*Algorithmica***16**(4–5) (1996), 543–547.Google Scholar - 3.Anderson, L.: Completing partial latin squares,
*Math. Fys. Meddelelser***41**(1985), 23–69.Google Scholar - 4.Bayardo, R.: Personal communications, 1999.Google Scholar
- 5.Bayardo, R. and Miranker, D.: A complexity analysis of space-bounded learning algorithms for the constraint satisfaction problem, in
*Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96)*, Portland, OR, 1996, pp. 558–562.Google Scholar - 6.Bayardo, R. and Schrag, R.: Using CSP look-back techniques to solve real-world SAT instances, in
*Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97)*, New Providence, RI, 1997, pp. 203–208.Google Scholar - 7.Brelaz, D.: New methods to color the vertices of a graph,
*Comm. ACM***22**(4) (1979), 251–256.Google Scholar - 8.Colbourn, C.: Embedding partial Steiner triple systems is NP-complete,
*J. Combin. Theory A***35**(1983), 100–105.Google Scholar - 9.Cook, S. and Mitchell, D.: Finding hard instances of the satisfiability problem: A survey, in Du, Gu, and Pardalos (eds),
*Satisfiability Problem: Theory and Applications*, Dimacs Series in Discrete Math. and Theoret. Comput. Sci. 35, 1998.Google Scholar - 10.Crato, N., Gomes, C. and Selman, B.: On estimating the index of stability with truncated data, Manuscript in preparation.Google Scholar
- 11.Crawford, J. and Baker, A.: Experimental results on the application of satisfiability algorithms to scheduling problems, in
*Proceedings of The Tenth National Conference on Artificial Intelligence (AAAI-94)*, Austin, TX, 1994, pp. 1092–1098.Google Scholar - 12.Davis, M., Logemann, G. and Loveland, D.: A machine program for theorem proving,
*Comm. ACM***5**(1979), 394–397.Google Scholar - 13.Davis, M. and Putman, H.: A computing procedure for quantification theory,
*J. ACM***7**(1960), 201–215.Google Scholar - 14.Denes, J. and Keedwell, A.:
*Latin Squares and Their Applications*, Akademiai Kiado, Budapest, and English Universities Press, 1974.Google Scholar - 15.Ertel, W. and Luby, M.: Optimal parallelization of Las Vegas algorithms, in
*Symp. on Theoretical Aspects of Computer Science*, Lecture Notes in Comput. Sci. 775, 1994, pp. 463–475.Google Scholar - 16.Feller,W.:
*An Introduction to Probability Theory and Its Applications*, Vol. I, Wiley, New York, 1968.Google Scholar - 17.Feller, W.:
*An Introduction to Probability Theory and Its Applications*, Vol. II, Wiley, New York, 1971.Google Scholar - 18.Frost, D., Rish, I. and Vila, L.: Summarizing CSP hardness with continuous probability distributions, in
*Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97)*, New Providence, RI, 1997, pp. 327–334.Google Scholar - 19.Fujita, M., Slaney, J. and Bennett, F.: Automatic generation of some results in finite algebra, in
*Proceedings of the International Joint Conference on Artificial Intelligence*, France, 1993, pp. 52–57.Google Scholar - 20.Gent, I. and Walsh, T.: Easy problems are sometimes hard,
*Artif. Intell.***70**(1993), 335–345.Google Scholar - 21.Gomes, C. P. and Selman, B.: Algorit portfolio design: Theory vs. practice, in
*Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence (UAI-97)*, Providence, RI, 1997, pp. 190–197.Google Scholar - 22.Gomes, C. P. and Selman, B.: Problem structure in the presence of perturbations, in
*Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97)*, New Providence, RI, 1997, pp. 221–227.Google Scholar - 23.Gomes, C. P., Selman, B. and Crato, N.: Heavy-tailed distributions in combinatorial search, in G. Smolka (ed.),
*Principles and Practice of Constraint Programming (CP97)*, Lecture Notes in Comput. Sci., Linz, Austria, 1997, pp. 121–135.Google Scholar - 24.Hall, P.: On some simple estimates of an exponent of regular variation,
*J. Roy. Statist. Soc.***44**(1982), 37–42.Google Scholar - 25.Hayes, B.: Computer science: Can't get no satisfaction,
*American Scientist***85**(2) (1996), 108–112.Google Scholar - 26.Hill, B.: A simple general approach to inference about the tail of a distribution,
*Ann. Statist.***3**(1975), 1163–1174.Google Scholar - 27.Hogg, T., Huberman, B. and Williams, C. (eds.): Phase transitions and complexity (Special Issue),
*Artif. Intell.***81**(1–2) (1996).Google Scholar - 28.Hogg, T. and Williams, C.: Expected gains from parallelizing constraint solving for hard problems, in
*Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94)*, Seattle, WA, 1994, pp. 1310–1315.Google Scholar - 29.Hoos, H.: Stochastic local search-methods, models, applications, Ph.D. Thesis, TU Darmstadt, 1998.Google Scholar
- 30.Huberman, B., Lukose, R. and Hogg, T.: An economics approach to hard computational problems,
*Science***275**(1993), 51–54.Google Scholar - 31.Johnson, D.: A theoretician's guide to the experimental analysis of algorithms, Preliminary manuscript. Invited talk presented at AAAI-96, Portland, OR, 1996.Google Scholar
- 32.Johnson, D. and Trick,M.: Cliques, coloring, and satisfiability: Second dimacs implementation challenge, in Dimacs Series in Discrete Math. and Theoret. Comput. Sci. 36, 1996.Google Scholar
- 33.Kamath, A., Karmarkar, N., Ramakrishnan, K. and Resende, M.: Computational experience with an interior point algorithm on the satisfiability problem, in
*Proceedings of Integer Programming and Combinatorial Optimization*, Waterloo, Canada, 1990, pp. 333–349.Google Scholar - 34.Kautz, H. and Selman, B.: Pushing the envelope: Planning, propositional logic, and stochastic search, in
*Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96)*, Portland, OR, 1996, pp. 1188–1194.Google Scholar - 35.Kirkpatrick, S. and Selman, B.: Critical behavior in the satisfiability of random Boolean expressions,
*Science***264**(1994), 1297–1301.Google Scholar - 36.Kwan, A.: Validity of normality assumption in CSP research, in
*Proceedings of the Pacific Rim International Conference on Artificial Intelligence*, 1995, pp. 459–465.Google Scholar - 37.Lam, C., Thiel, L. and Swiercz, S.: The non-existence of finite projective planes of order 10,
*Canad. J. Math.***XLI**(6) (1994), 1117–1123.Google Scholar - 38.Li, C. M.: Personal communications, 1999.Google Scholar
- 39.Li, C. M. and Anbulagan: Heuristics based on unit propagation for satisfiability problems, in
*Proceedings of the International Joint Conference on Artificial Intelligence*, 1997, pp. 366–371.Google Scholar - 40.Luby, M., Sinclair, A. and Zuckerman, D.: Optimal speedup of Las Vegas algorithms,
*Inform. Process. Lett.*, 1993, pp. 173–180.Google Scholar - 41.Macready, W., Siapas, A. and Kauffman, S.: Criticality and parallelism in combinatorial optimization,
*Science***271**(1996), 56–59.Google Scholar - 42.Mandelbrot, B.: The Pareto-Lévy law and the distribution of income,
*Internat. Econom. Rev.***1**(1960), 79–106.Google Scholar - 43.Mandelbrot, B.: The variation of certain speculative prices,
*J. Business***36**(1963), 394–419.Google Scholar - 44.Mandelbrot, B.:
*The Fractal Geometry of Nature*, Freeman, New York, 1983Google Scholar - 45.McAloon, K., Tretkoff, C. and Wetzel, G.: Sports league scheduling, in
*Proceedings of Third Ilog International Users Meeting*, 1997.Google Scholar - 46.Mitchell, D. and Levesque, H.: Some pitfalls for experimenters with random SAT,
*Artif. Intell.***81**(1–2) (1996), 111–125.Google Scholar - 47.Monasson, R., Zecchina, R., Kirkpatrick, S., Selman, B. and Troyansky, L.: Determining computational complexity from characteristic 'phase transitions',
*Nature***400**(8) (1999), 133–137.Google Scholar - 48.Nemhauser, G. and Trick, M.: Scheduling a major college basketball conference,
*Oper. Res.***46**(1) (1998), 1–8.Google Scholar - 49.Niemela, I.: Logic programs with stable model semantics as a constraint programming paradigm. Ext. version of paper presented at the
*Workshop on Computational Aspects of Nonmonotonic Reasoning*, 1998.Google Scholar - 50.Niemela, I.: Personal communications, 1999.Google Scholar
- 51.Nolan, J.: Stable distributions, Manuscript, in preparation, 1999.Google Scholar
- 52.Prosser, P.: Hybrid algorithms for the constraint satisfaction problem,
*Comput. Intell.***9**(3) (1993), 268–299.Google Scholar - 53.Puget, J. F. and Leconte, M.: Beyond the black box: Constraints as objects, in
*Proceedings of ILPS'95*, Lisbon, Portugal, 1995, pp. 513–527.Google Scholar - 54.Regin, J. C.: A filtering algorithm for constraints of difference in CSP, in
*Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94)*, Seattle, WA, 1994, pp. 362–367.Google Scholar - 55.Rish, I. and Frost, D.: Statistical analysis of backtracking on inconsistent CSPs, in
*Proceedings of Third International Conference on Principles and Practices of Constraint Programming*, 1997, pp. 150–162.Google Scholar - 56.Samorodnitsky, G. and Taqqu, M.:
*Stable Non-Gaussian Random Processes: Stochastic Models with Infinite Variance*, Chapman and Hall, 1994.Google Scholar - 57.Selman, B., Kautz, H. and Cohen, B.: Local search strategies for satisfiability testing, in D. Johnson and M. Trick (eds), Dimacs Series in Discrete Math. and Theoret. Comput. Sci. 26, 1993, pp. 521–532.Google Scholar
- 58.Selman, B. and Kirkpatrick, S.: Finite-size scaling of the computational cost of systematic search,
*Artif. Intell.***81**(1–2) (1996), 273–295.Google Scholar - 59.Shaw, P., Stergiou, K. and Walsh, T.: Arc consistency and quasigroup completion, in
*Proceedings of ECAI-98, Workshop on Binary Constraints*, 1998.Google Scholar - 60.Shonkwiler, R., E. and van Vleck, E.: Parallel speedup of Monte Carlo methods for global optimization,
*J. Complexity***10**(1994), 64–95.Google Scholar - 61.Slaney, J., Fujita, M. and Stickel, M.: Automated reasoning and exhaustive search: Quasigroup existence problems,
*Comput. Math. Appl.***29**(1995), 115–132.Google Scholar - 62.Smith, B. and Grant, S.: Sparse constraint graphs and exceptionally hard problems, in
*Proceedings of the International Joint Conference on Artificial Intelligence*, 1995, pp. 646–651.Google Scholar - 63.Stickel, M.: Personal communications, 1998.Google Scholar
- 64.Vandengriend and Culberson: The
*Gnm*phase transition is not hard for the Hamiltonian cycle problem,*J. Artif. Intell. Res.***9**(1999), 219–245.Google Scholar - 65.Walsh, T.: Search in a small world, in
*Proceedings of the International Joint Conference on Artificial Intelligence*, Stockholm, Sweden, 1999Google Scholar - 66.Zolotarev, V.:
*One-Dimensional Stable Distributions*, Transl. Math. Monographs 65, Amer. Math. Soc., 1986. Translation from the original 1983 Russian ed.Google Scholar