Theoretically Grounded Acceleration Techniques for Simulated Annealing

Part of the Intelligent Systems Reference Library book series (ISRL, volume 38)

Abstract

Simulated annealing (SA) is a generic optimization method whose popularity stems from its simplicity and its global convergence properties; it emulates the physical process of annealing whereby a solid is heated and then cooled down to eventually reach a minimum energy configuration. Although successfully applied to many difficult problems, SA is widely reported to converge very slowly, and it is common practice to relax some of its convergence conditions as well as to allow extra freedom in its design. However, variations on the theme of annealing usually come without optimal convergence guarantees.

In this paper, we review the fundamentals of SA and we focus on acceleration techniques that come with a rigorous mathematical justification. We discuss the design of the candidate-solution generation mechanism, the issue of finite-time cooling, and the technique of acceleration by concave distortion of the objective function. We also investigate a recently introduced generalization of SA—stochastic continuation—which significantly increases the design flexibility by allowing the candidate-solution generation mechanism and the objective function to vary with temperature.

Keywords

Simulated Annealing Energy Landscape Communication Mechanism Global Convergence Property Communication Matrix 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Discussion on the meeting on the Gibbs sampler and other Markov chain Monte Carlo methods. J. Roy. Statist. Soc. Ser. B 55(1), 53–102 (1993)Google Scholar
  2. 2.
    Azencott, R.: A common large deviations mathematical framework for sequential annealing and parallel annealing. In: Azencott, R. (ed.) Simulated Annealing: Parallelization Techniques, pp. 11–23. Wiley, New York (1992)Google Scholar
  3. 3.
    Azencott, R.: Sequential simulated annealing: speed of convergence and acceleration techniques. In: Azencott, R. (ed.) Simulated Annealing: Parallelization Techniques, pp. 1–10. Wiley, New York (1992)Google Scholar
  4. 4.
    Blake, A., Zisserman, A.: Visual reconstruction. The MIT Press (1987)Google Scholar
  5. 5.
    Catoni, O.: Large deviations and cooling schedules for the simulated annealing algorithm. C. R. Acad. Sci. Paris Sér. I Math. 307, 535–538 (1988) (in French)MathSciNetMATHGoogle Scholar
  6. 6.
    Catoni, O.: Rough large deviation estimates for simulated annealing: application to exponential schedules. Ann. Probab. 20(3), 1109–1146 (1992)MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    Catoni, O.: Metropolis, simulated annealing, and iterated energy transformation algorithms: theory and experiments. J. Complexity 12(4), 595–623 (1996)MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Catoni, O.: Simulated annealing algorithms and Markov chains with rare transitions. In: Séminaire de Probabilités XXXIII. Lecture Notes in Math., vol. 1709, pp. 69–119. Springer, New York (1999)CrossRefGoogle Scholar
  9. 9.
    Chiang, T.S., Chow, Y.: On the convergence rate of annealing processes. SIAM J. Control Optim. 26(6), 1455–1470 (1988)MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    Cohn, H., Fielding, M.: Simulated annealing: searching for an optimal temperature schedule. SIAM J. Optim. 9(3), 779–802 (1999)MathSciNetMATHCrossRefGoogle Scholar
  11. 11.
    Del Moral, P., Miclo, L.: On the convergence and applications of generalized simulated annealing. SIAM J. Control Optim. 37(4), 1222–1250 (1999)MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Desai, M.: Some results characterizing the finite time behaviour of the simulated annealing algorithm. Sādhanā 24(4-5), 317–337 (1999)MathSciNetMATHGoogle Scholar
  13. 13.
    Fielding, M.: Simulated annealing with an optimal fixed temperature. SIAM J. Optim. 11(2), 289–307 (2000)MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Frigerio, A., Grillo, G.: Simulated annealing with time-dependent energy function. Math. Z. 213, 97–116 (1993)MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Gidas, B.: Nonstationary Markov chains and convergence of the annealing algorithm. J. Statist. Phys. 39(1/2), 73–131 (1985)MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    Hajek, B.: Cooling schedules for optimal annealing. Math. Oper. Res. 13(2), 311–329 (1988)MathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Johnson, A., Jacobson, S.: On the convergence of generalized hill climbing algorithms. Discrete Appl. Math. 119(1-2), 37–57 (2002)MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    van Laarhoven, P.J.M., Aarts, E.H.L.: Simulated annealing: theory and practice. D. Reidel Publishing Company (1987)Google Scholar
  19. 19.
    Löwe, M.: Simulated annealing with time-dependent energy function via Sobolev inequalities. Stochastic Process. Appl. 63(2), 221–233 (1996)MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Nielsen, M.: Graduated nonconvexity by functional focusing. IEEE Trans. Pattern Anal. Machine Intell. 19(5), 521–525 (1997)CrossRefGoogle Scholar
  21. 21.
    Nikolova, M.: Markovian reconstruction using a GNC approach. IEEE Trans. Image Process. 8(9), 1204–1220 (1999)CrossRefGoogle Scholar
  22. 22.
    Orosz, J., Jacobson, S.: Analysis of static simulated annealing algorithms. J. Optim. Theory Appl. 115(1), 165–182 (2002)MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Orosz, J., Jacobson, S.: Finite-time performance analysis of static simulated annealing algorithms. Comput. Optim. Appl. 21(1), 21–53 (2002)MathSciNetMATHCrossRefGoogle Scholar
  24. 24.
    Robini, M.C., Lachal, A., Magnin, I.E.: A stochastic continuation approach to piecewise constant reconstruction. IEEE Trans. Image Process. 16(10), 2576–2589 (2007)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Robini, M.C., Magnin, I.E.: 3-D reconstruction from a few radiographs using the Metropolis dynamics with annealing. In: Proc. IEEE Int. Conf. Image Processing, Kobe, Japan, vol. 3, pp. 876–880 (1999)Google Scholar
  26. 26.
    Robini, M.C., Magnin, I.E.: Stochastic nonlinear image restoration using the wavelet transform. IEEE Trans. Image Process. 12(8), 890–905 (2003)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Robini, M.C., Magnin, I.E.: Optimization by stochastic continuation. SIAM J. Imaging Sci. 3(4), 1096–1121 (2010)MathSciNetMATHCrossRefGoogle Scholar
  28. 28.
    Robini, M.C., Rastello, T., Magnin, I.E.: Simulated annealing, acceleration techniques and image restoration. IEEE Trans. Image Process. 8(10), 1374–1387 (1999)MathSciNetMATHCrossRefGoogle Scholar
  29. 29.
    Robini, M.C., Reissman, P.J.: On simulated annealing with temperature-dependent energy and temperature-dependent communication. Statist. Probab. Lett. 81(8), 915–920 (2011)MathSciNetMATHCrossRefGoogle Scholar
  30. 30.
    Robini, M.C., Reissman, P.J.: From simulated annealing to stochastic continuation: a new trend in combinatorial optimization. J. Global Optim. (to appear, 2012)Google Scholar
  31. 31.
    Robini, M.C., Smekens, F., Sixou, B.: Optimal inverse treatment planning by stochastic continuation. In: Proc. 8th IEEE Int. Symp. Biomedical Imaging, Chicago, IL, pp. 1792–1796 (2011)Google Scholar
  32. 32.
    Schuur, P.: Classification of acceptance criteria for the simulated annealing algorithm. Math. Oper. Res. 22(2), 266–275 (1997)MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.CREATIS (CNRS UMR 5220; INSERM U1044), INSA-LyonVilleurbanne CedexFrance

Personalised recommendations