Skip to main content
SpringerLink
Log in

Simulated annealing applied to test generation: landscape characterization and stopping criteria

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

This paper investigates a measurement approach to support the implementation of Simulated Annealing (SA) applied to test generation. SA, like other metaheuristics, is a generic technique that must be tuned to the testing problem under consideration. Finding an adequate setting of SA parameters, that will offer good performance for the target problem, is known to be difficult. Our measurement approach is intended to guide the implementation choices to be made. It builds upon advanced research on how to characterize search problems and the dynamics of metaheuristic techniques applied to them. Central to this research is the concept of landscape. Existing measures of landscape have mainly been applied to combinatorial problems considered in complexity theory. We show that some of these measures can be useful for testing problems as well. The diameter and autocorrelation are retained to study the adequacy of alternative settings of SA parameters. A new measure, the Generation Rate of Better Solutions (GRBS), is introduced to monitor convergence of the search process and implement stopping criteria. The measurement approach is experimented on various case studies, and allows us to successfully revisit a problem issued from our previous work on testing control systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abdellatif-Kaddour O (2003) Property-oriented testing of control systems: stepwise construction of test scenarios generated by simulated annealing search. Doctoral Dissertation, Polytechnic National Institut of Toulouse, France, LAAS Report n° 03573, (In French)

  • Abdellatif-Kaddour O, Thevenod-Fosse P, Waeselynck H (2003a) Property-oriented testing: a strategy for exploring dangerous scenarios. Proc. ACM Symposium on Applied Computing (SAC'2003), Melbourne, USA, pp 1128–1134

  • Abdellatif-Kaddour O, Thevenod-Fosse P, Waeselynck H (2003b) An empirical investigation of simulated annealing applied to property-oriented testing. Proc. ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'03), Tunis, Tunisia

  • Abrial J-R, Börger E, Langmaar H (eds) (1996) Formal methods for industrial applications: specifying and programming the steam boiler control. Springer, Berlin Heidelberg New York

    Google Scholar 

  • Adamopoulos K, Harman M, Hierons RM (2004) How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. Proc. Genetic and Evolutionary Computation Conference (GECCO 2004), LNCS 3103, Springer, Berlin Heidelberg New York, pp 1338–1349

  • Angel E, Zissimopoulos V (1998) Autocorrelation coefficient for the graph bipartitioning problem. Theor Comp Sci 191:229–243

    Article  MathSciNet  MATH  Google Scholar 

  • Angel E, Zissimopoulos V (2000) On the classification of NP-complete problems in terms of their correlation coefficient. Discrete App Math 99(1–3):261–277

    Article  MathSciNet  MATH  Google Scholar 

  • Belaidouni M (2001) Metaheuristics and search landscapes. Doctoral Dissertation, University of Angers, France (In French)

  • Belaidouni M, Hao JK (2000) Landscapes of the maximal constraint satisfaction problem. Proc. 4th European Conference on Artificial Evolution (EA'99), LNCS 1829, Springer, Berlin Heidelberg New York, pp 244–255

  • Belaidouni M, Hao JK (2002) SAT, Local search dynamics and density of states. Proc. 5th European Conference on Artificial Evolution, LNCS 2310, Springer, Berlin Heidelberg New York, pp 192–204

  • Clarke J, Dolado JJ, Harman M, Hierons R, Jones B, Lumkin M, Mitchell B, Mancoridis S, Rees K, Roper M, Shepperd M (2003) Reformulating software engineering as a search problem. IEE Proc Softw 150(3):161–175

    Article  Google Scholar 

  • Connolly DT (1990) An improved annealing scheme for the QAP. Eur J Oper Res 46(1):93–100

    Article  MATH  MathSciNet  Google Scholar 

  • Eremeev AV, Reeves CR (2003) On confidence intervals for the number of local optima. Proc. EvoWorkshops 2003, LNCS 2611, Springer, Berlin Heidelberg New York, pp 224–235

  • Frank J, Cheeseman P, Stutz J (1997) When gravity fails: local search topology. J Artif Intell Res 7:249–281

    MathSciNet  MATH  Google Scholar 

  • Gross HG, Jones B, Eyres DE (2000) Structural performance measure of evolutionary testing applied to worst-case timing of real-time systems. IEE Proc Softw 147(2):161–175

    Article  Google Scholar 

  • Harman M, Jones BF (2001) Search-based software engineering. Inf Softw Technol 43(14):833–839

    Article  Google Scholar 

  • Holland J (1975) Adaptation in natural and artificial systems. University of Michigan

  • Hordijk W (1996) A measure of landscapes. Evol Comput 4(4):335–360

    Google Scholar 

  • Jones T, Forrest S (1995) Fitness distance correlation as a measure of problem difficulty for genetic algorithms. Proc Int Conf on Genetic Algorithms (ICGA'03), pp 184–192

  • Jones BF, Sthamer H-H, Eyres DE (1996) Automatic structural testing using genetic algorithms. Softw Eng J 11(5):299–306

    Article  Google Scholar 

  • Kirkpatrick S, Gellat CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680

    Article  MathSciNet  Google Scholar 

  • Lammermann F, Wegener J (2005) Test-goal-specific termination criteria for evolutionary white-box testing by means of software measures. Proc. 6th Metaheuristics International Conference (MIC'2005)

  • Lundy M, Mees AI (1986) Convergence of an annealing algorithm. Math Program 34(1):111–124

    Article  MathSciNet  MATH  Google Scholar 

  • Maniezzo V, Dorigo M, Colorni A (1995) Algodesk: an experimental comparison of eight evolutionary heuristics applied to the quadratic assignment problem. Eur J Oper Res 81(1):188–204

    Article  MATH  Google Scholar 

  • McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156

    Article  Google Scholar 

  • Merz P, Freisleben B (2000) Fitness landscape analysis and memetic algorithms for the quadratic assignment problem. IEEE Trans Evol Comput 4(4):337–352

    Article  Google Scholar 

  • Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27(12):1085–1110

    Article  Google Scholar 

  • Nugent CE, Vollman TE, Ruml J (1968) An experimental comparison of techniques for the assignment of facilities to locations. Oper Res 16:150–173

    Article  Google Scholar 

  • O'Sullivan M, Vössner S, Wegener J (1998) Testing temporal correctness of real-time systems—a new approach using genetic algorithms and cluster analysis. Proc. 6th European Conference on Software Testing, Analysis & Review (EuroSTAR 1998)

  • Pargas RP, Harrold M-J, Peck RR (1999) Test data generation using genetic algorithms. Softw Test Verif Reliab 9(4):263–282

    Article  Google Scholar 

  • Rayward-Smith VJ, Osman IH, Reeves CR, Smith GD (1996) Modern heuristic search methods. Wiley

  • Rosé H, Ebeling W, Asselmeyer T (1996) The density of states—a measure of the difficulty of optimisation problems. Proc. Parallel Problem Solving from Nature (PPSN IV), LNCS 1141, Springer, Berlin Heidelberg New York, pp 208–217

  • Schultz AC, Grefenstette JJ, De Jong KA (1995) Learning to break things: adaptative testing of intelligent controllers. Handbook on evolutionary computation, chapter G3.5. IOP and Oxford University Press

  • Stadler PF, Schnabl W (1992) The landscape of the traveling salesman problem. Phys Lett, A 161(4):337–344

    Article  MathSciNet  MATH  Google Scholar 

  • Taillard ED (1991) Robust tabu search for the quadratic assignment problem. Parallel Comput 17(4&5):443–455

    Article  MathSciNet  Google Scholar 

  • Tracey N (2000) A search-based automated test data generation framework for safety-critical software. PhD Dissertation, University of York, UK

  • Tracey N, Clark J, Mander K, McDermid J (1998a) An automated framework for structural test-data generation. Proc. 13th IEEE Conference on Automated Software Engineering (ASE), Hawaii, USA, pp 285–288

  • Tracey N, Clark J, Mander K (1998b) Automated program flaw finding using simulated annealing. Proc. ACM Int. Symp. on Software Testing and Analysis (ISSTA’98), Clearwater Beach, Florida, USA, pp 73–81

  • Tracey N, Clark J, Mander K, McDermid J (2000) Automated test-data generation for exception conditions. Softw Pract Exp 30(1):61–79

    Article  Google Scholar 

  • Vanneschi L, Tomassini M, Collard P, Clergue M (2003) Fitness distance correlation in structural mutation genetic programming. Proc. Europ. Conf. on Genetic Programming (EuroGP'03), LNCS 2610, Springer, Berlin Heidelberg New York, pp 455–464

  • Wegener J, Buehler O (2004) Evaluation of different fitness functions for the evolutionary testing of an autonomous parking system. Proc. Genetic and Evolutionary Computation Conference (GECCO-2004), LNCS 3103, Springer, Berlin Heidelberg New York, pp 1400–1412

  • Wegener J, Sthamer HH, Jones BF, Eyres DE (1997) Testing real-time systems using genetic algorithms. Softw Qual J 6(2):127–135

    Article  Google Scholar 

  • Wegener J, Buhr K, Pohlheim H (2002) Automatic test data generation for structural testing of embedded software systems by evolutionary testing. Proc. Genetic and Evolutionary Computation Conference (GECCO-2002), New York, USA, pp 1233–1240

  • Weinberger E (1990) Correlated and uncorrelated landscapes and how to tell the difference. Biol Cybern 63:325–336

    Article  MATH  Google Scholar 

  • Yokoo M (1997) Why adding more constraints makes a problem easier for hill-climbing algorithms: analyzing landscapes of CSPs. Proc. Int. Conf. on Principles and Practice of Constraint Programming (CP'97), LNCS 1330, Springer, Berlin Heidelberg New York, pp 356–370

Download references

Author information

Authors and Affiliations

  1. LAAS-CNRS, 7, Av du Colonel Roche, 31077, Toulouse, Cedex 4, France

    Hélène Waeselynck, Pascale Thévenod-Fosse & Olfa Abdellatif-Kaddour

Authors
  1. Hélène Waeselynck
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pascale Thévenod-Fosse
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olfa Abdellatif-Kaddour
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hélène Waeselynck.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Waeselynck, H., Thévenod-Fosse, P. & Abdellatif-Kaddour, O. Simulated annealing applied to test generation: landscape characterization and stopping criteria. Empir Software Eng 12, 35–63 (2007). https://doi.org/10.1007/s10664-006-7551-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-006-7551-5

Keywords

Search

Navigation