Theoretical Analysis of Local Search in Software Testing

  • Andrea Arcuri
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5792)

Abstract

The field of search based software engineering lacks of theoretical foundations. In this paper we theoretically analyse local search algorithms applied to software testing. We consider an infinitely large class of software that has an easy search landscape. Although the search landscape is easy, the software can be arbitrarily complex and large. We prove that Hill Climbing asymptotically has a strictly better runtime than Random Search. However, we prove that a very fast variant of Hill Climbing on reasonable size of software actually does not scale up properly. Although that variant has an exponential runtime, we prove that asymptotically it is still better than Random Search. We show that even on the easiest software testing problems, more sophisticated algorithms than local search are still required to get better performance.

References

  1. 1.
    McMinn, P.: Search-based software test data generation: A survey. Software Testing, Verification and Reliability 14(2), 105–156 (2004)CrossRefGoogle Scholar
  2. 2.
    Lehre, P.K., Yao, X.: Runtime analysis of search heuristics on software engineering problems. Frontiers of Computer Science in China 3(1), 64–72 (2009)CrossRefGoogle Scholar
  3. 3.
    Lehre, P.K., Yao, X.: Runtime analysis of (1+1) ea on computing unique input output sequences. In: IEEE Congress on Evolutionary Computation (CEC), pp. 1882–1889 (2007)Google Scholar
  4. 4.
    Lehre, P.K., Yao, X.: Crossover can be constructive when computing unique input output sequences. In: Li, X., Kirley, M., Zhang, M., Green, D., Ciesielski, V., Abbass, H.A., Michalewicz, Z., Hendtlass, T., Deb, K., Tan, K.C., Branke, J., Shi, Y. (eds.) SEAL 2008. LNCS, vol. 5361, pp. 595–604. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Harman, M., McMinn, P.: A theoretical and empirical study of search based testing: Local, global and hybrid search. IEEE Transactions on Software Engineering (to appear)Google Scholar
  6. 6.
    Harman, M., McMinn, P.: A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation. In: Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pp. 73–83 (2007)Google Scholar
  7. 7.
    Arcuri, A., Lehre, P., Yao, X.: Theoretical runtime analysis in search based software engineering. Technical Report CSR-09-04, University of BirminghamGoogle Scholar
  8. 8.
    Arcuri, A., Lehre, P.K., Yao, X.: Theoretical runtime analyses of search algorithms on the test data generation for the triangle classification problem. In: International Workshop on Search-Based Software Testing (SBST), pp. 161–169 (2008)Google Scholar
  9. 9.
    Arcuri, A.: Full theoretical runtime analysis of alternating variable method on the triangle classification problem. In: International Symposium on Search Based Software Engineering (SSBSE), pp. 113–121 (2009)Google Scholar
  10. 10.
    Arcuri, A.: Longer is better: On the role of test sequence length in software testing. Technical Report CSR-09-03, University of Birmingham (2009)Google Scholar
  11. 11.
    Droste, S., Jansen, T., Wegener, I.: Upper and lower bounds for randomized search heuristics in black-box optimization. Theory of Computing Systems 39(4), 525–544 (2006)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Droste, S., Jansen, T., Wegener, I.: On the analysis of the (1+1) evolutionary algorithm. Theoretical Computer Science 276, 51–81 (2002)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    He, J., Yao, X.: A study of drift analysis for estimating computation time of evolutionary algorithms. Natural Computing 3(1), 21–35 (2004)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. MIT Press and McGraw-Hill (2001)Google Scholar
  15. 15.
    Jansen, T.: On the brittleness of evolutionary algorithms. In: Stephens, C.R., Toussaint, M., Whitley, L.D., Stadler, P.F. (eds.) FOGA 2007. LNCS, vol. 4436, pp. 54–69. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  16. 16.
    R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2008) ISBN 3-900051-07-0Google Scholar
  17. 17.
    Arcuri, A.: Insight knowledge in search based software testing. In: Genetic and Evolutionary Computation Conference (GECCO), pp. 1649–1656 (2009)Google Scholar
  18. 18.
    Oliveto, P.S., He, J., Yao, X.: Time complexity of evolutionary algorithms for combinatorial optimization: A decade of results. International Journal of Automation and Computing 4(3), 281–293 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Andrea Arcuri
    • 1
  1. 1.Simula Research LaboratoryLysakerNorway

Personalised recommendations