Skip to main content
SpringerLink
Log in

Constructing Search Spaces for Search-Based Software Testing Using Neural Networks

  • Conference paper
  • First Online:
Search-Based Software Engineering (SSBSE 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11664))

Included in the following conference series:

Abstract

A central requirement for any Search-Based Software Testing (SBST) technique is a convenient and meaningful fitness landscape. Whether one follows a targeted or a diversification driven strategy, a search landscape needs to be large, continuous, easy to construct and representative of the underlying property of interest. Constructing such a landscape is not a trivial task often requiring a significant manual effort by an expert.

We present an approach for constructing meaningful and convenient fitness landscapes using neural networks (NN) – for targeted and diversification strategies alike. We suggest that output of an NN predictor can be interpreted as a fitness for a targeted strategy. The NN is trained on a corpus of execution traces and various properties of interest, prior to searching. During search, the trained NN is queried to predict an estimate of a property given an execution trace. The outputs of the NN form a convenient search space which is strongly representative of a number of properties. We believe that such a search space can be readily used for driving a search towards specific properties of interest.

For a diversification strategy, we propose the use of an autoencoder; a mechanism for compacting data into an n-dimensional “latent” space. In it, datapoints are arranged according to the similarity of their salient features. We show that a latent space of execution traces possesses characteristics of a convenient search landscape: it is continuous, large and crucially, it defines a notion of similarity to arbitrary observations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The dimensionality is arbitrary, three is chosen here so that the space can be plotted for qualitative analysis.

  2. 2.

    Böhme et al. showed that enforcing diversity on AFL’s search is beneficial [3]. In future work we intend to investigate how the effectiveness using their notion of diversity compares with that proposed here.

References

  1. Aleti, A., Moser, I., Grunske, L.: Analysing the fitness landscape of search-based software testing problems. Autom. Softw. Eng. 24(3), 603–621 (2017)

    Article  Google Scholar 

  2. Alshahwan, N., Harman, M.: Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, pp. 181–192. ACM (2014)

    Google Scholar 

  3. Böhme, M., Pham, V.T., Roychoudhury, A.: Coverage-based greybox fuzzing as Markov chain. IEEE Trans. Softw. Eng. 45, 489–506 (2017)

    Article  Google Scholar 

  4. Bonzini, P.: sed(1) - Linux man page (2019). https://linux.die.net/man/1/sed

  5. Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: Maher, M.J. (ed.) ASIAN 2004. LNCS, vol. 3321, pp. 320–329. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30502-6_23

    Chapter  Google Scholar 

  6. Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: ARTOO: adaptive random testing for object-oriented software. In: Proceedings of the 30th International Conference on Software Engineering, pp. 71–80. ACM (2008)

    Google Scholar 

  7. Cortes, C., Mohri, M.: AUC optimization vs. error rate minimization. In: Advances in Neural Information Processing Systems, pp. 313–320 (2004)

    Google Scholar 

  8. Doersch, C.: Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 (2016)

  9. Dumoulin, V., Visin, F.: A guide to convolution arithmetic for deep learning. arXiv preprint arXiv:1603.07285 (2016)

  10. Feldt, R., Poulding, S., Clark, D., Yoo, S.: Test set diameter: quantifying the diversity of sets of test cases. In: 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 223–233. IEEE (2016)

    Google Scholar 

  11. Gay, G., Staats, M., Whalen, M., Heimdahl, M.P.: The risks of coverage-directed test case generation. IEEE Trans. Softw. Eng. 41(8), 803–819 (2015)

    Article  Google Scholar 

  12. Glasmachers, T.: Limits of end-to-end learning. arXiv preprint arXiv:1704.08305 (2017)

  13. Gordon, A.D., Melham, T.: Five axioms of alpha-conversion. In: Goos, G., Hartmanis, J., van Leeuwen, J., von Wright, J., Grundy, J., Harrison, J. (eds.) TPHOLs 1996. LNCS, vol. 1125, pp. 173–190. Springer, Heidelberg (1996). https://doi.org/10.1007/BFb0105404

    Chapter  Google Scholar 

  14. Harman, M., Clark, J.: Metrics are fitness functions too. In: Proceedings of 10th International Symposium on Software Metrics, pp. 58–69. IEEE (2004)

    Google Scholar 

  15. Harman, M., Jones, B.F.: Search-based software engineering. Inf. Softw. Technol. 43(14), 833–839 (2001)

    Article  Google Scholar 

  16. Harman, M., Mansouri, S.A., Zhang, Y.: Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. (CSUR) 45(1), 11 (2012)

    Article  Google Scholar 

  17. Heimdahl, M.P.E., George, D., Weber, R.: Specification test coverage adequacy criteria = specification test generation inadequacy criteria. In: Proceedings of Eighth IEEE International Symposium on High Assurance Systems Engineering, pp. 178–186. IEEE (2004)

    Google Scholar 

  18. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  19. Huang, J., Ling, C.X.: Using auc and accuracy in evaluating learning algorithms. IEEE Trans. Knowl. Data Eng. 17(3), 299–310 (2005)

    Article  Google Scholar 

  20. Inozemtseva, L., Holmes, R.: Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International Conference on Software Engineering, pp. 435–445. ACM (2014)

    Google Scholar 

  21. Jones, D.: Sparse - a semantic parser for C (2019). https://sparse.wiki.kernel.org/index.php/Main_Page

  22. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188 (2014)

  23. Karpathy, A.: Cs231n convolutional neural networks for visual recognition (2016). http://cs231n.github.io/neural-networks-1/

  24. Kawaguchi, K.: Deep learning without poor local minima. In: Advances in Neural Information Processing Systems, pp. 586–594 (2016)

    Google Scholar 

  25. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  26. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  27. Kochhar, P.S., Thung, F., Lo, D.: Code coverage and test suite effectiveness: empirical study with real bugs in large systems. In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 560–564. IEEE (2015)

    Google Scholar 

  28. Lane, T., et al.: libjpeg 6b (1998). http://libjpeg.sourceforge.net/

  29. Luk, C.K., et al.: Pin: building customized program analysis tools with dynamic instrumentation. In: ACM SIGPLAN Notices, vol. 40, pp. 190–200. ACM (2005)

    Google Scholar 

  30. McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)

    Article  Google Scholar 

  31. Meyering, J., Gordon, A.: GNU sed (2019). https://www.gnu.org/software/sed/

  32. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  33. Miller, B.P., Fredriksen, L., So, B.: An empirical study of the reliability of unix utilities. Commun. ACM 33(12), 32–44 (1990)

    Article  Google Scholar 

  34. Nethercote, N., Seward, J.: Valgrind: a framework for heavyweight dynamic binary instrumentation. In: ACM SIGPLAN Notices, vol. 42, pp. 89–100. ACM (2007)

    Google Scholar 

  35. Nguyen, Q., Hein, M.: The loss surface of deep and wide neural networks. arXiv preprint arXiv:1704.08045 (2017)

  36. Nguyen, Q., Hein, M.: Optimization landscape and expressivity of deep CNNs. In: International Conference on Machine Learning, pp. 3727–3736 (2018)

    Google Scholar 

  37. Salimans, T., Kingma, D.P.: Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems, pp. 901–909 (2016)

    Google Scholar 

  38. Shepperd, M.: Fundamentals of Software Measurement. Prentice-Hall, Upper Saddle River (1995)

    MATH  Google Scholar 

  39. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  40. Staats, M., Gay, G., Whalen, M., Heimdahl, M.: On the danger of coverage directed test case generation. In: de Lara, J., Zisman, A. (eds.) FASE 2012. LNCS, vol. 7212, pp. 409–424. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28872-2_28

    Chapter  Google Scholar 

  41. Swirszcz, G., Czarnecki, W.M., Pascanu, R.: Local minima in training of neural networks. arXiv preprint arXiv:1611.06310 (2016)

  42. Tan, S.H., Yi, J., Mechtaev, S., Roychoudhury, A., et al.: Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In: Proceedings of the 39th International Conference on Software Engineering Companion, pp. 180–182. IEEE Press (2017)

    Google Scholar 

  43. VP Users (2017). http://valgrind.org/gallery/users.html

  44. Veillard, D.: The XML C parser and toolkit of Gnome (2019). http://xmlsoft.org/

  45. Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43(14), 841–854 (2001)

    Article  Google Scholar 

  46. Zalewski, M.: American fuzzy lop (2007). http://lcamtuf.coredump.cx/afl/

Download references

Author information

Authors and Affiliations

  1. University College London, Gower Street, London, WC1E 6BT, UK

    Leonid Joffe & David Clark

Authors
  1. Leonid Joffe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonid Joffe .

Editor information

Editors and Affiliations

  1. SnT/University of Luxembourg, Luxembourg, Luxembourg

    Shiva Nejati

  2. University of South Carolina, Columbia, SC, USA

    Gregory Gay

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Joffe, L., Clark, D. (2019). Constructing Search Spaces for Search-Based Software Testing Using Neural Networks. In: Nejati, S., Gay, G. (eds) Search-Based Software Engineering. SSBSE 2019. Lecture Notes in Computer Science(), vol 11664. Springer, Cham. https://doi.org/10.1007/978-3-030-27455-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-27455-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-27454-2

  • Online ISBN: 978-3-030-27455-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation