Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

An empirical study on constraint optimization techniques for test generation

约束优化技术在测试用例生成中的实证研究

  • 146 Accesses

  • 6 Citations

Abstract

Constraint solving is a frequent, but expensive operation with symbolic execution to generate tests for a program. To improve the efficiency of test generation using constraint solving, four optimization techniques are usually applied to existing constraint solvers, which are constraint independence, constraint set simplification, constraint caching, and expression rewriting. In this paper, we conducted an empirical study, using these four constraint optimization techniques in a well known test generation tool KLEE with 77 GNU Coreutils applications, to systematically investigate how these optimization techniques affect the efficiency of test generation. The experimental results show that these constraint optimization techniques as well as their combinations cannot improve the efficiency of test generation significantly for ALL-SIZED programs. Moreover, we studied the constraint optimization techniques with respect to two static metrics, lines of code (LOC) and cyclomatic complexity (CC), of programs. The experimental results show that the “constraint set simplification” technique can improve the efficiency of test generation significantly for the programs with high LOC and CC values. The “constraint caching” optimization technique can improve the efficiency of test generation significantly for the programs with low LOC and CC values. Finally, we propose four hybrid optimization strategies and practical guidelines based on different static metrics.

创新点

我们使用KLEE执行了77个GNU Coreutils程序,用于系统的研究4种流行的约束优化技术如何影响测试用例生成的效率。结果表明,对于所有程序,这些约束优化技术及它们的组合不能大幅提高测试用例生成的效率。 此外,我们研究了程序的两个静态指标(代码行和圈复杂度)跟约束优化技术的关系。结果表明,对于大规模及高圈复杂度程序,使用“约束集简化”技术,对于小规模及低圈复杂度程序,使用“约束缓存”技术,都可以显著提高测试生成效率。最后,基于不同的静态指标,我们提出了四种混合优化策略和指导方针。

This is a preview of subscription content, log in to check access.

References

  1. 1

    Beizer B. Software Testing Techniques. 2nd ed. New York: International Thomson Computer Press, 1990

  2. 2

    Fang C R, Chen Z Y, Xu B W. Comparing logic coverage criteria on test case prioritization. Sci China Inf Sci, 2012, 55: 2826–2840

  3. 3

    Yang R, Chen Z Y, Zhang Z Y, et al. Efsm-based test case generation: sequence, data, and oracle. Int J Softw Eng Knowl Eng, 2015, 25: 633–667

  4. 4

    Orso A, Rothermel G. Software testing: a research travelogue (2000–2014). In: Proceedings of the IEEE International Conference on Future of Software Engineering (ICSE). New York: ACM, 2014. 117–132

  5. 5

    King J C. Symbolic execution and program testing. Commun ACM, 1976, 19: 385–394

  6. 6

    Chen T, Zhang X-S, Guo S-Z, et al. State of the art: dynamic symbolic execution for automated test generation. Future Gener Comput Syst, 2013, 29: 1758–1773

  7. 7

    Anand S, Burke E K, Chen T Y, et al. An orchestrated survey of methodologies for automated software test case generation. J Syst Softw, 2013, 86: 1978–2001

  8. 8

    Cadar C, Dunbar D, Engler D R. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley: USENIX Association, 2008. 209–224

  9. 9

    Cadar C, Ganesh V, Pawlowski P M, et al. Exe: automatically generating inputs of death. ACM Trans Inf Syst Secur, 2008, 12: 10

  10. 10

    Godefroid P, Klarlund N, Sen K. Dart: directed automated random testing. In: Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. New York: ACM, 2005. 40: 213–223

  11. 11

    Sen K, Marinov D, Agha G. CUTE: a concolic unit testing engine for C. In: Proceedings of the 10th European Software Engineering Conference Held Jointly With 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering. New York: ACM, 2005. 263–272

  12. 12

    Barrett C, de Moura L, Stump A. Design and results of the first satisfiability modulo theories competition (smt-comp 2005). J Autom Reasoning, 2005, 35: 373–390

  13. 13

    Schittkowski K. NLPQL: a fortran subroutine solving constrained nonlinear programming problems. Ann Oper Res, 1986, 5: 485–500

  14. 14

    Cadar C, Engler D. Execution generated test cases: how to make systems code crash itself. In: Model Checking Software. Berlin: Springer, 2005. 2–23

  15. 15

    Godefroid P, Levin M Y, Molnar D A, et al. Automated whitebox fuzz testing. In: Proceedings of the Network and Distributed System Security Symposium, San Diego, 2008. 8: 151–166

  16. 16

    Brumley D, Newsome J, Song D, et al. Towards automatic generation of vulnerability-based signatures. In: Proceedings of IEEE Symposium on Security and Privacy, Berkeley/Oakland, 2006. 15–16

  17. 17

    Brumley D, Newsome J, Song D, et al. Theory and techniques for automatic generation of vulnerability-based signatures. IEEE Trans Depend Secure Comput, 2008, 5: 224–241

  18. 18

    Brumley D, Wang H, Jha S, et al. Creating vulnerability signatures using weakest preconditions. In: Proceedings of the 20th IEEE Computer Security Foundations Symposium, Venice, 2007. 311–325

  19. 19

    Liang Z K, Sekar R. Fast and automated generation of attack signatures: a basis for building self-protecting servers. In: Proceedings of the 12th ACM Conference on Computer and Communications Security. New York: ACM, 2005. 213–222

  20. 20

    Newsome J, Brumley D, Song D. Vulnerability-specific execution filtering for exploit prevention on commodity software. In: Proceedings of the Network and Distributed System Security Symposium, San Diego, 2006. 58–71

  21. 21

    Brumley D, Hartwig C, Kang M G, et al. Bitscope: Automatically Dissecting Malicious Binaries. School of Computer Science, Carnegie Mellon University, Technology Report CMU-CS-07-133. 2007

  22. 22

    Brumley D, Hartwig C, Liang Z K, et al. Automatically identifying trigger-based behavior in malware. In: Botnet Detection. Berlin: Springer, 2008. 65–88

  23. 23

    Moser A, Kruegel C, Kirda E. Exploring multiple execution paths for malware analysis. In: Proceedings of IEEE Symposium on Security and Privacy, Berkeley, 2007. 231–245

  24. 24

    Song D, Brumley D, Yin H, et al. Bitblaze: a new approach to computer security via binary analysis. In: Information Systems Security. Berlin: Springer, 2008. 1–25

  25. 25

    Chandra A K, Iyengar V S. Constraint solving for test case generation: a technique for high-level design verification. In: Proceedings of IEEE International Conference on Computer Design: VLSI in Computers and Processors, Cambridge, 1992. 245–248

  26. 26

    De Milli R A, Offutt A J. Constraint-based automatic test data generation. IEEE Trans Softw Eng, 1991, 17: 900–910

  27. 27

    Gotlieb A, Botella B, Rueher M. Automatic test data generation using constraint solving techniques. ACM SIGSOFT Softw Eng Notes, 1998, 23: 53–62

  28. 28

    Tovey C A. A simplified np-complete satisfiability problem. Discrete Appl Math, 1984, 8: 85–89

  29. 29

    Ganesh V, Dill D L. A decision procedure for bit-vectors and arrays. In: Computer Aided Verification. Berlin: Springer, 2007. 519–531

  30. 30

    de Moura L, Bjørner N. Z3: an efficient smt solver. In: Tools and Algorithms for the Construction and Analysis of Systems. Berlin: Springer, 2008. 337–340

  31. 31

    Barrett C, Tinelli C. Cvc3. In: Computer Aided Verification. Berlin: Springer, 2007. 298–302

  32. 32

    Palikareva H, Cadar C. Multi-solver support in symbolic execution. In: Computer Aided Verification. Berlin: Springer, 2013. 53–68

  33. 33

    Jones C. Software metrics: good, bad and missing. Computer, 1994, 27: 98–100

  34. 34

    Shepperd M. A critique of cyclomatic complexity as a software metric. Softw Eng J, 1988, 3: 30–36

  35. 35

    Ferguson R, Korel B. The chaining approach for software test data generation. ACM Trans Softw Eng Meth, 1996, 5: 63–86

  36. 36

    Offutt A J, Hayes J H. A semantic model of program faults. ACM SIGSOFT Soft Eng Notes, 1996, 21: 195–200

  37. 37

    Brummayer R, Biere A. Boolector: an efficient smt solver for bit-vectors and arrays. In: Tools and Algorithms for the Construction and Analysis of Systems. Berlin: Springer, 2009. 174–177

  38. 38

    Erete I, Orso A. Optimizing constraint solving to better support symbolic execution. In: Proceedings of IEEE 4th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Berlin, 2011. 310–315

Download references

Author information

Correspondence to Zhenyu Chen.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Chen, Z., Gao, R. et al. An empirical study on constraint optimization techniques for test generation. Sci. China Inf. Sci. 60, 012105 (2017). https://doi.org/10.1007/s11432-015-0450-5

Download citation

Keywords

  • test generation
  • symbolic execution
  • constraint solving
  • constraint optimization
  • static metric
  • 012105
  • 测试用例生成
  • 符号执行
  • 约束求解
  • 约束优化
  • 静态度量