Abstract
Differential software testing is important for software quality assurance as it aims to automatically generate test inputs that reveal behavioral differences in software. Detecting regression bugs in software evolution, analyzing side-channels in programs, maximizing the execution cost of a program over multiple executions, and evaluating the robustness of neural networks are instances of differential software analysis to generate diverging executions of program paths. The key challenge thereby is to simultaneously reason about multiple program paths, often across program variants, in an efficient way. Existing work in differential testing is often not (specifically) directed to reveal a different behavior or is limited to a subset of the search space. This work proposes the concept of Hybrid Differential Software Testing (HyDiff) as a hybrid analysis technique to generate difference revealing inputs. HyDiff consists of two components that operate in a parallel setup: (1) a search-based technique that inexpensively generates inputs and (2) a systematic exploration technique to also exercise deeper program behaviors. HyDiff’s search-based component uses differential fuzzing directed by differential heuristics. HyDiff’s systematic exploration component is based on differential dynamic symbolic execution that allows to incorporate concrete inputs in its analysis. HyDiff is evaluated experimentally with applications specific for differential testing. The results show that HyDiff is effective in all considered categories and outperforms its components in isolation.
Please note that this book chapter is a condensed version of the original dissertation [1] and the corresponding publications [2,3,4,5,6,7].
Chapter PDF
Similar content being viewed by others
References
Noller, Y.: Hybrid differential software testing. Ph.D. Thesis, Humboldt-Universität zu Berlin, Mathematisch-Naturwissenschaftliche Fakultät (2020). https://doi.org/10.18452/21968
Nilizadeh, S., Noller, Y., Păsăreanu, C.S.: Diffuzz: differential fuzzing for side-channel analysis. In: Proceedings of the 41st International Conference on Software Engineering, ICSE ’19, pp. 176–187. IEEE Press, Piscataway (2019). https://doi.org/10.1109/ICSE.2019.00034
Noller, Y.: Differential program analysis with fuzzing and symbolic execution. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, pp. 944–947. ACM, New York (2018). https://doi.org/10.1145/3238147.3241537
Noller, Y., Kersten, R., Păsăreanu, C.S.: Badger: complexity analysis with fuzzing and symbolic execution. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, pp. 322–332. ACM, New York (2018). https://doi.org/10.1145/3213846.3213868
Noller, Y., Nguyen, H.L., Tang, M., Kehrer, T.: Shadow symbolic execution with java pathfinder. SIGSOFT Softw. Eng. Notes 42(4), 1–5 (2018). https://doi.org/10.1145/3149485.3149492
Noller, Y., Nguyen, H.L., Tang, M., Kehrer, T., Grunske, L.: Complete shadow symbolic execution with java pathfinder. SIGSOFT Softw. Eng. Notes 44(4), 15–16 (2019). https://doi.org/10.1145/3364452.33644558
Noller, Y., Păsăreanu, C.S., Böhme, M., Sun, Y., Nguyen, H.L., Grunske, L.: Hydiff: Hybrid differential software analysis. In: Will appear in: Proceedings of the 42nd International Conference on Software Engineering, ICSE ’20 (2020)
ISO/IEC/IEEE International Standard—Systems and software engineering–Vocabulary. ISO/IEC/IEEE 24765:2017(E), pp. 1–541 (2017). https://doi.org/10.1109/IEEESTD.2017.8016712
DeRemer, F., Kron, H.H.: Programming-in-the-large versus programming-in-the-small. IEEE Trans. Softw. Eng. 2(2), 80–86 (1976). https://doi.org/10.1109/TSE.1976.233534
Vliet, H.V.: Software Engineering: Principles and Practice, 3rd edn. Wiley, London (2008)
Orso, A., Rothermel, G.: Software testing: A research travelogue (2000–2014). In: Proceedings of the on Future of Software Engineering, FOSE 2014, pp. 117–132. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2593882.2593885
Fraser, G., Arcuri, A.: A large-scale evaluation of automated unit test generation using evosuite. ACM Trans. Softw. Eng. Methodol. 24(2) (2014). https://doi.org/10.1145/2685612
Palikareva, H., Kuchta, T., Cadar, C.: Shadow of a doubt: testing for divergences between software versions. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 1181–1192 (2016). https://doi.org/10.1145/2884781.2884845
Person, S., Dwyer, M.B., Elbaum, S., Păsăreanu, C.S.: Differential symbolic execution. In: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, SIGSOFT ’08/FSE-16, pp. 226–237. ACM, New York (2008). https://doi.org/10.1145/1453101.1453131
Luckow, K., Kersten, R., Păsăreanu, C.S.: Symbolic complexity analysis using context-preserving histories. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 58–68 (2017). https://doi.org/10.1109/ICST.2017.13
Petsios, T., Zhao, J., Keromytis, A.D., Jana, S.: Slowfuzz: Automated domain-independent detection of algorithmic complexity vulnerabilities. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, pp. 2155–2168. ACM, New York (2017). https://doi.org/10.1145/3133956.3134073
Antonopoulos, T., Gazzillo, P., Hicks, M., Koskinen, E., Terauchi, T., Wei, S.: Decomposition instead of self-composition for proving the absence of timing channels. SIGPLAN Not. 52(6), 362–375 (2017). https://doi.org/10.1145/3140587.3062378
Chen, J., Feng, Y., Dillig, I.: Precise detection of side-channel vulnerabilities using quantitative Cartesian Hoare logic. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, pp. 875–890. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3133956.3134058
Păsăreanu, C.S., Phan, Q.S., Malacaria, P.: Multi-run side-channel analysis using symbolic execution and Max-SMT. In: 2016 IEEE 29th Computer Security Foundations Symposium (CSF), pp. 387–400 (2016). https://doi.org/10.1109/CSF.2016.34
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., Kroening, D.: Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, pp. 109–119. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3238147.3238172
Tramer, F., Boneh, D.: Adversarial training and robustness for multiple perturbations. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 5858–5868. Curran Associates, Red Hook (2019)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples (2014). arXiv preprint arXiv:1412.6572
Cha, S.K., Avgerinos, T., Rebert, A., Brumley, D.: Unleashing mayhem on binary code. In: 2012 IEEE Symposium on Security and Privacy, pp. 380–394 (2012). https://doi.org/10.1109/SP.2012.31
Stephens, N., Grosen, J., Salls, C., Dutcher, A., Wang, R., Corbetta, J., Shoshitaishvili, Y., Kruegel, C., Vigna, G.: Driller: augmenting fuzzing through selective symbolic execution. In: 23nd Annual Network and Distributed System Security Symposium, NDSS 2016, San Diego, California, USA, February 21–24, 2016 (2016). https://doi.org/10.14722/ndss.2016.23368
Harrold, M.J.: Testing evolving software. J. Syst. Softw. 47(2), 173–181 (1999). https://doi.org/10.1016/S0164-1212(99)00037-0
Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Testing Verif. Reliab. 22(2), 67–120 (2012). https://doi.org/10.1002/stvr.430
Brennan, T., Saha, S., Bultan, T., Păsăreanu, C.S.: Symbolic path cost analysis for side-channel detection. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, pp. 27–37. ACM, New York (2018). https://doi.org/10.1145/3213846.3213867
Miller, B.P., Fredriksen, L., So, B.: An empirical study of the reliability of Unix utilities. Commun. ACM 33(12), 32–44 (1990). https://doi.org/10.1145/96267.96279
Godefroid, P., Klarlund, N., Sen, K.: Dart: Directed automated random testing. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’05, pp. 213–223. Association for Computing Machinery, New York (2005). https://doi.org/10.1145/1065010.1065036
King, J.C.: Symbolic execution and program testing. Commun. ACM 19(7), 385–394 (1976). https://doi.org/10.1145/360248.360252
Böhme, M., Pham, V.T., Roychoudhury, A.: Coverage-based greybox fuzzing as Markov chain. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, pp. 1032–1043. ACM, New York (2016). https://doi.org/10.1145/2976749.2978428
Pham, V.T., Böhme, M., Santosa, A.E., Căciulescu, A.R., Roychoudhury, A.: Smart greybox fuzzing. IEEE Trans. Softw. Eng. 1–17 (2019)
Böhme, M., Pham, V.T., Nguyen, M.D., Roychoudhury, A.: Directed greybox fuzzing. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, pp. 2329–2344. ACM, New York (2017). https://doi.org/10.1145/3133956.3134020
Website: American fuzzy lop (AFL)—a security-oriented fuzzer (2014). http://lcamtuf.coredump.cx/afl/
Zeller, A., Gopinath, R., Böhme, M., Fraser, G., Holler, C.: The fuzzing book. In: The Fuzzing Book. Saarland University (2019). https://www.fuzzingbook.org/
Person, S., Yang, G., Rungta, N., Khurshid, S.: Directed incremental symbolic execution. In: Proceedings of the 32Nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’11, pp. 504–515. ACM, New York (2011). https://doi.org/10.1145/1993498.1993558
Yang, G., Păsăreanu, C.S., Khurshid, S.: Memoized symbolic execution. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA 2012, pp. 144–154. ACM, New York (2012). https://doi.org/10.1145/2338965.2336771
Majumdar, R., Sen, K.: Hybrid concolic testing. In: 29th International Conference on Software Engineering (ICSE’07), pp. 416–426. IEEE Computer Society, Los Alamitos (2007). https://doi.org/10.1109/ICSE.2007.41
Website: Software-artifact infrastructure repository (2019). http://sir.unl.edu
Just, R., Jalali, D., Ernst, M.D.: Defects4j: A database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437–440. ACM, New York (2014). https://doi.org/10.1145/2610384.2628055
Website: Commons CLI (2019). https://commons.apache.org/proper/commons-cli/
Website: DARPA’s Space/Time Analysis for Cybersecurity (STAC) program (2015). https://www.darpa.mil/program/space-time-analysis-for-cybersecurity
Website: Debian bug report log 800564—php5: trivial hash complexity DoS attack (2015). https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800564
Lipp, M., Schwarz, M., Gruss, D., Prescher, T., Haas, W., Fogh, A., Horn, J., Mangard, S., Kocher, P., Genkin, D., Yarom, Y., Hamburg, M.: Meltdown: reading kernel memory from user space. In: 27th USENIX Security Symposium (USENIX Security 18), pp. 973–990. USENIX Association, Baltimore (2018)
Kocher, P., Horn, J., Fogh, A., Genkin, D., Gruss, D., Haas, W., Hamburg, M., Lipp, M., Mangard, S., Prescher, T., Schwarz, M., Yarom, Y.: Spectre attacks: exploiting speculative execution. In: 2019 IEEE Symposium on Security and Privacy (SP), pp. 1–19 (2019). https://doi.org/10.1109/SP.2019.00002
Barthe, G., D’Argenio, P.R., Rezk, T.: Secure information flow by self-composition. In: Proceedings. 17th IEEE Computer Security Foundations Workshop, 2004, pp. 100–114 (2004). https://doi.org/10.1109/CSFW.2004.1310735
Kocher, P.C.: Timing attacks on implementations of Diffie-Hellman, RSA, DSS, and other systems. In: Koblitz, N. (ed.) Advances in Cryptology—CRYPTO ’96, pp. 104–113. Springer, Berlin, Heidelberg (1996)
Website: Deep learning test toolset (2020). https://github.com/theyoucheng/DLTT
Website: SideFuzz: Fuzzing for side-channel vulnerabilities (2018). https://github.com/phayes/sidefuzz
Acknowledgements
I want to thank my family and friends for supporting me during my PhD. My great appreciation goes to all my co-authors and my particular gratitude to Lars Grunske and Corina Păsăreanu for their great support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2022 The Author(s)
About this chapter
Cite this chapter
Noller, Y. (2022). Hybrid Differential Software Testing. In: Felderer, M., et al. Ernst Denert Award for Software Engineering 2020. Springer, Cham. https://doi.org/10.1007/978-3-030-83128-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-83128-8_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-83127-1
Online ISBN: 978-3-030-83128-8
eBook Packages: Computer ScienceComputer Science (R0)