Abstract
We introduce a novel automated testing technique that combines LLM and search-based fuzzing. We use ChatGPT to parameterise C programs. We compile the resultant code snippets, and feed compilable ones to SearchGEM5, our extension to AFL++ fuzzer with customised new mutation operators. We run thus created 4005 binaries through our system under test, gem5, increasing its existing test coverage by more than 1000 lines. We discover 244 instances where gem5 simulation of the binary differs from the binary’s expected behaviour.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
AFL compiler fuzzer. https://github.com/agroce/afl-compiler-fuzzer
Artifact of SearchGEM5. Zenodo (2023). https://doi.org/10.5281/zenodo.8316685
Aschermann, C., Frassetto, T., Holz, T., Jauernig, P., Sadeghi, A.R., Teuchert, D.: NAUTILUS: fishing for deep bugs with grammars. In: NDSS (2019)
Binkert, N., et al.: The gem5 simulator. ACM SIGARCH Comput. Archit. News 39(2), 1–7 (2011)
Biswas, S.: Role of ChatGPT in computer programming: ChatGPT in computer programming. Mesopotamian J. Comput. Sci. 2023, 8–16 (2023)
Bossuet, L., Grosso, V., Lara-Nino, C.A.: Emulating side channel attacks on gem5: lessons learned. In: EuroS &PW, pp. 287–295. IEEE (2023)
Bruns, N., Herdt, V., Große, D., Drechsler, R.: Toward RISC-V CSR compliance testing. IEEE Embed. Syst. Lett. 13(4), 202–205 (2021)
Destefanis, G., Bartolucci, S., Ortu, M.: A preliminary analysis on the code generation capabilities of GPT-3.5 and Bard AI models for java functions (2023)
Fioraldi, A., et al.: AFL++ : Combining incremental steps of fuzzing research. In: USENIX Workshop at WOOT 20. USENIX Association (2020)
Git repository of gfauto. https://github.com/google/graphicsfuzz.git
Git repository of searchGEM5. https://github.com/karineek/SearchGEM5/
Kersten, R., Luckow, K., Păsăreanu, C.S.: POSTER: AFL-based fuzzing for java with kelinci. In: SIGSAC, pp. 2511–2513. CCS 2017. ACM (2017)
Langdon, W.B., Yoo, S., Harman, M.: Inferring automatic test oracles. In: SBST, pp. 5–6. Buenos Aires, Argentina (2017)
Lund, B.D., Wang, T.: Chatting about ChatGPT: how may AI and GPT impact academia and libraries? Library Hi Tech News 40(3), 26–29 (2023)
Menéndez, H.D., Suárez-Tangil, G.: ObfSec: measuring the security of obfuscations from a testing perspective. Expert Syst. Appl. 210, 118298 (2022)
Pham, V.T., Böhme, M., Roychoudhury, A.: AFLNET: a greybox fuzzer for network protocols. In: ICST, pp. 460–465 (2020)
AFL’s’ fork for fuzzing pure Python. https://github.com/jwilk/python-afl
Zalewski, M.: Technical “whitepaper” for afl-fuzz. Accessed 21 Apr 2023. http://lcamtuf.coredump.cx/afl/technical_details.txt
Acknowledgments
Authors are listed in alphabetical order. This work was supported by the UKRI EPSRC grant no. EP/P023991/1 and the UKRI TAS Hub grant no. EP/V00784X/1 and EP/V026801/2.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dakhama, A., Even-Mendoza, K., Langdon, W., Menendez, H., Petke, J. (2024). SearchGEM5: Towards Reliable Gem5 with Search Based Software Testing and Large Language Models. In: Arcaini, P., Yue, T., Fredericks, E.M. (eds) Search-Based Software Engineering. SSBSE 2023. Lecture Notes in Computer Science, vol 14415. Springer, Cham. https://doi.org/10.1007/978-3-031-48796-5_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-48796-5_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48795-8
Online ISBN: 978-3-031-48796-5
eBook Packages: Computer ScienceComputer Science (R0)