Skip to main content
SpringerLink
Log in

Syntax-Aware Mutation for Testing the Solidity Compiler

  • Conference paper
  • First Online:
Computer Security – ESORICS 2023 (ESORICS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14346))

Included in the following conference series:

  • 283 Accesses

Abstract

We introduce fuzzol, the first syntax-aware mutation fuzzer for systematically testing the security and reliability of solc, the standard Solidity compiler. fuzzol addresses a challenge of existing fuzzers when dealing with structured inputs: the generation of inputs that get past the parser checks of the system under test. To do so, fuzzol introduces a novel syntax-aware mutation that breaks into three strategies, each of them making different kind of changes in the inputs. Contrary to existing mutations, our mutation is able to change constructs, statements, and entire pieces of code, in a fine-grained manner that conforms to the syntactic rules of the Solidity grammar. Moreover, to explore new paths in the compiler’s codebase faster, we introduce a mutation strategy prioritization algorithm that allows fuzzol to identify and apply only those mutation strategies that are most effective in exercising new interesting paths. To evaluate fuzzol, we test 33 of the latest solc stable releases, and compare fuzzol with (1) Superion, a grammar-aware fuzzer, (2) AFL-compiler-fuzzer, a text-mutation fuzzer and (3) two grammar-blind fuzzers with advanced test input generation schedules: AFLFast and MO pt-AFL. fuzzol identified 19 bugs in total (7 of which were previously unknown to Solidity developers), while the other fuzzers missed half of these bugs. Also, fuzzol outperforms all fuzzers in terms of line, function, and branch coverage (from 3.75% to 408.8% improvement), while it is the most effective one when it comes to test input generation. Finally, our experiments indicate that our prioritization algorithm makes fuzzol explore new paths roughly one day (\(\sim \)24 h) faster.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. The Counterparty financial platform. https://counterparty.io/. Accessed 15 Jan 2023

  2. Hedera hashgraph. Accessed 15 Jan 2023

    Google Scholar 

  3. Optimized contract crash. https://github.com/ethereum/solidity/issues/12840. Accessed 05 Jan 2023

  4. Optimized contract freeze. https://github.com/ethereum/solidity/issues/12848. Accessed 03 Jan 2023

  5. Solidity. https://docs.soliditylang.org/en/v0.8.0/. Accessed 03 Jan 2023

  6. Solidity compiler - issues catalog. https://github.com/ethereum/solidity/issues. Accessed 15 Jan 2023

  7. The Solidity contract-oriented programming language Github repository. https://github.com/ethereum/solidity. Accessed 05 Jan 2023

  8. Z3 GitHub repository (2021). https://github.com/Z3Prover/z3. Accessed 20 Jan 2023

  9. Aschermann, C., Frassetto, T., Holz, T., Jauernig, P., Sadeghi, A., Teuchert, D.: NAUTILUS: fishing for deep bugs with grammars. In: Proceedings of the 26th Annual Network and Distributed System Security Symposium (NDSS) (2019)

    Google Scholar 

  10. Atlidakis, V., Godefroid, P., Polishchuk, M.: Restler: stateful rest API fuzzing. In: Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, pp. 748–758. IEEE Press (2019)

    Google Scholar 

  11. Blazytko, T., et al.: Grimoire: synthesizing structure while fuzzing. In: Proceedings of the 28th USENIX Conference on Security Symposium, pp. 1985–2002. USENIX Association, USA (2019)

    Google Scholar 

  12. Böhme, M., Pham, V.T., Nguyen, M.D., Roychoudhury, A.: Directed greybox fuzzing. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, pp. 2329–2344. Association for Computing Machinery, New York (2017)

    Google Scholar 

  13. Böhme, M., Pham, V.T., Roychoudhury, A.: Coverage-based greybox fuzzing as Markov chain, CCS 2016, pp. 1032–1043. Association for Computing Machinery, New York (2016)

    Google Scholar 

  14. Bounimova, E., Godefroid, P., Molnar, D.: Billions and billions of constraints: whitebox fuzz testing in production. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE 2013, pp. 122–131. IEEE Press (2013)

    Google Scholar 

  15. Brent, L., Grech, N., Lagouvardos, S., Scholz, B., Smaragdakis, Y.: Ethainter: a smart contract security analyzer for composite vulnerabilities. In: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2020, pp. 454–469. Association for Computing Machinery, New York (2020)

    Google Scholar 

  16. Browne, R.: Ether, the world’s second-biggest cryptocurrency, is closing in on an all-time high (2021). https://www.cnbc.com/2021/01/19/bitcoin-ethereum-eth-cryptocurrency-nears-all-time-high.html. Accessed 20 Jan 2023

  17. Cha, S.K., Woo, M., Brumley, D.: Program-adaptive mutational fuzzing. In: Proceedings of the 2015 IEEE Symposium on Security and Privacy, SP 2015, pp. 725–741. IEEE Computer Society, USA (2015)

    Google Scholar 

  18. Chaliasos, S., Gervais, A., Livshits, B.: A study of inline assembly in solidity smart contracts. Proc. ACM Program. Lang. 6(OOPSLA2) (2022)

    Google Scholar 

  19. Chaliasos, S., Sotiropoulos, T., Spinellis, D., Gervais, A., Livshits, B., Mitropoulos, D.: Finding typing compiler bugs. In: Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2022, pp. 183–198. ACM, New York (2022)

    Google Scholar 

  20. Chen, H., et al.: Hawkeye: towards a desired directed grey-box fuzzer. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, pp. 2095–2108. Association for Computing Machinery, New York (2018)

    Google Scholar 

  21. Chen, J., et al.: A survey of compiler testing. ACM Comput. Surv. 53(1), 1–36 (2020)

    Article  MathSciNet  Google Scholar 

  22. Cordeiro, L., Fischer, B., Marques-Silva, J.: SMT-based bounded model checking for embedded ANSI-C software. In: Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, ASE 2009, pp. 137–148. IEEE Computer Society, USA (2009)

    Google Scholar 

  23. de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  24. Ghaleb, A., Pattabiraman, K.: How effective are smart contract analysis tools? Evaluating smart contract static analysis tools using bug injection. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020, pp. 415–427. ACM, New York (2020)

    Google Scholar 

  25. Godefroid, P.: Fuzzing: hack, art, and science. Commun. ACM 63(2), 70–76 (2020)

    Article  Google Scholar 

  26. Godefroid, P., Levin, M.Y., Molnar, D.: Sage: whitebox fuzzing for security testing: Sage has had a remarkable impact at Microsoft. Queue 10(1), 20–27 (2012)

    Article  Google Scholar 

  27. Grech, N., Kong, M., Jurisevic, A., Brent, L., Scholz, B., Smaragdakis, Y.: Madmax: analyzing the out-of-gas world of smart contracts. Commun. ACM 63(10), 87–95 (2020)

    Article  Google Scholar 

  28. Groce, A., van Tonder, R., Kalburgi, G.T., Le Goues, C.: Making no-fuss compiler fuzzing effective. In: Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction, CC 2022, pp. 194–204. Association for Computing Machinery, New York (2022)

    Google Scholar 

  29. Holler, C., Herzig, K., Zeller, A.: Fuzzing with code fragments. In: Proceedings of the 21st USENIX Conference on Security Symposium, Security 2012, p. 38. USENIX Association, USA (2012)

    Google Scholar 

  30. Jiang, B., Liu, Y., Chan, W.K.: Contractfuzzer: fuzzing smart contracts for vulnerability detection. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, pp. 259–269. Association for Computing Machinery, New York (2018)

    Google Scholar 

  31. Klees, G., Ruef, A., Cooper, B., Wei, S., Hicks, M.: Evaluating fuzz testing. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, pp. 2123–2138. Association for Computing Machinery, New York (2018)

    Google Scholar 

  32. Lemieux, C., Sen, K.: Fairfuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, pp. 475–485. Association for Computing Machinery, New York (2018)

    Google Scholar 

  33. Li, Y., et al.: Cerebro: context-aware adaptive fuzzing for effective vulnerability detection. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, pp. 533–544. ACM, New York (2019)

    Google Scholar 

  34. Livinskii, V., Babokin, D., Regehr, J.: Random testing for C and C++ compilers with YARPGen. Proc. ACM Program. Lang. 4(OOPSLA) (2020)

    Google Scholar 

  35. Lyu, C., et al.: MOPT: optimized mutation scheduling for fuzzers. In: Proceedings of the 28th USENIX Conference on Security Symposium, pp. 1949–1966. USENIX Association, USA (2019)

    Google Scholar 

  36. Lyu, C., et al.: EMS: history-driven mutation for coverage-based fuzzing. In: 29th Annual Network and Distributed System Security Symposium (2022)

    Google Scholar 

  37. Zalewski, M.: American fuzzy lop (2013). https://lcamtuf.coredump.cx/afl/. Accessed 13 Jan 2023

  38. Rash, M.: AFL-COV - AFL fuzzing code coverage (2021). https://github.com/mrash/afl-cov. Accessed 06 Jan 2023

  39. Rigger, M., Su, Z.: Testing database engines via pivoted query synthesis. In: 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020), pp. 667–682. USENIX Association (2020)

    Google Scholar 

  40. Sotiropoulos, T., Chaliasos, S., Atlidakis, V., Mitropoulos, D., Spinellis, D.: Data-oriented differential testing of object-relational mapping systems. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1535–1547 (2021)

    Google Scholar 

  41. Sun, C., Le, V., Su, Z.: Finding and analyzing compiler warning defects. In: Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, pp. 203–213. Association for Computing Machinery, New York (2016)

    Google Scholar 

  42. Sun, C., Le, V., Zhang, Q., Su, Z.: Toward understanding compiler bugs in GCC and LLVM. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, pp. 294–305. Association for Computing Machinery, New York (2016)

    Google Scholar 

  43. Veggalam, S., Rawat, S., Haller, I., Bos, H.: IFuzzer: an evolutionary interpreter fuzzer using genetic programming. In: Askoxylakis, I., Ioannidis, S., Katsikas, S., Meadows, C. (eds.) ESORICS 2016. LNCS, vol. 9878, pp. 581–601. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45744-4_29

    Chapter  Google Scholar 

  44. Wang, J., Chen, B., Wei, L., Liu, Y.: Superion: Grammar-aware greybox fuzzing. In: Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, pp. 724–735. IEEE Press (2019)

    Google Scholar 

  45. Weimer, W., Nguyen, T., Le Goues, C., Forrest, S.: Automatically finding patches using genetic programming. In: Proceedings of the 31st International Conference on Software Engineering, ICSE 2009, pp. 364–374. IEEE, USA (2009)

    Google Scholar 

  46. Wüstholz, V., Christakis, M.: Harvey: a greybox fuzzer for smart contracts. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pp. 1398–1409. Association for Computing Machinery, New York (2020)

    Google Scholar 

  47. Yan, S., Wu, C., Li, H., Shao, W., Jia, C.: Pathafl: path-coverage assisted fuzzing. In: Proceedings of the 15th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2020, pp. 598–609. Association for Computing Machinery, New York (2020)

    Google Scholar 

  48. Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in C compilers. SIGPLAN Not. 46(6), 283–294 (2011)

    Article  Google Scholar 

  49. Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in c compilers. In: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, pp. 283–294. Association for Computing Machinery, New York (2011)

    Google Scholar 

  50. Zubairy, R.: Create a blockchain app for loyalty points with Hyperledger Fabric Ethereum Virtual Machine (2018). Accessed 06 Jan 2023

    Google Scholar 

Download references

Acknowledgments

We thank the anonymous reviewers and the shepherd for their constructive feedback. This work was supported by the European Union programme under grant agreements No. 82735 (Cybersecpro) and No. 82886 (Sentinel).

Author information

Authors and Affiliations

  1. Technical University of Crete, Chania, Greece

    Charalambos Mitropoulos & Sotiris Ioannidis

  2. ETH Zurich, Zürich, Switzerland

    Thodoris Sotiropoulos

  3. University of Athens, Athens, Greece

    Dimitris Mitropoulos

Authors
  1. Charalambos Mitropoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thodoris Sotiropoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sotiris Ioannidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dimitris Mitropoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dimitris Mitropoulos .

Editor information

Editors and Affiliations

  1. University of California, Irvine, CA, USA

    Gene Tsudik

  2. University of Padua, Padua, Italy

    Mauro Conti

  3. Delft University of Technology, Delft, The Netherlands

    Kaitai Liang

  4. Delft University of Technology, Delft, The Netherlands

    Georgios Smaragdakis

Appendices

A Bugs Previously Unknown to Solidity Developers

In the table below, we enumerate all bugs that (1) fuzzol identified and (2) were unknown to Solidity developers.

Table 3. Category and references of the bugs that were unknown to Solidity developers.
Full size table

B Differences Between FUZZOL and Fuzzers Included in Our Evaluation

In the following table, we present the main design differences between fuzzol and the related fuzzers included in our evaluation. Note that all fuzzers are AFL-based.

Table 4. Point-to-point comparison between fuzzol and the fuzzers included in our evaluation. GB: grammar-blind, GA: grammar-aware, TM: text-mutation
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mitropoulos, C., Sotiropoulos, T., Ioannidis, S., Mitropoulos, D. (2024). Syntax-Aware Mutation for Testing the Solidity Compiler. In: Tsudik, G., Conti, M., Liang, K., Smaragdakis, G. (eds) Computer Security – ESORICS 2023. ESORICS 2023. Lecture Notes in Computer Science, vol 14346. Springer, Cham. https://doi.org/10.1007/978-3-031-51479-1_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-51479-1_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-51478-4

  • Online ISBN: 978-3-031-51479-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation