Skip to main content
SpringerLink
Log in

ReSuMo: a regression strategy and tool for mutation testing of solidity smart contracts

  • Research
  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Mutation testing is a powerful test adequacy assessment technique that can guarantee the deployment of deeply scrutinized and more reliable Smart Contract code. However, regularly evaluating the test suite during the evolution of a project can be unfeasible due to the high costs of the assessment. This paper presents ReSuMo, the first regression mutation testing approach and tool for Solidity Smart Contracts. ReSuMo uses a static and file-level technique to select a subset of Smart Contracts to mutate, and a subset of test files to re-run during a regression mutation testing campaign. After each mutation testing run, ReSuMo incrementally updates the results using the test outcomes registered on the previous program revision. In this way, it can speed up the mutation testing process for evolving projects, while providing a comprehensive adequacy assessment of the whole test suite.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

The tool that permits the reproduction of the experiments reported in this paper is available from the following Github repository: https://github.com/MorenaBarboni/ReSuMo.

Notes

  1. Proxy pattern:  https://docs.openzeppelin.com/upgrades-plugins/1.x/proxies 

  2. Solidity documentation: https://docs.soliditylang.org/

  3. SuMo repository: https://github.com/MorenaBarboni/SuMo-SOlidity-MUtator

  4. Dependency graph: https://www.npmjs.com/package/dependency-graph

  5. ReSuMo repository: https://github.com/MorenaBarboni/ReSuMo

  6. Mochawesome: https://www.npmjs.com/package/mochawesome

  7. Source Code https://github.com/ConsenSys/solidity-parser-antlr

  8. Liquidity Protocol repository: https://github.com/1inch/liquidity-protocol

  9. Farming repository: https://github.com/1inch/farming

  10. Limit Order Protocol repository: https://github.com/1inch/limit-order-protocol

  11. Flaky test method of Limit Order Protocol: https://github.com/1inch/limit-order-protocol/blob/0c0036df78799680d932b7c097de7d1f23cb63a4/test/Interactions.js#L77

  12. Solidity Coverage repository: https://github.com/sc-forks/solidity-coverage.git

References

  • Andesta, E., Faghih, F., & Fooladgar, M. (2020). Testing smart contracts gets smarter. In: 2020 10th International Conference on Computer and Knowledge Engineering (ICCKE), pp. 405–412. IEEE

  • Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006). Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering, 32(8), 608–624.

    Article  Google Scholar 

  • Barboni, M., Casoni, F., Morichetta, A., & Polini, A. (2022). Resumo: Regression mutation testing for solidity smart contracts. In: International Conference on the Quality of Information and Communications Technology (QUATIC), pp. 61–76. Springer

  • Barboni, M., Morichetta, A., & Polini, A. (2022). Sumo: A mutation testing approach and tool for the ethereum blockchain. Journal of Systems and Software, 193, 111445.

    Article  Google Scholar 

  • Buterin, V. (2013). Ethereum white paper: A next generation smart contract & decentralized application platform

  • Cachia, M. A., Micallef, M., & Colombo, C. (2013). Towards incremental mutation testing. Electronic Notes in Theoretical Computer Science, 294, 2–11.

    Article  Google Scholar 

  • Chapman, P., Xu, D., Deng, L., & Xiong, Y. (2019). Deviant: A mutation testing tool for solidity smart contracts. In: 2019 IEEE International Conference on Blockchain (Blockchain), pp. 319–324. IEEE

  • Chakraborty, P., Shahriyar, R., Iqbal, A., & Bosu, A. (2018). Understanding the software development practices of blockchain projects: a survey. In: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1–10

  • Chen, L., & Zhang, L. (2018). Speeding up mutation testing via regression test selection: An extensive study. In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pp. 58–69. IEEE

  • Corradini, F., Marcelletti, A., Morichetta, A., Polini, A., Re, B., & Tiezzi, F. (2020). Engineering trustable choreography-based systems using blockchain. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 1470–1479

  • Corradini, F., Marcelletti, A., Morichetta, A., Polini, A., Re, B., Scala, E., & Tiezzi, F. (2021). Model-driven engineering for multi-party business processes on multiple blockchains. Blockchain: Research and Applications, 2(3), 100018

  • Frankl, P. G., Weiss, S. N., & Hu, C. (1997). All-uses vs mutation testing: An experimental comparison of effectiveness. Journal of Systems and Software, 38(3), 235–253.

    Article  Google Scholar 

  • Gligoric, M., Eloussi, L., & Marinov, D. (2015). Ekstazi: Lightweight test selection. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, pp. 713–716. IEEE

  • Hartel, P., & Schumi, R. (2020). Mutation testing of smart contracts at scale. In: International Conference on Tests and Proofs, pp. 23–42 (2020). Springer

  • Honig, J. J., Everts, M. H., & Huisman, M. (2019). Practical mutation testing for smart contracts. In: Data Privacy Management, Cryptocurrencies and Blockchain Technology, pp. 289–303. Springer, Cham

  • Inozemtseva, L., & Holmes, R. (2014). Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International Conference on Software Engineering, pp. 435–445

  • Legunsen, O., Shi, A., & Marinov, D. (2017). Starts: Static regression test selection. In: 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 949–954. IEEE

  • Li, Z., Wu, H., Xu, J., Wang, X., Zhang, L., & Chen, Z. (2019). Musc: A tool for mutation testing of ethereum smart contract. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1198–1201. IEEE

  • Ma, W., Laurent, T., Ojdanić, M., Chekam, T. T., Ventresque, A., & Papadakis, M. (2020). Commit-aware mutation testing. In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 394–405. IEEE

  • Palechor, L., & Bezemer, C. P. (2022). How are solidity smart contracts tested in open source projects? an exploratory study. In: Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test, pp. 165–169

  • Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. L., & Harman, M. (2019). Mutation testing advances: An analysis and survey. Advances in Computers, vol. 112, pp. 275–378. Elsevier

  • Papadakis, M., Jia, Y., Harman, M., & Le Traon, Y. (2015). Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1, pp. 936–946. IEEE

  • Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2), 131–164.

    Article  Google Scholar 

  • Sánchez, A. B., Delgado-Pérez, P., Medina-Bulo, I., & Segura, S. (2022). Mutation testing in the wild: findings from github. Empirical Software Engineering, 27(6), 132.

    Article  Google Scholar 

  • Shi, A., Bell, J., & Marinov, D. (2019). Mitigating the effects of flaky tests on mutation testing. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 112–122

  • Tengeri, D., Vidács, L., Beszédes, Á., Jász, J., Balogh, G., Vancsics, B., & Gyimóthy, T. (2016). Relating code coverage, mutation score and test suite reducibility to defect density. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 174–179. IEEE

  • Vacca, A., Di Sorbo, A., Visaggio, C. A., & Canfora, G. (2021). A systematic literature review of blockchain and smart contract development: Techniques, tools, and open challenges. Journal of Systems and Software, 174, 110891.

    Article  Google Scholar 

  • Zhang, L., Kim, M., & Khurshid, S. (2011). Localizing failure-inducing program edits based on spectrum information. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM), pp. 23–32. IEEE

  • Zhang, L., Marinov, D., Zhang, L., & Khurshid, S. (2012). Regression mutation testing. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 331–341

Download references

Author information

Authors and Affiliations

  1. University of Camerino, Camerino, 62032, Italy

    Morena Barboni, Andrea Morichetta, Andrea Polini & Francesco Casoni

Authors
  1. Morena Barboni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Morichetta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Polini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francesco Casoni
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors confirm their contribution to the paper as follows: Morena Barboni, Andrea Morichetta, and Andrea Polini designed the study, analyzed the results, and prepared the manuscript. Morena Barboni implemented the ReSuMo tool, while Francesco Casoni devised the test and contract selection strategy. All authors reviewed the results and approved the final version of the manuscript.

Corresponding author

Correspondence to Morena Barboni.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barboni, M., Morichetta, A., Polini, A. et al. ReSuMo: a regression strategy and tool for mutation testing of solidity smart contracts. Software Qual J 32, 225–253 (2024). https://doi.org/10.1007/s11219-023-09637-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-023-09637-1

Keywords

Search

Navigation