Skip to main content
SpringerLink
Log in

Assessing Black-box Test Case Generation Techniques for Microservices

  • Conference paper
  • First Online:
Quality of Information and Communications Technology (QUATIC 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1621))

  • 703 Accesses

Abstract

Testing of microservices architectures (MSA) – today a popular software architectural style - demands for automation in its several tasks, like tests generation, prioritization and execution. Automated black-box generation of test cases for MSA currently borrows techniques and tools from the testing of RESTful Web Services.

This paper: i) proposes the uTest stateless pairwise combinatorial technique (and its automation tool) for test cases generation for functional and robustness microservices testing, and ii) experimentally compares - with three open-source MSA used as subjects - four state-of-the-art black-box tools conceived for Web Services, adopting evolutionary-, dependencies- and mutation-based generation techniques, and the proposed uTest combinatorial tool.

The comparison shows little differences in coverage values; uTest pairwise testing achieves better average failure rate with a considerably lower number of tests. Web Services tools do not perform for MSA as well as a tester might expect, highlighting the need for MSA-specific techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.openapis.org.

  2. 2.

    Our comparison does not include the techniques in references [6, 9, 10], and [11], whose tools are probably still at proof-of-concept stage. For instance, like Corradini et al. [12] we did not manage to run RESTest on our case studies. However, differently from [12], our comparison includes Evomaster, besides RestTestGen, Restler, bBOXRT.

  3. 3.

    https://github.com/uDEVOPS2020/uTest.

  4. 4.

    https://github.com/FudanSELab/train-ticket.

  5. 5.

    https://github.com/microservices-demo/microservices-demo.

  6. 6.

    https://github.com/microservices-patterns/ftgo-application.

References

  1. Lewis, J., Fowler, M.: Microservices - a definition of this new architectural term. http://martinfowler.com/articles/microservices.html (2014)

  2. Arcuri, A.: Automated black- and white-box testing of RESTful APIs with EvoMaster. IEEE Softw. 38(3), 72–78 (2021)

    Article  Google Scholar 

  3. Ma, S., Fan, C., Chuang, Y., Lee, W., Lee, S., Hsueh, N.: Using service dependency graph to analyze and test microservices. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), vol. 02, pp. 81–86 (2018)

    Google Scholar 

  4. Corradini, D., Zampieri, A., Pasqua, M., Viglianisi, E., Dallago, M., Ceccato, M.: Automated black-box testing of nominal and error scenarios in RESTful APIs. Softw. Test. Verification Reliab. 32, e1808 (2022)

    Google Scholar 

  5. Atlidakis, V., Godefroid, P., Polishchuk, M.: RESTler: Stateful rest API fuzzing. In: IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 748–758. IEEE (2019)

    Google Scholar 

  6. Martin-Lopez, A., Segura, S., Ruiz-Cortés, A.: RESTest: black-box constraint-based testing of RESTful web APIs. In: Kafeza, E., Benatallah, B., Martinelli, F., Hacid, H., Bouguettaya, A., Motahari, H. (eds.) ICSOC 2020. LNCS, vol. 12571, pp. 459–475. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65310-1_33

    Chapter  Google Scholar 

  7. Martin-Lopez, A., Segura, S., Ruiz-Cortés, A.: Test coverage criteria for RESTful web APIs. In: Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (A-TEST), pp. 15–21. ACM (2019)

    Google Scholar 

  8. Laranjeiro, N., Agnelo, J., Bernardino, J.: A black box tool for robustness testing of rest services. IEEE Access 9, 24738–24754 (2021)

    Article  Google Scholar 

  9. Ed-douibi, H., Izquierdo, J.L.C., Cabot, J.: Automatic generation of test cases for REST APIs: a specification-based approach. In: IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC), pp. 181–190. IEEE (2018)

    Google Scholar 

  10. Karlsson, S., Čaušević, A., Sundmark. D.: QuickREST: property-based test generation of OpenAPI-described RESTful APIs. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 131–141. IEEE (2020)

    Google Scholar 

  11. Bania, O., Florea, D., Gyalai, R., Curiac, D.: Automated specification-based testing of REST APIs. Sensors 21(16), 5375 (2021)

    Article  Google Scholar 

  12. Corradini, D., Zampieri, A., Pasqua, M., Ceccato, M.: Empirical comparison of black-box test case generation tools for RESTful APIs. In: 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM), pp. 226–236. IEEE (2021)

    Google Scholar 

  13. Cohen, D.M., Dalal, S.R., Parelius, J., Patton, G.C.: The combinatorial design approach to automatic test generation. IEEE Software 13(5), 83–88 (1996)

    Article  Google Scholar 

  14. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 1–29 (2011)

    Article  Google Scholar 

  15. Hu, L., Wong, W.E., Kuhn, D.R., Kacker, R.N.: How does combinatorial testing perform in the real world: an empirical study. Empirical Software Eng. 25(4), 2661–2693 (2020). https://doi.org/10.1007/s10664-019-09799-2

    Article  Google Scholar 

  16. Pezzè, M., Young, M.: Software Testing and Analysis - Process, Principles and Techniques. Wiley, Hoboken (2007)

    MATH  Google Scholar 

  17. Bertolino, A., De Angelis, G., Guerriero, A., Miranda, B., Pietrantuono, R.. Russo, S.: DevOpRET: continuous reliability testing in DevOps. J. Softw. Evol. Process. e2298 (2020). smr.2298

    Google Scholar 

  18. Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., Ding, D.: Fault analysis and debugging of microservice systems: industrial survey, benchmark system, and empirical study. IEEE Trans. Softw. Eng. 47(2), 243–260 (2021)

    Article  Google Scholar 

  19. Richardson, C.: Microservices Patterns. Manning Publications, Shelter Island (2018)

    Google Scholar 

  20. Portswigger: burp suite. https://portswigger.net/burp

  21. Corradini, D., Zampieri, A.. Pasqua, M., Ceccato, M.: Restats: a test coverage tool for RESTful APIs. CoRR, abs/2108.08209 (2021)

    Google Scholar 

  22. Indrasiri, K., Siriwardena, P.: Microservices for the Enterprise: Designing, Developing, and Deploying, 1st edn. Apress, USA (2018)

    Google Scholar 

  23. Waseem, M., Liang, P., Shahin, M., Di Salle, A., Márquez, G.: Design, monitoring, and testing of microservices systems: the practitioners’ perspective. J. Syst. Softw. 182, 111061 (2021)

    Article  Google Scholar 

  24. Cinque, M., Della Corte, R., Pecchia, A.: Microservices monitoring with event logs and black box execution tracing. IEEE Trans. Serv. Comput. 15(1), 294–307 (2022)

    Article  Google Scholar 

  25. Zhou, X., et al.: Poster: benchmarking microservice systems for software engineering research. In: 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion), pp. 323–324. IEEE (2018)

    Google Scholar 

Download references

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 871342.

Author information

Authors and Affiliations

  1. DIETI, Università Degli Studi di Napoli Federico II, Napoli, Italy

    Luca Giamattei, Antonio Guerriero, Roberto Pietrantuono & Stefano Russo

Authors
  1. Luca Giamattei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Guerriero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roberto Pietrantuono
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefano Russo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luca Giamattei .

Editor information

Editors and Affiliations

  1. University of Malaga, Málaga, Spain

    Antonio Vallecillo

  2. Leiden University, Leiden, The Netherlands

    Joost Visser

  3. University of Castila-La Mancha, Ciudad Real, Spain

    Ricardo Pérez-Castillo

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Giamattei, L., Guerriero, A., Pietrantuono, R., Russo, S. (2022). Assessing Black-box Test Case Generation Techniques for Microservices. In: Vallecillo, A., Visser, J., Pérez-Castillo, R. (eds) Quality of Information and Communications Technology. QUATIC 2022. Communications in Computer and Information Science, vol 1621. Springer, Cham. https://doi.org/10.1007/978-3-031-14179-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14179-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14178-2

  • Online ISBN: 978-3-031-14179-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation