Advertisement

Comparing Graph-Based Algorithms to Generate Test Cases from Finite State Machines

  • Matheus Monteiro MarianoEmail author
  • Érica Ferreira de Souza
  • André Takeshi Endo
  • Nandamudi Lankalapalli Vijaykumar
Article
  • 31 Downloads

Abstract

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Keywords

Software testing Model based testing Finite state machine Graph-based algorithms 

Notes

Acknowledgments

The authors would like to thank CAPES for the financial support and Dr. Rafael Santos for his help with the analyses. A.T. Endo and E.F. Souza are partially financially supported by CNPq/Brazil (grant numbers 420363/2018-1 and 432247/2018-1, respectively).

References

  1. 1.
    Ali S, Briand LC, Rehman MJ-U, Asghar H, Iqbal MZZ, Nadeem A (2007) A state-based approach to integration testing based on uml models. Inf Softw Technol 49(11-12):1087–1106CrossRefGoogle Scholar
  2. 2.
    Amaral A (2005) Geração de casos de testes para sistemas especificados em statecharts. (INPE-14215-TDI/1116)Google Scholar
  3. 3.
    Ammann P, Offutt J (2008) Introduction to Software Testing. Cambridge University Press, Cambridge. [S.l.]CrossRefGoogle Scholar
  4. 4.
    Anbunathan R, Basu A (2013) Dataflow test case generation from uml class diagrams. In: Proc. IEEE international conference on computational intelligence and computing research, 2013. IEEE, pp 1–?9Google Scholar
  5. 5.
    Arantes AO, de Santiago VA, Vijaykumar NL, De Souza EF (2014) Tool support for generating model-based test cases via web. Int J Web Eng Technol iiWAS 9(1):62–96CrossRefGoogle Scholar
  6. 6.
    Barmi ZA, Ebrahimi AH, Feldt R (2011) Alignment of requirements specification and testing: a systematic mapping study. In: Proc. international conference on software testing, verification and validation workshops, 2011. IEEE, pp 476?-485Google Scholar
  7. 7.
    Belli F, Beyazit M, Takagi T, Furukawa Z (2012) Model-based mutation testing using pushdown automata. IEICE Trans Inf Syst 95(9):2211–2218CrossRefGoogle Scholar
  8. 8.
    Belli F, Beyazit M, Takagi T, Furukawa Z (2013) Mutation testing of “go-back” functions based on pushdown automata. In: Proc. international conference on software testing, verification and validation, 2011. IEEE, pp 249–258Google Scholar
  9. 9.
    Belli F, Endo AT, Linschulte M, Simao A (2014) A holistic approach to model-based testing of web service compositions. Software: Practice and Experience 44(2):201–234Google Scholar
  10. 10.
    Belli F, Hollmann A, Kleinselbeck M (2009) A graph-model-based testing method compared with the classification tree method for test case generation. In: Proc. international conference on secure software integration and reliability improvement, 3., 2009. IEEE, pp 193–200Google Scholar
  11. 11.
    Bondy J, Murty U (2008) Graduate texts in mathematics: graph theory. Springer, USAGoogle Scholar
  12. 12.
    Brito RC, Martendal DM, de Oliveira HEM (2003) Máquinas de estados finitos de mealy e mooreGoogle Scholar
  13. 13.
    Broy M, Jonsson B, Katoen JP, Leucker M, Pretschner A (2005) Model-based testing of reactive systems. Springer, BerlinCrossRefGoogle Scholar
  14. 14.
    Cartaxo EG, Machado PD, Neto FGO (2011) On the use of a similarity function for test case selection in the context of model-based testing. Software Testing, Verification and Reliability 21(2):75–100CrossRefGoogle Scholar
  15. 15.
    Chow TS (1978) Testing software design modeled by finite-state machines. IEEE Trans Softw Eng SE-4 (3):178–187CrossRefGoogle Scholar
  16. 16.
    Dang T, Nahhal T (2009) Coverage-guided test generation for continuous and hybrid systems. Formal Methods in System Design 34(2):183–213CrossRefGoogle Scholar
  17. 17.
    Devroey X, Perrouin G, Schobbens P-Y (2014) Abstract test case generation for behavioural testing of software product lines, 86–93Google Scholar
  18. 18.
    Dorofeeva R, El-Fakih K, Maag S, Cavalli AR, Yevtushenko N (2010) Fsm-based conformance testing methods: a survey annotated with experimental evaluation. Inf Softw Technol 52(12):1286–1297CrossRefGoogle Scholar
  19. 19.
    Dorofeeva R, El-Fakih K, Yevtushenko N (2005) An improved conformance testing method. In: Proc. international conference on formal techniques for networked and distributed systems, 2005, pp 204–218Google Scholar
  20. 20.
    Edmonds J, Johnson EL (1973) Matching, euler tours and the chinese postman. Math Program 5(1):88–124MathSciNetCrossRefGoogle Scholar
  21. 21.
    Endo AT, Simao A (2013) Evaluating test suite characteristics, cost, and effectiveness of fsm-based testing methods. Inf Softw Technol 55(6):1045–1062CrossRefGoogle Scholar
  22. 22.
    Felizardo KR, Mendes E, Kalinowski M, Souza EF, Vijaykumar NL (2016) Using forward snowballing to update systematic reviews in software engineering. In: International symposium on empirical software engineering and measurement (ESEM)Google Scholar
  23. 23.
    Gill A, et al. (1962) Introduction to the theory of finite-state machines. [S.I.] McGraw-Hill, New YorkzbMATHGoogle Scholar
  24. 24.
    Gonenc G (1970) A method for the design of fault detection experiments. IEEE Trans Comput 100(6):551–558CrossRefGoogle Scholar
  25. 25.
    Gurbuz HG, Tekinerdogan B (2017) Model-based testing for software safety: a systematic mapping study. Softw Qual J 26(4):1–46Google Scholar
  26. 26.
    Hessel A, Pettersson P (2007) A global algorithm for model-based test suite generation. Electronic Notes in Theoretical Computer Science 190(2):47–59CrossRefGoogle Scholar
  27. 27.
    IEEE (2005) Ieee standard for software verification and validation. IEEE Std 1012-2004 (Revision of IEEE Std 1012-1998), pp 1–110Google Scholar
  28. 28.
    Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. Trans Softw Eng 37(5):649–678CrossRefGoogle Scholar
  29. 29.
    Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014. ACM, pp 654– 665Google Scholar
  30. 30.
    Khan MU, Iftikhar S, Iqbal MZ, Sherin S (2017) Empirical studies omit reporting necessary details: a systematic literature review of reporting quality in model based testing. Computer Standards & InterfacesGoogle Scholar
  31. 31.
    Kitchenham BA (2007) Guidelines for performing systematic literature reviews in software engineering. Keele University, Durham. Technical Report EBSE-2007-01Google Scholar
  32. 32.
    Lee D, Yannakakis M (1996) Principles and methods of testing finite state machines-a survey. Proc IEEE 84(8):1090–1123CrossRefGoogle Scholar
  33. 33.
    Li N, Offutt J (2017) Test oracle strategies for model-based testing. IEEE Trans Softw Eng 43(4):372–395CrossRefGoogle Scholar
  34. 34.
    Majeed S, Ryu M (2016) Model-based replay testing for event-driven software. In: Proceedings..., pages 1527–1533. Annual ACM Symposium on Applied Computing, 31., 2016, ACMGoogle Scholar
  35. 35.
    Mariano MM, Souza ÉF, Endo AT, Vijaykumar NL (2016) A comparative study of algorithms for generating switch cover test sets. In: Anais..., Maceió. Simpósio Brasileiro de Qualidade de Software, 15., 2016Google Scholar
  36. 36.
    Mariano MM, Souza EF, Endo AT, Vijaykumar NL (2019a) Analyzing graph-based algorithms employed to generate test cases from finite state machines. In: 2019 IEEE Latin American test symposium (LATS), pp 1–6Google Scholar
  37. 37.
    Mariano MM, Souza EF, Endo AT, Vijaykumar NL (2019) Identifying approaches to generate test cases in model-based testing: a systematic mapping.. In: Proceedings of the Iberoamerican conference on software engineering, 22., 2019, pp 1–14, HavanaGoogle Scholar
  38. 38.
    Mathur AP (2008) Foundations of software testing. [S.l.] Pearson Education, LondonGoogle Scholar
  39. 39.
    Myers GJ, Sandler C, Badgett T (1976) The art of software testing. [S.l.] Wiley, New YorkGoogle Scholar
  40. 40.
    Naito S (1981) Fault detection for sequential machines by transition-toursGoogle Scholar
  41. 41.
    Neumann F (2004) Expected runtimes of evolutionary algorithms for the eulerian cycle problem. In: Proceedings..., pages 904–910. Congress on Evolutionary Computation, 2001. IEEEGoogle Scholar
  42. 42.
    Papadakis M, Shin D, Yoo S, Bae D (2018) Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults. In: Proceedings of the 40th international conference on software engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pp 537–548. ACMGoogle Scholar
  43. 43.
    Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. IEEE Trans Vis Comput Graph 64:1–18Google Scholar
  44. 44.
    Pimont S, Rault J-C (1976) A software reliability assessment based on a structural and behavioral analysis of programs. In: Proc. international conference on software engineering, 2., 1976, IEEE, pp 486–491Google Scholar
  45. 45.
    Pinheiro AC, Simão A, Ambrosio AM (2014) Fsm-based test case generation methods applied to test the communication software on board the itasat university satellite: a case study. J Aerosp Technol Manag 6(4):447–461CrossRefGoogle Scholar
  46. 46.
    Pressman RS (2005) Software engineering: a practitioner’s approach. [S.l.]: Palgrave MacmillanGoogle Scholar
  47. 47.
    Proch S, Mishra P (2014) Directed test generation for hybrid systems. In: Proceedings..., pages 156–162. International Symposium on Quality Electronic Design, 15., 2014Google Scholar
  48. 48.
    Santiago V, Vijaykumar NL, Guimarães D, Amaral AS, Ferreira É (2008) An environment for automated test case generation from statechart-based and finite state machine-based behavioral models. In: Proc. Software Testing Verification and Validation Workshop, 2008. ICSTW’08. IEEE International Conference on, pp 63–72Google Scholar
  49. 49.
    Satpathy M, Yeolekar A, Peranandam P, Ramesh S (2012) Efficient coverage of parallel and hierarchical stateflow models for test case generation. Software Testing, Verification and Reliability 22(7):457–479CrossRefGoogle Scholar
  50. 50.
    Siavashi F, Truscan D (2014) Environment modeling in model-based testing: concepts, prospects and research challenges: a systematic literature review. In: Proceedings..., pages 30:1–30:6. Proc. International Conference on Evaluation and Assessment in Software Engineering, 19., 2015. ACM, New YorkGoogle Scholar
  51. 51.
    Sidhu DP, Leung T-K (1989) Formal methods for protocol testing: a detailed study. IEEE Trans Softw Eng 15(4):413–426CrossRefGoogle Scholar
  52. 52.
    Simão A., Petrenko A, Maldonado J (2009) Comparing finite state machine test coverage criteria. 3(2): 91–105Google Scholar
  53. 53.
    Souza ÉF, Santiago VA JR, Guimaraes D, Vijaykumar NL (2008) Evaluation of test criteria for space application software modeling in statecharts. In: Proc. international conference on computational intelligence for modelling control and automation, p 2008Google Scholar
  54. 54.
    Souza ÉF, Santiago VA JR, Vijaykumar NL (2017) H-switch cover: a new test criterion to generate test case from finite state machines. Softw Qual J 25(2):373–405CrossRefGoogle Scholar
  55. 55.
    Souza SDRSD (2000) Validação de especificações de sistemas reativos: definição e análise de critérios de testeGoogle Scholar
  56. 56.
    Vu T-D, Hung PN, Nguyen V-H (2015) A method for automated test data generation from sequence diagrams and object constraint language, pp 335–341Google Scholar
  57. 57.
    Wang W, Sampath S, Lei Y, Kacker R, Kuhn R, Lawrence J (2016) Using combinatorial testing to build navigation graphs for dynamic web applications. Software testing. Verification and Reliability 26(4):318–346CrossRefGoogle Scholar
  58. 58.
    Yao J, Wang Z, Yin X, Shiyz X, Wu J (2014) Formal modeling and systematic black-box testing of sdn data plane. In: Proc. international conference on network protocols, 22., 2014, IEEE, pp 179–190Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Authors and Affiliations

  1. 1.Laboratory of Computing and Applied MathematicsNational Institute for Space ResearchSão José dos CamposBrazil
  2. 2.Department of ComputerFederal University of TechnologyCornélio ProcópioBrazil
  3. 3.ICT - Institute of Science & TechnologyFederal University of São PauloSão José dos CamposBrazil
  4. 4.LABACNational Institute for Space ResearchSão José dos CamposBrazil

Personalised recommendations