Software & Systems Modeling

, Volume 15, Issue 3, pp 631–645 | Cite as

Comprehensive analysis of FBD test coverage criteria using mutants

  • Donghwan Shin
  • Eunkyoung Jee
  • Doo-Hwan Bae
Regular Paper


Function block diagram (FBD), a graphical modeling language for programmable logic controllers, has been widely used to implement safety critical system software such as nuclear reactor protection systems. With the growing importance of structural testing for FBD models, structural test coverage criteria for FBD models have been proposed and evaluated using mutation analysis in our previous work. We extend the previous work by comprehensively analyzing the relationships among fault detection effectiveness, test suite size, and coverage level through several research questions. We generate a large number of test suites achieving an FBD test coverage ranging from 0 to 100 %, and we also generate many artificial faults (i.e. mutants) for the FBD models. Our analysis results show that the fault detection effectiveness of the FBD coverage criteria increases with increasing coverage levels, and the coverage criteria are highly effective at detecting faults in all subject models. Furthermore, the test suites generated with the FBD coverage criteria are more effective and efficient than the randomly generated test suites. The FBD coverage criteria are strong at detecting faults in Boolean edges, while relatively weak at detecting wrong constants in FBD models. Empirical knowledge regarding our experiments provide the validity of using the FBD coverage criteria, and therefore, of FBD model-based testing.


FBD model-based testing Test coverage criteria Mutation analysis 



The authors thank the anonymous reviewers for their valuable comments and suggestions to improve the quality of the paper. This research was partially supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2013R1A1A2058618) and the IT R&D program of MSIP/KEIT [10044457, Development of Autonomous Intelligent Collaboration Framework for Knowledge Bases and Smart Devices].


  1. 1.
    Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, New York (2011)Google Scholar
  2. 2.
    Ould, M.A., Unwin, C.: Testing in Software Development. Cambridge University Press, Cambridge (1986)Google Scholar
  3. 3.
    Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, UK (2008)CrossRefzbMATHGoogle Scholar
  4. 4.
    Beizer, B.: Software Testing Techniques. Dreamtech Press, India (2003)Google Scholar
  5. 5.
    IEC Standard 1131–3, Part3: Programming Languages (1993)Google Scholar
  6. 6.
    Doosan Heavy Industry and Construction: Software Design Specification for the Bistable Processor of the Reactor Protection System. KNICS-RPS-SDS231-01, Rev. 01 (2006)Google Scholar
  7. 7.
    NRC. Regulatory guide 1.171, software unit testing for digital computer software used in safety systems of nuclear power plants. Tech. rep. (1997)Google Scholar
  8. 8.
    NRC, Draft regulatory guide dg-1208, proposed revision 1 of regulatory guide 1.171. Tech. rep. (2012)Google Scholar
  9. 9.
    Jee, E., Yoo, J., Cha, S., Bae, D.: A data flow-based structural testing technique for fbd programs. Inf. Softw. Technol. 51(7), 1131–1139 (2009)CrossRefGoogle Scholar
  10. 10.
    Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S.: Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32(8), 608–624 (2006)CrossRefGoogle Scholar
  11. 11.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)CrossRefGoogle Scholar
  12. 12.
    King, K.N., Offutt, A.J.: A fortran language system for mutation-based software testing. Softw. Pract. Exp. 21(7), 685–718 (1991)CrossRefGoogle Scholar
  13. 13.
    Shin, D., Jee, E., Bae, D.-H.: Empirical evaluation on fbd model-based test coverage criteria using mutation analysis. In: Model Driven Engineering Languages and Systems, pp. 465–479. Springer, Berlin (2012)Google Scholar
  14. 14.
    Hutchins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments of the effectiveness of dataflow-and controlflow-based test adequacy criteria. In: Proceedings of the 16th international conference on Software engineering, pp. 191–200. IEEE Computer Society Press, Silver Spring (1994)Google Scholar
  15. 15.
    Li, N., Praphamontripong, U., Offutt, J.: An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In: International Conference on Software Testing, Verification and Validation Workshops, 2009. ICSTW’09, pp. 220–229. IEEE, New York (2009)Google Scholar
  16. 16.
    Offutt, J., Abdurazik, A.: Generating tests from uml specifications. In: UML99-The Unified Modeling Language, pp. 416–429. Springer, Berlin (1999)Google Scholar
  17. 17.
    Kim, Y.G., Hong, H.S., Bae, D.-H., Cha, S.-D.: Test cases generation from uml state diagrams. In: Software, IEE Proceedings-, vol. 146, pp. 187–192. IET (1999) Google Scholar
  18. 18.
    Rountev, A., Kagan, S., Sawin, J.: Coverage criteria for testing of object interactions in sequence diagrams. In: Fundamental Approaches to Software Engineering, pp. 289–304. Springer, Berlin (2005)Google Scholar
  19. 19.
    Briand, L., Labiche, Y., Wang, Y.: Using simulation to empirically investigate test coverage criteria based on statechart. In: Proceedings of the 26th International Conference on Software Engineering, pp. 86–95. IEEE Computer Society, Berlin (2004)Google Scholar
  20. 20.
    Lakehal, A., Parissis, I.: Structural coverage criteria for lustre/scade programs. Softw. Test. Verif. Reliab. 19(2), 133–154 (2009)CrossRefGoogle Scholar
  21. 21.
    Papailiopoulou, V., Rajan, A., Parissis, I.: Structural test coverage criteria for integration testing of lustre/scade programs. In: Salaün, G., Schätz, B. (eds.) Formal Methods for Industrial Critical Systems, pp. 85–101. Springer, Berlin (2011)Google Scholar
  22. 22.
    Alyokhin, V., Elbel, B., Rothfelder, M., Pretschner, A.: Coverage metrics for continuous function charts. In: 15th International Symposium on Software Reliability Engineering, 2004. ISSRE 2004, pp. 257–268. IEEE, New York (2004)Google Scholar
  23. 23.
    Frankl, P.G., Weyuker, E.J.: A formal analysis of the fault-detecting ability of testing methods. IEEE Trans. Softw. Eng. 19(3), 202–213 (1993)CrossRefGoogle Scholar
  24. 24.
    Zhu, H.: A formal analysis of the subsume relation between software test adequacy criteria. IEEE Trans. Softw. Eng. 22(4), 248–255 (1996)CrossRefGoogle Scholar
  25. 25.
    Zhu, H., Hall, P.A., May, J.H.: Software unit test coverage and adequacy. ACM Comput. Surv. (CSUR) 29(4), 366–427 (1997)CrossRefGoogle Scholar
  26. 26.
    Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In Software Engineering? ESEC/FSE99, pp. 146–162. Springer, Berlin (1999)Google Scholar
  27. 27.
    Rayadurgam, S., Heimdahl, M.P.E.: Coverage based test-case generation using model checkers. In: Engineering of Computer Based Systems, 2001. ECBS 2001. Proceedings. Eighth Annual IEEE International Conference and Workshop on the, pp. 83–91. IEEE, New York (2001)Google Scholar
  28. 28.
    Staats, M., Gay, G., Whalen, M., Heimdahl, M.: On the danger of coverage directed test case generation. In: Fundamental Approaches to Software Engineering, pp. 409–424. Springer, Berlin (2012)Google Scholar
  29. 29.
    Heimdahl, M.P., George, D.: Test-suite reduction for model based tests: Effects on test quality and implications for testing. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering, pp. 176–185. IEEE Computer Society, Silver Spring (2004)Google Scholar
  30. 30.
    Arcuri, A.: A theoretical and empirical analysis of the role of test sequence length in software testing for structural coverage. IEEE Trans. Softw. Eng. 38(3), 497–519 (2012)CrossRefGoogle Scholar
  31. 31.
    Mader, A.: A classification of plc models and applications. In: Boel, R., Stremersch, G. (eds.) Discrete Event Systems, pp. 239–246. Springer, Berlin (2000)Google Scholar
  32. 32.
    Oh, Y., Yoo, J., Cha, S., Seong Son, H.: Software safety analysis of function block diagrams using fault trees. Reliab. Eng. Syst. Saf. 88(3), 215–228 (2005)CrossRefGoogle Scholar
  33. 33.
    Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. (TOSEM) 5(2), 99–118 (1996)CrossRefGoogle Scholar
  34. 34.
    De Moura, L., Bjørner, N.: Z3: An efficient smt solver. In: Tools and Algorithms for the Construction and Analysis of Systems, pp. 337–340. Springer, Berlin (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Department of Computer ScienceKAISTDaejeonRepublic of Korea

Personalised recommendations