Empirical Evaluation on FBD Model-Based Test Coverage Criteria Using Mutation Analysis

  • Donghwan Shin
  • Eunkyoung Jee
  • Doo-Hwan Bae
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7590)


Function Block Diagram (FBD), one of the PLC programming languages, is a graphical modeling language which has been increasingly used to implement safety-critical software such as nuclear reactor protection software. With increased importance of structural testing for FBD models, FBD model-based test coverage criteria have been introduced. In this paper, we empirically evaluate the fault detection effectiveness of the FBD coverage criteria using mutation analysis. We produce 1800 test suites satisfying the FBD criteria and generate more than 600 mutants automatically for the target industrial FBD models. Then we evaluate mutant detection of the test suites to assess the fault detection effectiveness of the coverage criteria. Based on the experimental results, we analyze strengths and weaknesses of the FBD coverage criteria, and suggest possible improvements for the test coverage criteria.


Function block diagram mutation analysis test coverage criteria 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    International Electrotechnical Commission: Programmable controllers : Part 3: Programming languages. IEC, Geneva (2003)Google Scholar
  2. 2.
    Doosan Heavy Industry and Construction: KNICS-RPS-SDS231-01, Rev. 01., Software Design Specification for the Bistable Processor of the Reactor Protection System (2006)Google Scholar
  3. 3.
    USNRC: Software Unit Testing for Digital Computer Software Used in Safety Systems of Nuclear Power Plants, Regulatory Guide 1.171 (1997)Google Scholar
  4. 4.
    Jee, E., Yoo, J., Cha, S., Bae, D.: A data flow-based structural testing technique for FBD programs. Information and Software Technology 51, 1131–1139 (2009)CrossRefGoogle Scholar
  5. 5.
    Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S.: Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. IEEE Transactions on Software Engineering 32, 608–624 (2006)CrossRefGoogle Scholar
  6. 6.
    King, K.N., Offutt, A.J.: A fortran language system for mutation-based software testing. Software: Practice and Experience 21, 685–718 (1991)CrossRefGoogle Scholar
  7. 7.
    Jia, Y., Harman, M.: An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 649–678 (2011)CrossRefGoogle Scholar
  8. 8.
    Hutchins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In: 16th International Conference on Software Engineering, pp. 191–200. IEEE Computer Society Press, Los Alamitos (1994)CrossRefGoogle Scholar
  9. 9.
    Li, N., Praphamontripong, U., Offutt, A.J.: An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In: International Conference on Software Testing, Verification and Validation Workshops, pp. 220–229. IEEE Press, New York (2009)CrossRefGoogle Scholar
  10. 10.
    Offutt, J., Abdurazik, A.: Generating Tests from UML Specifications. In: France, R., Rumpe, B. (eds.) UML 1999. LNCS, vol. 1723, pp. 416–429. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  11. 11.
    Andrews, A., France, R., Ghosh, S., Craig, G.: Test adequacy criteria for UML design models. Software Testing, Verification and Reliability 13, 95–127 (2003)CrossRefGoogle Scholar
  12. 12.
    Mcquillan, J.A., Power, J.F.: A survey of UML-based coverage criteria for software testing. Department of Computer Science. NUI Maynooth, Co. Kildare, Ireland (2005)Google Scholar
  13. 13.
    Lakehal, A., Parissis, I.: Structural test coverage criteria for lustre programs. In: 10th International Workshop on Formal Methods for Industrial Critical Systems, pp. 35–43. ACM Press, New York (2005)CrossRefGoogle Scholar
  14. 14.
    Mader, A.: A classification of PLC models and applications. In: 5th International Workshop on Discrete Event Systems –Discrete Event Systems, Analysis and Control, pp. 239–247. Springer, Heidelberg (2000)Google Scholar
  15. 15.
    Jee, E.: A Data Flow-Based Structural Testing Technique for FBD Programs. Ph.D Thesis. KAIST Press, Republic of Korea (2009)Google Scholar
  16. 16.
    Yices SMT solver,
  17. 17.
    Oh, Y., Yoo, J., Cha, S., Seong Son, H.: Software safety analysis of function block diagrams using fault trees. Reliability Engineering & System Safety 88, 215–228 (2005)CrossRefGoogle Scholar
  18. 18.
    Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology 5, 99–118 (1996)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Donghwan Shin
    • 1
  • Eunkyoung Jee
    • 1
  • Doo-Hwan Bae
    • 1
  1. 1.Dept. of Computer ScienceKAISTDaejeonRepublic of Korea

Personalised recommendations