Empirical Evaluation on FBD Model-Based Test Coverage Criteria Using Mutation Analysis
Function Block Diagram (FBD), one of the PLC programming languages, is a graphical modeling language which has been increasingly used to implement safety-critical software such as nuclear reactor protection software. With increased importance of structural testing for FBD models, FBD model-based test coverage criteria have been introduced. In this paper, we empirically evaluate the fault detection effectiveness of the FBD coverage criteria using mutation analysis. We produce 1800 test suites satisfying the FBD criteria and generate more than 600 mutants automatically for the target industrial FBD models. Then we evaluate mutant detection of the test suites to assess the fault detection effectiveness of the coverage criteria. Based on the experimental results, we analyze strengths and weaknesses of the FBD coverage criteria, and suggest possible improvements for the test coverage criteria.
KeywordsFunction block diagram mutation analysis test coverage criteria
Unable to display preview. Download preview PDF.
- 1.International Electrotechnical Commission: Programmable controllers : Part 3: Programming languages. IEC, Geneva (2003)Google Scholar
- 2.Doosan Heavy Industry and Construction: KNICS-RPS-SDS231-01, Rev. 01., Software Design Specification for the Bistable Processor of the Reactor Protection System (2006)Google Scholar
- 3.USNRC: Software Unit Testing for Digital Computer Software Used in Safety Systems of Nuclear Power Plants, Regulatory Guide 1.171 (1997)Google Scholar
- 9.Li, N., Praphamontripong, U., Offutt, A.J.: An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In: International Conference on Software Testing, Verification and Validation Workshops, pp. 220–229. IEEE Press, New York (2009)CrossRefGoogle Scholar
- 12.Mcquillan, J.A., Power, J.F.: A survey of UML-based coverage criteria for software testing. Department of Computer Science. NUI Maynooth, Co. Kildare, Ireland (2005)Google Scholar
- 14.Mader, A.: A classification of PLC models and applications. In: 5th International Workshop on Discrete Event Systems –Discrete Event Systems, Analysis and Control, pp. 239–247. Springer, Heidelberg (2000)Google Scholar
- 15.Jee, E.: A Data Flow-Based Structural Testing Technique for FBD Programs. Ph.D Thesis. KAIST Press, Republic of Korea (2009)Google Scholar
- 16.Yices SMT solver, http://yices.csl.sri.com