Faulty Interaction Identification via Constraint Solving and Optimization
Abstract
Combinatorial testing (CT) is an important black-box testing method. In CT, the behavior of the system under test (SUT) is affected by several parameters/components. Then CT generates a combinatorial test suite. After the user executes a test suite and starts debugging, some test cases fail and some pass. From the perspective of a black box, the failures are caused by interaction of several parameters. It will be helpful if we can identify a small set of interacting parameters that caused the failures. This paper proposes a new automatic approach to identifying faulty interactions. It uses (pseudo-Boolean) constraint solving and optimization techniques to analyze the execution results of the combinatorial test suite. Experimental results show that the method is quite efficient and it can find faulty combinatorial interactions quickly. They also shed some light on the relation between the size of test suite and the ability of fault localization.
Keywords
Test Suite System Under Test Combinatorial Testing Covering Array Combinatorial InteractionPreview
Unable to display preview. Download preview PDF.
References
- 1.Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: An approach to testing based on combinatorial design. IEEE Trans. on Softw. Eng. 23(7), 437–444 (1997)CrossRefGoogle Scholar
- 2.Colbourn, C., McClary, D.: Locating and detecting arrays for interaction faults. J. of Combinatorial Optimization 15(1), 17–48 (2011)MathSciNetCrossRefGoogle Scholar
- 3.Czerwonka, J.: Pairwise testing in realworld: Practical extensions to test case generator. In: Proc. PNSQC 2006, pp. 419–430 (2006)Google Scholar
- 4.Gebser, M., Kaufmann, B., Neumann, A., Schaub, T.: Conflict-driven answer set solving. In: Proc. IJCAI 2007, pp. 386–392 (2007), http://www.cs.uni-potsdam.de/clasp/
- 5.Ghandehari, L.S.G., Lei, Y., Xie, T., Kuhn, D.R., Kacker, R.: Identifying failure-inducing combinations in a combinatorial test set. In: Proc. ICST 2012, pp. 370–379 (2012)Google Scholar
- 6.Kuhn, D.R., Okun, V.: Pseudo-exhaustive testing for software. In: Proc. SEW 2006, pp. 153–158 (2006)Google Scholar
- 7.Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. on Softw. Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
- 8.Martínez, C., Moura, L., Panario, D., Stevens, B.: Algorithms to Locate Errors Using Covering Arrays. In: Laber, E.S., Bornstein, C., Nogueira, L.T., Faria, L. (eds.) LATIN 2008. LNCS, vol. 4957, pp. 504–519. Springer, Heidelberg (2008)CrossRefGoogle Scholar
- 9.Nie, C., Leung, H.: A survey of combinatorial testing. ACM Computing Surveys 43(2) (2011)Google Scholar
- 10.Shi, L., Nie, C., Xu, B.: A Software Debugging Method Based on Pairwise Testing. In: Sunderam, V.S., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2005, Part III. LNCS, vol. 3516, pp. 1088–1091. Springer, Heidelberg (2005)CrossRefGoogle Scholar
- 11.Wang, Z., Xu, B., Chen, L., Xu, L.: Adaptive interaction fault location based on combinatorial testing. In: Proc. QSIC 2010, pp. 495–502 (2010)Google Scholar
- 12.Yilmaz, C., Cohen, M.B., Porter, A.A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. on Softw. Eng. 32(1), 20–34 (2006)CrossRefGoogle Scholar
- 13.Zeller, A., Hildebrandt, R.: Simplifying and isolating failure-inducing input. IEEE Trans. on Soft. Eng., 183–200 (2002)Google Scholar
- 14.Zhang, Z., Zhang, J.: Characterizing failure-causing parameter interactions by adaptive testing. In: Proc. ISSTA 2011, pp. 331–341 (2011)Google Scholar