Test input reduction for result inspection to facilitate fault localization

  • Dan Hao
  • Tao Xie
  • Lu Zhang
  • Xiaoyin Wang
  • Jiasu Sun
  • Hong Mei


Testing-based fault-localization (TBFL) approaches often require the availability of high-statement-coverage test suites that sufficiently exercise the areas around the faults. However, in practice, fault localization often starts with a test suite whose quality may not be sufficient to apply TBFL approaches. Recent capture/replay or traditional test-generation tools can be used to acquire a high-statement-coverage test collection (i.e., test inputs only) without expected outputs. But it is expensive or even infeasible for developers to manually inspect the results of so many test inputs. To enable practical application of TBFL approaches, we propose three strategies to reduce the test inputs in an existing test collection for result inspection. These three strategies are based on the execution traces of test runs using the test inputs. With the three strategies, developers can select only a representative subset of the test inputs for result inspection and fault localization. We implemented and applied the three test-input-reduction strategies to a series of benchmarks: the Siemens programs, DC, and TCC. The experimental results show that our approach can help developers inspect the results of a smaller subset (less than 10%) of test inputs, whose fault-localization effectiveness is close to that of the whole test collection.


Test suite reduction Testing Debugging Fault localization 


  1. Agrawal, H., Horgan, J., London, S., Wong, W.: Fault localization using execution slices and dataflow. In: Proc. 6th International Symposium on Software Reliability Engineering, pp. 143–151, October 1995 Google Scholar
  2. Barbosa, E., Maldonado, J., Vincenzi, A.: Toward the determination of sufficient mutant operators for C. Softw. Test. Verif. Reliab. 11(2), 113–136 (2001) CrossRefGoogle Scholar
  3. Baresi, L., Young, M.: Test oracles. Technical Report CIS-TR-01-02, University of Oregon, Dept. of Computer and Information Science, Eugene, Oregon, USA, August 2001 Google Scholar
  4. Baudry, B., Fleurey, F., Traon, Y.L.: Improving test suites for efficient fault localization. In: Proc. 28th International Conference on Software Engineering, pp. 82–91, May 2006 Google Scholar
  5. Cleve, H., Zeller, A.: Locating causes of program failure. In: Proc. 27th International Conference on Software Engineering, pp. 342–351, May 2005 Google Scholar
  6. Do, H., Elbaum, S., Rothermel, G.: Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Emp. Softw. Eng. 10(4), 405–435 (2005) CrossRefGoogle Scholar
  7. Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: A family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002) CrossRefGoogle Scholar
  8. Enderton, H.B.: Elements of Set Theory. Academic Press, San Diego (1977) zbMATHGoogle Scholar
  9. Graves, T.L., Harrold, M.J., Kim, J.-M., Porter, A., Rothermel, G.: An empirical study of regression test selection techniques. ACM Trans. Softw. Eng. Methodol. 10(2), 184–208 (2001) zbMATHCrossRefGoogle Scholar
  10. Guo, L., Roychoudhury, A., Wang, T.: Accurately choosing execution runs for software fault localization. In: Proc. International Conference on Compiler Construction, pp. 80–95, Mar. 2006 Google Scholar
  11. Hao, D.: Testing-based interactive fault localization. In: Proc. 28th International Conference on Software Engineering, Doctoral Symposium Track, pp. 957–960, May 2006 Google Scholar
  12. Hao, D., Pan, Y., Zhang, L., Zhao, W., Mei, H., Sun, J.: A similarity-aware approach to testing based fault localization. In: Proc. 20th IEEE International Conference on Automated Software Engineering, pp. 291–294 (2005a) Google Scholar
  13. Hao, D., Zhang, L., Zhong, H., Mei, H., Sun, J.: Eliminating harmful redundancy for testing-based fault localization using test suite reduction: An experimental study. In: Proc. 21st IEEE International Conference on Software Maintenance, pp. 683–686 (2005b) Google Scholar
  14. Hao, D., Zhang, L., Mei, H., Sun, J.: Towards interactive fault localization using test information. In: Proc. 13th Asia Pacific Software Engineering Conference, pp. 277–284 (2006) Google Scholar
  15. Hao, D., Zhang, L., Pan, Y., Mei, H., Sun, J.: On similarity-awareness in testing-based fault localization. Autom. Softw. Eng. J. 15(2), 207–249 (2008) CrossRefGoogle Scholar
  16. Hao, D., Zhang, L., Zhang, L., Sun, J., Mei, H.: VIDA: Visual interactive debugging. In: Proc. 31st International Conference on Software Engineering, Formal Research Demonstrations, pp. 583–586, May 2009 Google Scholar
  17. Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2(3), 270–285 (1993) CrossRefGoogle Scholar
  18. Hutchins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In: Proc. 16th International Conference on Software Engineering, pp. 191–200, May 1994 Google Scholar
  19. Jeffrey, D., Gupta, N.: Test suite reduction with selective redundancy. In: Proc. 21st IEEE International Conference on Software Maintenance, pp. 549–558, Sept. 2005 Google Scholar
  20. Jeffrey, D., Gupta, N.: Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Trans. Softw. Eng. 33(2), 108–123 (2007) CrossRefGoogle Scholar
  21. Jones, J.A., Harrold, M.J.: Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng. 29(3), 195–209 (2003) CrossRefGoogle Scholar
  22. Jones, J.A., Harrold, M.J.: Empirical evaluation of tarantula automatic fault-localization technique. In: Proc. 20th International Conference on Automated Software Engineering, pp. 273–282 (2005) Google Scholar
  23. Jones, J.A., Harrold, M.J., Stasko, J.: Visualization of test information to assist fault localization. In: Proc. 24th International Conference on Software Engineering, pp. 467–477, May 2002 Google Scholar
  24. Joseph, N.: Statistics and Probability in Modern Life. Saunders, Philadelphia (1992) Google Scholar
  25. Liblit, B., Aiken, A., Zheng, A.X., Jordan, M.I.: Bug isolation via remote program sampling. In: Proc. ACM SIGPLAN 2003 Conference on Programming Languages Design and Implementation, pp. 141–154 (2003) Google Scholar
  26. Liu, C., Yuan, X., Fei, L., Han, J., Midkiff, S.P.: SOBER: Statistical model-based bug localization. In: Proc. 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 286–295, September 2005 Google Scholar
  27. Ma, X.y., He, Z.f., Sheng, B.k., Ye, C.q.: A genetic algorithm for test-suite reduction. In: Proc. the International Conference on Systems, Man and Cybernetics, pp. 133–139, October 2005 Google Scholar
  28. Pacheco, C., Ernst, M.D.: Eclat: Automatic generation and classification of test inputs. In: Proc. 19th European Conference on Object-Oriented Programming, pp. 504–527 (2005) Google Scholar
  29. Renieris, M., Reiss, S.P.: Fault localization with nearest neighbor queries. In: Proc. 18th International Conference on Automated Software Engineering, pp. 30–39 (2003) Google Scholar
  30. Rothermel, G., Harrold, M.J.: A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Methodol. 6(2), 173–210 (1997) CrossRefGoogle Scholar
  31. Rothermel, G., Harrold, M.J.: Experimental studies of a safe regression test selection technique. IEEE Trans. Softw. Eng. 24(6), 401–419 (1998) CrossRefGoogle Scholar
  32. Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001) CrossRefGoogle Scholar
  33. Rothermel, G., Harrold, M.J., von Ronne, J., Hong, C.: Empirical studies of test-suite reduction. Softw. Test. Verif. Reliab. 12(4), 219–249 (2002) CrossRefGoogle Scholar
  34. Saff, D., Artzi, S., Perkins, J.H., Ernst, M.D.: Automatic test factoring for Java. In: Proc. 20th International Conference on Automated Software Engineering, pp. 114–123, November 2005 Google Scholar
  35. Sen, K., Marinov, D., Agha, G.: CUTE: A concolic unit testing engine for C. In: Proc. 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp. 263–272 (2005) Google Scholar
  36. Sprenkle, S., Sampath, S., Gibson, E., Pollock, L., Souter, A.: An empirical comparison of test suite reduction techniques for user-session-based testing of web applications. In: Proc. 21st IEEE International Conference on Software Maintenance, pp. 587–596, Sept. 2005 Google Scholar
  37. Srivastava, A., Thiagarajan, J.: Effectively prioritizing tests in development environment. In: Proc. 2002 International Symposium on Software Testing and Analysis, pp. 97–106, July 2002 Google Scholar
  38. Trochim, W.M.K.: Research Methods Knowledge Base. Thomson Custom Pub., New York (2007) Google Scholar
  39. Walcott, K.R., Kapfhammer, G.M., Soffa, M.L., Roos, R.S.: Time-aware test suite prioritization. In: Proc. International Symposium on Software Testing and Analysis, pp. 1–12 (2006) Google Scholar
  40. Wang, T., Roychoudhury, A.: Automated path generation for software fault localization. In: Proc. 20th International Conference on Automated Software Engineering, pp. 347–351 (2005) Google Scholar
  41. Wohlin, C., Runeson, P., Host, M.: Experimentation in Software Engineering: An Introduction. Springer, Berlin (1999) Google Scholar
  42. Xie, T., Notkin, D.: Tool-assisted unit-test generation and selection based on operational abstractions. Autom. Softw. Eng. J. 13(3), 345–371 (2006) CrossRefGoogle Scholar
  43. Xie, Y., Engler, D.: Using redundancies to find errors. IEEE Trans. Softw. Eng. 29(10), 915–928 (2003) CrossRefGoogle Scholar
  44. Yu, Y., Jones, J.A., Harrold, M.J.: An empirical study of the effects of test-suite reduction on fault localization. In: Proc. 30th International Conference on Software Engineering, pp. 201–210, May 2008 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Dan Hao
    • 1
  • Tao Xie
    • 2
  • Lu Zhang
    • 1
  • Xiaoyin Wang
    • 1
  • Jiasu Sun
    • 1
  • Hong Mei
    • 1
  1. 1.Key Laboratory of High Confidence Software Technologies, Ministry of Education, Institute of Software, School of Electronics Engineering and Computer SciencePeking UniversityBeijingPeople’s Republic of China
  2. 2.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA

Personalised recommendations