Advertisement

Software Quality Journal

, Volume 16, Issue 3, pp 303–327 | Cite as

Enhancing adaptive random testing for programs with high dimensional input domains or failure-unrelated parameters

  • Fei-Ching KuoEmail author
  • Tsong Yueh Chen
  • Huai Liu
  • Wing Kwong Chan
Article

Abstract

Adaptive random testing (ART), an enhancement of random testing (RT), aims to both randomly select and evenly spread test cases. Recently, it has been observed that the effectiveness of some ART algorithms may deteriorate as the number of program input parameters (dimensionality) increases. In this article, we analyse various problems of one ART algorithm, namely fixed-sized-candidate-set ART (FSCS-ART), in the high dimensional input domain setting, and study how FSCS-ART can be further enhanced to address these problems. We propose to add a filtering process of inputs into FSCS-ART to achieve a more even-spread of test cases and better failure detection effectiveness in high dimensional space. Our study shows that this solution, termed as FSCS-ART-FE, can improve FSCS-ART not only in the case of high dimensional space, but also in the case of having failure-unrelated parameters. Both cases are common in real life programs. Therefore, we recommend using FSCS-ART-FE instead of FSCS-ART whenever possible. Other ART algorithms may face similar problems as FSCS-ART; hence our study also brings insight into the improvement of other ART algorithms in high dimensional space.

Keywords

Software testing Random testing Adaptive random testing Fixed-sized-candidate-set ART High dimension problem Failure-unrelated parameters 

Notes

Acknowledgment

This research project is supported by an Australian Research Council Discovery Grant (DP0880295).

References

  1. Ammann, P. E., & Knight, J. C. (1988). Data diversity: An approach to software fault tolerance. IEEE Transactions on Computers, 37(4), 418–425.CrossRefGoogle Scholar
  2. Bellman, R. (1957). Dynamic programming. New Jersey: Princeton University Press.Google Scholar
  3. Bird, D. L., & Munoz, C. U. (1983). Automatic generation of random self-checking test cases. IBM Systems Journal, 22(3), 229–245.CrossRefGoogle Scholar
  4. Bishop, P. G. (1993). The variation of software survival times for different operational input profiles. In Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23) (pp. 98–107). IEEE Computer Society Press.Google Scholar
  5. Branicky, M. S., LaValle, S. M., Olson, K., & Yang, L. (2001). Quasi-randomized path planning. In Proceedings of the 2001 IEEE International Conference on Robotics and Automation (pp. 1481–1487).Google Scholar
  6. Chan, K. P., Chen, T. Y., & Towey, D. (2006). Restricted random testing: Adaptive random testing by exclusion. International Journal of Software Engineering and Knowledge Engineering, 16(4), 553–584.CrossRefGoogle Scholar
  7. Chen, T. Y., Kuo, F.-C., & Liu, H. (2007). On test case distributions of adaptive random testing. In Proceedings of the 19th International Conference on Software Engineering and Knowledge Engineering (SEKE’07) (pp. 141–144). Boston.Google Scholar
  8. Chen, T. Y., Kuo, F.-C., & Merkel, R. (2006). On the statistical properties of testing effectiveness measures. Journal of Systems and Software, 79(5), 591–601.CrossRefGoogle Scholar
  9. Chen, T. Y., Kuo, F.-C., Merkel, R. G., & Ng, S. P. (2004). Mirror adaptive random testing. Information and Software Technology, 46(15), 1001–1010.CrossRefGoogle Scholar
  10. Chen, T. Y., Kuo, F.-C., & Zhou, Z. Q. (2005). On the Relationships between the distribution of failure-causing inputs and effectiveness of adaptive random testing. In Proceedings of the 17th International Conference on Software Engineering and Knowledge Engineering (SEKE’05) (pp. 306–311). Taipei, Taiwan.Google Scholar
  11. Chen, T. Y., & Merkel, R. (2008). An upper bound on software testing effectiveness. ACM Transaction on Software Engineering Methodologies.Google Scholar
  12. Chen, T. Y., Tse, T. H., & Yu, Y. T. (2001). Proportional sampling strategy: A compendium and some insights. Journal of Systems and Software, 58(1), 65–81.CrossRefGoogle Scholar
  13. Cobb, R., & Mills, H. D. (1990). Engineering software under statistical quality control. IEEE Software, 7(6), 45–54.CrossRefGoogle Scholar
  14. Dabóczi, T., Kollár, I., Simon, G., & Megyeri, T. (2003). Automatic testing of graphical user interfaces. In Proceedings of the 20th IEEE Instrumentation and Measurement Technology Conference 2003 (IMTC’03) (pp. 441–445). Vail, CO.Google Scholar
  15. Finelli, G. B. (1991). NASA software failure characterization experiments. Reliability Engineering and System Safety, 32(1–2), 155–169.CrossRefGoogle Scholar
  16. Forrester, J. E., & Miller, B. P. (2000). An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th USENIX Windows Systems Symposium (pp. 59–68). Seattle.Google Scholar
  17. Godefroid, P., Klarlund, N., & Sen, K. (2005). Dart: Directed automated random testing. In Proceedings of ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’05) (pp. 213–223).Google Scholar
  18. Hamlet, R. (2002). Random testing. In J. Marciniak (Ed.), Encyclopedia of software engineering (2nd edn.). Wiley.Google Scholar
  19. Kuo, F.-C., Chen, T. Y., Liu, H., & Chan, W. K. (2007). Enhancing adaptive random testing in high dimensional input domain. In Proceedings of the 22nd Annual ACM Symposium on Applied Computing (SAC’07) (pp. 1467–1472). ACM Press.Google Scholar
  20. Mak, I. K. (1997). On the effectiveness of random testing. Master’s thesis, Department of Computer Science, University of Melbourne.Google Scholar
  21. Matsumoto, M., & Nishimura, T. (1998). Mersenne twister: A 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Simulation, 8(1), 3–30.zbMATHCrossRefGoogle Scholar
  22. Mayer, J. (2005). Lattice-based adaptive random testing. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE’05) (pp. 333–336). New York: ACM Press.Google Scholar
  23. Miller, E. (2005). Website testing, http://www.soft.com/eValid/Technology/White.Papers/website.testing.html, Software Research, Inc. Accessed 21 Feb 2008.
  24. Miller, B. P., Fredriksen, L., & So, B. (1990). An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12), 32–44.CrossRefGoogle Scholar
  25. Miller, B. P., Koski, D., Lee, C. P., Maganty, V., Murthy, R., Natarajan, A., & Steidl, J. (1995). Fuzz revisited: A re-examination of the reliability of UNIX utilities and services. Tech. Rep. CS-TR-1995-1268, University of Wisconsin.Google Scholar
  26. Myers, G. J., Sandler, C., Badgett, T., & Thomas, T. M. (2004). The art of software testing (2nd edn.). New Jersey: Wiley.Google Scholar
  27. Nyman, N. In defense of monkey testing: Random testing can find bugs, even in well engineered software, http://www.automationjunkies.com/resources/nyman_monkey.rtf, Microsoft Corporation. Accessed 21 Feb 2008.
  28. Regehr, J. (2005). Random testing of interrupt-driven software. In Proceedings of the 5th ACM International Conference on Embedded software (EMSOFT’05) (pp. 290–298). New York, NY: ACM Press.Google Scholar
  29. Sen, K., Marinov, D., & Agha, G. (2005). CUTE: A concolic unit testing engine for C. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13) (pp. 263–272). New York, NY: ACM Press.Google Scholar
  30. Slutz, D. (1998). Massive stochastic testing of SQL. In Proceedings of the 24th International Conference on Very Large Databases (VLDB’98) (pp. 618–622).Google Scholar
  31. Yoshikawa, T., Shimura, K., & Ozawa, T. (2003). Random program generator for Java JIT compiler test system. In Proceedings of the 3rd International Conference on Quality Software (QSIC’03) (pp. 20–24). IEEE Computer Society Press.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Fei-Ching Kuo
    • 1
    Email author
  • Tsong Yueh Chen
    • 1
  • Huai Liu
    • 1
  • Wing Kwong Chan
    • 2
  1. 1.Faculty of Information and Communication TechnologiesSwinburne University of TechnologyHawthornAustralia
  2. 2.Department of Computer ScienceCity University of Hong KongHong KongChina

Personalised recommendations