Advertisement

Science China Information Sciences

, 60:092106 | Cite as

Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing

Research Paper

Abstract

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing by increasing the diversity of test cases. However, the intention of ensuring even spread of test cases inevitably causes an overhead problem. Although two basic forgetting strategies (i.e. random forgetting and consecutive retention) were proposed to reduce the computation cost of ART, they only considered the temporal distribution of test cases. In the paper, we presented a distance-aware forgetting strategy for the fixed size candidate set version of ART (DF-FSCS), in which the spatial distribution of test cases is taken into consideration. For a given candidate, the test cases out of its “sight” are ignored to reduce the distance computation cost. At the same time, the dynamic adjustment for partitioning and the second-round forgetting are adopted to ensure the linear complexity of DF-FSCS algorithm. Both simulation analysis and empirical study are employed to investigate the efficiency and effectiveness of DF-FSCS. The experimental results show that DF-FSCS significantly outperforms the classical ART algorithm FSCS-ART in efficiency, and has comparable failure-detection effectiveness. Com-pared with two basic forgetting methods, DF-FSCS is better in both efficiency and effectiveness. In contrast with a typical linear-time ART algorithm RBCVT-Fast, our algorithm requires less computational overhead and exhibits the similar failure-detection capability. In addition, DF-FSCS has more reliable performance than RBCVT-Fast in detecting failures for the programs with high-dimensional input domain.

Keywords

adaptive random testing software testing test cases computational overhead diversity 

Notes

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 61462030), Australian Research Council Linkage Grant (Grant No. LP100200208), Natural Science Foun-dation of Jiangxi Province (Grant Nos. 20162BCB23036, 20151BAB207018), and Science Foundation of Jiangxi Educational Committee (Grant No. GJJ150465).

References

  1. 1.
    Hamlet R. Random testing. In: Marciniak J J, ed. Encyclopedia of Software Engineering. 2nd ed. Chichester: John Wiley and Sons, 2002. 1507–1513Google Scholar
  2. 2.
    Chen T Y, Leung H, Mak I K. Adaptive random testing. In: Proceedings of the 9th Asian Computing Science Conference, Chiang Mai, 2004. 320–329MATHGoogle Scholar
  3. 3.
    Chen T Y, Kuo F-C, Merkel R G, et al. Adaptive random testing: the ART of test case diversity. J Syst Softw, 2010, 83: 60–66CrossRefGoogle Scholar
  4. 4.
    Orso A, Rothermel G. Software testing: a research travelogue (2000-2014). In: Proceedings of Future of Software Engineering (FOSE’14), Hyderabad, 2014. 117–132Google Scholar
  5. 5.
    Chen T Y, Kuo F-C, Towey D, et al. A revisit of three studies related to random testing. Sci China Inf Sci, 2015, 58: 052104Google Scholar
  6. 6.
    Shi Q, Chen Z, Fang C, et al. Measuring the diversity of a test set with distance entropy. IEEE Trans Reliab, 2016, 65: 19–27CrossRefGoogle Scholar
  7. 7.
    Feldt R, Poulding S, Clark D, et al. Test set diameter: quantifying the diversity of sets of test cases. In: Proceedings of the 9th IEEE International Conference on Software Testing, Verification and Validation (ICST’16), Chicago, 2016. 223–233Google Scholar
  8. 8.
    Chen T Y, Merkel R G. An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol, 2008, 17: 1–27CrossRefGoogle Scholar
  9. 9.
    Chen T Y. Fundamentals of test case selection: diversity, diversity, diversity. In: Proceedings of the 2nd International Conference on Software Engineering and Data Mining (SEDM’10), Chengdu, 2010. 723–724Google Scholar
  10. 10.
    Mariani L, Pezzè M, Zuddas D. Recent advances in automatic black-box testing. Adv Comput, 2015, 99: 157–193CrossRefGoogle Scholar
  11. 11.
    Arcuri A, Briand L. Adaptive random testing: an illusion of effectiveness? In: Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA’11), Toronto, 2011. 265–275Google Scholar
  12. 12.
    Chan K P, Chen T Y, Towey D. Forgetting test cases. In: Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Chicago, 2006. 485–494Google Scholar
  13. 13.
    Rigaux P, Scholl M, Voisard A. Spatial Databases: With Application to GIS. San Francisco: Morgan Kaufmann Publishers, 2002. 267–309CrossRefGoogle Scholar
  14. 14.
    Shahbazi A, Tappenden A F, Miller J. Centroidal voronoi tessellations—a new approach to random testing. IEEE Trans Softw Eng, 2013, 39: 163–183CrossRefGoogle Scholar
  15. 15.
    Finelli G B. NASA software failure characterization experiments. Reliab Eng Syst Saf, 1991, 32: 155–169CrossRefGoogle Scholar
  16. 16.
    Bishop P G. The variation of software survival time for different operational input profiles. In: Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23), Toulouse, 1993. 98–107Google Scholar
  17. 17.
    Schneckenburger C, Mayer J. Towards the determination of typical failure patterns. In: Proceedings of the 4th International Workshop on Software Quality Assurance (SOQUA’07), Dubrovnik, 2007. 90–93Google Scholar
  18. 18.
    Chen T Y, Tse T H, Yu Y T. Proportional sampling strategy: a compendium and some insights. J Syst Softw, 2001, 58: 65–81CrossRefGoogle Scholar
  19. 19.
    Chen T Y, Huang D H. Adaptive random testing by localization. In: Proceedings of the 11th Asia-Pacific Software Engineering Conference (APSEC’04), Busan, 2004. 292–298Google Scholar
  20. 20.
    Chen T Y, Kuo F-C, Merkel R. On the statistical properties of testing effectiveness measures. J Syst Softw, 2006, 79: 591–601CrossRefGoogle Scholar
  21. 21.
    Kuo F-C. On adaptive random testing. Dissertation for the Doctoral Degree. Melbourne: Swinburne University of Technology, 2006. 24–26Google Scholar
  22. 22.
    ACM. Collected Algorithms from ACM: Volume 1, Algorithms 1–220. New York: Association for Computer Machinery, 1980Google Scholar
  23. 23.
    Press W H, Teukolsky S A, Vetterling W T, et al. Numerical Recipes: The Art of Scientific Computing. 3rd ed. Cambridge: Cambridge University Press, 2007MATHGoogle Scholar
  24. 24.
    Jia Y, Harman M. An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng, 2011, 37: 649–678CrossRefGoogle Scholar
  25. 25.
    Chen T Y, Kuo F-C, Merkel R G, et al. Mirror adaptive random testing. Inform Softw Tech, 2004, 46: 1001–1010CrossRefGoogle Scholar
  26. 26.
    Chen T Y, Merkel R, Eddy G, et al. Adaptive random testing through dynamic partitioning. In: Proceedings of the 4th International Conference on Quality Software (QSIC’04), Braunschweig, 2004. 79–86Google Scholar
  27. 27.
    Chen T Y, Huang D, Zhou Z. On adaptive random testing through iterative partitioning. J Inf Sci Eng, 2011, 27: 1449–1472Google Scholar
  28. 28.
    Chow C, Chen T Y, Tse T H. The ART of divide and conquer: an innovative approach to improving the efficiency of adaptive random testing. In: Proceedings of the 13th International Conference on Quality Software (QSIC’13), Nanjing, 2013. 268–275Google Scholar
  29. 29.
    Barus A C, Chen T Y, Kuo F-C, et al. A cost-effective random testing method for programs with non-numeric inputs. IEEE Trans Comput, 2016, 65: 3509–3523MathSciNetGoogle Scholar

Copyright information

© Science China Press and Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Chengying Mao
    • 1
  • Tsong Yueh Chen
    • 2
  • Fei-Ching Kuo
    • 2
  1. 1.School of Software and Communication EngineeringJiangxi University of Finance and EconomicsNanchangChina
  2. 2.Department of Computer Science and Software EngineeringSwinburne University of TechnologyMelbourneAustralia

Personalised recommendations