Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing

Abstract

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing by increasing the diversity of test cases. However, the intention of ensuring even spread of test cases inevitably causes an overhead problem. Although two basic forgetting strategies (i.e. random forgetting and consecutive retention) were proposed to reduce the computation cost of ART, they only considered the temporal distribution of test cases. In the paper, we presented a distance-aware forgetting strategy for the fixed size candidate set version of ART (DF-FSCS), in which the spatial distribution of test cases is taken into consideration. For a given candidate, the test cases out of its “sight” are ignored to reduce the distance computation cost. At the same time, the dynamic adjustment for partitioning and the second-round forgetting are adopted to ensure the linear complexity of DF-FSCS algorithm. Both simulation analysis and empirical study are employed to investigate the efficiency and effectiveness of DF-FSCS. The experimental results show that DF-FSCS significantly outperforms the classical ART algorithm FSCS-ART in efficiency, and has comparable failure-detection effectiveness. Com-pared with two basic forgetting methods, DF-FSCS is better in both efficiency and effectiveness. In contrast with a typical linear-time ART algorithm RBCVT-Fast, our algorithm requires less computational overhead and exhibits the similar failure-detection capability. In addition, DF-FSCS has more reliable performance than RBCVT-Fast in detecting failures for the programs with high-dimensional input domain.

This is a preview of subscription content, log in to check access.

References

  1. 1

    Hamlet R. Random testing. In: Marciniak J J, ed. Encyclopedia of Software Engineering. 2nd ed. Chichester: John Wiley and Sons, 2002. 1507–1513

  2. 2

    Chen T Y, Leung H, Mak I K. Adaptive random testing. In: Proceedings of the 9th Asian Computing Science Conference, Chiang Mai, 2004. 320–329

  3. 3

    Chen T Y, Kuo F-C, Merkel R G, et al. Adaptive random testing: the ART of test case diversity. J Syst Softw, 2010, 83: 60–66

  4. 4

    Orso A, Rothermel G. Software testing: a research travelogue (2000-2014). In: Proceedings of Future of Software Engineering (FOSE’14), Hyderabad, 2014. 117–132

  5. 5

    Chen T Y, Kuo F-C, Towey D, et al. A revisit of three studies related to random testing. Sci China Inf Sci, 2015, 58: 052104

  6. 6

    Shi Q, Chen Z, Fang C, et al. Measuring the diversity of a test set with distance entropy. IEEE Trans Reliab, 2016, 65: 19–27

  7. 7

    Feldt R, Poulding S, Clark D, et al. Test set diameter: quantifying the diversity of sets of test cases. In: Proceedings of the 9th IEEE International Conference on Software Testing, Verification and Validation (ICST’16), Chicago, 2016. 223–233

  8. 8

    Chen T Y, Merkel R G. An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol, 2008, 17: 1–27

  9. 9

    Chen T Y. Fundamentals of test case selection: diversity, diversity, diversity. In: Proceedings of the 2nd International Conference on Software Engineering and Data Mining (SEDM’10), Chengdu, 2010. 723–724

  10. 10

    Mariani L, Pezzè M, Zuddas D. Recent advances in automatic black-box testing. Adv Comput, 2015, 99: 157–193

  11. 11

    Arcuri A, Briand L. Adaptive random testing: an illusion of effectiveness? In: Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA’11), Toronto, 2011. 265–275

  12. 12

    Chan K P, Chen T Y, Towey D. Forgetting test cases. In: Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Chicago, 2006. 485–494

  13. 13

    Rigaux P, Scholl M, Voisard A. Spatial Databases: With Application to GIS. San Francisco: Morgan Kaufmann Publishers, 2002. 267–309

  14. 14

    Shahbazi A, Tappenden A F, Miller J. Centroidal voronoi tessellations—a new approach to random testing. IEEE Trans Softw Eng, 2013, 39: 163–183

  15. 15

    Finelli G B. NASA software failure characterization experiments. Reliab Eng Syst Saf, 1991, 32: 155–169

  16. 16

    Bishop P G. The variation of software survival time for different operational input profiles. In: Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23), Toulouse, 1993. 98–107

  17. 17

    Schneckenburger C, Mayer J. Towards the determination of typical failure patterns. In: Proceedings of the 4th International Workshop on Software Quality Assurance (SOQUA’07), Dubrovnik, 2007. 90–93

  18. 18

    Chen T Y, Tse T H, Yu Y T. Proportional sampling strategy: a compendium and some insights. J Syst Softw, 2001, 58: 65–81

  19. 19

    Chen T Y, Huang D H. Adaptive random testing by localization. In: Proceedings of the 11th Asia-Pacific Software Engineering Conference (APSEC’04), Busan, 2004. 292–298

  20. 20

    Chen T Y, Kuo F-C, Merkel R. On the statistical properties of testing effectiveness measures. J Syst Softw, 2006, 79: 591–601

  21. 21

    Kuo F-C. On adaptive random testing. Dissertation for the Doctoral Degree. Melbourne: Swinburne University of Technology, 2006. 24–26

  22. 22

    ACM. Collected Algorithms from ACM: Volume 1, Algorithms 1–220. New York: Association for Computer Machinery, 1980

  23. 23

    Press W H, Teukolsky S A, Vetterling W T, et al. Numerical Recipes: The Art of Scientific Computing. 3rd ed. Cambridge: Cambridge University Press, 2007

  24. 24

    Jia Y, Harman M. An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng, 2011, 37: 649–678

  25. 25

    Chen T Y, Kuo F-C, Merkel R G, et al. Mirror adaptive random testing. Inform Softw Tech, 2004, 46: 1001–1010

  26. 26

    Chen T Y, Merkel R, Eddy G, et al. Adaptive random testing through dynamic partitioning. In: Proceedings of the 4th International Conference on Quality Software (QSIC’04), Braunschweig, 2004. 79–86

  27. 27

    Chen T Y, Huang D, Zhou Z. On adaptive random testing through iterative partitioning. J Inf Sci Eng, 2011, 27: 1449–1472

  28. 28

    Chow C, Chen T Y, Tse T H. The ART of divide and conquer: an innovative approach to improving the efficiency of adaptive random testing. In: Proceedings of the 13th International Conference on Quality Software (QSIC’13), Nanjing, 2013. 268–275

  29. 29

    Barus A C, Chen T Y, Kuo F-C, et al. A cost-effective random testing method for programs with non-numeric inputs. IEEE Trans Comput, 2016, 65: 3509–3523

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 61462030), Australian Research Council Linkage Grant (Grant No. LP100200208), Natural Science Foun-dation of Jiangxi Province (Grant Nos. 20162BCB23036, 20151BAB207018), and Science Foundation of Jiangxi Educational Committee (Grant No. GJJ150465).

Author information

Correspondence to Chengying Mao.

Additional information

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mao, C., Chen, T.Y. & Kuo, F. Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing. Sci. China Inf. Sci. 60, 092106 (2017). https://doi.org/10.1007/s11432-016-0087-0

Download citation

Keywords

  • adaptive random testing
  • software testing
  • test cases
  • computational overhead
  • diversity