Skip to main content
SpringerLink
Log in

Deep Learning System and It’s Automatic Testing: An Approach

  • Published:
Annals of Data Science Aims and scope Submit manuscript

Abstract

The process of testing conventional programs is quite easy as compared to the programs using Deep Learning approach. The term Deep learning (DL) is used for a novel programming approach that is highly data centric and where the governing rules and logic are primarily dependent on the data used for training. Conventionally, Deep Learning models are evaluated by using a test dataset to evaluate their performance against set parameters. The difference in data and logic handling between programs using conventional methods and programs using the DL approach makes it difficult to apply the traditional approaches of testing directly to DL based programs. The accuracy of test data is currently the best measure of the adequacy of testing in the DL based systems. This poses a problem because of the difficulty in availability of test data that is of sufficient quality. This in turn restricts the level of confidence that can be established on the adequacy of testing of DL based systems. Unlike conventional applications, using the conventional programming approaches the lack of quality test data and the lack of interpretability makes the system analysis and detection of defects a difficult task in DL based systems. So testing of DL based models can be done automatically with a different approach compared to normal software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data Availability

This is a review paper so no data is needed.

Code avialability

This is a review paper so no code is needed.

References

  1. Ma L, Zhang F, Sun J, Xue M, Li B, Juefei-Xu F, Wang Y (2018) Deepmutation: mutation testing of deep learning systems. In: 2018 IEEE 29th international symposium on software reliability engineering (ISSRE), pp 100–111. https://doi.org/10.1109/ISSRE.2018.00021

  2. Dhar V (2013) Data science and prediction. Commun ACM 56(12):64–73. https://doi.org/10.1145/2500499

    Article  Google Scholar 

  3. Van Der Aalst W (2016) Data science in action. Process mining. Springer, Berlin, pp 3–23. https://doi.org/10.1007/978-3-662-49851-4_1

  4. Garousi V, Felderer M, Karapıçak ÇM, Yılmaz U (2018) Testing embedded software: a survey of the literature. Inf Softw Technol 104:14–45. https://doi.org/10.1016/j.infsof.2018.06.016

    Article  Google Scholar 

  5. Takanen A, Demott JD, Miller C, Kettunen A (2018) Fuzzing for software security testing and quality assurance. Artech House

  6. Asplund F (2019) Exploratory testing: Do contextual factors influence software fault identification? Inf Softw Technol 107:101–111. https://doi.org/10.1016/j.infsof.2018.11.003

    Article  Google Scholar 

  7. Monperrus M (2018) Automatic software repair: a bibliography. ACM Comput Surv (CSUR) 51(1):1–24. https://doi.org/10.1145/3105906

    Article  Google Scholar 

  8. Felderer M, Russo B, Auer F (2019) On testing data-intensive software systems. In: Security and quality in cyber-physical systems engineering. Springer, Cham, pp 129–148. https://doi.org/10.1007/978-3-030-25312-7_6

  9. Kim Y, Hong S, Ko B, Phan DL, Kim M (2018) Invasive software testing: mutating target programs to diversify test exploration for high test coverage. In: 2018 IEEE 11th international conference on software testing, verification and validation (ICST), pp 239–249. https://doi.org/10.1109/ICST.2018.00032

  10. Budgen D, Tomayko JE (2003) Norm Gibbs and his contribution to software engineering education through the SEI curriculum modules. In: Proceedings 16th conference on software engineering education and training, 2003 (CSEE&T 2003), pp 3–13. https://doi.org/10.1109/CSEE.2003.1191340

  11. Scatalon LP, Fioravanti ML, Prates JM, Garcia RE, Barbosa EF (2018) A survey on graduates’ curriculum-based knowledge gaps in software testing. In: 2018 IEEE frontiers in education conference (FIE), pp 1–8. https://doi.org/10.1109/FIE.2018.8658688

  12. Drave I, Hillemacher S, Greifenberg T, Rumpe B, Wortmann A, Markthaler M, Kriebel S (2018) Model-based testing of software-based system functions. In: 2018 44th Euromicro conference on software engineering and advanced applications (SEAA), pp 146–153. https://doi.org/10.1109/SEAA.2018.00032

  13. Böhme M, Paul S (2015) A probabilistic analysis of the efficiency of automated software testing. IEEE Trans Software Eng 42(4):345–360. https://doi.org/10.1109/TSE.2015.2487274

    Article  Google Scholar 

  14. Chen TY, Kuo FC, Liu H, Poon PL, Towey D, Tse TH, Zhou ZQ (2018) Metamorphic testing: a review of challenges and opportunities. ACM Comput Surv (CSUR) 51(1):1–27. https://doi.org/10.1145/3143561

    Article  Google Scholar 

  15. Kuchta T, Palikareva H, Cadar C (2018) Shadow symbolic execution for testing software patches. ACM Trans Software Eng Methodol (TOSEM) 27(3):1–32. https://doi.org/10.1145/3208952

    Article  Google Scholar 

  16. Alghamdi AM, Eassa FE (2019) Software testing techniques for parallel systems: a survey. Int J Comput Sci Netw Secur 19(4):176–186

    Google Scholar 

  17. Wu H, Petke J, Jia Y, Harman M (2018) An empirical comparison of combinatorial testing, random testing and adaptive random testing. IEEE Trans Software Eng. https://doi.org/10.1109/TSE.2018.2852744

    Article  Google Scholar 

  18. Markthaler M, Kriebel S, Salman KS, Greifenberg T, Hillemacher S, Rumpe B, Richenhagen J (2018) Improving model- based testing in automotive software engineering. In: 2018 IEEE/ACM 40th international conference on software engineering: software engineering in practice track (ICSE-SEIP), pp 172–180

  19. Van Deursen A, Aniche M, Boone C, Cunha ML, Nadeem A (2019) Software quality and testing. Delft University of Technology

  20. Lemieux C, Sen K (2018) Fairfuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, pp 475–485. https://doi.org/10.1145/3238147.3238176

  21. Harman M, Jia Y, Zhang Y (2015) Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th international conference on software testing, verification and validation (ICST), pp 1–12. https://doi.org/10.1109/ICST.2015.7102580

  22. Majumdar R (2010) Paul Ammann and Jeff Offutt Introduction to Software Testing. Cambridge University Press (2008). ISBN: 978-0-521-88038-1.£ 32.99. Hardcover. Comput J 53(5):615–615. https://doi.org/10.1093/comjnl/bxp017

  23. Maciel CP, Souza EF, Vijaykumar NL, Falbo RA, Meinerz GV, Felizardo KR (2018) An empirical study on the knowledge management practice in software testing. In: Experimental Software Engineering Latin American Workshop (ESELAW’18). XXI Ibero-American Conference on Software Engineering (CIBSE)

  24. Xie T, Tillmann N, Lakshman P (2016) Advances in unit testing: theory and practice. In: Proceedings of the 38th international conference on software engineering companion, pp 904–905. https://doi.org/10.1145/2889160.2891056

  25. Clegg BS, Rojas JM, Fraser G (2017) Teaching software testing concepts using a mutation testing game. In: 2017 IEEE/ACM 39th international conference on software engineering: software engineering education and training track (ICSE-SEET), pp 33–36. https://doi.org/10.1109/ICSE-SEET.2017.1

  26. Angell R, Johnson B, Brun Y, Meliou A (2018) Themis: automatically testing software for discrimination. In: Proceedings of the 2018 26th ACM Joint Meeting on European software engineering conference and symposium on the foundations of software engineering, pp 871–875. https://doi.org/10.1145/3236024.3264590

  27. Xiu M, Ming Z, Adams B (2019) An exploratory study on machine learning model stores. https://doi.org/10.1109/MS.2020.2975159

  28. Kamilaris A, Prenafeta-Boldú FX (2018) Deep learning in agriculture: a survey. Comput Electron Agric 147:70–90. https://doi.org/10.1016/j.compag.2018.02.016

    Article  Google Scholar 

  29. Ferentinos KP (2018) Deep learning models for plant disease detection and diagnosis. Comput Electron Agric 145:311–318. https://doi.org/10.1016/j.compag.2018.01.009

    Article  Google Scholar 

  30. Lusch B, Kutz JN, Brunton SL (2018) Deep learning for universal linear embeddings of nonlinear dynamics. Nat Commun 9(1):1–10. https://doi.org/10.1038/s41467-018-07210-0

    Article  Google Scholar 

  31. Singh AK, Ganapathy subramanian B, Sarkar S, Singh A (2018) Deep learning for plant stress phenotyping: trends and future perspectives. Trends Plant Sci 23(10):883–898. https://doi.org/10.1016/j.tplants.2018.07.004

    Article  Google Scholar 

  32. Raissi M, Wang Z, Triantafyllou MS, Karniadakis GE (2019) Deep learning of vortex-induced vibrations. J Fluid Mech 861:119–137. https://doi.org/10.1017/jfm.2018.872

    Article  Google Scholar 

  33. Kulin M, Kazaz T, Moerman I, De Poorter E (2018) End-to-end learning from spectrum data: a deep learning approach for wireless signal identification in spectrum monitoring applications. IEEE Access 6:18484–18501. https://doi.org/10.1109/ACCESS.2018.2818794

    Article  Google Scholar 

  34. Shallue CJ, Vanderburg A (2018) Identifying exoplanets with deep learning: a five-planet resonant chain around kepler-80 and an eighth planet around kepler-90. Astron J 155(2):94

    Article  Google Scholar 

  35. Finlayson SG, Chung HW, Kohane IS, Beam AL (2018) Adversarial attacks against medical deep learning systems. arXiv preprint arXiv:1804.05296

  36. Stadie BC, Yang G, Houthooft R, Chen X, Duan Y, Wu Y, Sutskever I (2018) Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01118.

  37. Raissi M, Perdikaris P, Karniadakis GE (2018) Multistep neural networks for data-driven discovery of nonlinear dynamical systems. arXiv preprint arXiv:1801.01236.

  38. Gurovich Y, Hanani Y, Bar O, Nadav G, Fleischer N, Gelbman D, Bird LM (2019) Identifying facial phenotypes of genetic disorders using deep learning. Nat Med 25(1):60–64. https://doi.org/10.1038/s41591-018-0279-0

    Article  Google Scholar 

  39. Ahmed M, Najmul Islam AKM (2020) Deep learning: hope or hype. Ann Data Sci 7:427–432. https://doi.org/10.1007/s40745-019-00237-0

    Article  Google Scholar 

  40. Hassani H, Huang X, Ghodsi SE, M (2020) Deep learning and implementations in banking. Ann Data Sci 7:433–446. https://doi.org/10.1007/s40745-020-00300-1

    Article  Google Scholar 

  41. Olson DL, Shi Y (2007) Introduction to business data mining. McGraw-Hill/Irwin, New York

    Google Scholar 

  42. Shi Y, Tian YJ, Kou G, Peng Y, Li JP (2011) Optimization based data mining: theory and applications. Springer, Berlin

    Book  Google Scholar 

  43. Tien JM (2017) Internet of things, real-time decision making, and artificial intelligence. Ann Data Sci 4(2):149–178. https://doi.org/10.1007/s40745-017-0112-5

    Article  Google Scholar 

  44. Grano G, Titov TV, Panichella, S., & Gall, H. C. (2018, March). How high will it be? using machine learning models to predict branch coverage in automated testing. In 2018 IEEE workshop on machine learning techniques for software quality evaluation (MaLTeSQuE), pp 19–24. https://doi.org/10.1109/MALTESQUE.2018.8368454

  45. King TM, Arbon J, Santiago D, Adamo D, Chin W, Shanmugam R (2019) AI for testing today and tomorrow: industry perspectives. In: 2019 IEEE international conference on Artificial Intelligence Testing (AITest), pp 81–88. https://doi.org/10.1109/AITest.2019.000-3

  46. Marcus G (2018) Deep learning: a critical appraisal. arXiv preprint arXiv:1801.00631

  47. Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Zhao J (2018) Deepgauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, pp. 120–131. https://doi.org/10.1145/3238147.3238202

  48. Ma L, Zhang F, Xue M, Li B, Liu Y, Zhao J, Wang Y (2018) Combinatorial testing for deep learning systems. arXiv preprint arXiv:1806.07723

  49. Dwarakanath A, Ahuja M, Sikand S, Rao RM, Bose RJC, Dubash N, Podder S (2018) Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, pp 118–128. https://doi.org/10.1145/3213846.3213858

  50. Kim J, Feldt R, Yoo S (2019) Guiding deep learning system testing using surprise adequacy. In: 2019 IEEE/ACM 41st international conference on software engineering (ICSE), pp 1039–1049. https://doi.org/10.1109/ICSE.2019.00108

  51. Zhang JM, Harman M, Ma L, Liu Y (2019) Machine learning testing: survey, landscapes and horizons. https://doi.org/10.1109/TSE.2019.2962027

  52. Srisakaokul S, Wu Z, Astorga A, Alebiosu O, Xie T (2018) Multiple-implementation testing of supervised learning software. In: Workshops at the thirty-second AAAI conference on artificial intelligence

  53. Sun Y, Huang X, Kroening D (2018) Testing deep neural networks. arXiv preprint arXiv:1803.04792

  54. Al-Hajjaji M, Thüm T, Lochau M, Meinicke J, Saake G (2019) Effective product-line testing using similarity-based product prioritization. Softw Syst Model 18(1):499–521. https://doi.org/10.1007/s10270-016-0569-2

    Article  Google Scholar 

Download references

Funding

No Funding is available for this reasearch.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, ABES Institute of Technology, Ghaziabad, UP, India

    Rijwan Khan

Authors
  1. Rijwan Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rijwan Khan.

Ethics declarations

Conflict of Interest

In this research I am sole author. I do not have any conflict of interest with anyone involve with review or editorial work.

Authors’ contributions

I have done research for this manuscript. I have written this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, R. Deep Learning System and It’s Automatic Testing: An Approach. Ann. Data. Sci. 10, 1019–1033 (2023). https://doi.org/10.1007/s40745-021-00361-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40745-021-00361-w

Keywords

Search

Navigation