Skip to main content
SpringerLink
Log in

A study of the interactive role of metamorphic testing and machine learning in the quality assurance of a deep learning forecasting application

  • Original Research
  • Published:
International Journal of Information Technology Aims and scope Submit manuscript

Abstract

One of the major problems in testing software code and assuring its quality is deriving test oracles. This problem becomes more evident when testing machine learning models. To overcome such problems, metamorphic testing is introduced. Metamorphic testing is built to test the presence of metamorphic relations; required relations that should hold for the program to be valid. Those metamorphic requirements are usually articulated by domain experts since they are application dependent which is a costly, complex, and error prone process. Therefore, the automatic detection of such relations can be more efficient for code verification. The novelty of this work is first emphasizing the two-way relationship between metamorphic testing and machine learning; where metamorphic testing is used to assess the effectiveness of machine learning models, while machine learning is used to automatically detect the main metamorphic relations in software code. Secondly, literature metamorphic relations are categorized into the eight main categories defined in literature, and three new metamorphic relations are introduced for the first time. Finally, all literature metamorphic relations are customized to fit the time series forecasting domain. An application is proposed consisting of a deep learning model for forecasting fresh produce yields along with a generalization framework to enable the proposed models to forecast other similar fresh produce yields. The interactive role played by metamorphic testing and machine learning is investigated through the quality assurance of the forecasting application. The datasets used to train and test the deep learning forecasting models as well as the forecasting models are verified using metamorphic tests and the metamorphic relations in the generalization code are automatically detected using support vector machine (SVM) models. Testing revealed the unmatched requirements that are fixed to have a valid application with sound data, effective models, and valid generalization code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The data used in this paper is available for open access on the referenced website source.

References

  1. Vineeta AS, Bansal A (2014) A study of various automated test oracle methods. In: 2014 5th international conference-confluence the next generation information technology summit. pp 753–760

  2. Barr E, Harman M, McMinn M, Phil S, Muzammil YS (2014) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41:1

    Google Scholar 

  3. Anwar N, Kar S (2019) Review paper on various software testing techniques and strategies. Glob J Comput Sci Technol 19:43–49

    Article  Google Scholar 

  4. Lin X, Simon M, Niu N (2018) Exploratory metamorphic testing for scientific software. Comput Sci Eng 22(2):78–87

    Article  Google Scholar 

  5. Murphy C, Kaiser G, Hu L, Wu L (2008) Properties of machine learning applications for use in metamorphic testing. 867–872

  6. Kanewala U, Lundgren A, Bieman JM (2016) Automated metamorphic testing of scientific software. Taylor & Francis EBooks, Oxford

    Google Scholar 

  7. Singh B, Jaiswal R (2023) TConvRec: temporal convolutional-recurrent fusion model with additional pattern learning. Int J Inf Technol 15:17–27

    Google Scholar 

  8. Thakur N, Karmakar S, Soni S (2022) Time series forecasting for univariant data using hybrid GA-OLSTM model and performance evaluations. Int J Inf Technol 14:1961–1966

    Google Scholar 

  9. Nasr I, Nassar L, Karray F (2022) Transfer learning framework for forecasting fresh produce yield and price. IJCNN

  10. Shashidhar R, Patilkulkarni S, Puneeth SB (2022) Combining audio and visual speech recognition using LSTM and deep convolutional neural network. Int J Inf Technol 14:3425–3436

    Google Scholar 

  11. Mishra PK, Karmakar S (2023) Performance of back-propagation neural network in chaotic data time series forecasting and evaluation over parametric forecast: a case study for rainfall-runoff modelling over a river basin. Int J Inf Technol 15:917–935

    Google Scholar 

  12. Lei M et al (2018) Secure deep learning engineering: a software quality assurance perspective. arXiv

  13. Du X, Xie X, Li Y, Ma L, Zhao J, Liu Y (2018) DeepCruiser: automated guided testing for stateful deep learning systems. arXiv e-prints, arXiv:1812.05339

  14. Segura S, Fraser G, Sánchez AB, Ruiz-Cortés A (2016) A survey on metamorphic testing. IEEE Trans Softw Eng 42:1–1

    Article  Google Scholar 

  15. Tsong YC et al (2019) Metamorphic testing a review of challenges and opportunities. ACM Comput Surv 51(1):Article 4

    Google Scholar 

  16. Srinivasan M, Kanewala U (2022) Metamorphic relation prioritization for effective regression testing. Softw Test Verif Reliab 32:e1807

    Article  Google Scholar 

  17. Aravind N, Karl M, Sigrid E (2019) Leveraging mutants for automatic prediction of metamorphic relations using machine learning. In: Proceedings of the 3rd ACM SIGSOFT international workshop on machine learning techniques for software quality evaluation (MaLTeSQuE 2019). Association for Computing Machinery, New York, pp 1–6

  18. Lin X, Simon M, Peng Z, Niu N (2020) Discovering metamorphic relations for scientific software from user forums. Comput Sci Eng 23(2):65–72

    Article  Google Scholar 

  19. Arianna G et al (2021) MeMo: automatically identifying metamorphic relations in Javadoc comments for test automation. J Syst Softw 181:111041

    Article  Google Scholar 

  20. Chen TY, Cheung SC, Yiu S (2020) Metamorphic testing: a new approach for generating next test cases

  21. Jameel T, Lin M, Chao L (2015) Test oracles based on metamorphic relations for image processing applications. IEEE/ACIS 16th international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD). pp 1–6

  22. Wildandyawan A, Nishi Y (2020) Object-based metamorphic testing through image structuring

  23. Murphy C, Kaiser G, Hu L, Wu L (2008) Properties of machine learning applications for use in metamorphic testing. 867–872

  24. Santos S, Silveira B, Andrade S, Delamaro M, Souza S (2020) An experimental study on applying metamorphic testing in machine learning applications. In: Proceedings of the 5th Brazilian symposium on systematic and automated software testing. Association for Computing Machinery, New York, pp 98–106

  25. Dwarakanath A, Ahuja M, Podder S, Vinu S, Naskar A, Koushik M (2019) Metamorphic Testing of a Deep Learning Based Forecaster. 2019 IEEE/ACM 4th international workshop on metamorphic testing (MET). pp 40–47

  26. Just R, Schweiggert F (2010) Automating software tests with partial oracles in integrated environments. In: Proceedings of the 5th workshop on automation of software test, ser. AST ’10. ACM, New York, pp 91–94

  27. Janiesch C, Zschech P, Heinrich K (2021) Machine learning and deep learning. Electron Mark 31:685–695

    Article  Google Scholar 

  28. Vargas R, Mosavi A, Ruiz L (2017) Deep learning: a review. Adv Intell Syst Comput 1:11

  29. Tan J, Yang J, Wu S, Chen G, Zhao J (2021) A critical look at the current train/test split in machine learning

  30. Sharma A, Wehrheim H (2019) Testing machine learning algorithms for balanced data usage. In: 2019 12th IEEE conference on software testing, validation and verification (ICST). pp 125–135

  31. Marijan D, Gotlieb A, Ahuja M (2019) Challenges of testing machine learning based systems. 101–102

  32. Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: automated white-box testing of deep learning systems, 26th symposium on operating systems principles. ACM, pp 1–18

  33. Tian Y, Pei K, Jana S, Ray B (2018), Deeptest: automated testing of deepneural-network-driven autonomous cars. International conference on software engineering. ACM, pp 303–314

  34. Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) Deep-road: Gan-based metamorphic autonomous driving system testing. International conference on automated software engineering. pp 132–142

  35. Allen FE (1970) Control flow analysis. Sigplan Not 5(7):1–19. https://doi.org/10.1145/390013.808479

    Article  Google Scholar 

  36. Li C, Liu J, Yang X, Yan S, Li M (2022) Metamorphic relation recognition method based on control flow graph features. J Phys Conf Ser 2219(1):012058

    Article  Google Scholar 

  37. Kanewala U (2014) Techniques for automatic detection of metamorphic relations. In: 2014 IEEE seventh international conference on software testing, verification and validation workshops. pp 237–238

  38. Kanewala U, Lundgren A, Bieman JM (2016) Chapter seven-automated metamorphic testing of scientific software, Software engineering for science. CRC Press, Boca Raton, pp 149–174

    Google Scholar 

  39. National Oceanic and Atmospheric Administration. https://www.noaa.gov/. Accessed 17 Oct 2021

  40. The California Strawberry Commission. https://www.calstrawberry.com/en-us/. Accessed 17 Oct 2021

  41. Saad M, Chaudhary M, Nassar L, Karray F, Gaudet V (2021) Versatile deep learning based application for time series imputation. IJCNN

  42. Pedregosa F et al (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830

    MathSciNet  Google Scholar 

  43. Chaudhary M, Gastli M, Nassar L, Karray F (2021) Deep learning approaches for forecasting Strawberry yields and prices using satellite images and station-based soil parameters. AAAI-MAKE

  44. Xie Y, Wiltgen B (2010) Adaptive feature based dynamic time warping, vol 10(1). IJCSNS

  45. Jafari F, Nassar L, Karray F (2021) Time series similarity analysis framework in fresh produce yield forecast domain. IEEE SMC

  46. Curreri F, Patanè L, Xibilia MG (2021) Soft sensor transferability: a survey. Appl Sci 1(16):7710

    Article  Google Scholar 

  47. Vishwanathan SVN, Schraudolph NN, Kondor R, Borgwardt KM (2010) Graph Kernels. J Mach Learn Res 11(3/1/2010):1201–1242

    MathSciNet  Google Scholar 

  48. Urry MJ, Sollich P (2013) Random walk kernels and learning curves for Gaussian process regression on random graphs. J Mach Learn Res 14:1801–1835

    MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the financial support provided by Loblaws corporation, NSERC CRD program, Mitacs and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI).

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering, University of Waterloo, 200 University Ave W, Waterloo, ON, N2L3G1, Canada

    Islam Nasr, Lobna Nassar & Fakhri Karray

Authors
  1. Islam Nasr
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lobna Nassar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fakhri Karray
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Islam Nasr.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nasr, I., Nassar, L. & Karray, F. A study of the interactive role of metamorphic testing and machine learning in the quality assurance of a deep learning forecasting application. Int. j. inf. tecnol. 16, 105–120 (2024). https://doi.org/10.1007/s41870-023-01390-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41870-023-01390-9

Keywords

Search

Navigation