Abstract
One of the grand challenges for adequately testing complex software is due to the oracle problem. Metamorphic Testing (MT) is a promising technique to alleviate the oracle problem through using one or multiple Metamorphic Relations (MRs) as test oracles. MT checks the satisfaction of every MR among the outputs of the MR-related tests instead of the correctness of individual test outputs. In practice, it is fairly easy to find MRs for testing any program, but it is very difficult to develop “good” MRs and evaluate their adequacy. A systematic approach for developing MRs and evaluating their adequacy in MT remains to be developed. In this paper, we propose a framework for evaluating MT and iteratively developing adequate MRs monitored by MT adequacy evaluation. The MT adequacy is measured by program coverages, mutation testing, and testing MRs with mutation tests. The MT evaluation results are used for guiding the iterative development of MRs, generating tests, and analyzing test outputs. We explain the framework through a testing example on an image processing program that is used for building the 3-dimensional structure of a biology cell based on its confocal image sections. In order to demonstrate the effectiveness of the proposed framework, we reported a case study of testing a complex scientific program: a Monte Carlo modeling program that simulates photon propagations in turbid tissue phantoms for accurate and efficient generation of reflectance images from biological tissues. The case study has shown the effectiveness of proposed MT framework for testing scientific software in general and the necessity of the MT enhancement in the development of adequate MRs. The case study results can be easily adapted for testing other software.
Similar content being viewed by others
References
Arridge, S. R. (1999). Optical tomography in medical imaging. Inverse Problems, 15, R41–R93.
Asrafi, M., Liu, H., & Kuo, F-C. (2011). On testing effectiveness of metamorphic relations: A Case study, In 5th International Conference on Secure Software Integration and Reliability Improvement (SSIRI), pp. 147–156.
Baresi L., & Young, M. (2001). Test oracles. Technical Report CIS-TR01 -02, Department of Computer and Information Science, University of Oregon.
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5), 507–525.
Chan, W. K., Cheung, S. C., & Leung, K. R. (2007). A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research, 4(1), 60–80.
Chen, T. Y., Cheung, S. C., & Yiu, S. (1998). Metamorphic testing: A new approach for generating next test cases, Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology.
Chen, T.Y., Feng, J., Tse, T.H. (2002). Metamorphic testing of programs on partial differential equations: a case study. In Proceedings of 26th Annual International Computer Software and Applications Conference (COMPSAC), pp. 327–333.
Chen, T. Y., Ho, J. W. K., Liu, H., & Xie, X. (2009). An innovattive approach for testing bioinformatics programs using metamorphic testing, BMC Bioinformatics, pp. 10–24.
Chen, T. Y., Huang, D. H., Tse, T. H., & Zhou, Z. Q. (2004). Case studies on the selection of useful relations in metamorphic testing”, In Proceedings of the 4 th Ibero-American Symposium on Software Engineering and Knowledge Engineering, pp. 569–583.
Chen, C., Lu, J. Q., Li, K., Zhao, S., Brock, R. S., & Hu, X. H. (2007). Numerical study of reflectance imaging using a parallel Monte Carlo method. Medical Physics, 34, 2939–2948.
Chen, T. Y., Poon, P. L., & Xie, X. (2015). METRIC: METamorphic relation identification based on the category-choice framework. Journal of Systems and Software, 116(C), 177–190.
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002). Semi-proving: An integrated method based on global symbolic evaluation and metamorphic testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pp. 191–195.
Confocal microscope, https://en.wikipedia.org/wiki/Confocal_microscopyWikipedia, last accessed on April 30, 2016.
Ding, J., & Zhang D. (2016). A machine learning approach for developing test Oracles for testing scientific software,In the 28 th SEKE (SEKE 2016), San Francisco, July 1–3.
Ding, J., Clarke, P. J., Argote-Garcia, G., & He, X. (2009). A methodology for evaluating test coverage criteria of high level Petri nets. Information and Software Technology, 51(11), 1520–1533.
Ding, J., Wu, T., Lu, J. Q., Hu, X. (2010). Self-checked metamorphic Testing of an image processing program, The 4th IEEE International Conference on Security Software Integration and Reliability Improvement, Singapore, June 9–11.
Ding, J., Zhang, D., Hu, X. (2016). An application of metamorphic testing for testing scientific software, In 1 st workshop on metamorphic testing with ICSE, Austin, TX, May 16.
Farrell, P. E., Pigott, M. D., Gorman, G. J., Ham, D. A., Wilson, C. R., & Bond, T. M. (2011). Automated continuous verification for numerical simulation. Geoscientific Model Development, 4(2), 435–449.
Gotleib, A. & Botella, B. (2003). Automated metamorphic testing, In Proceedings of 27 th Annual International Computer Software and Applications Conference, (pp. 34–40).
Gotlieb, A., Botella, B., & Rueher, M. (1998). Automatic test data generation using constraint solving techniques. In ACM International Symposium on Software Testing and Analysis (ISSTA). Software Engineering Notes, 23(2):53–62.
Guderlei, R., & Mayer, J. (2007). Statistical metamorphic testing—testing programs with random output by means of statistical hypothesis tests and metamorphic testing”, In Proceedings of the 7 th International Conference on Quality Software. pp. 404–409, 2007.
Hu, P., Zhang, Z., Chan, W. K., & Tse, T. H. (2006). An empirical comparison between direct and indirect test result checking approaches. In Proceedings of the 3 rd International Workshop on Soft. Quality Assurance, pp. 6–13.
Jameel, T., Lin, M., & Chao, L. (2015). Test oracles based on metamorphic relations for image processing applications. In 16 th International Conference on SE, AI, Networking and Parallel/Distributed Computing (SNPD), pp. 1–6.
Jia, Y., & Harman, M. (2011). An analysis and survey of the development of mutation testing. In IEEE Transactions on Software Engineering, 37(5), pp. 649–678, September–October 2011.
Kanewala, U., & Bieman, J. M. (2014). Testing scientific software: A systematic literature review. Information and Software Technology, 56(10), 1219–1232.
Kanewala, U., Bieman, J. M., & Ben-Hur, A. (2015). Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels. Journal of Software Testing, Verification and Reliability, 26(3), 245–269.
Keijzer, M., Jacques, S. L., Prahl, S. A., & Welch, A. J. (1989). Light distributions in artery tissure: Monte Carlo simulations for finite-diameter laser beams. Lasers in Surgery and Medicine, 9, 148–154.
Le, V., Afshari, M., & Su, Z. (2014). Compiler validation via equivalence modulo inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ‘14). ACM, New York, NY, USA, pp.216–226.
Lindvall, M., Ganesan, D., Árdal, R., & Wiegand, R. E. (2015). “Metamorphic model-based testing applied on NASA DAT: an experience report”. In Proceedings of the 37 th International Conference on Software Engineering, Vol. 2 (ICSE ‘15), Vol. 2. pp. 129–138.
Liu, H., Kuo, F., Towey, D., & Chen, T. Y. (2014). How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering, 40(1), 4–22.
Mayer, J., Guderlei, R. (2006). On random testing of image processing applications. In Proceedings of 6 th International Conference on Quality Software (QSIC’06), pp. 85–92.
Mayer, J. & Guderlei, R. (2006). An empirical study on the selection of good metamorphic relations. In Proceedings of the 30 th Annual International Computer Software and Applications Conference (COMPSAC), pp. 475–484.
Mayer, J., Infor, A. A., Ulm, U. (2005). On testing image procesing applications with staticstical methods. Software Engineering (SE 2005), Lecture Notes in Informatics, pp. 69–78.
Murphy, C. Kaiser, G., Hu, L. & Wu, L. (2008). Properties of machine learning applications for use in metamorphic testing”. In Proceedings of the 20 th International conference on software engineering and knowledge engineering (SEKE), pp. 867–872.
Murphy, C. (2010). Metamorphic testing techniques to detect defects in applications without test Oracles. Doctoral dissertation, Columbia University.
Murphy, C., Shen, K.,& Kaiser, G. (2009). Automatic system testing of program without test oracles. In Proceedings of 2009 ACM International Symposium of Software Testing and Analysis (ISSTA).
Murphy, C., Shen, K., & Kaiser, G. (2009). Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles, In Proceedings of the 2 nd IEEE International Conference on Software Testing, Verification and Validation (ICST).
Nguyen-Hoan, L., Flint, S., Sankara, R. (2010). A survey of scienfitic software development, In 2010 ACM International Symposium on Empiricial Software Engineering and Measurement (ESSM’10), pp. 12:1–12:10.
Nie, C., & Leung, H. (2011). A survey of combinatorial testing. ACM Computing Survey, 43(2), 11.
Obayashi, M., Kubota, H., McCarron, S. P., & Mallet L. (1998). The assertion based testing tool for OOP: ADL2. http://adl.xopen.org/exgr/icse/icse98.htm May 1998.
Pacheco, P. (1996). Parallel Programming with MPI. Morgan Kaufmann; 1st edition.
Pezzè, M., & Young, M. (2007). Software testing and analysis: Process, principles, and techniques. New Jersey: Wiley.
Segura, S., Fraser, G., Sanchez, A., & Ruiz-Cortes, A. (2016). A Survey on Metamorphic Testing. In IEEE Transaction on Software Engineering (vol. PP, no. 99). doi:10.1109/TSE.2016.2532875.
Sanders, R., Kelly, D. (2008). The challenge of testing scientific software.In Proceedings of the Conference for the Association for Software Testing (CAST), pp. 30–36, Toronto, July.
Shan, L. & Zhu, H. (2009). Generating structually complex test cases by data mutation: A case study of testing an automated modelling tool. The Computer Journal, 52(5).
Weyuker, E. J. (1982). On testing non-testable program. Computer Journal, 25(4), 465–470.
Xie, X., Ho, J., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011). Testing and validating machine learning classifiers by metamorphic testing. J. System and Software., 84(4), 544–558.
Zhou, Z. Q., Chan, W. K., Chan, W. K., Tse, T. H., & HU, P. (2009). Experimental study to compare the use of metamorphictesting and assertion checking. Journal of Software, 20(10), 2637–2654.
Zhou, Z.Q., Xiang, S., Chen, T.Y. (2015). Metamorphic testing for software quality assessment: A study of search engines. In IEEE Transactions on Software Engineering, PrePrints, doi:10.1109/TSE.2015.2478001.
Zhu, H., Hall, P. A., & May, J. H. (1997). Software unit test coverage and adequacy. ACM Computing Surveys, 29(4), 366–427.
Zhu, H., & He, X. (2002). A methodology of testing high-level petri nets. Journal of Information and Software Technology, 44, 473–489.
Acknowledgments
We thank Dr. Tong Wu for his implementation and testing of the Monte Carlo simulation program. This research is supported in part by Grant #1262933 and #1560037 from the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ding, J., Hu, XH. Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program. Software Qual J 25, 841–869 (2017). https://doi.org/10.1007/s11219-016-9337-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-016-9337-3