Bioinformatics is the application of computational, mathematical and statistical techniques to solve problems in biology and medicine. Bioinformatics programs developed for computational simulation and large-scale data analysis are widely used in almost all areas of biophysics. The appropriate choice of algorithms and correct implementation of these algorithms are critical for obtaining reliable computational results. Nonetheless, it is often very difficult to systematically test these programs as it is often hard to verify the correctness of the output, and to effectively generate failure-revealing test cases. Software testing is an important process of verification and validation of scientific software, but very few studies have directly dealt with the issues of bioinformatics software testing. In this work, we review important concepts and state-of-the-art methods in the field of software testing. We also discuss recent reports on adapting and implementing software testing methodologies in the bioinformatics field, with specific examples drawn from systems biology and genomic medicine.
KeywordsSoftware testing Bioinformatics Quality assurance Automated testing Cloud-based testing
Compliance with ethical standards
This work was supported in part by funds from the New South Wales Ministry of Health, a New South Wales Genomics Collaborative Grant, an Australian Research Council Grant, and a Microsoft Azure Research Award.
Conflict of interests
All authors (AHK, EG, TYC, MAC, ALME and JWKH) declare that they do not have any conflict of interest.
This article does not contain any studies with human or animal subjects performed by any of the authors.
- Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, New YorkGoogle Scholar
- Chen L, Avizienis A (1978) N-Version Programming: A fault-tolerance approach to reliability of software operation. In: Proc. 8th IEEE Int. Symp. on Fault-Tolerant Computing (FTCS-8), Toulouse, France, June 1978, pp 3–9Google Scholar
- Chen TY, Cheung SC, Yiu S (1998) Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Dept. of Computer Science, Hong Kong University of Science and TechnologyGoogle Scholar
- Chen TY, Merkel RG, Eddy G, Wong P (2004) Adaptive Random Testing Through Dynamic Partitioning. In: Proc Fourth Int Conf Quality Software, Braunschweig, Germany, Sept 2004, pp 79–86Google Scholar
- Chen TY, Leung H, Mak I (2005) Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference: Higher-Level Decision Making (ASIAN 2004), Springer-Verlag, Berlin Heidelberg, pp 320–329Google Scholar
- Chen TY, Ho JW, Liu H, Xie X (2009) An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, 24. This paper provides the first case study of using metamorphic testing to test bioinformatics programs.Google Scholar
- Chu M, Murphy C, Kaiser G (2008) Distributed in vivo testing of software applications. In: Proc 1st Int Conf Software Testing, Verification, and Validation, Lillehammer, Norway, 2008, pp 509–512Google Scholar
- Gao J, Bai X, Tsai W-T (2011) Cloud testing-issues, challenges, needs and practice. Software Engineering: An International Journal 1:9–23Google Scholar
- Hamlet R. (1994) Random testing. Encyclopedia of Software Engineering. Wiley, New York, pp 970–978Google Scholar
- Hannay JE, MacLeod C, Singer J, Langtangen HP, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software? In: Proceedings of Workshop on Software Engineering for Computational Science and Engineering., pp 1–8Google Scholar
- IEEE (1990) IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990 1–84.Google Scholar
- IEEE (2013). Software and systems engineering — Software testing — Part 1: Concepts and definitions. ISO/IEC/IEEE 29119 1–84.Google Scholar
- ISTQB I (2015) Glossary of Testing Terms. ISTQB Glossary http://www.istqb.org/downloads/finish/20/193.html.
- Kuo F-C, Chen TY, Tam WK (2011) Testing embedded software by metamorphic testing: A wireless metering system case study. In: 36th IEEE Conference on Local Computer Networks (LCN), Bonn, Germany, 2011, pp 291–294Google Scholar
- Lanubile F, Shull F, Basili VR (1998) Experimenting with error abstraction in requirements documents. In: Proceedings of Fifth International Software Metrics Symposium, Bethesda, Maryland, 2008, pp 114–121Google Scholar
- Leavitt N (2009) Is Cloud Computing Really Ready for Prime Time? Computer 42:15–20Google Scholar
- Murphy C, Kaiser GE, Hu L (2008) Properties of machine learning applications for use in metamorphic testing. In: Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE, San Francisco, CA, pp 867–872Google Scholar
- Murphy C, Shen K, Kaiser G (2009a) Automatic system testing of programs without test oracles. In: Proceedings of the eighteenth international symposium on Software testing and analysis, Chicago, IL, 2009, pp 189–200Google Scholar
- Murphy C, Kaiser G, Vo I, Chu M (2009b) Quality assurance of software applications using the in vivo testing approach. In: Proceedings of International Conference on Software Testing Verification and Validation, Denver CO, 2009, pp 111–120Google Scholar
- Myers GJ, Sandler C, Badgett T (2011) The art of software testing. Wiley, New YorkGoogle Scholar
- O’Rawe J, Jiang T, Sun G, Wu Y, Wang W, Hu J, Bodily P, Tian L, Hakonarson H, Johnson WE (2013) Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Med 5:28. This paper shows there is a low concordance among five widely used variant calling pipelines, suggesting the importance of improving the quality of these pipelinesGoogle Scholar
- Parveen T, Tilley S (2010) When to migrate software testing to the cloud? In: Proceedings of Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), Washington, DC, 2010. pp 424–427Google Scholar
- Segura S, Hierons RM, Benavides D, Ruiz-Cortés A (2010) Automated test data generation on the analyses of feature models: A metamorphic testing approach. In: Proceedings of the Third International Conference on Software Testing, Verification and Validation (ICST), Washington, DC, 2010, pp 35–44Google Scholar
- Zhou ZQ, Huang D, Tse T, Yang Z, Huang H, Chen T (2004) Metamorphic testing and its applications. In: Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004), Xi'An, China, 2004, pp 346–351Google Scholar