Bioinformatics is the application of computational, mathematical and statistical techniques to solve problems in biology and medicine. Bioinformatics programs developed for computational simulation and large-scale data analysis are widely used in almost all areas of biophysics. The appropriate choice of algorithms and correct implementation of these algorithms are critical for obtaining reliable computational results. Nonetheless, it is often very difficult to systematically test these programs as it is often hard to verify the correctness of the output, and to effectively generate failure-revealing test cases. Software testing is an important process of verification and validation of scientific software, but very few studies have directly dealt with the issues of bioinformatics software testing. In this work, we review important concepts and state-of-the-art methods in the field of software testing. We also discuss recent reports on adapting and implementing software testing methodologies in the bioinformatics field, with specific examples drawn from systems biology and genomic medicine.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Alden K, Read M (2013) Computing: Scientific software needs quality control. Nature 502:448
Baxter SM, Day SW, Fetrow JS, Reisinger SJ (2006) Scientific software development is not an oxymoron. PLoS Comput Biol 2, e87
Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, New York
Bergmann FT, Sauro HM (2008) Comparing simulation results of SBML capable simulators. Bioinformatics 24:1963–1965
Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur Gener Comput Syst 25:599–616
Candea G, Bucur S, Zamfir C (2010) Automated software testing as a service. In: Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, Indianapolis, Indiana, USA, pp 155–160
Chan FT, Chen TY, Mak IK, Yu YT (1996) Proportional sampling strategy: guidelines for software testing practitioners. Inf Softw Technol 38:775–782
Chan WK, Cheung S, Leung KR (2007) A metamorphic testing approach for online testing of service-oriented software applications. Int J Web Serv Res 4:61–81
Check Hayden E (2013) Mozilla plan seeks to debug scientific code. Nature 501:472
Check Hayden E (2015) Journal buoys code-review push. Nature 520:276–277
Chen L, Avizienis A (1978) N-Version Programming: A fault-tolerance approach to reliability of software operation. In: Proc. 8th IEEE Int. Symp. on Fault-Tolerant Computing (FTCS-8), Toulouse, France, June 1978, pp 3–9
Chen TY, Merkel R (2008) An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol 17:16
Chen TY, Cheung SC, Yiu S (1998) Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Dept. of Computer Science, Hong Kong University of Science and Technology
Chen TY, Tse T, Zhou ZQ (2003) Fault-based testing without the need of oracles. Inf Softw Technol 45:1–9
Chen TY, Merkel RG, Eddy G, Wong P (2004) Adaptive Random Testing Through Dynamic Partitioning. In: Proc Fourth Int Conf Quality Software, Braunschweig, Germany, Sept 2004, pp 79–86
Chen TY, Leung H, Mak I (2005) Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference: Higher-Level Decision Making (ASIAN 2004), Springer-Verlag, Berlin Heidelberg, pp 320–329
Chen TY, Ho JW, Liu H, Xie X (2009) An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, 24. This paper provides the first case study of using metamorphic testing to test bioinformatics programs.
Chen TY, Kuo F-C, Merkel RG, Tse T (2010) Adaptive random testing: The ART of test case diversity. J Syst Softw 83:60–66
Chu M, Murphy C, Kaiser G (2008) Distributed in vivo testing of software applications. In: Proc 1st Int Conf Software Testing, Verification, and Validation, Lillehammer, Norway, 2008, pp 509–512
Dai H, Murphy C, Kaiser G (2010) Confu: Configuration fuzzing testing framework for software vulnerability detection. Int J Secure Softw Eng 1:41–55
DeMilli R, Offutt AJ (1991) Constraint-based automatic test data generation. IEEE Trans Softw Eng 17:900–910
Evans TW, Gillespie CS, Wilkinson DJ (2008) The SBML discrete stochastic models test suite. Bioinformatics 24:285–286
Gao J, Bai X, Tsai W-T (2011) Cloud testing-issues, challenges, needs and practice. Software Engineering: An International Journal 1:9–23
Giannoulatou E, Park S-H, Humphreys DT, Ho JW (2014) Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie. BMC Bioinformatics 15:S15
Gray R, Kelly D (2010) Investigating test selection techniques for scientific software using Hook’s mutation sensitivity testing. Procedia Comput Sci 1:1487–1494
Hamlet RG (1977) Testing programs with the aid of a compiler. IEEE Trans Softw Eng SE-3:279–290
Hamlet R. (1994) Random testing. Encyclopedia of Software Engineering. Wiley, New York, pp 970–978
Hannay JE, MacLeod C, Singer J, Langtangen HP, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software? In: Proceedings of Workshop on Software Engineering for Computational Science and Engineering., pp 1–8
Hook D, Kelly D (2009) Mutation sensitivity testing. Comput Sci Eng 11:40–47
Howden WE (1976) Reliability of the path analysis testing strategy. IEEE Trans Softw Eng SE-2:208–215
IEEE (1990) IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990 1–84.
IEEE (2013). Software and systems engineering — Software testing — Part 1: Concepts and definitions. ISO/IEC/IEEE 29119 1–84.
ISTQB I (2015) Glossary of Testing Terms. ISTQB Glossary http://www.istqb.org/downloads/finish/20/193.html.
Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37:649–678
Joppa LN, McInerny G, Harper R, Salido L, Takeda K, O’Hara K, Gavaghan D, Emmott S (2013) Troubling trends in scientific software use. Science 340:814–815
Knight JC, Leveson NG (1986) An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans Softw Eng 12:96–109
Kuo F-C, Chen TY, Tam WK (2011) Testing embedded software by metamorphic testing: A wireless metering system case study. In: 36th IEEE Conference on Local Computer Networks (LCN), Bonn, Germany, 2011, pp 291–294
Langmead B, Salzberg SL (2012) Fast gapped-read alignment with Bowtie 2. Nat Methods 9:357–359
Lanubile F, Shull F, Basili VR (1998) Experimenting with error abstraction in requirements documents. In: Proceedings of Fifth International Software Metrics Symposium, Bethesda, Maryland, 2008, pp 114–121
Leavitt N (2009) Is Cloud Computing Really Ready for Prime Time? Computer 42:15–20
Li H, Durbin R (2009) Fast and accurate short read alignment with Burrows–Wheeler transform. Bioinformatics 25:1754–1760
Liu H, Kuo F-C, Towey D, Chen TY (2014) How effectively does metamorphic testing alleviate the oracle problem? IEEE Trans Softw Eng 40:4–22
McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Cazier J-B, Donnelly P (2014) Choice of transcripts and software has a large effect on variant annotation. Genome Med 6:26
Merali Z (2010) Computational science: Error, why scientific programming does not compute. Nature 467:775–777
Murphy C, Kaiser GE, Hu L (2008) Properties of machine learning applications for use in metamorphic testing. In: Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE, San Francisco, CA, pp 867–872
Murphy C, Shen K, Kaiser G (2009a) Automatic system testing of programs without test oracles. In: Proceedings of the eighteenth international symposium on Software testing and analysis, Chicago, IL, 2009, pp 189–200
Murphy C, Kaiser G, Vo I, Chu M (2009b) Quality assurance of software applications using the in vivo testing approach. In: Proceedings of International Conference on Software Testing Verification and Validation, Denver CO, 2009, pp 111–120
Myers GJ, Sandler C, Badgett T (2011) The art of software testing. Wiley, New York
O’Rawe J, Jiang T, Sun G, Wu Y, Wang W, Hu J, Bodily P, Tian L, Hakonarson H, Johnson WE (2013) Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Med 5:28. This paper shows there is a low concordance among five widely used variant calling pipelines, suggesting the importance of improving the quality of these pipelines
Otero C, Peter A (2015) Research Directions for Engineering Big Data Analytics Software. IEEE Intell Syst 30:13–19
Parveen T, Tilley S (2010) When to migrate software testing to the cloud? In: Proceedings of Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), Washington, DC, 2010. pp 424–427
Riungu-Kalliosaari L, Taipale O, Smolander K (2012) Testing in the cloud: Exploring the practice. IEEE Softw 29:46–51
Segura S, Hierons RM, Benavides D, Ruiz-Cortés A (2010) Automated test data generation on the analyses of feature models: A metamorphic testing approach. In: Proceedings of the Third International Conference on Software Testing, Verification and Validation (ICST), Washington, DC, 2010, pp 35–44
Weyuker EJ (1982) On testing non-testable programs. Comput J 25:465–470
Xie X, Ho JW, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84:544–558
Zhou ZQ, Huang D, Tse T, Yang Z, Huang H, Chen T (2004) Metamorphic testing and its applications. In: Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004), Xi'An, China, 2004, pp 346–351
Compliance with ethical standards
This work was supported in part by funds from the New South Wales Ministry of Health, a New South Wales Genomics Collaborative Grant, an Australian Research Council Grant, and a Microsoft Azure Research Award.
Conflict of interests
All authors (AHK, EG, TYC, MAC, ALME and JWKH) declare that they do not have any conflict of interest.
This article does not contain any studies with human or animal subjects performed by any of the authors.
About this article
Cite this article
Kamali, A.H., Giannoulatou, E., Chen, T.Y. et al. How to test bioinformatics software?. Biophys Rev 7, 343–352 (2015). https://doi.org/10.1007/s12551-015-0177-3
- Software testing
- Quality assurance
- Automated testing
- Cloud-based testing