Skip to main content

How to test bioinformatics software?


Bioinformatics is the application of computational, mathematical and statistical techniques to solve problems in biology and medicine. Bioinformatics programs developed for computational simulation and large-scale data analysis are widely used in almost all areas of biophysics. The appropriate choice of algorithms and correct implementation of these algorithms are critical for obtaining reliable computational results. Nonetheless, it is often very difficult to systematically test these programs as it is often hard to verify the correctness of the output, and to effectively generate failure-revealing test cases. Software testing is an important process of verification and validation of scientific software, but very few studies have directly dealt with the issues of bioinformatics software testing. In this work, we review important concepts and state-of-the-art methods in the field of software testing. We also discuss recent reports on adapting and implementing software testing methodologies in the bioinformatics field, with specific examples drawn from systems biology and genomic medicine.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2


  1. Alden K, Read M (2013) Computing: Scientific software needs quality control. Nature 502:448

    CAS  Article  PubMed  Google Scholar 

  2. Baxter SM, Day SW, Fetrow JS, Reisinger SJ (2006) Scientific software development is not an oxymoron. PLoS Comput Biol 2, e87

    PubMed Central  Article  PubMed  Google Scholar 

  3. Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, New York

  4. Bergmann FT, Sauro HM (2008) Comparing simulation results of SBML capable simulators. Bioinformatics 24:1963–1965

    PubMed Central  CAS  Article  PubMed  Google Scholar 

  5. Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur Gener Comput Syst 25:599–616

    Article  Google Scholar 

  6. Candea G, Bucur S, Zamfir C (2010) Automated software testing as a service. In: Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, Indianapolis, Indiana, USA, pp 155–160

    Chapter  Google Scholar 

  7. Chan FT, Chen TY, Mak IK, Yu YT (1996) Proportional sampling strategy: guidelines for software testing practitioners. Inf Softw Technol 38:775–782

    Article  Google Scholar 

  8. Chan WK, Cheung S, Leung KR (2007) A metamorphic testing approach for online testing of service-oriented software applications. Int J Web Serv Res 4:61–81

    Article  Google Scholar 

  9. Check Hayden E (2013) Mozilla plan seeks to debug scientific code. Nature 501:472

    CAS  Article  Google Scholar 

  10. Check Hayden E (2015) Journal buoys code-review push. Nature 520:276–277

    CAS  Article  PubMed  Google Scholar 

  11. Chen L, Avizienis A (1978) N-Version Programming: A fault-tolerance approach to reliability of software operation. In: Proc. 8th IEEE Int. Symp. on Fault-Tolerant Computing (FTCS-8), Toulouse, France, June 1978, pp 3–9

  12. Chen TY, Merkel R (2008) An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol 17:16

    Article  Google Scholar 

  13. Chen TY, Cheung SC, Yiu S (1998) Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Dept. of Computer Science, Hong Kong University of Science and Technology

  14. Chen TY, Tse T, Zhou ZQ (2003) Fault-based testing without the need of oracles. Inf Softw Technol 45:1–9

    Article  Google Scholar 

  15. Chen TY, Merkel RG, Eddy G, Wong P (2004) Adaptive Random Testing Through Dynamic Partitioning. In: Proc Fourth Int Conf Quality Software, Braunschweig, Germany, Sept 2004, pp 79–86

  16. Chen TY, Leung H, Mak I (2005) Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference: Higher-Level Decision Making (ASIAN 2004), Springer-Verlag, Berlin Heidelberg, pp 320–329

  17. Chen TY, Ho JW, Liu H, Xie X (2009) An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, 24. This paper provides the first case study of using metamorphic testing to test bioinformatics programs.

  18. Chen TY, Kuo F-C, Merkel RG, Tse T (2010) Adaptive random testing: The ART of test case diversity. J Syst Softw 83:60–66

    Article  Google Scholar 

  19. Chu M, Murphy C, Kaiser G (2008) Distributed in vivo testing of software applications. In: Proc 1st Int Conf Software Testing, Verification, and Validation, Lillehammer, Norway, 2008, pp 509–512

  20. Dai H, Murphy C, Kaiser G (2010) Confu: Configuration fuzzing testing framework for software vulnerability detection. Int J Secure Softw Eng 1:41–55

    Article  Google Scholar 

  21. DeMilli R, Offutt AJ (1991) Constraint-based automatic test data generation. IEEE Trans Softw Eng 17:900–910

    Article  Google Scholar 

  22. Evans TW, Gillespie CS, Wilkinson DJ (2008) The SBML discrete stochastic models test suite. Bioinformatics 24:285–286

    CAS  Article  PubMed  Google Scholar 

  23. Gao J, Bai X, Tsai W-T (2011) Cloud testing-issues, challenges, needs and practice. Software Engineering: An International Journal 1:9–23

    Google Scholar 

  24. Giannoulatou E, Park S-H, Humphreys DT, Ho JW (2014) Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie. BMC Bioinformatics 15:S15

    PubMed Central  Article  PubMed  Google Scholar 

  25. Gray R, Kelly D (2010) Investigating test selection techniques for scientific software using Hook’s mutation sensitivity testing. Procedia Comput Sci 1:1487–1494

    Article  Google Scholar 

  26. Hamlet RG (1977) Testing programs with the aid of a compiler. IEEE Trans Softw Eng SE-3:279–290

    Article  Google Scholar 

  27. Hamlet R. (1994) Random testing. Encyclopedia of Software Engineering. Wiley, New York, pp 970–978

  28. Hannay JE, MacLeod C, Singer J, Langtangen HP, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software? In: Proceedings of Workshop on Software Engineering for Computational Science and Engineering., pp 1–8

    Google Scholar 

  29. Hook D, Kelly D (2009) Mutation sensitivity testing. Comput Sci Eng 11:40–47

    Article  Google Scholar 

  30. Howden WE (1976) Reliability of the path analysis testing strategy. IEEE Trans Softw Eng SE-2:208–215

    Article  Google Scholar 

  31. IEEE (1990) IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990 1–84.

  32. IEEE (2013). Software and systems engineering — Software testing — Part 1: Concepts and definitions. ISO/IEC/IEEE 29119 1–84.

  33. ISTQB I (2015) Glossary of Testing Terms. ISTQB Glossary

  34. Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37:649–678

    CAS  Article  Google Scholar 

  35. Joppa LN, McInerny G, Harper R, Salido L, Takeda K, O’Hara K, Gavaghan D, Emmott S (2013) Troubling trends in scientific software use. Science 340:814–815

    CAS  Article  PubMed  Google Scholar 

  36. Knight JC, Leveson NG (1986) An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans Softw Eng 12:96–109

    Article  Google Scholar 

  37. Kuo F-C, Chen TY, Tam WK (2011) Testing embedded software by metamorphic testing: A wireless metering system case study. In: 36th IEEE Conference on Local Computer Networks (LCN), Bonn, Germany, 2011, pp 291–294

  38. Langmead B, Salzberg SL (2012) Fast gapped-read alignment with Bowtie 2. Nat Methods 9:357–359

    PubMed Central  CAS  Article  PubMed  Google Scholar 

  39. Lanubile F, Shull F, Basili VR (1998) Experimenting with error abstraction in requirements documents. In: Proceedings of Fifth International Software Metrics Symposium, Bethesda, Maryland, 2008, pp 114–121

  40. Leavitt N (2009) Is Cloud Computing Really Ready for Prime Time? Computer 42:15–20

    Google Scholar 

  41. Li H, Durbin R (2009) Fast and accurate short read alignment with Burrows–Wheeler transform. Bioinformatics 25:1754–1760

    PubMed Central  CAS  Article  PubMed  Google Scholar 

  42. Liu H, Kuo F-C, Towey D, Chen TY (2014) How effectively does metamorphic testing alleviate the oracle problem? IEEE Trans Softw Eng 40:4–22

    Article  Google Scholar 

  43. McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Cazier J-B, Donnelly P (2014) Choice of transcripts and software has a large effect on variant annotation. Genome Med 6:26

    PubMed Central  Article  PubMed  Google Scholar 

  44. Merali Z (2010) Computational science: Error, why scientific programming does not compute. Nature 467:775–777

    CAS  Article  PubMed  Google Scholar 

  45. Murphy C, Kaiser GE, Hu L (2008) Properties of machine learning applications for use in metamorphic testing. In: Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE, San Francisco, CA, pp 867–872

  46. Murphy C, Shen K, Kaiser G (2009a) Automatic system testing of programs without test oracles. In: Proceedings of the eighteenth international symposium on Software testing and analysis, Chicago, IL, 2009, pp 189–200

  47. Murphy C, Kaiser G, Vo I, Chu M (2009b) Quality assurance of software applications using the in vivo testing approach. In: Proceedings of International Conference on Software Testing Verification and Validation, Denver CO, 2009, pp 111–120

  48. Myers GJ, Sandler C, Badgett T (2011) The art of software testing. Wiley, New York

  49. O’Rawe J, Jiang T, Sun G, Wu Y, Wang W, Hu J, Bodily P, Tian L, Hakonarson H, Johnson WE (2013) Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Med 5:28. This paper shows there is a low concordance among five widely used variant calling pipelines, suggesting the importance of improving the quality of these pipelines

  50. Otero C, Peter A (2015) Research Directions for Engineering Big Data Analytics Software. IEEE Intell Syst 30:13–19

    Article  Google Scholar 

  51. Parveen T, Tilley S (2010) When to migrate software testing to the cloud? In: Proceedings of Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), Washington, DC, 2010. pp 424–427

  52. Riungu-Kalliosaari L, Taipale O, Smolander K (2012) Testing in the cloud: Exploring the practice. IEEE Softw 29:46–51

    Article  Google Scholar 

  53. Segura S, Hierons RM, Benavides D, Ruiz-Cortés A (2010) Automated test data generation on the analyses of feature models: A metamorphic testing approach. In: Proceedings of the Third International Conference on Software Testing, Verification and Validation (ICST), Washington, DC, 2010, pp 35–44

  54. Weyuker EJ (1982) On testing non-testable programs. Comput J 25:465–470

    Article  Google Scholar 

  55. Xie X, Ho JW, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84:544–558

    PubMed Central  Article  PubMed  Google Scholar 

  56. Zhou ZQ, Huang D, Tse T, Yang Z, Huang H, Chen T (2004) Metamorphic testing and its applications. In: Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004), Xi'An, China, 2004, pp 346–351

Download references

Compliance with ethical standards


This work was supported in part by funds from the New South Wales Ministry of Health, a New South Wales Genomics Collaborative Grant, an Australian Research Council Grant, and a Microsoft Azure Research Award.

Conflict of interests

All authors (AHK, EG, TYC, MAC, ALME and JWKH) declare that they do not have any conflict of interest.

Ethical approval

This article does not contain any studies with human or animal subjects performed by any of the authors.

Author information



Corresponding author

Correspondence to Joshua W. K. Ho.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kamali, A.H., Giannoulatou, E., Chen, T.Y. et al. How to test bioinformatics software?. Biophys Rev 7, 343–352 (2015).

Download citation


  • Software testing
  • Bioinformatics
  • Quality assurance
  • Automated testing
  • Cloud-based testing