Biophysical Reviews

, Volume 7, Issue 3, pp 343–352 | Cite as

How to test bioinformatics software?

  • Amir Hossein Kamali
  • Eleni Giannoulatou
  • Tsong Yueh Chen
  • Michael A. Charleston
  • Alistair L. McEwan
  • Joshua W. K. Ho


Bioinformatics is the application of computational, mathematical and statistical techniques to solve problems in biology and medicine. Bioinformatics programs developed for computational simulation and large-scale data analysis are widely used in almost all areas of biophysics. The appropriate choice of algorithms and correct implementation of these algorithms are critical for obtaining reliable computational results. Nonetheless, it is often very difficult to systematically test these programs as it is often hard to verify the correctness of the output, and to effectively generate failure-revealing test cases. Software testing is an important process of verification and validation of scientific software, but very few studies have directly dealt with the issues of bioinformatics software testing. In this work, we review important concepts and state-of-the-art methods in the field of software testing. We also discuss recent reports on adapting and implementing software testing methodologies in the bioinformatics field, with specific examples drawn from systems biology and genomic medicine.


Software testing Bioinformatics Quality assurance Automated testing Cloud-based testing 


Compliance with ethical standards


This work was supported in part by funds from the New South Wales Ministry of Health, a New South Wales Genomics Collaborative Grant, an Australian Research Council Grant, and a Microsoft Azure Research Award.

Conflict of interests

All authors (AHK, EG, TYC, MAC, ALME and JWKH) declare that they do not have any conflict of interest.

Ethical approval

This article does not contain any studies with human or animal subjects performed by any of the authors.


  1. Alden K, Read M (2013) Computing: Scientific software needs quality control. Nature 502:448CrossRefPubMedGoogle Scholar
  2. Baxter SM, Day SW, Fetrow JS, Reisinger SJ (2006) Scientific software development is not an oxymoron. PLoS Comput Biol 2, e87PubMedCentralCrossRefPubMedGoogle Scholar
  3. Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, New YorkGoogle Scholar
  4. Bergmann FT, Sauro HM (2008) Comparing simulation results of SBML capable simulators. Bioinformatics 24:1963–1965PubMedCentralCrossRefPubMedGoogle Scholar
  5. Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur Gener Comput Syst 25:599–616CrossRefGoogle Scholar
  6. Candea G, Bucur S, Zamfir C (2010) Automated software testing as a service. In: Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, Indianapolis, Indiana, USA, pp 155–160CrossRefGoogle Scholar
  7. Chan FT, Chen TY, Mak IK, Yu YT (1996) Proportional sampling strategy: guidelines for software testing practitioners. Inf Softw Technol 38:775–782CrossRefGoogle Scholar
  8. Chan WK, Cheung S, Leung KR (2007) A metamorphic testing approach for online testing of service-oriented software applications. Int J Web Serv Res 4:61–81CrossRefGoogle Scholar
  9. Check Hayden E (2013) Mozilla plan seeks to debug scientific code. Nature 501:472CrossRefGoogle Scholar
  10. Check Hayden E (2015) Journal buoys code-review push. Nature 520:276–277CrossRefPubMedGoogle Scholar
  11. Chen L, Avizienis A (1978) N-Version Programming: A fault-tolerance approach to reliability of software operation. In: Proc. 8th IEEE Int. Symp. on Fault-Tolerant Computing (FTCS-8), Toulouse, France, June 1978, pp 3–9Google Scholar
  12. Chen TY, Merkel R (2008) An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol 17:16CrossRefGoogle Scholar
  13. Chen TY, Cheung SC, Yiu S (1998) Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Dept. of Computer Science, Hong Kong University of Science and TechnologyGoogle Scholar
  14. Chen TY, Tse T, Zhou ZQ (2003) Fault-based testing without the need of oracles. Inf Softw Technol 45:1–9CrossRefGoogle Scholar
  15. Chen TY, Merkel RG, Eddy G, Wong P (2004) Adaptive Random Testing Through Dynamic Partitioning. In: Proc Fourth Int Conf Quality Software, Braunschweig, Germany, Sept 2004, pp 79–86Google Scholar
  16. Chen TY, Leung H, Mak I (2005) Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference: Higher-Level Decision Making (ASIAN 2004), Springer-Verlag, Berlin Heidelberg, pp 320–329Google Scholar
  17. Chen TY, Ho JW, Liu H, Xie X (2009) An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, 24. This paper provides the first case study of using metamorphic testing to test bioinformatics programs.Google Scholar
  18. Chen TY, Kuo F-C, Merkel RG, Tse T (2010) Adaptive random testing: The ART of test case diversity. J Syst Softw 83:60–66CrossRefGoogle Scholar
  19. Chu M, Murphy C, Kaiser G (2008) Distributed in vivo testing of software applications. In: Proc 1st Int Conf Software Testing, Verification, and Validation, Lillehammer, Norway, 2008, pp 509–512Google Scholar
  20. Dai H, Murphy C, Kaiser G (2010) Confu: Configuration fuzzing testing framework for software vulnerability detection. Int J Secure Softw Eng 1:41–55CrossRefGoogle Scholar
  21. DeMilli R, Offutt AJ (1991) Constraint-based automatic test data generation. IEEE Trans Softw Eng 17:900–910CrossRefGoogle Scholar
  22. Evans TW, Gillespie CS, Wilkinson DJ (2008) The SBML discrete stochastic models test suite. Bioinformatics 24:285–286CrossRefPubMedGoogle Scholar
  23. Gao J, Bai X, Tsai W-T (2011) Cloud testing-issues, challenges, needs and practice. Software Engineering: An International Journal 1:9–23Google Scholar
  24. Giannoulatou E, Park S-H, Humphreys DT, Ho JW (2014) Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie. BMC Bioinformatics 15:S15PubMedCentralCrossRefPubMedGoogle Scholar
  25. Gray R, Kelly D (2010) Investigating test selection techniques for scientific software using Hook’s mutation sensitivity testing. Procedia Comput Sci 1:1487–1494CrossRefGoogle Scholar
  26. Hamlet RG (1977) Testing programs with the aid of a compiler. IEEE Trans Softw Eng SE-3:279–290CrossRefGoogle Scholar
  27. Hamlet R. (1994) Random testing. Encyclopedia of Software Engineering. Wiley, New York, pp 970–978Google Scholar
  28. Hannay JE, MacLeod C, Singer J, Langtangen HP, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software? In: Proceedings of Workshop on Software Engineering for Computational Science and Engineering., pp 1–8Google Scholar
  29. Hook D, Kelly D (2009) Mutation sensitivity testing. Comput Sci Eng 11:40–47CrossRefGoogle Scholar
  30. Howden WE (1976) Reliability of the path analysis testing strategy. IEEE Trans Softw Eng SE-2:208–215CrossRefGoogle Scholar
  31. IEEE (1990) IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990 1–84.Google Scholar
  32. IEEE (2013). Software and systems engineering — Software testing — Part 1: Concepts and definitions. ISO/IEC/IEEE 29119 1–84.Google Scholar
  33. ISTQB I (2015) Glossary of Testing Terms. ISTQB Glossary
  34. Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37:649–678CrossRefGoogle Scholar
  35. Joppa LN, McInerny G, Harper R, Salido L, Takeda K, O’Hara K, Gavaghan D, Emmott S (2013) Troubling trends in scientific software use. Science 340:814–815CrossRefPubMedGoogle Scholar
  36. Knight JC, Leveson NG (1986) An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans Softw Eng 12:96–109CrossRefGoogle Scholar
  37. Kuo F-C, Chen TY, Tam WK (2011) Testing embedded software by metamorphic testing: A wireless metering system case study. In: 36th IEEE Conference on Local Computer Networks (LCN), Bonn, Germany, 2011, pp 291–294Google Scholar
  38. Langmead B, Salzberg SL (2012) Fast gapped-read alignment with Bowtie 2. Nat Methods 9:357–359PubMedCentralCrossRefPubMedGoogle Scholar
  39. Lanubile F, Shull F, Basili VR (1998) Experimenting with error abstraction in requirements documents. In: Proceedings of Fifth International Software Metrics Symposium, Bethesda, Maryland, 2008, pp 114–121Google Scholar
  40. Leavitt N (2009) Is Cloud Computing Really Ready for Prime Time? Computer 42:15–20Google Scholar
  41. Li H, Durbin R (2009) Fast and accurate short read alignment with Burrows–Wheeler transform. Bioinformatics 25:1754–1760PubMedCentralCrossRefPubMedGoogle Scholar
  42. Liu H, Kuo F-C, Towey D, Chen TY (2014) How effectively does metamorphic testing alleviate the oracle problem? IEEE Trans Softw Eng 40:4–22CrossRefGoogle Scholar
  43. McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Cazier J-B, Donnelly P (2014) Choice of transcripts and software has a large effect on variant annotation. Genome Med 6:26PubMedCentralCrossRefPubMedGoogle Scholar
  44. Merali Z (2010) Computational science: Error, why scientific programming does not compute. Nature 467:775–777CrossRefPubMedGoogle Scholar
  45. Murphy C, Kaiser GE, Hu L (2008) Properties of machine learning applications for use in metamorphic testing. In: Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE, San Francisco, CA, pp 867–872Google Scholar
  46. Murphy C, Shen K, Kaiser G (2009a) Automatic system testing of programs without test oracles. In: Proceedings of the eighteenth international symposium on Software testing and analysis, Chicago, IL, 2009, pp 189–200Google Scholar
  47. Murphy C, Kaiser G, Vo I, Chu M (2009b) Quality assurance of software applications using the in vivo testing approach. In: Proceedings of International Conference on Software Testing Verification and Validation, Denver CO, 2009, pp 111–120Google Scholar
  48. Myers GJ, Sandler C, Badgett T (2011) The art of software testing. Wiley, New YorkGoogle Scholar
  49. O’Rawe J, Jiang T, Sun G, Wu Y, Wang W, Hu J, Bodily P, Tian L, Hakonarson H, Johnson WE (2013) Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Med 5:28. This paper shows there is a low concordance among five widely used variant calling pipelines, suggesting the importance of improving the quality of these pipelinesGoogle Scholar
  50. Otero C, Peter A (2015) Research Directions for Engineering Big Data Analytics Software. IEEE Intell Syst 30:13–19CrossRefGoogle Scholar
  51. Parveen T, Tilley S (2010) When to migrate software testing to the cloud? In: Proceedings of Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), Washington, DC, 2010. pp 424–427Google Scholar
  52. Riungu-Kalliosaari L, Taipale O, Smolander K (2012) Testing in the cloud: Exploring the practice. IEEE Softw 29:46–51CrossRefGoogle Scholar
  53. Segura S, Hierons RM, Benavides D, Ruiz-Cortés A (2010) Automated test data generation on the analyses of feature models: A metamorphic testing approach. In: Proceedings of the Third International Conference on Software Testing, Verification and Validation (ICST), Washington, DC, 2010, pp 35–44Google Scholar
  54. Weyuker EJ (1982) On testing non-testable programs. Comput J 25:465–470CrossRefGoogle Scholar
  55. Xie X, Ho JW, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84:544–558PubMedCentralCrossRefPubMedGoogle Scholar
  56. Zhou ZQ, Huang D, Tse T, Yang Z, Huang H, Chen T (2004) Metamorphic testing and its applications. In: Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004), Xi'An, China, 2004, pp 346–351Google Scholar

Copyright information

© International Union for Pure and Applied Biophysics (IUPAB) and Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Amir Hossein Kamali
    • 1
    • 2
  • Eleni Giannoulatou
    • 1
    • 3
  • Tsong Yueh Chen
    • 4
  • Michael A. Charleston
    • 5
  • Alistair L. McEwan
    • 2
  • Joshua W. K. Ho
    • 1
    • 3
  1. 1.Victor Chang Cardiac Research InstituteDarlinghurstAustralia
  2. 2.School of Electrical and Information EngineeringThe University of SydneySydneyAustralia
  3. 3.St. Vincent’s Clinical School, The University of New South WalesSydneyAustralia
  4. 4.Department of Computer Science and Software EngineeringSwinburne University of TechnologyMelbourneAustralia
  5. 5.School of Physical SciencesThe University of TasmaniaHobartAustralia

Personalised recommendations