Skip to main content
SpringerLink
Log in

How to test bioinformatics software?

  • Review
  • Published:
Biophysical Reviews Aims and scope Submit manuscript

Abstract

Bioinformatics is the application of computational, mathematical and statistical techniques to solve problems in biology and medicine. Bioinformatics programs developed for computational simulation and large-scale data analysis are widely used in almost all areas of biophysics. The appropriate choice of algorithms and correct implementation of these algorithms are critical for obtaining reliable computational results. Nonetheless, it is often very difficult to systematically test these programs as it is often hard to verify the correctness of the output, and to effectively generate failure-revealing test cases. Software testing is an important process of verification and validation of scientific software, but very few studies have directly dealt with the issues of bioinformatics software testing. In this work, we review important concepts and state-of-the-art methods in the field of software testing. We also discuss recent reports on adapting and implementing software testing methodologies in the bioinformatics field, with specific examples drawn from systems biology and genomic medicine.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Alden K, Read M (2013) Computing: Scientific software needs quality control. Nature 502:448

    Article  CAS  PubMed  Google Scholar 

  • Baxter SM, Day SW, Fetrow JS, Reisinger SJ (2006) Scientific software development is not an oxymoron. PLoS Comput Biol 2, e87

    Article  PubMed Central  PubMed  Google Scholar 

  • Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, New York

  • Bergmann FT, Sauro HM (2008) Comparing simulation results of SBML capable simulators. Bioinformatics 24:1963–1965

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur Gener Comput Syst 25:599–616

    Article  Google Scholar 

  • Candea G, Bucur S, Zamfir C (2010) Automated software testing as a service. In: Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, Indianapolis, Indiana, USA, pp 155–160

    Chapter  Google Scholar 

  • Chan FT, Chen TY, Mak IK, Yu YT (1996) Proportional sampling strategy: guidelines for software testing practitioners. Inf Softw Technol 38:775–782

    Article  Google Scholar 

  • Chan WK, Cheung S, Leung KR (2007) A metamorphic testing approach for online testing of service-oriented software applications. Int J Web Serv Res 4:61–81

    Article  Google Scholar 

  • Check Hayden E (2013) Mozilla plan seeks to debug scientific code. Nature 501:472

    Article  CAS  Google Scholar 

  • Check Hayden E (2015) Journal buoys code-review push. Nature 520:276–277

    Article  CAS  PubMed  Google Scholar 

  • Chen L, Avizienis A (1978) N-Version Programming: A fault-tolerance approach to reliability of software operation. In: Proc. 8th IEEE Int. Symp. on Fault-Tolerant Computing (FTCS-8), Toulouse, France, June 1978, pp 3–9

  • Chen TY, Merkel R (2008) An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol 17:16

    Article  Google Scholar 

  • Chen TY, Cheung SC, Yiu S (1998) Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Dept. of Computer Science, Hong Kong University of Science and Technology

  • Chen TY, Tse T, Zhou ZQ (2003) Fault-based testing without the need of oracles. Inf Softw Technol 45:1–9

    Article  Google Scholar 

  • Chen TY, Merkel RG, Eddy G, Wong P (2004) Adaptive Random Testing Through Dynamic Partitioning. In: Proc Fourth Int Conf Quality Software, Braunschweig, Germany, Sept 2004, pp 79–86

  • Chen TY, Leung H, Mak I (2005) Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference: Higher-Level Decision Making (ASIAN 2004), Springer-Verlag, Berlin Heidelberg, pp 320–329

  • Chen TY, Ho JW, Liu H, Xie X (2009) An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, 24. This paper provides the first case study of using metamorphic testing to test bioinformatics programs.

  • Chen TY, Kuo F-C, Merkel RG, Tse T (2010) Adaptive random testing: The ART of test case diversity. J Syst Softw 83:60–66

    Article  Google Scholar 

  • Chu M, Murphy C, Kaiser G (2008) Distributed in vivo testing of software applications. In: Proc 1st Int Conf Software Testing, Verification, and Validation, Lillehammer, Norway, 2008, pp 509–512

  • Dai H, Murphy C, Kaiser G (2010) Confu: Configuration fuzzing testing framework for software vulnerability detection. Int J Secure Softw Eng 1:41–55

    Article  Google Scholar 

  • DeMilli R, Offutt AJ (1991) Constraint-based automatic test data generation. IEEE Trans Softw Eng 17:900–910

    Article  Google Scholar 

  • Evans TW, Gillespie CS, Wilkinson DJ (2008) The SBML discrete stochastic models test suite. Bioinformatics 24:285–286

    Article  CAS  PubMed  Google Scholar 

  • Gao J, Bai X, Tsai W-T (2011) Cloud testing-issues, challenges, needs and practice. Software Engineering: An International Journal 1:9–23

    Google Scholar 

  • Giannoulatou E, Park S-H, Humphreys DT, Ho JW (2014) Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie. BMC Bioinformatics 15:S15

    Article  PubMed Central  PubMed  Google Scholar 

  • Gray R, Kelly D (2010) Investigating test selection techniques for scientific software using Hook’s mutation sensitivity testing. Procedia Comput Sci 1:1487–1494

    Article  Google Scholar 

  • Hamlet RG (1977) Testing programs with the aid of a compiler. IEEE Trans Softw Eng SE-3:279–290

    Article  Google Scholar 

  • Hamlet R. (1994) Random testing. Encyclopedia of Software Engineering. Wiley, New York, pp 970–978

  • Hannay JE, MacLeod C, Singer J, Langtangen HP, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software? In: Proceedings of Workshop on Software Engineering for Computational Science and Engineering., pp 1–8

    Google Scholar 

  • Hook D, Kelly D (2009) Mutation sensitivity testing. Comput Sci Eng 11:40–47

    Article  Google Scholar 

  • Howden WE (1976) Reliability of the path analysis testing strategy. IEEE Trans Softw Eng SE-2:208–215

    Article  Google Scholar 

  • IEEE (1990) IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990 1–84.

  • IEEE (2013). Software and systems engineering — Software testing — Part 1: Concepts and definitions. ISO/IEC/IEEE 29119 1–84.

  • ISTQB I (2015) Glossary of Testing Terms. ISTQB Glossary http://www.istqb.org/downloads/finish/20/193.html.

  • Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37:649–678

    Article  CAS  Google Scholar 

  • Joppa LN, McInerny G, Harper R, Salido L, Takeda K, O’Hara K, Gavaghan D, Emmott S (2013) Troubling trends in scientific software use. Science 340:814–815

    Article  CAS  PubMed  Google Scholar 

  • Knight JC, Leveson NG (1986) An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans Softw Eng 12:96–109

    Article  Google Scholar 

  • Kuo F-C, Chen TY, Tam WK (2011) Testing embedded software by metamorphic testing: A wireless metering system case study. In: 36th IEEE Conference on Local Computer Networks (LCN), Bonn, Germany, 2011, pp 291–294

  • Langmead B, Salzberg SL (2012) Fast gapped-read alignment with Bowtie 2. Nat Methods 9:357–359

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Lanubile F, Shull F, Basili VR (1998) Experimenting with error abstraction in requirements documents. In: Proceedings of Fifth International Software Metrics Symposium, Bethesda, Maryland, 2008, pp 114–121

  • Leavitt N (2009) Is Cloud Computing Really Ready for Prime Time? Computer 42:15–20

    Google Scholar 

  • Li H, Durbin R (2009) Fast and accurate short read alignment with Burrows–Wheeler transform. Bioinformatics 25:1754–1760

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Liu H, Kuo F-C, Towey D, Chen TY (2014) How effectively does metamorphic testing alleviate the oracle problem? IEEE Trans Softw Eng 40:4–22

    Article  Google Scholar 

  • McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Cazier J-B, Donnelly P (2014) Choice of transcripts and software has a large effect on variant annotation. Genome Med 6:26

    Article  PubMed Central  PubMed  Google Scholar 

  • Merali Z (2010) Computational science: Error, why scientific programming does not compute. Nature 467:775–777

    Article  CAS  PubMed  Google Scholar 

  • Murphy C, Kaiser GE, Hu L (2008) Properties of machine learning applications for use in metamorphic testing. In: Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE, San Francisco, CA, pp 867–872

  • Murphy C, Shen K, Kaiser G (2009a) Automatic system testing of programs without test oracles. In: Proceedings of the eighteenth international symposium on Software testing and analysis, Chicago, IL, 2009, pp 189–200

  • Murphy C, Kaiser G, Vo I, Chu M (2009b) Quality assurance of software applications using the in vivo testing approach. In: Proceedings of International Conference on Software Testing Verification and Validation, Denver CO, 2009, pp 111–120

  • Myers GJ, Sandler C, Badgett T (2011) The art of software testing. Wiley, New York

  • O’Rawe J, Jiang T, Sun G, Wu Y, Wang W, Hu J, Bodily P, Tian L, Hakonarson H, Johnson WE (2013) Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Med 5:28. This paper shows there is a low concordance among five widely used variant calling pipelines, suggesting the importance of improving the quality of these pipelines

  • Otero C, Peter A (2015) Research Directions for Engineering Big Data Analytics Software. IEEE Intell Syst 30:13–19

    Article  Google Scholar 

  • Parveen T, Tilley S (2010) When to migrate software testing to the cloud? In: Proceedings of Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), Washington, DC, 2010. pp 424–427

  • Riungu-Kalliosaari L, Taipale O, Smolander K (2012) Testing in the cloud: Exploring the practice. IEEE Softw 29:46–51

    Article  Google Scholar 

  • Segura S, Hierons RM, Benavides D, Ruiz-Cortés A (2010) Automated test data generation on the analyses of feature models: A metamorphic testing approach. In: Proceedings of the Third International Conference on Software Testing, Verification and Validation (ICST), Washington, DC, 2010, pp 35–44

  • Weyuker EJ (1982) On testing non-testable programs. Comput J 25:465–470

    Article  Google Scholar 

  • Xie X, Ho JW, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84:544–558

    Article  PubMed Central  PubMed  Google Scholar 

  • Zhou ZQ, Huang D, Tse T, Yang Z, Huang H, Chen T (2004) Metamorphic testing and its applications. In: Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004), Xi'An, China, 2004, pp 346–351

Download references

Compliance with ethical standards

ᅟ

Funding

This work was supported in part by funds from the New South Wales Ministry of Health, a New South Wales Genomics Collaborative Grant, an Australian Research Council Grant, and a Microsoft Azure Research Award.

Conflict of interests

All authors (AHK, EG, TYC, MAC, ALME and JWKH) declare that they do not have any conflict of interest.

Ethical approval

This article does not contain any studies with human or animal subjects performed by any of the authors.

Author information

Authors and Affiliations

  1. Victor Chang Cardiac Research Institute, Darlinghurst, NSW, 2010, Australia

    Amir Hossein Kamali, Eleni Giannoulatou & Joshua W. K. Ho

  2. School of Electrical and Information Engineering, The University of Sydney, Sydney, NSW, 2006, Australia

    Amir Hossein Kamali & Alistair L. McEwan

  3. St. Vincent’s Clinical School, The University of New South Wales, Sydney, NSW, 2010, Australia

    Eleni Giannoulatou & Joshua W. K. Ho

  4. Department of Computer Science and Software Engineering, Swinburne University of Technology, Melbourne, VIC, Australia

    Tsong Yueh Chen

  5. School of Physical Sciences, The University of Tasmania, Hobart, TAS, Australia

    Michael A. Charleston

Authors
  1. Amir Hossein Kamali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eleni Giannoulatou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tsong Yueh Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael A. Charleston
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alistair L. McEwan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Joshua W. K. Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joshua W. K. Ho.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kamali, A.H., Giannoulatou, E., Chen, T.Y. et al. How to test bioinformatics software?. Biophys Rev 7, 343–352 (2015). https://doi.org/10.1007/s12551-015-0177-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12551-015-0177-3

Keywords

Search

Navigation