Skip to main content
View expanded cover

International Workshop on Eternal Systems

EternalS 2012: Trustworthy Eternal Systems via Evolving Software, Data and Knowledge pp 77–90Cite as

Assessment of Software Testing and Quality Assurance in Natural Language Processing Applications and a Linguistically Inspired Approach to Improving It

  • Conference paper

Part of the Communications in Computer and Information Science book series (CCIS,volume 379)

Abstract

Significant progress has been made in addressing the scientific challenges of biomedical text mining. However, the transition from a demonstration of scientific progress to the production of tools on which a broader community can rely requires that fundamental software engineering requirements be addressed. In this paper we characterize the state of biomedical text mining software with respect to software testing and quality assurance. Biomedical natural language processing software was chosen because it frequently specifically claims to offer production-quality services, rather than just research prototypes.

We examined twenty web sites offering a variety of text mining services. On each web site, we performed the most basic software test known to us and classified the results. Seven out of twenty web sites returned either bad results or the worst class of results in response to this simple test. We conclude that biomedical natural language processing tools require greater attention to software quality.

We suggest a linguistically motivated approach to granular evaluation of natural language processing applications, and show how it can be used to detect performance errors of several systems and to predict overall performance on specific equivalence classes of inputs.

We also assess the ability of linguistically-motivated test suites to provide good software testing, as compared to large corpora of naturally-occurring data. We measure code coverage and find that it is considerably higher when even small structured test suites are utilized than when large corpora are used.

Keywords

  • Test Suite
  • Natural Language Processing
  • Software Testing
  • Code Coverage
  • Natural Language Processing System

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-45260-4_6
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-45260-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Baumgartner Jr., W.A., Lu, Z., Johnson, H.L., Gregory Caporaso, J., Paquette, J., Lindemann, A., White, E.K., Medvedeva, O., Bretonnel Cohen, K., Hunter, L.E.: Concept recognition for extraction protein interaction relations from biomedical text. Genome Biology 9(suppl. 2), S9 (2008)

    Google Scholar 

  2. Beizer, B.: Software testing techniques, 2nd edn. International Thomson Computer Press (1990)

    Google Scholar 

  3. Beizer, B.: Black-box testing: Techniques for functional testing of software and systems. Wiley (1995)

    Google Scholar 

  4. Binder, R.V.: Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Professional (1999)

    Google Scholar 

  5. Chang, G., Roth, C.R., Reyes, C.L., Pornillos, O., Chen, Y.-J., Chen, A.P.: Letters: Retraction. Science 314, 1875 (2006)

    CrossRef  Google Scholar 

  6. Bretonnel Cohen, K., Tanabe, L., Kinoshita, S., Hunter, L.: A resource for constructing customized test suites for molecular biology entity identification systems. In: BioLINK 2004: Linking Biological Literature, Ontologies, and Databases: Tools for Users, pp. 1–8. Association for Computational Linguistics (2004)

    Google Scholar 

  7. Bretonnel Cohen, K., Baumgartner Jr., W.A., Hunter, L.: Software testing and the naturally occurring data assumption in natural language processing. In: Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pp. 23–30. Association for Computational Linguistics (2008)

    Google Scholar 

  8. Food and Drug Administration, US Department of Health and Human Services, General principles of software validation: Final guidance for industry and FDA staff (2002)

    Google Scholar 

  9. Hakenberg, J., Leaman, R., Vo, N.H., Jonnalagadda, S., Sullivan, R., Miller, C., Tari, L., Baral, C., Gonzalez, G.: Efficient extraction of protein-protein interactions from full-text articles. IEEE/ACM Transactions on Computational Biology and Bioinformatics (July 2010)

    Google Scholar 

  10. Hirschman, L., Mani, I.: Evaluation. In: Mitkov, R. (ed.) The Oxford Handbook of Computational Linguistics, ch. 23. Oxford University Press (2003)

    Google Scholar 

  11. Hunter, L.E., Lu, Z., Firby, J., Baumgartner Jr., W.A., Johnson, H.L., Ogren, P.V., Bretonnel Cohen, K.: OpenDMAP: An open-source, ontology driven concept analysis engine, with applications to capturing knowledge regarding protein transport, protein interactions and cell-specific gene expression. BMC Bioinformatics 9(78) (2008)

    Google Scholar 

  12. Johnson, H.L., Bretonnel Cohen, K., Hunter, L.: A fault model for ontology mapping, alignment, and linking systems. In: Pacific Symposium on Biocomputing 2007, pp. 233–244. World Scientific Publishing (2007)

    Google Scholar 

  13. Kaner, C., Nguyen, H.Q., Falk, J.: Testing computer software, 2nd edn. John Wiley and Sons (1999)

    Google Scholar 

  14. Marick, B.: The craft of software testing: subsystem testing including object-based and object-oriented testing. Prentice Hall (1997)

    Google Scholar 

  15. McConnell, S.: Code complete, 2nd edn. Microsoft Press (2004)

    Google Scholar 

  16. Miller, G.: A scientist’s nightmare: software problem leads to five retractions. Science 314, 1856–1857 (2006)

    CrossRef  Google Scholar 

  17. Oepen, S., Netter, K., Klein, J.: TSNLP – test suites for natural language processing. In: Nerbonne, J. (ed.) Linguistic Databases, ch. 2, pp. 13–36. CSLI Publications (1998)

    Google Scholar 

  18. Palmer, M., Finin, T.: Workshop on the evaluation of natural language processing systems. Computational Linguistics 16(3), 175–181 (1990)

    Google Scholar 

  19. Partee, B.H., ter Meulen, A., Wall, R.E.: Mathematical methods in linguistics. Springer (1990)

    Google Scholar 

  20. Rebholz-Schuhmann, D., Kirsch, H., Couto, F.: Facts from text—is text mining ready to deliver? PLoS Biology 3(2), 188–191 (2005)

    CrossRef  Google Scholar 

  21. Steedman, M.: On becoming a discipline. Computational Linguistics 34(1), 137–144 (2008)

    CrossRef  Google Scholar 

  22. Tanabe, L., John Wilbur, W.: Tagging gene and protein names in biomedical text. Bioinformatics 18(8), 1124–1132 (2002)

    CrossRef  Google Scholar 

  23. Tanabe, L., Xie, N., Thom, L.H., Matten, W., John Wilbur, W.: GENETAG: a tagged corpus for gene/protein name recognition. BMC Bioinformatics 6(suppl. 1), S4 (2005)

    Google Scholar 

  24. Verspoor, K., Bretonnel Cohen, K., Mani, I., Goertzel, B.: Introduction to BioNLP 2006. In: Linking Natural Language Processing and Biology: Towards Deeper Biological Literature Analysis, pp. iii–iv. Association for Computational Linguistics (2006)

    Google Scholar 

  25. Zweigenbaum, P., Demner-Fushman, D., Yu, H., Bretonnel Cohen, K.: Frontiers for biomedical text mining: current progress. Briefings in Bioinformatics 8(5) (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computational Bioscience Program, University of Colorado School of Medicine, Aurora, Colorado, USA

    K. Bretonnel Cohen, Lawrence E. Hunter & Martha Palmer

  2. Department of Linguistics, University of Colorado at Boulder, Boulder, Colorado, USA

    K. Bretonnel Cohen, Lawrence E. Hunter & Martha Palmer

Authors
  1. K. Bretonnel Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lawrence E. Hunter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martha Palmer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Information Engineering and Computer Science, University of Trento, Via Sommarive 5, 38123, Povo, Trento, Italy

    Alessandro Moschitti & Barbara Plank & 

Rights and permissions

Reprints and Permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Cohen, K.B., Hunter, L.E., Palmer, M. (2013). Assessment of Software Testing and Quality Assurance in Natural Language Processing Applications and a Linguistically Inspired Approach to Improving It. In: Moschitti, A., Plank, B. (eds) Trustworthy Eternal Systems via Evolving Software, Data and Knowledge. EternalS 2012. Communications in Computer and Information Science, vol 379. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45260-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-45260-4_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-45259-8

  • Online ISBN: 978-3-642-45260-4

  • eBook Packages: Computer ScienceComputer Science (R0)