Journal of Medical Systems

, 41:32 | Cite as

An Evolving Ecosystem for Natural Language Processing in Department of Veterans Affairs

  • Jennifer H. Garvin
  • Megha Kalsy
  • Cynthia Brandt
  • Stephen L. Luther
  • Guy Divita
  • Gregory Coronado
  • Doug Redd
  • Carrie Christensen
  • Brent Hill
  • Natalie Kelly
  • Qing Zeng Treitler
Systems-Level Quality Improvement
Part of the following topical collections:
  1. Systems-Level Quality Improvement

Abstract

In an ideal clinical Natural Language Processing (NLP) ecosystem, researchers and developers would be able to collaborate with others, undertake validation of NLP systems, components, and related resources, and disseminate them. We captured requirements and formative evaluation data from the Veterans Affairs (VA) Clinical NLP Ecosystem stakeholders using semi-structured interviews and meeting discussions. We developed a coding rubric to code interviews. We assessed inter-coder reliability using percent agreement and the kappa statistic. We undertook 15 interviews and held two workshop discussions. The main areas of requirements related to; design and functionality, resources, and information. Stakeholders also confirmed the vision of the second generation of the Ecosystem and recommendations included; adding mechanisms to better understand terms, measuring collaboration to demonstrate value, and datasets/tools to navigate spelling errors with consumer language, among others. Stakeholders also recommended capability to: communicate with developers working on the next version of the VA electronic health record (VistA Evolution), provide a mechanism to automatically monitor download of tools and to automatically provide a summary of the downloads to Ecosystem contributors and funders. After three rounds of coding and discussion, we determined the percent agreement of two coders to be 97.2% and the kappa to be 0.7851. The vision of the VA Clinical NLP Ecosystem met stakeholder needs. Interviews and discussion provided key requirements that inform the design of the VA Clinical NLP Ecosystem.

Keywords

Natural language processing Formative evaluation 

References

  1. 1.
    I2b2.org. i2b2: informatics for integrating biology & the bedside. 2016. https://www.i2b2.org/. Accessed 4 Jan 2016.
  2. 2.
    Murphy, S., and Wilcox, A., Mission and sustainability of informatics for integrating biology and the bedside (i2b2). Egems (Generating Evidence & Methods To Improve Patient Outcomes) 2(2), 2014.Google Scholar
  3. 3.
    Zeng, Q. T., Redd, D., and Divita, G., Characterizing clinical text and sublanguage: A case study of the VA clinical notes. J Health MedInform S3:001, 2011.Google Scholar
  4. 4.
    Collaboration between VINCI and CHIR. 2016. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/chir.cfm. Accessed 28 April 2015.
  5. 5.
    VA Informatics and Computing Infrastructure (VINCI). 2009. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/. Accessed 1 June 2011.
  6. 6.
    Welcome to eMerge. 2016. Emerge.mc.vanderbilt.edu. http://emerge.mc.vanderbilt.edu/about-emerge/. Accessed 15 Jan 2016
  7. 7.
    Open Health Natural Language Processing (OHNLP) Consortium. 2016. Ohnlp.org. http://www.ohnlp.org/index.php/Main_Page. Accessed 16 May 2016.
  8. 8.
    Ohno-Machado, L., et al., iDASH: Integrating data for analysis, anonymization, and sharing. J. Am. Med. Inform. Assoc. 19(2):196–201, 2012.CrossRefPubMedGoogle Scholar
  9. 9.
    Rea, S., et al., Building a robust, scalable and standards-driven infrastructure for secondary use of EHR data: The SHARPn project. J. Biomed. Inform. 45(4):763–771, 2012.CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Faems, D., Open innovation: Researching a new paradigm—by H. Chesbrough, W. Vanhaverbeke and J. West. Creat Innov Manag 17(4):334–335, 2008.CrossRefGoogle Scholar
  11. 11.
    Saxenian, A., Regional advantage. Harvard University Press, Cambridge, 1994.Google Scholar
  12. 12.
    HL7 Clinical Document Architecture. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=7. Accessed 12 Aug 2016.
  13. 13.
    Lane, N., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., and Campbell, A., A survey of mobile phone sensing. IEEE Commun. Mag. 48(9):140–150, 2010.CrossRefGoogle Scholar
  14. 14.
    Messerschmitt, D., and Szyperski, C., Software ecosystem: understanding an indispensable technology and industry. MIT Press, 2003.Google Scholar
  15. 15.
    Brandenburger, A., and Nalebuff, B., Co-opetition. Doubleday, New York, 1996.Google Scholar
  16. 16.
    Lam, T., Gotz, A., Leveraging the eclipse ecosystem for the scientific community. 2005. https://accelconf.web.cern.ch/accelconf/ica05/proceedings/pdf/O4_009.pdf. Accessed 2 Oct 2015.
  17. 17.
    Iansiti, M., and Levien, R., The keystone advantage. Harvard Business School Press, Boston, 2004.Google Scholar
  18. 18.
    Gawer, A. The organization of technological platforms. Technology and organization: essays in honour of Joan Woodward, 287–296, 2010.Google Scholar
  19. 19.
    Herbsleb, J. D., Muller-Birn, C., and Towne, W. B., The VistA ecosystem: current status and future directions. Carnegie Mellon University, Pittsburgh, 2010.Google Scholar
  20. 20.
    Wallace, C., and Legro, M., Using formative evaluation in an implementation project to increase vaccination rates in high-risk veterans: QUERI series. Implement. Sci. 3(1):22, 2008.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Flick, U., The SAGE qualitative research kit. Eight volume set. Sage Publications, London, 2007.Google Scholar
  22. 22.
    Schiller, C., Winters, M., Hanson, H., and Ashe, M., A framework for stakeholder identification in concept mapping and health research: A novel process and its application to older adult mobility and the built environment. BMC Public Health 13(1):428, 2013.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Sittig, D., and Singh, H., A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 19(Suppl 3):i68–i74, 2010.CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Guest, G., MacQueen, K., and Namey, E., Applied thematic analysis. Sage Publications, Los Angeles, 2012.CrossRefGoogle Scholar
  25. 25.
    Braun, V., and Clarke, V., Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101, 2006.CrossRefGoogle Scholar
  26. 26.
    McHugh, M. L., Interrater reliability: The kappa statistic. Biochem Med 22(3):276–282, 2012.CrossRefGoogle Scholar
  27. 27.
    Lang, J. P., Redmine. 2006. http://www.redmine.org. Accessed 8 Sept 2016.
  28. 28.
    Department of Veterans Affairs Blueprint for Excellence, VHA Blueprint for Excellence. 2014. http://www.va.gov/HEALTH/docs/VHA_Blueprint_for_Excellence.pdf. Accessed 12 July 2016.

Copyright information

© Springer Science+Business Media New York (outside the USA) 2016

Authors and Affiliations

  • Jennifer H. Garvin
    • 1
    • 2
    • 3
    • 4
  • Megha Kalsy
    • 1
    • 4
  • Cynthia Brandt
    • 5
    • 6
  • Stephen L. Luther
    • 7
  • Guy Divita
    • 1
    • 4
  • Gregory Coronado
    • 1
  • Doug Redd
    • 1
    • 4
    • 8
  • Carrie Christensen
    • 1
    • 4
  • Brent Hill
    • 1
    • 4
  • Natalie Kelly
    • 1
  • Qing Zeng Treitler
    • 1
    • 4
    • 8
  1. 1.IDEAS Center SLC VA Healthcare SystemSalt Lake CityUSA
  2. 2.GRECC SLC VA Healthcare SystemSalt Lake CityUSA
  3. 3.Division of EpidemiologyUniversity of Utah School of MedicineSalt Lake CityUSA
  4. 4.Department of Biomedical InformaticsUniversity of Utah School of MedicineSalt Lake CityUSA
  5. 5.VA Connecticut Healthcare SystemWest HavenUSA
  6. 6.Yale School of MedicineNew HavenUSA
  7. 7.James A Haley Veterans HospitalTampaUSA
  8. 8.Department of Clinical Research and LeadershipGeorge Washington University School of Medicine and Health SciencesWashingtonUSA

Personalised recommendations