An Evolving Ecosystem for Natural Language Processing in Department of Veterans Affairs
In an ideal clinical Natural Language Processing (NLP) ecosystem, researchers and developers would be able to collaborate with others, undertake validation of NLP systems, components, and related resources, and disseminate them. We captured requirements and formative evaluation data from the Veterans Affairs (VA) Clinical NLP Ecosystem stakeholders using semi-structured interviews and meeting discussions. We developed a coding rubric to code interviews. We assessed inter-coder reliability using percent agreement and the kappa statistic. We undertook 15 interviews and held two workshop discussions. The main areas of requirements related to; design and functionality, resources, and information. Stakeholders also confirmed the vision of the second generation of the Ecosystem and recommendations included; adding mechanisms to better understand terms, measuring collaboration to demonstrate value, and datasets/tools to navigate spelling errors with consumer language, among others. Stakeholders also recommended capability to: communicate with developers working on the next version of the VA electronic health record (VistA Evolution), provide a mechanism to automatically monitor download of tools and to automatically provide a summary of the downloads to Ecosystem contributors and funders. After three rounds of coding and discussion, we determined the percent agreement of two coders to be 97.2% and the kappa to be 0.7851. The vision of the VA Clinical NLP Ecosystem met stakeholder needs. Interviews and discussion provided key requirements that inform the design of the VA Clinical NLP Ecosystem.
KeywordsNatural language processing Formative evaluation
- 1.I2b2.org. i2b2: informatics for integrating biology & the bedside. 2016. https://www.i2b2.org/. Accessed 4 Jan 2016.
- 2.Murphy, S., and Wilcox, A., Mission and sustainability of informatics for integrating biology and the bedside (i2b2). Egems (Generating Evidence & Methods To Improve Patient Outcomes) 2(2), 2014.Google Scholar
- 3.Zeng, Q. T., Redd, D., and Divita, G., Characterizing clinical text and sublanguage: A case study of the VA clinical notes. J Health MedInform S3:001, 2011.Google Scholar
- 4.Collaboration between VINCI and CHIR. 2016. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/chir.cfm. Accessed 28 April 2015.
- 5.VA Informatics and Computing Infrastructure (VINCI). 2009. Hsrd.research.va.gov. http://www.hsrd.research.va.gov/for_researchers/vinci/. Accessed 1 June 2011.
- 6.Welcome to eMerge. 2016. Emerge.mc.vanderbilt.edu. http://emerge.mc.vanderbilt.edu/about-emerge/. Accessed 15 Jan 2016
- 7.Open Health Natural Language Processing (OHNLP) Consortium. 2016. Ohnlp.org. http://www.ohnlp.org/index.php/Main_Page. Accessed 16 May 2016.
- 11.Saxenian, A., Regional advantage. Harvard University Press, Cambridge, 1994.Google Scholar
- 12.HL7 Clinical Document Architecture. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=7. Accessed 12 Aug 2016.
- 14.Messerschmitt, D., and Szyperski, C., Software ecosystem: understanding an indispensable technology and industry. MIT Press, 2003.Google Scholar
- 15.Brandenburger, A., and Nalebuff, B., Co-opetition. Doubleday, New York, 1996.Google Scholar
- 16.Lam, T., Gotz, A., Leveraging the eclipse ecosystem for the scientific community. 2005. https://accelconf.web.cern.ch/accelconf/ica05/proceedings/pdf/O4_009.pdf. Accessed 2 Oct 2015.
- 17.Iansiti, M., and Levien, R., The keystone advantage. Harvard Business School Press, Boston, 2004.Google Scholar
- 18.Gawer, A. The organization of technological platforms. Technology and organization: essays in honour of Joan Woodward, 287–296, 2010.Google Scholar
- 19.Herbsleb, J. D., Muller-Birn, C., and Towne, W. B., The VistA ecosystem: current status and future directions. Carnegie Mellon University, Pittsburgh, 2010.Google Scholar
- 21.Flick, U., The SAGE qualitative research kit. Eight volume set. Sage Publications, London, 2007.Google Scholar
- 27.Lang, J. P., Redmine. 2006. http://www.redmine.org. Accessed 8 Sept 2016.
- 28.Department of Veterans Affairs Blueprint for Excellence, VHA Blueprint for Excellence. 2014. http://www.va.gov/HEALTH/docs/VHA_Blueprint_for_Excellence.pdf. Accessed 12 July 2016.