Recognition of medication information from discharge summaries using ensembles of classifiers
Extraction of clinical information such as medications or problems from clinical text is an important task of clinical natural language processing (NLP). Rule-based methods are often used in clinical NLP systems because they are easy to adapt and customize. Recently, supervised machine learning methods have proven to be effective in clinical NLP as well. However, combining different classifiers to further improve the performance of clinical entity recognition systems has not been investigated extensively. Combining classifiers into an ensemble classifier presents both challenges and opportunities to improve performance in such NLP tasks.
We investigated ensemble classifiers that used different voting strategies to combine outputs from three individual classifiers: a rule-based system, a support vector machine (SVM) based system, and a conditional random field (CRF) based system. Three voting methods were proposed and evaluated using the annotated data sets from the 2009 i2b2 NLP challenge: simple majority, local SVM-based voting, and local CRF-based voting.
Evaluation on 268 manually annotated discharge summaries from the i2b2 challenge showed that the local CRF-based voting method achieved the best F-score of 90.84% (94.11% Precision, 87.81% Recall) for 10-fold cross-validation. We then compared our systems with the first-ranked system in the challenge by using the same training and test sets. Our system based on majority voting achieved a better F-score of 89.65% (93.91% Precision, 85.76% Recall) than the previously reported F-score of 89.19% (93.78% Precision, 85.03% Recall) by the first-ranked system in the challenge.
Our experimental results using the 2009 i2b2 challenge datasets showed that ensemble classifiers that combine individual classifiers into a voting system could achieve better performance than a single classifier in recognizing medication information from clinical text. It suggests that simple strategies that can be easily implemented such as majority voting could have the potential to significantly improve clinical entity recognition.
- Hirschman, L, Morgan, AA, Yeh, AS (2002) Rutabaga by any other name: extracting biological names. J Biomed Inform 35: pp. 247-259 CrossRef
- Krauthammer, M, Nenadic, G (2004) Term identification in the biomedical literature. J Biomed Inform 37: pp. 512-526 CrossRef
- Kazama, J, Makino, T, Ohta, Y, Tsujii, T (2002) Tuning Support Vector Machines for biomedical named entity recognition. Proceedings of the ACL-02 Workshop on Natural Language Processing in the Biomedical Domain. pp. 1-8 CrossRef
- Smith, L, Tanabe, L, Ando, R, Kuo, C, Chung, I, Hsu, C, Lin, Y, Klinger, R, Friedrich, C, Ganchev, K, Torii, M, Liu, H, Haddow, B, Struble, C, Povinelli, R, Vlachos, A, Baumgartner, W, Hunter, L, Carpenter, B, Tsai, R, Dai, H, Liu, F, Chen, Y, Sun, C, Katrenko, S, Adriaans, P, Blaschke, C, Torres, R, Neves, M, Nakov, P, Divoli, M, Mana-Lopez, A, Mata-Vazquez, J, Wilbur, W (2008) Overview of BioCreative II gene mention recognition. Genome Biol 9: pp. S2 CrossRef
- Takeuchi, K, Collier, N (2003) Bio-medical entity extraction using Support Vector Machines. Proceedings of the ACL 2003 workshop on Natural language processing in biomedicine. pp. 57-64 CrossRef
- Torii, M, Hu, Z, Wu, CH, Liu, H (2009) BioTagger-GM: a gene/protein name recognition system. J Am Med Inform Assoc 16: pp. 247-255 CrossRef
- Yamamoto, K, Kudo, T, Konagaya, A, Matsumoto, Y (2003) Protein name tagging for biomedical annotation in text. Proceedings of ACL,workshop on Natural Language Processing in Biomedicine. pp. 65-72 CrossRef
- Zhou, G, Shen, D, Zhang, J, Su, J, Tan, S (2005) Recognition of protein/gene names from text using an ensemble of classifiers. BMC Bioinformatics 6: pp. S7 CrossRef
- Uzuner, Ö, Solti, I, Cadag, E (2010) Extracting medication information from clinical text. J Am Med Inform Assoc 17: pp. 514-518 CrossRef
- Patrick, J, Li, M (2010) High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge. J Am Med Inform Assoc 17: pp. 524-527 CrossRef
- Doan, S, Bastarache, L, Klimkowski, S, Denny, JC, Xu, H (2010) Integrating existing natural language processing tools for medication extraction from discharge summaries. J Am Med Inform Assoc 17: pp. 528-531 CrossRef
- Halgrim, S, Xia, F, Solti, I, Cadag, E, Uzuner, Ö (2010) Statistical extraction of medication information from clinical records. Proceedings of AMIA Summit on Translational Bioinformatics. pp. 10-12
- Halgrim, S, Xia, F, Solti, I, Cadag, E, Uzuner, Ö (2011) A cascade of classifiers for extracting medication information from discharge summaries. Journal of Biomedical Semantics 2: pp. S2 CrossRef
- Doan, S, Xu, H (2010) Recognizing medication related entities in hospital discharge summaries using support vector machine. Proceedings of the 23th Conference on Computational Linguistics. pp. 259-266
- Uzuner, Ö, Solti, I, Xia, F, Cadag, E (2010) Community annotation experiment for ground truth generation for the i2b2 medication challenge. J Am Med Inform Assoc 17: pp. 519-523 CrossRef
- Xu, H, Stenner, SP, Doan, S, Johnson, KB, Waitman, LR, Denny, JC (2010) MedEx: a medication information extraction system for clinical narratives. J Am Med Inform Assoc 17: pp. 19-24 CrossRef
- Settles, B (2004) Biomedical named entity recognition using conditional random fields and rich feature sets. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP). pp. 104-107 CrossRef
- Burges, CJC (1998) A Tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2: pp. 121-167 CrossRef
- Vapnik, VN (1995) The Nature of Statistical Learning.
- TinySVM toolkit[ http://chasen.org/~taku/software/TinySVM/]
- Yamcha toolkit[ http://chasen.org/~taku/software/YamCha/]
- Li, Z, Liu, F, Antieau, L, Cao, Y, Yu, H (2010) Lancet: a high precision medication event extraction system for clinical text. J Am Med Inform Assoc 17: pp. 563-567 CrossRef
- CRF++ toolkit. [ http://crfpp.sourceforge.net/]
- NLTK package. [ www.nltk.org]
- Collier, N, Nobata, C, Tsujii, J (2000) Extracting the names of genes and gene products with a hidden Markov model. Proceedings of the 18th Conference on Computational Linguistics. pp. 201-207 CrossRef
- Breiman, L (1996) Bagging predictors. Machine Learning 24: pp. 123-140
- Schapire, RE, Singer, Y (1998) Improved boosting algorithms using confidence-rated predictions. Proceedings of the 11th Annual ACM Conference on Computational Learning Theory.
- ConllEval evaluation script. [ http://www.cnts.ua.ac.be/conll2002/ner/bin/conlleval.txt]
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/12/36/prepub
- Recognition of medication information from discharge summaries using ensembles of classifiers
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
BMC Medical Informatics and Decision Making
- Online Date
- May 2012
- Online ISSN
- BioMed Central
- Additional Links
- Author Affiliations
- 1. National Institute of Informatics, Hitotsubashi, Chiyoda, Tokyo, Japan
- 2. Department of Biomedical Informatics, School of Medicine, Vanderbilt University, Nashville, TN, USA
- 3. Department of Computer Science, Posts and Telecommunications Institute of Technology, Hanoi, Vietnam