Advertisement

Information Retrieval Journal

, Volume 19, Issue 1–2, pp 113–148 | Cite as

State-of-the-art in biomedical literature retrieval for clinical cases: a survey of the TREC 2014 CDS track

  • Kirk RobertsEmail author
  • Matthew Simpson
  • Dina Demner-Fushman
  • Ellen Voorhees
  • William Hersh
Medical Information Retrieval

Abstract

Providing access to relevant biomedical literature in a clinical setting has the potential to bridge a critical gap in evidence-based medicine. Here, our goal is specifically to provide relevant articles to clinicians to improve their decision-making in diagnosing, treating, and testing patients. To this end, the TREC 2014 Clinical Decision Support Track evaluated a system’s ability to retrieve relevant articles in one of three categories (Diagnosis, Treatment, Test) using an idealized form of a patient medical record . Over 100 submissions from over 25 participants were evaluated on 30 topics, resulting in over 37k relevance judgments. In this article, we provide an overview of the task, a survey of the information retrieval methods employed by the participants, an analysis of the results, and a discussion on the future directions for this challenging yet important task.

Keywords

Biomedical information retrieval Clinical decision support Information retrieval evaluation 

Notes

Acknowledgments

Kirk Roberts, Matthew Simpson, and Dina Demner-Fushman were supported by the intramural research program at the U.S. National Library of Medicine, National Institutes of Health. The authors would also like to thank the following participants for providing feedback and clarifications: Raymond Wan, Paul McNamee, Jean Garcia-Gathright, Joao Palotti, Eva D’hondt, Dawit Girmay, Afshin Deroie, Sungbin Choi, Luca Soldaini, Joe McCarthy, and Yi-Shu Wei.

References

  1. Amati, G., & Van Rijsbergen, C. J. (2002). Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems, 20(4), 357–389.CrossRefGoogle Scholar
  2. An, X., & Cercone, N. (2014). How complementary are different information retrieval techniques? a study in biomedicine domain. In Proceedings of Computational Linguistics and Intelligent Text Processing (CICLing), pp. 367–380.Google Scholar
  3. Arighi, C. N., Roberts, P. M., Agarwal, S., Bhattacharya, S., Cesareni, G., Chatr-aryamontri, A., et al. (2011). BioCreative III interactive task: An overview. BMC Bioinformatics, 12(Suppl 8), S4.CrossRefGoogle Scholar
  4. Aronson, A., & Lang, F. M. (2010). An overview of MetaMap: Historical perspective and recent advances. Journal of the American Medical Informatics Association, 17, 229–236.CrossRefGoogle Scholar
  5. Aronson, A. R., Mork, J. G., Gay, C. W., Humphrey, S. M., & Rogers, W. J. (2004). The NLM indexing initiative’s medical text indexer. In Studies in Health Technology and Informatics (MEDINFO), pp. 268–272.Google Scholar
  6. Banks, D., Over, P., & Zhang, N. F. (1999). Blind men and elephants: Six approaches to TREC data. Information Retrieval, 1, 7–34.CrossRefGoogle Scholar
  7. Bhandari, A., Kulkarni, A. (2014). San Francisco State University at TREC 2014: Clinical Decision Support Track. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  8. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.zbMATHGoogle Scholar
  9. Chapman, W. W., & Cohen, K. B. (2009). Current issues in biomedical text mining and natural language processing. Journal of Biomedical Informatics, 42(5), 757–759.CrossRefGoogle Scholar
  10. Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001). A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics, 34(5), 301–310.CrossRefGoogle Scholar
  11. Choi, S., & Choi, J. (2014). SNUMedinfo at TREC CDSS track 2014: Medical case-based retrieval task. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  12. Clinchant, S., & Gaussier, E. (2010). Information-based Models for Ad Hoc IR. In Proceedings of the 33rd Annual ACM International Conference on Research and Development in Information Retrieval.Google Scholar
  13. Del Fiol, G., Workman, T. E., & Gorman, P. N. (2014). Clinical questions raised by clinicians at the point of care: A systematic review. JAMA Internal Medicine, 174(5), 710–718. pMID24663331.CrossRefGoogle Scholar
  14. Demner-Fushman, D., Chapman, W. W., & McDonald, C. J. (2009). What can natural language processing do for clinical decision support? Journal of Biomedical Informatics, 42(5), 760–772.CrossRefGoogle Scholar
  15. D’hondt, E., Grau, B., Darmoni, S., Névéol, A., Schuers, M., & Zweigenbaum, P. (2014). LIMSI @ TREC clinical decision support track. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  16. Dinh, D., & Ben Abacha, A. (2014). CRP Henri Tudor at TREC 2014: Combining Search Results for Clinical Decision Support. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  17. Fellbaum, C. (1998). WordNet: An electronic lexical database. Cambridge: MIT Press.zbMATHGoogle Scholar
  18. Garcia-Gathright, J., Meng, F., & Hsu, W. (2014). UCLA at TREC 2014 Clinical Decision Support Track: Exploring Language Models, Query Expansion, and Boosting. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  19. Girmay, D., & Deroie, A. (2014). Query expansion using SNOMED-CT and weighing schemes. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  20. Gobeill, J., Gaudinat, A., Pasche, E., & Ruch, P. (2014). Full-texts representation with Medical Subjects Headings, and co-citations network reranking strategies for TREC 2014 Clinical Decision Support Track. In Proceedings of the 2014 Text Retrieval Conference Google Scholar
  21. Goeuriot, L., Jones, G. J., Kelly, L., Leveling, J., Hanbury, A., Müller, H., Salanterä, S., Suominen, H., & Zuccon, G. (2013). ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: Information retrieval to address patients’ questions when reading clinical reports. In CLEF 2013 Working Notes.Google Scholar
  22. Goeuriot, L., Kelly, L., Li, W., Palotti, J., Pecina, P., Zuccon, G., Hanbury, A., Jones, G. J., & Müller, H. (2014). ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred health information retrieval. In CLEF 2014 Working Notes, pp. 43–61Google Scholar
  23. Goodwin, T., & Harabagiu, S. (2014). UTD at TREC 2014: Query expansion for clinical decision support. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  24. Hasan, S. A., Zhu, X., Dong, Y., Liu, J., & Farri, O. (2014). A hybrid approach to clinical question answering. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  25. Hatcher, E., & Gospodnetic, O. (2004). Lucene in action. Greenwich: Manning Publications.Google Scholar
  26. Haynes, R. B., McKibbon, K. A., Wilczynski, N. L., Walter, S. D., & Werre, S. R., Hedges Team (2005). Optimal search strategies for retrieving scientifically strong studies of treatment from medline: Analytical survey. BMJ, 330, 1179–1185.CrossRefGoogle Scholar
  27. Hersh, W., & Bhupatiraju, R. T. (2003). TREC genomics track overview. In Proceedings of the Twelfth Text Retrieval Conference.Google Scholar
  28. Hersh, W., & Kim, E. (2006). The impact of relevance judgments and data fusion on results of image retrieval test collections. In Proceedings of the Second MUSCLE/ImageCLEF Workshop on Image and Video Retrieval Evaluation, pp. 29–38Google Scholar
  29. Hersh, W., Buckley, C., Leone, T., & Hickam, D. (1994a). OHSUMED: An interactive retrieval evaluation and new large test collection for research. In Proceedings of the 17th Annual ACM International Conference on Research and Development in Information Retrieval, pp. 192–201Google Scholar
  30. Hersh, W., Bhupatiraju, R. T., Ross, L., Johnson, P., Cohen, A. M., & Kraemer, D. F. (2004). TREC 2004 genomics track overview. In Proceedings of the Thirteenth Text Retrieval Conference.Google Scholar
  31. Hersh, W., Cohen, A., Yang, J., Bhupatiraju, R. T., Roberts, P., & Hearst, M. (2005). TREC 2005 genomics track overview. In Proceedings of the Fourteenth Text Retrieval Conference.Google Scholar
  32. Hersh, W., Cohen, A. M., Roberts, P., & Rekapalli, H. K. (2006). TREC 2006 genomics track overview. In Proceedings of the Fifteenth Text Retrieval Conference.Google Scholar
  33. Hersh, W., Cohen, A., Ruslen, L., & Roberts, P. (2007). TREC 2007 genomics track overview. In Proceedings of the Sixteenth Text Retrieval Conference.Google Scholar
  34. Hersh, W. R., & Hickam, D. H. (1995). An evaluation of interactive boolean and natural language searching with an online medical textbook. Journal of the American Society for Information Science, 46(7), 478–489.CrossRefGoogle Scholar
  35. Hersh, W. R., Hickam, D. H., Haynes, R. B., & McKibbon, K. A. (1994b). A performance and failure analysis of SAPHIRE with a MEDLINE test collection. Journal of American Biomedical Informatics, 1(1), 51–60.Google Scholar
  36. Hu, Q., Huang, J. X., & Miao, J. (2011). A robust approach to optimizing multi-source information for enhancing genomics retrieval performance. BMC Bioinformatics, 12(Suppl 5), S6.CrossRefGoogle Scholar
  37. Jelinek, F., & Mercer, R. (1980). Interpolated estimation of markovsource parameters from sparse data. Pattern Recognition in Practice pp. 381–402.Google Scholar
  38. Joo, H., & Sohn, K. (2014). TREC2014 clinical decision support: Concept-based clinical information retrieval using MetaMap. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  39. Kalpathy-Cramer, J., de Herrera, A. G. S., Demner-Fushman, D., Antani, S., Bedrick, S., & Müller, H. (2014). Evaluating performance of biomedical image retrieval systems: An overview of the medical image retrieval task at ImageCLEF 2004–2013. Computerized Medical Imaging and Graphics.Google Scholar
  40. Kim, J. D., & Cohen, K. B. (2013). Natural language query processing for SPARQL generation—A prototype system for SNOMEDCT. In Proceedings of BioLINK, pp. 32–36.Google Scholar
  41. Li, M., Song, Y., He, Y., Hu, Q., He, L., & Haacke, E. M. (2014). ECNU at TREC 2014: Clinical decision support track. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  42. Lindberg, D. A., Humphreys, B. L., & McCray, A. T. (1993). The unified medical language system. Methods of Information in Medicine, 32(4), 281–291.Google Scholar
  43. Liu, S., Ma, W., Moore, R., Ganesan, V., & Nelson, S. (2005). RxNorm: Prescription for electronic drug information exchange. IT Professional, 7(5), 17–23.Google Scholar
  44. Lu, Z. (2011). PubMed and beyond: A survey of web tools for searching biomedical literature. Database 2011Google Scholar
  45. McDonald, C. J., Huff, S. M., Suico, J. G., Hill, G., Leavelle, D., Aller, R., et al. (2003). LOINC, a universal standard for identifying laboratory observations: A 5-year update. Clinical Chemistry, 49(4), 624–633.CrossRefGoogle Scholar
  46. McNamee, P., Mayfield, J., & Piatko, C. (2002). HAIRCUT: A system for multilingual text retrieval in Java. Journal of Computing Sciences in Colleges, 17(3), 8–22.Google Scholar
  47. Mourão, A., Martins, F., & Magalhães, J. (2014). NovaSearch at TREC 2014 clinical decision support track. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  48. NCBI (2010). Entrez programming utilities help. National Center for Biotechnology InformationGoogle Scholar
  49. Oh, H. S., & Jung, Y. (2014). KISTI at TREC 2014 clinical decision support track: Concept-based document re-ranking to biomedical document retrieval. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  50. Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., & Lioma, C. (2006). Terrier: A high performance and scalable information retrieval platform. In SIGIR Open Source Workshop.Google Scholar
  51. Palotti, J., Rekabsaz, N., Anderson, L., & Hanbury, A. (2014). TUW @ TREC clinical decision support track. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  52. Ryu, B., & Choi, J. (2013). Biomedical test collection with multiple query representation. In Proceedings of the Fifth International Workshop on Evaluating Information Access, pp. 33–36.Google Scholar
  53. Sankhavara, J., Thakrar, F., Sarkar, S., & Majumder, P. (2014). Fusing manual and machine feedback in biomedical domain. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  54. Scott, D., Lee, J., Silva, I., Park, S., Moody, G., Celi, L., & Mark, R. (2013). Accessing the public MIMIC-II intensive care relational database for clinical research. BMC Medical Informatics and Decision Making, 13(9). http://www.biomedcentral.com/1472-6947/13/9.
  55. Singh, A., & Chowdary, C. R. (2014). Centrality based document ranking. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  56. Soldaini, L., Cohan, A., Yates, A., Goharian, N., & Frieder, O. (2014). Query reformulation for clinical decision support search. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  57. Spärck Jones, K., Walker, S., & Robertson, S. E. (2000). A probabilistic model of information retrieval: Development and comparative experiments. Information Processing and Management, 36(6), 779–840.CrossRefGoogle Scholar
  58. Stearns, M. Q., Price, C., Spackman, K. A., & Yang, A. Y. (2001). SNOMED clinical terms: Overview of the development process and project status. In Proceedings of the AMIA Annual Symposium, pp. 662–666.Google Scholar
  59. Strohman, T., Metzler, D., Turtle, H., & Croft, W. B. (2005). Indri: A language-model based search engine for complex queries. In International Conference on Intelligence Analysis Google Scholar
  60. Voorhees, E. M. (2014). The effect of sampling strategy on inferred measures. In Proceedings of the 37th Annual ACM International Conference on Research and Development in Information Retrieval, pp. 1119–1122.Google Scholar
  61. Voorhees, E. M., & Hersh, W. (2012). Overview of the TREC 2012 medical records track. In Proceedings of the 11th Text REtrieval Conference.Google Scholar
  62. Voorhees, E. M., & Tong, R. M. (2011). Overview of the TREC 2011 medical records track. In Proceedings of the 10th Text REtrieval Conference.Google Scholar
  63. Wan, R., Man, J. H. K., & Chan, T. F. (2014). Query modification through external sources to support clinical decisions. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  64. Wang, X., Chused, A., Elhadad, N., Friedman, C., & Markatou, M. (2008). Automated knowledge acquisition from clinical narrative reports. In Proceedings of the AMIA Annual Symposium, pp. 783–787.Google Scholar
  65. Wang, X., Thompson, P., Tsujii, J., & Ananiadou, S. (2012). Biomedical Chinese-English CLIR using an extended CMeSH resource to expand queries. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pp. 1148–1155.Google Scholar
  66. Wang, Y., & Fang, H. (2014). Explore the query expansion methods for concept based representation. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  67. Wei, Y., Hsu, C., Thomas, A., & McCarthy, J. F. (2014). Atigeo at TREC 2014 clinical decision support task. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  68. Weinreich, S. S., Mangon, R., Sikkens, J., Teeuw, M., & Cornel, M. (2008). OrphaNet: A european database for rare diseases. Nederlands tijdschrift voor geneeskunde, 152(9), 518–519.Google Scholar
  69. Wilbur, W. J., & Kim, W. (2014). Stochastic gradient descent and the prediction of MeSH for PubMed records. InProceedings of the AMIA Annual Symposium, pp. 1198–1207.Google Scholar
  70. Wing, C., & Yang, H. (2014). Query refinement: Negation detection and proximity learning: Georgetown at TREC 2014 clinical decision support track. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  71. Wu, Y., Denny, J. C., Rosenbloom, S. T., Miller, R. A., Giuse, D. A., & Xu, H. (2012). A comparative study on current clinical natural language processing systems on handling abbreviations in discharge summaries. In Proceedings of the AMIA Annual Symposium, pp. 997–1003.Google Scholar
  72. Xu, T., McNamee, P., & Oard, D. W. (2014). HLTCOE at TREC 2014: Microblog and clinical decision support. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  73. Xue, T., Fu, Q., Gu, H., Zhang, S., & Wang, C. (2014). Clinical decision support track of 2014. In Proceedings of the 2014 Text Retrieval Conference.Google Scholar
  74. Yilmaz, E., Kanoulas, E., & Aslam, J. A. (2008). A simple and efficient sampling method for estimating AP and NDCG. In Proceedings of the 31st Annual ACM International Conference on Research and Development in Information Retrieval, pp. 603–610.Google Scholar
  75. Zhang, X., Cole, M., & Belkin, N. (2011). Predicting users’ domain knowledge from search behaviors. In Proceedings of the 34th Annual ACM International Conference on Research and Development in Information Retrieval, pp. 1225–1226.Google Scholar

Copyright information

© Springer Science+Business Media New York (outside the USA) 2015

Authors and Affiliations

  • Kirk Roberts
    • 1
    Email author
  • Matthew Simpson
    • 1
  • Dina Demner-Fushman
    • 1
  • Ellen Voorhees
    • 2
  • William Hersh
    • 3
  1. 1.Lister Hill National Center for Biomedical Communications, National Library of MedicineNational Institutes of HealthBethesdaUSA
  2. 2.National Institute of Standards and TechnologyGaithersburgUSA
  3. 3.Department of Medical Informatics and Clinical EpidemiologyOregon Health & Science UniversityPortlandUSA

Personalised recommendations