Automated Detection of Radiology Reports that Require Follow-up Imaging Using Natural Language Processing Feature Engineering and Machine Learning Classification

  • Robert LouEmail author
  • Darco Lalevic
  • Charles Chambers
  • Hanna M. Zafar
  • Tessa S. Cook
Original Paper


While radiologists regularly issue follow-up recommendations, our preliminary research has shown that anywhere from 35 to 50% of patients who receive follow-up recommendations for findings of possible cancer on abdominopelvic imaging do not return for follow-up. As such, they remain at risk for adverse outcomes related to missed or delayed cancer diagnosis. In this study, we develop an algorithm to automatically detect free text radiology reports that have a follow-up recommendation using natural language processing (NLP) techniques and machine learning models. The data set used in this study consists of 6000 free text reports from the author’s institution. NLP techniques are used to engineer 1500 features, which include the most informative unigrams, bigrams, and trigrams in the training corpus after performing tokenization and Porter stemming. On this data set, we train naive Bayes, decision tree, and maximum entropy models. The decision tree model, with an F1 score of 0.458 and accuracy of 0.862, outperforms both the naive Bayes (F1 score of 0.381) and maximum entropy (F1 score of 0.387) models. The models were analyzed to determine predictive features, with term frequency of n-grams such as “renal neoplasm” and “evalu with enhanc” being most predictive of a follow-up recommendation. Key to maximizing performance was feature engineering that extracts predictive information and appropriate selection of machine learning algorithms based on the feature set.


Artificial intelligence Binary classification Follow-up Machine learning Natural language processing Structured reporting 



  1. 1.
    Dutta S, Long WJ, Brown DF, Reisner AT: Automated detection using natural language processing of radiologists recommendations for additional imaging of incidental findings. Ann Emerg Med 62(2):162–169, 2013CrossRefGoogle Scholar
  2. 2.
    Yetisgen-Yildiz M, Gunn ML, Xia F, Payne TH: Automatic identification of critical follow-up recommendation sentences in radiology reports. AMIA Annu Symp Proc 2011:1593–1602, 2011Google Scholar
  3. 3.
    Cook TS, Lalevic D, Sloan C, Chadalavada SC, Langlotz CP, Schnall MD, Zafar HM: Implementation of an automated radiology recommendation-tracking engine for abdominal imaging findings of possible cancer. J Am Coll Radiol. 14(5):629–636, 2017CrossRefGoogle Scholar
  4. 4.
    Langlotz CP: Structured radiology reporting: are we there yet? Radiology. 253(1):23–25, 2009CrossRefGoogle Scholar
  5. 5.
    Bosmans JM, Peremans L, Menni M, De Schepper AM, Duyck PO, Parizel PM: Structured reporting: if, why, when, how-and at what expense? Results of a focus group meeting of radiology professionals from eight countries. Insights Imaging. 3(3):295–302, 2012CrossRefGoogle Scholar
  6. 6.
    Pons E, Braun LM, Hunink MG, Kors JA: Natural language processing in radiology: a systematic review. Radiology. 279(2):329–343, 2016CrossRefGoogle Scholar
  7. 7.
    Xu Y, Tsujii J, Chang EI: Named entity recognition of follow-up and time information in 20,000 radiology reports. J Am Med Inform Assoc. 19(5):792–799, 2012CrossRefGoogle Scholar
  8. 8.
    Yetisgen-Yildiz M, Gunn ML, Xia F, Payne TH: A text processing pipeline to extract recommendations from radiology reports. J Biomed Inform 46(2):354–362, 2013CrossRefGoogle Scholar
  9. 9.
    Zafar HM, Chadalavada SC, Kahn CE, Cook TS, Sloan CE, Lalevic D et al.: Code abdomen: an assessment coding scheme for abdominal imaging findings possibly representing cancer. J Am Coll Radiol JACR. 12(9):947–950, 2015CrossRefGoogle Scholar
  10. 10.
    The Porter Stemming Algorithm. Accessed June 1, 2017.
  11. 11.
    Peng HC, Long F, Ding C: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence. 27(8):1226–1238, 2005CrossRefGoogle Scholar
  12. 12.
    MEGA Model Optimization Package. Accessed June 1, 2017.
  13. 13.
    Pennington J, Socher R, Manning C: Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543CrossRefGoogle Scholar
  14. 14.
    Chen MC, Ball RL, Yang L, Moradzadeh N, Chapman BE, Larson DB, Langlotz CP, Amrhein TJ, Lungren MP: Deep learning to classify radiology free-text reports. Radiology. 286(3):845–852, 2018CrossRefGoogle Scholar
  15. 15.
    Chen PH, Zafar H, Galperin-Aizenberg M, Cook T: Integrating natural language processing and machine learning algorithms to categorize oncologic response in radiology reports. J Digit Imaging. 31(2):178–184, 2018CrossRefGoogle Scholar
  16. 16.
    Hassanpour S, Langlotz CP, Amrhein TJ, Befera NT, Lungren MP: Performance of a machine learning classifier of knee MRI reports in two large academic radiology practices: a tool to estimate diagnostic yield. AJR Am J Roentgenol. 208(4):750–753, 2017CrossRefGoogle Scholar
  17. 17.
    Hassanpour S, Bay G, Langlotz CP: Characterization of change and significance for clinical findings in radiology reports through natural language processing. J Digit Imaging. 30(3):314–322, 2017CrossRefGoogle Scholar
  18. 18.
    Bird, Steven, Ewan, Klein, and Loper, Edward (2009), Natural language processing with Python, O’Reilly Media..Google Scholar
  19. 19.
    Garla V, Taylor C, Brandt C: Semi-supervised clinical text classification with Laplacian SVMs: an application to cancer case management. J Biomed Inform 46(5):869–875, 2013CrossRefGoogle Scholar
  20. 20.
    Reiner BI: Quantitative analysis of uncertainty in medical reporting: creating a standardized and objective methodology. J Digit Imaging. 31(2):145–149, 2018CrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  1. 1.Perelman School of Medicine at the University of PennsylvaniaPhiladelphiaUSA
  2. 2.Hospital of the University of PennsylvaniaPhiladelphiaUSA

Personalised recommendations