Journal of Digital Imaging

, Volume 25, Issue 1, pp 30–36 | Cite as

Automated Detection of Critical Results in Radiology Reports

  • Paras LakhaniEmail author
  • Woojin Kim
  • Curtis P. Langlotz


The goal of this study was to develop and validate text-mining algorithms to automatically identify radiology reports containing critical results including tension or increasing/new large pneumothorax, acute pulmonary embolism, acute cholecystitis, acute appendicitis, ectopic pregnancy, scrotal torsion, unexplained free intraperitoneal air, new or increasing intracranial hemorrhage, and malpositioned tubes and lines. The algorithms were developed using rule-based approaches and designed to search for common words and phrases in radiology reports that indicate critical results. Certain text-mining features were utilized such as wildcards, stemming, negation detection, proximity matching, and expanded searches with applicable synonyms. To further improve accuracy, the algorithms utilized modality and exam-specific queries, searched under the “Impression” field of the radiology report, and excluded reports with a low level of diagnostic certainty. Algorithm accuracy was determined using precision, recall, and F-measure using human review as the reference standard. The overall accuracy (F-measure) of the algorithms ranged from 81% to 100%, with a mean precision and recall of 96% and 91%, respectively. These algorithms can be applied to radiology report databases for quality assurance and accreditation, integrated with existing dashboards for display and monitoring, and ported to other institutions for their own use.


Algorithms Communication Critical Results Reporting Data Mining Natural Language Processing Quality Assurance Quality Control 



This research was funded by the Society for Imaging Informatics (SIIM) Research Grant.


  1. 1.
    Patient safety requirement 2 C (standard NPSG.2a). In: The 2007 Comprehensive Accreditation Manual for Hospitals: The Official Handbook. Oakbrook Terrace, IL: Joint Commission Resources, 2007, pp NPSG-3–NPSG-4Google Scholar
  2. 2.
    American College of Radiology: ACR Practice Guideline for Communication of Diagnostic Imaging Findings. In: Practice Guidelines & Technical Standards 2005. Reston, VA: American College of Radiology, 2005Google Scholar
  3. 3.
    Lakhani P, Langlotz CP: Automated detection of radiology reports that document non-routine communication of critical or significant radiology results. J Digital Imaging 23:647–57, 2010. Dec (Epub 2008 Oct)CrossRefGoogle Scholar
  4. 4.
    Hersh WR, Detmer WM, Frisse ME: Information-retrieval systems. In: Medical Informatics. Springer, New York, NY, 2001, pp 539–572Google Scholar
  5. 5.
    Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG: A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 34:301–310, 2001PubMedCrossRefGoogle Scholar
  6. 6.
    Hripcsak G, Rothschild AS: Agreement, the F-measure, and reliability in information retrieval. J Am Med Inform Assoc. 12:296–298, 2005PubMedCrossRefGoogle Scholar
  7. 7.
    Lacson R, Khorasani R: Natural language processing: the basics (part 1). J Am Coll Radiol. 8(6):436–7, 2011PubMedCrossRefGoogle Scholar
  8. 8.
    Goldstein I, Arzrumtsyan A, Uzuner O: Three approaches to automatic assignment of ICD-9-CM codes to radiology reports. AMIA Annu Symp Proc 11:279–283, 2007Google Scholar
  9. 9.
    Imai T, Aramaki E, Kajino M, Miyo K, Onogi Y, Ohe K: Finding malignant findings from radiological reports using medical attributes and syntactic information. Stud Health Technol Inform 129:540–544, 2007PubMedGoogle Scholar
  10. 10.
    Mamlin BW, Heinze DT, McDonald CJ: Automated extraction and normalization of findings from cancer-related free-text radiology reports. AMIA Annu Symp Proc 420–424, 2003Google Scholar
  11. 11.
    Hripcsak G, Austin JH, Alderson PO, Friedman C: Use of natural language processing to translate clinical information from a database of 889,921 chest radiographic reports. Radiology 224:157–163, 2002PubMedCrossRefGoogle Scholar
  12. 12.
    Taira RK, Soderland SG: A statistical natural language processor for medical reports. Proc AMIA Symp 970–974, 1999Google Scholar
  13. 13.
    Dreyer KJ, Kalra MK, Maher MM, Hurier AM, Asfaw BA, Schultz T, Halpern EF, Thrall JH: Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: validation study. Radiology 234:323–329, 2005PubMedCrossRefGoogle Scholar
  14. 14.
    Cheng LT, Zheng J, Savova GK, Erickson BJ: Discerning tumor status from unstructured MRI reports—completeness of information in existing reports and utility of automated natural language processing. J Digit Imaging 23(2):119–132, 2010. Apr (Epub 2009 May 30)PubMedCrossRefGoogle Scholar
  15. 15.
    Hayes-Roth F: Rule-based systems. Commun ACM 28:921–932, 1985CrossRefGoogle Scholar
  16. 16.
    Baud R, Lovis C, Rassinoux AM, Michel PA, Scherrer JR: Automatic extraction of linguistic knowledge from an international classification. Stud Health Technol Inform 52(Pt 1):581–585, 1998PubMedGoogle Scholar
  17. 17.
    Wilcox AB, Hripcsak G: The role of domain knowledge in automating medical text report classification. J Am Med Inform Assoc 10(4):330–338, 2003PubMedCrossRefGoogle Scholar
  18. 18.
    Lakhani P, Langlotz CP: Documentation of nonroutine communications of critical or significant radiology results: a multiyear experience at a tertiary hospital. J Am Coll Radiol 7(10):782–790, 2010PubMedCrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2011

Authors and Affiliations

  • Paras Lakhani
    • 1
    Email author
  • Woojin Kim
    • 1
  • Curtis P. Langlotz
    • 1
  1. 1.Department of RadiologyHospital of the University of PennsylvaniaPhiladelphiaUSA

Personalised recommendations