Journal of Digital Imaging

, Volume 32, Issue 5, pp 685–692 | Cite as

Comprehensive Word-Level Classification of Screening Mammography Reports Using a Neural Network Sequence Labeling Approach

  • Ryan G. ShortEmail author
  • John Bralich
  • Dave Bogaty
  • Nicholas T. Befera


Radiology reports contain a large amount of potentially valuable unstructured data. Recently, neural networks have been employed to perform classification of radiology reports over a few classes at the document level. The success of neural networks in sequence-labeling problems such as named entity recognition and part of speech tagging suggests that they could be used to classify radiology report text with greater granularity. We employed a neural network architecture to comprehensively classify mammography report text at the word level using a sequence labeling approach. Two radiologists devised a comprehensive classification system for screening mammography reports. Each word in each report was manually categorized by a radiologist into one of 33 categories according to the classification system. Tagged words referencing the same finding were grouped into unique sets. We pre-labeled reports with a rule-based algorithm and then manually edited these annotations for 6705 screening mammography reports (25.1%, 66.8%, and 8.1% BI-RADS 0, 1, and 2, respectively). A combined convolutional and recurrent neural network model was used to label words in each sentence of the individual reports. A siamese recurrent neural network was then used to group findings into sets. Performance of the neural network-based method was compared to a rule-based algorithm and a conditional random field (CRF) model. Global accuracy (percentage of documents where all word tags were predicted correctly) and keyword accuracy (percentage of all words that were labeled correctly, excluding words tagged as unimportant) were calculated on an unseen 519 report test set. Two-tailed t tests were used to assess differences between algorithm performance, and p < 0.05 was used to determine statistical significance. The neural network-based approach showed significantly higher global accuracy compared to both the rule-based algorithm (88.3 vs 57.0%, p < 0.001) and the CRF model (88.3% vs. 75.8%, p < 0.001). The neural network also showed significantly higher keyword level accuracy compared to the rule-based algorithm (95.5% vs. 80.9% p < 0.001) and CRF model (95.5% vs. 76.9%, p < 0.001). We demonstrate the potential of neural networks to accurately perform word-level multilabel classification of free text radiology reports across 33 classes, thus showing the utility of a sequence labeling approach to NLP of radiology reports. We found that a neural network classifier outperforms a rule-based algorithm and a CRF classifier for comprehensive multilabel classification of free text screening mammography reports at the word level. By approaching radiology report classification as a sequence-labeling problem, we demonstrate the ability of neural networks to extract data from free text radiology reports at a level of granularity not previously reported.


Natural language processing NLP Deep learning Radiology reporting 


Compliance with Ethical Standards

Conflicts of Interest

Ryan G. Short is co-founder and CMO of Scanslated, Inc.

John Bralich is an employee of Scanslated, Inc.

Dave Bogaty is an employee of Scanslated, Inc.

Nicholas T. Befera is co-founder and CEO of Scanslated, Inc.


  1. 1.
    Friedman PJ: Radiologic reporting: structure. AJR Am J Roentgenol 140:171–172, 1983CrossRefGoogle Scholar
  2. 2.
    Bruno MA, Petscavage-Thomas JM, Mohr MJ, Bell SK, Brown SD: The “Open Letter”: Radiologists’ Reports in the Era of Patient Web Portals. J Am Coll Radiol 11:863–867, 2014CrossRefGoogle Scholar
  3. 3.
    Cai T, Giannopoulos AA, Yu S, Kelil T, Ripley B, Kumamaru KK, Rybicki FJ, Mitsouras D: Natural Language Processing Technologies in Radiology Research and Clinical Applications. Radiographics 36:176–191, 2016CrossRefGoogle Scholar
  4. 4.
    Pons E, Braun LMM, Hunink MGM, Kors JA: Natural Language Processing in Radiology: A Systematic Review. Radiology 279:329–343, 2016CrossRefGoogle Scholar
  5. 5.
    Hassanpour S, Bay G, Langlotz CP: Characterization of Change and Significance for Clinical Findings in Radiology Reports Through Natural Language Processing. J Digit Imaging 30:314–322, 2017CrossRefGoogle Scholar
  6. 6.
    Masino AJ, Grundmeier RW, Pennington JW, Germiller JA, Crenshaw, 3rd. EB: Temporal bone radiology report classification using open source machine learning and natural langue processing libraries. BMC Med Inform Decis Mak 16:65, 2016CrossRefGoogle Scholar
  7. 7.
    Chen P-H, Zafar H, Galperin-Aizenberg M, Cook T: Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports. J Digit Imaging 31:178–184, 2017. CrossRefGoogle Scholar
  8. 8.
    Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J: Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst:3111–3119, 2013Google Scholar
  9. 9.
    Shin B, Chokshi FH, Lee T, Choi JD: Classification of radiology reports using neural attention models. 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, 2017, pp 4363–4370. .
  10. 10.
    Chokshi F, Shin B, Lee T, Lemmon A, Necessary S, Choi J. Natural Langeuage Processing for Classification of Acute, Communicable Findings on Unstructured Head CT Reports: Comparison of Neural Network and Non-Neural Machine Learning Techniques. bioRxiv 2017:173310. .
  11. 11.
    Chen MC, Ball RL, Yang L, Moradzadeh N, Chapman BE, Larson DB et al.: Deep Learning to Classify Radiology Free-Text Reports. Radiology 171115, 2017Google Scholar
  12. 12.
    Banerjee I, Chen MC, Lungren MP, Rubin DL: Radiology report annotation using intelligent word embeddings: Applied to multi-institutional chest CT cohort. J Biomed Inform 77:11–20, 2018CrossRefGoogle Scholar
  13. 13.
    Wang P, Qian Y, Soong FK, He L, Zhao H. Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network. arXiv [csCL] 2015.Google Scholar
  14. 14.
    Dernoncourt F, Lee JY, Szolovits P. NeuroNER: an easy-to-use program for named-entity recognition based on neural networks. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2017. .
  15. 15.
    Cornegruta S, Bakewell R, Withey S, Montana G. Modelling Radiological Language with Bidirectional Long Short-Term Memory Networks. arXiv [csCL] 2016.Google Scholar
  16. 16.
    Short RG, Middleton D, Befera NT, Gondalia R, Tailor TD: Patient-Centered Radiology Reporting: Using Online Crowdsourcing to Assess the Effectiveness of a Web-Based Interactive Radiology Report. J Am Coll Radiol 14:1489–1497, 2017CrossRefGoogle Scholar
  17. 17.
    Gale W, Oakden-Rayner L, Carneiro G, Bradley AP, Palmer LJ. Detecting hip fractures with radiologist-level performance using deep neural networks. arXiv [csCV] 2017.Google Scholar
  18. 18.
    Bird S, Klein E, Loper E. Natural Language Processing with Python. “O’Reilly Media, Inc.”; 2009.Google Scholar
  19. 19.
    Welcome to n.d. (accessed March 12, 2018).
  20. 20.
  21. 21.
    American College of Radiology. ACR BI-RADS atlas: Breast Imaging Reporting and Data System ; Mammography, Ultrasound, Magnetic Resonance Imaging, Follow-up and Outcome Monitoring, Data Dictionary. 2013.Google Scholar
  22. 22.
    Lafferty JD, Mc Callum A, Pereira FCN. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning, Morgan Kaufmann Publishers Inc.; 2001, p. 282–9.Google Scholar
  23. 23.
    Finkel JR, Grenager T, Manning C: Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2005, pp. 363–370Google Scholar
  24. 24.
    Silfverberg M, Ruokolainen T, Linden K, Kurimo M: Others. Part-of-speech tagging using conditional random fields: Exploiting sub-label dependencies for improved accuracy, 2014Google Scholar
  25. 25.
  26. 26.
    Andrew G, Gao J. Scalable training ofL1-regularized log-linear models. Proceedings of the 24th international conference on Machine learning - ICML ‘07, 2007.
  27. 27.
    Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv [csCL] 2015.Google Scholar
  28. 28.
    Řehůřek R, Sojka P: Software Framework for Topic Modelling with Large Corpora. Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Valletta, ELRA, 2010, pp. 45–50Google Scholar
  29. 29.
    Chollet F. Keras 2015.Google Scholar
  30. 30.
    Cho K, van Merrienboer B, Bahdanau D, Bengio Y. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. arXiv [csCL] 2014.Google Scholar
  31. 31.
    Mueller J, Thyagarajan A. Siamese Recurrent Architectures for Learning Sentence Similarity. AAAI 2016.Google Scholar
  32. 32.
    Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. arXiv [csLG] 2014.Google Scholar
  33. 33.
    Liu F, Baldwin T, Cohn T. Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields. arXiv [csCL] 2017.Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2018

Authors and Affiliations

  1. 1.Department of RadiologyDuke University Medical CenterDurhamUSA
  2. 2.Scanslated, Inc.DurhamUSA

Personalised recommendations