Skip to main content

Advertisement

SpringerLink
Log in

Deep Reinforcement Learning with Automated Label Extraction from Clinical Reports Accurately Classifies 3D MRI Brain Volumes

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Image classification is probably the most fundamental task in radiology artificial intelligence. To reduce the burden of acquiring and labeling data sets, we employed a two-pronged strategy. We automatically extracted labels from radiology reports in Part 1. In Part 2, we used the labels to train a data-efficient reinforcement learning (RL) classifier. We applied the approach to a small set of patient images and radiology reports from our institution. For Part 1, we trained sentence-BERT (SBERT) on 90 radiology reports. In Part 2, we used the labels from the trained SBERT to train an RL-based classifier. We trained the classifier on a training set of \(40\) images. We tested on a separate collection of \(24\) images. For comparison, we also trained and tested a supervised deep learning (SDL) classification network on the same set of training and testing images using the same labels. Part 1: The trained SBERT model improved from 82 to \(100\%\) accuracy. Part 2: Using Part 1’s computed labels, SDL quickly overfitted the small training set. Whereas SDL showed the worst possible testing set accuracy of 50%, RL achieved \(100\%\) testing set accuracy, with a \(p\)-value of \(4.9\times {10}^{-4}\). We have shown the proof-of-principle application of automated label extraction from radiological reports. Additionally, we have built on prior work applying RL to classification using these labels, extending from 2D slices to entire 3D image volumes. RL has again demonstrated a remarkable ability to train effectively, in a generalized manner, and based on small training sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. McBee, M. P. et al. Deep learning in radiology. Academic radiology 25, 1472–1480 (2018).

    Article  Google Scholar 

  2. Saba, L. et al. The present and future of deep learning in radiology. European journal of radiology 114, 14–24 (2019).

    Article  Google Scholar 

  3. Mazurowski, M. A., Buda, M., Saha, A. & Bashir, M. R. Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI. Journal of magnetic resonance imaging 49, 939–954 (2019).

    Article  Google Scholar 

  4. Parekh, V. S., Braverman, V., Jacobs, M. A., et al. Multitask radiological modality invariant landmark localization using deep reinforcement learning in Medical Imaging with Deep Learning. Proceedings of the Third Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research (2020), 588–600.

  5. Alansary, A. et al. Evaluating reinforcement learning agents for anatomical landmark detection. Medical image analysis 53, 156–164 (2019).

    Article  Google Scholar 

  6. Ghesu, F.-C. et al. Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans. IEEE transactions on pattern analysis and machine intelligence 41, 176–189 (2017).

    Article  Google Scholar 

  7. Zhou, S. K., Le, H. N., Luu, K., Nguyen, H. V. & Ayache, N. Deep rein-forcement learning in medical imaging: A literature review. arXiv preprint arXiv:2103.05115 (2021).

  8. Al, W. A. & Yun, I. D. Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images. IEEE transactions on medical imaging 39, 1245–1255 (2019).

    Google Scholar 

  9. Blair, S. I. A. S. A., White, C. & Moses, L. D. D. Localization of lumbar and thoracic vertebrae in 3d CT datasets by combining deep reinforcement learning with imitation learning (2018).

  10. Maicas, G., Carneiro, G., Bradley, A. P., Nascimento, J. C. & Reid, I. Deep reinforcement learning for active breast lesion detection from DCE-MRI. International conference on medical image computing and computer-assisted intervention (2017), 665–673.

  11. Ali, I. et al. Lung nodule detection via deep reinforcement learning. Fron-tiers in oncology 8, 108 (2018).

    Article  Google Scholar 

  12. Jang, Y. & Jeon, B. Deep Reinforcement Learning with Explicit Spatio-Sequential Encoding Network for Coronary Ostia Identification in CT Im- ages. Sensors 21, 6187 (2021).

    Article  Google Scholar 

  13. Codari, M. et al. Deep reinforcement learning for localization of the aortic annulus in patients with aortic dissection. International Workshop on Thoracic Image Analysis (2020), 94–105.

  14. Zhang, P., Wang, F. & Zheng, Y. Deep reinforcement learning for vessel centerline tracing in multi-modality 3D volumes. International Confer-ence on Medical Image Computing and Computer-Assisted Intervention (2018), 755–763.

  15. Winkel, D. J. et al. Validation of a fully automated liver segmentation algorithm using multi-scale deep reinforcement learning and comparison versus manual segmentation. European journal of radiology 126, 108918 (2020).

    Article  Google Scholar 

  16. Winkel, D. J., Breit, H.-C., Weikert, T. J. & Stieltjes, B. Building large-scale quantitative imaging databases with multi-scale deep reinforcement learning: initial experience with whole-body organ volumetric analyses. Journal of Digital Imaging 34, 124–133 (2021).

    Article  Google Scholar 

  17. Li, Z. & Xia, Y. Deep reinforcement learning for weakly-supervised lymph node segmentation in CT images. IEEE Journal of Biomedical and Health Informatics 25, 774–783 (2020).

    Article  Google Scholar 

  18. Yin, S., Han, Y. & Li, S. Left Ventricle Contouring in Cardiac Images Based on Deep Reinforcement Learning. arXiv preprint arXiv:2106.04127 (2021).

  19. Si, X. et al. Multi-step segmentation for prostate MR image based on re-inforcement learning in Medical Imaging 2020: Image-Guided Procedures. Robotic Interventions, and Modeling 11315 (2020), 113152R.

    Google Scholar 

  20. Xiong, J. et al. Edge-Sensitive Left Ventricle Segmentation Using Deep Reinforcement Learning. Sensors 21, 2375 (2021).

    Article  Google Scholar 

  21. Zhang, D., Chen, B. & Li, S. Sequential conditional reinforcement learning for simultaneous vertebral body detection and segmentation with modeling the spine anatomy. Medical Image Analysis 67, 101861 (2021).

    Article  Google Scholar 

  22. Kooi, T. et al. A comparison between a deep convolutional neural network and radiologists for classifying regions of interest in mammography. In-ternational Workshop on Breast Imaging (2016), 51–56.

  23. Stember, J. & Shalu, H. Deep reinforcement learning-based image classifi-cation achieves perfect testing set accuracy for MRI brain tumors with a training set of only 30 images. arXiv preprint arXiv:2102.02895 (2021).

  24. Stember, J. Comparison of Contextual Bandits versus Markov Decision Process Reinforcement Learning for MRI brain classification (Aug. 2021).

  25. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

  26. Vaswani, A. et al. Attention is all you need. arXiv preprint arXiv:1706.03762 (2017).

  27. Jawahar, G., Sagot, B. & Seddah, D. What does BERT learn about the structure of language? ACL 2019–57th Annual Meeting of the Associa-tion for Computational Linguistics (2019).

  28. Liu, Y. et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).

  29. Reimers, N. & Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).

  30. Stember, J. & Shalu, H. Deep reinforcement learning to detect brain le-sions on MRI: a proof-of-concept application of reinforcement learning to medical images. arXiv preprint arXiv:2008.02708 (2020).

  31. Stember, J. & Shalu, H. Unsupervised deep clustering and reinforcement learning can accurately segment MRI brain tumors with very small training sets. arXiv preprint arXiv:2012.13321 (2020).

  32. Stember, J. N. & Shalu, H. Reinforcement learning using Deep Q Networks and Q learning accurately localizes brain tumors on MRI with very small training sets. arXiv preprint arXiv:2010.10763 (2020).

  33. Dietterich, T. G. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation 10, 1895–1923 (1998).

    Article  CAS  Google Scholar 

Download references

Funding

We gratefully acknowledge support from the following sources: American Society of Neuroradiology Research Grant in Artificial Intelligence, Canon Medical Systems USA, Inc./Radiological Society of North America Research Seed Grant, Memorial Sloan Kettering Cancer Center Radiology Developmental Project Fund.

Author information

Authors and Affiliations

  1. Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA

    Joseph Nathaniel Stember

  2. Indian Institute of Technology, Madras, Chennai, India, 600036

    Hrithwik Shalu

Authors
  1. Joseph Nathaniel Stember
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hrithwik Shalu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors contributed to the study conception and design, data curation/processing, and all components of the analysis. Both authors contributed to the manuscript. Both read and approved the final/submitted version of the manuscript.

Corresponding author

Correspondence to Joseph Nathaniel Stember.

Ethics declarations

Conflict of Interest

The authors have pursued a provisional patent and conversion to full patent based on work including that described here.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stember, J.N., Shalu, H. Deep Reinforcement Learning with Automated Label Extraction from Clinical Reports Accurately Classifies 3D MRI Brain Volumes. J Digit Imaging 35, 1143–1152 (2022). https://doi.org/10.1007/s10278-022-00644-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-022-00644-5

Keywords

Search

Navigation