Abstract
Deep learning models have demonstrated great potential in medical imaging but are limited by the expensive, large volume of annotations required. To address this, we compared different active learning strategies by training models on subsets of the most informative images using real-world clinical datasets for brain tumor segmentation and proposing a framework that minimizes the data needed while maintaining performance. Then, 638 multi-institutional brain tumor magnetic resonance imaging scans were used to train three-dimensional U-net models and compare active learning strategies. Uncertainty estimation techniques including Bayesian estimation with dropout, bootstrapping, and margins sampling were compared to random query. Strategies to avoid annotating similar images were also considered. We determined the minimum data necessary to achieve performance equivalent to the model trained on the full dataset (α = 0.05). Bayesian approximation with dropout at training and testing showed results equivalent to that of the full data model (target) with around 30% of the training data needed by random query to achieve target performance (p = 0.018). Annotation redundancy restriction techniques can reduce the training data needed by random query to achieve target performance by 20%. We investigated various active learning strategies to minimize the annotation burden for three-dimensional brain tumor segmentation. Dropout uncertainty estimation achieved target performance with the least annotated data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bakas S, Reyes M, Jakab A, et al. Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge. arXiv:1811.02629.
Budd S, Robinson EC and Kainz B. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Med Image Anal 2021; 71: 102062. 2021/04/27. https://doi.org/10.1016/j.media.2021.102062.
Yousaf T, Dervenoulas G and Politis M. Chapter Two - Advances in MRI Methodology. In: Politis M (ed) International Review of Neurobiology. Academic Press, 2018, pp.31–76.
Cristobal-Huerta A, Poot DHJ, Vogel MW, et al. Compressed Sensing 3D-GRASE for faster High-Resolution MRI. Magn Reson Med 2019; 82: 984–999. 20190502.https://doi.org/10.1002/mrm.27789
Banerjee I, Bhattacharjee K, Burns JL, et al. “Shortcuts” Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and Mitigation. J Am Coll Radiol 2023; 20: 842–851. 20230727. https://doi.org/10.1016/j.jacr.2023.06.025
Wang K, Zhang D, Li Y, et al. Cost-Effective Active Learning for Deep Image Classification. arXiv:1701.03551.
Yang L, Zhang Y, Chen J, et al. Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation. arXiv:1706.04737.
Gal Y and Ghahramani Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arXiv:1506.02142.
Gal Y, Islam R and Ghahramani Z. Deep Bayesian Active Learning with Image Data. arXiv:1703.02910.
Smailagic A, Noh HY, Costa P, et al. MedAL: Deep Active Learning Sampling Method for Medical Image Analysis. arXiv:1809.09287.
Kim ST, Mushtaq F and Navab N. Confident Coreset for Active Learning in Medical Image Analysis. arXiv preprint arXiv:200402200 2020.
Warren KE, Vezina G, Poussaint TY, et al. Response assessment in medulloblastoma and leptomeningeal seeding tumors: recommendations from the Response Assessment in Pediatric Neuro-Oncology committee. Neuro Oncol 2018; 20: 13–23. 2017/04/28. https://doi.org/10.1093/neuonc/nox087.
Sharma D, Shanis Z, Reddy CK, et al. Active Learning Technique for Multimodal Brain Tumor Segmentation Using Limited Labeled Images. Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data. Cham: Springer International Publishing, 2019, pp.148–156.
Peng J, Kim DD, Patel JB, et al. Deep Learning-Based Automatic Tumor Burden Assessment of Pediatric High-Grade Gliomas, Medulloblastomas, and Other Leptomeningeal Seeding Tumors. Neuro Oncol 2021 2021/06/27. https://doi.org/10.1093/neuonc/noab151.
Ronneberger O, Fischer P and Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv:1505.04597.
Ahmed S, Srinivasu PN, Alhumam A, et al. AAL and Internet of Medical Things for Monitoring Type-2 Diabetic Patients. Diagnostics (Basel) 2022; 12 20221109. https://doi.org/10.3390/diagnostics12112739.
Settles B. Active learning literature survey. 2009.
Lakens D. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. Soc Psychol Personal Sci 2017; 8: 355–362. 20170505.https://doi.org/10.1177/1948550617697177
Magadza T and Viriri S. Deep Learning for Brain Tumor Segmentation: A Survey of State-of-the-Art. J Imaging 2021; 7. https://doi.org/10.3390/jimaging7020019.
Sailunaz K, Alhajj S, Özyer T, et al. A survey on brain tumor image analysis. Med Biol Eng Comput 2023. https://doi.org/10.1007/s11517-023-02873-4.
Ahamed MF, Hossain MM, Nahiduzzaman M, et al. A review on brain tumor segmentation based on deep learning methods with federated learning techniques. Comput Med Imaging Graph 2023; 10. https://doi.org/10.1016/j.compmedimag.2023.102313.
Wang J, Yan Y, Zhang Y, et al. Deep Reinforcement Active Learning for Medical Image Classification. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Cham: Springer International Publishing, 2020, pp.33–42.
Li H and Yin Z. Attention, Suggestion and Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Cham: Springer International Publishing, 2020, pp.3–13.
Li Y, Chen J, Xie X, et al. Self-Loop Uncertainty: A Novel Pseudo-Label for Semi-Supervised Medical Image Segmentation. arXiv:2007.09854.
Last F, Klein T, Ravanbakhsh M, et al. Human-Machine Collaboration for Medical Image Segmentation. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2020: 1040–1044.
Venturini L, Papageorghiou AT, Noble JA, et al. Uncertainty Estimates as Data Selection Criteria to Boost Omni-Supervised Learning. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I. Lima, Peru: Springer-Verlag, 2020, p. 689–698.
Mahapatra D, Bozorgtabar B, Thiran J-P, et al. Efficient Active Learning for Image Classification and Segmentation Using a Sample Selection and Conditional Generative Adversarial Network. In: Cham, 2018, pp.580–588. Springer International Publishing.
Wang H, Rivenson Y, Jin Y, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nature Methods 2019; 16: 103–110. https://doi.org/10.1038/s41592-018-0239-0.
Zhang Y, Liu S, Li C, et al. Rethinking the Dice Loss for Deep Learning Lesion Segmentation in Medical Images. Journal of Shanghai Jiaotong University (Science) 2021; 26: 93–102. https://doi.org/10.1007/s12204-021-2264-x.
Funding
This project was supported by Alpert Medical School Summer Assistantship award to DDK. This work was supported by National Science Foundation of Hunan Province, China (2022JJ30762), International Science and Technology Innovation Joint Base of Machine Vision and Medical Image Processing in Hunan Providence, China (2021CB1013), and the 111 project (B18059) to CZ. This work was supported by Hunan Province Key Areas Research and Development Program, China (2022SK2054) to BZ. This work was supported by Huxiang High-level Talent Gathering Project-Innovation Talent, China (2021RC5003) to WL. This work was supported by the Natural Science Foundation of China (81971696 to LY), Natural Science Foundation of Hunan Province (2022JJ30861 to LY), and Sheng Hua Yu Ying Project of Central South University to LY.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
RSC reports personal fees from Roivant Sciences and personal fees from Sumitovant Biopharma, outside the submitted work. CB is a consultant for Depuy-Synthes, Bionaut Labs, Galectin Therapeutics, Haystack Oncology, and Privo Technologies. CB is a co-founder of Belay Diagnostics and OrisDx.
Additional information
Pre-print
This paper has been uploaded to Arxiv for preprint at https://arxiv.org/abs/2302.10185. Since the pre-print, the submitted manuscript has been augmented with new statistical analysis. Briefly, we investigated various active learning techniques to reduce the amount of training data needed for a model without compromising model performance. In the original analysis, we compared techniques and model performance by seeing when two models had no significant difference in performance. The analysis has been now improved by investigating statistically equal performance by using two one-sided T tests. Furthermore, the preprint includes a section on algorithmically selecting the best initial dataset to perform active learning as opposed to random query. After checking for statistically equal performance, this method demonstrated no contribution to active learning techniques in our project and was therefore removed from the submitted manuscript.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Kim, D.D., Chandra, R.S., Yang, L. et al. Active Learning in Brain Tumor Segmentation with Uncertainty Sampling and Annotation Redundancy Restriction. J Digit Imaging. Inform. med. (2024). https://doi.org/10.1007/s10278-024-01037-6
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10278-024-01037-6