Abstract
Ischemic stroke segmentation at an acute stage is vital in assessing the severity of patients’ impairment and guiding therapeutic decision-making for reperfusion. Although many deep learning studies have shown attractive performance in medical segmentation, it is difficult to use these models trained on public data with private hospitals’ datasets. Here, we demonstrate an ensemble model that employs two different multimodal approaches for generalization, a more effective way to perform on external datasets. First, after we jointly train a segmentation model on diffusion-weighted imaging (DWI) and apparent diffusion coefficient (ADC) MR modalities, the model is inferred on the DWI images. Second, a channel-wise segmentation model is trained by concatenating the DWI and ADC images as input, and then is inferred using both MR modalities. Before training with ischemic stroke data, we utilized BraTS 2021, a public brain tumor dataset, for transfer learning. An extensive ablation study evaluates which strategy learns better representations for ischemic stroke segmentation. In our study, nnU-Net well-known for robustness is selected as our baseline model. Our proposed method is evaluated on three different datasets: the Asan Medical Center (AMC) I and II, and the 2022 Ischemic Stroke Lesion Segmentation (ISLES). Our experiments are widely validated over a large, multi-center, and multi-scanner dataset with a huge amount of 846 scans. Not only stroke lesion models can benefit from transfer learning using brain tumor data, but combining the MR modalities using different training schemes also highly improves segmentation performance. The method achieved a top-1 ranking in the ongoing ISLES’22 challenge and performed particularly well on lesion-wise metrics of interest to neuroradiologists, achieving a Dice coefficient of 78.69% and a lesion-wise F1 score of 82.46%. Also, the method was relatively robust on the AMC I (Dice, 60.35%; lesion-wise F1, 68.30%) and II (Dice; 74.12%; lesion-wise F1, 67.53%) datasets in different settings. The high segmentation accuracy of our proposed method could improve radiologists’ ability to detect ischemic stroke lesions in MRI images. Our model weights and inference code are available on https://github.com/MDOpx/ISLES22-model-inference.
Similar content being viewed by others
Data Availability
Available upon request.
References
R.L. Sacco, S.E. Kasner, J.P. Broderick, L.R. Caplan, J.J. Connors, A. Culebras, M.S. Elkind, M.G. George, A.D. Hamdan, R.T. Higashida, B.L. Hoh, L.S. Janis, C.S. Kase, D.O. Kleindorfer, J.M. Lee, M.E. Moseley, E.D. Peterson, T.N. Turan, A.L. Valderrama, H.V. Vinters, C.o.C.S. American Heart Association Stroke Council, Anesthesia, R. Council on Cardiovascular, Intervention, C. Council on, N. Stroke, E. Council on, Prevention, D. Council on Peripheral Vascular, P.A. Council on Nutrition, Metabolism, An updated definition of stroke for the 21st century: a statement for healthcare professionals from the American Heart Association/American Stroke Association, Stroke, 44 (2013) 2064–2089.
J.L. Saver, Time is brain--quantified, Stroke, 37 (2006) 263-266.
G. Zaharchuk, I.S. El Mogy, N.J. Fischbein, G.W. Albers, Comparison of arterial spin labeling and bolus perfusion-weighted imaging for detecting mismatch in acute stroke, Stroke, 43 (2012) 1843-1848.
C. Yoon, S. Misra, K.-J. Kim, C. Kim, B.J. Kim, Collaborative multi-modal deep learning and radiomic features for classification of strokes within 6 h, Expert Systems with Applications, 228 (2023) 120473.
J. Vymazal, A.M. Rulseh, J. Keller, L. Janouskova, Comparison of CT and MR imaging in ischemic stroke, Insights Imaging, 3 (2012) 619-627.
B.L. Edlow, S. Hurwitz, J.A. Edlow, Diagnosis of DWI-negative acute ischemic stroke: A meta-analysis, Neurology, 89 (2017) 256-262.
C.Z. Simonsen, M.H. Madsen, M.L. Schmitz, I.K. Mikkelsen, M. Fisher, G. Andersen, Sensitivity of diffusion- and perfusion-weighted imaging for diagnosing acute ischemic stroke is 97.5%, Stroke, 46 (2015) 98–101.
E.C. Jauch, J.L. Saver, H.P. Adams, Jr., A. Bruno, J.J. Connors, B.M. Demaerschalk, P. Khatri, P.W. McMullan, Jr., A.I. Qureshi, K. Rosenfield, P.A. Scott, D.R. Summers, D.Z. Wang, M. Wintermark, H. Yonas, C. American Heart Association Stroke, N. Council on Cardiovascular, D. Council on Peripheral Vascular, C. Council on Clinical, Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association, Stroke, 44 (2013) 870–947.
A.K. Rana, J.M. Wardlaw, P.A. Armitage, M.E. Bastin, Apparent diffusion coefficient (ADC) measurements may be more reliable and reproducible than lesion volume on diffusion-weighted images from patients with acute ischaemic stroke-implications for study design, Magn Reson Imaging, 21 (2003) 617-624.
B.T. Bratane, B. Bastan, M. Fisher, J. Bouley, N. Henninger, Ischemic lesion volume determination on diffusion weighted images vs. apparent diffusion coefficient maps, Brain Res, 1279 (2009) 182–188.
R. Zhang, L. Zhao, W. Lou, J.M. Abrigo, V.C.T. Mok, W.C.W. Chu, D. Wang, L. Shi, Automatic Segmentation of Acute Ischemic Stroke From DWI Using 3-D Fully Convolutional DenseNets, IEEE Trans Med Imaging, 37 (2018) 2149-2160.
I. Woo, A. Lee, S.C. Jung, H. Lee, N. Kim, S.J. Cho, D. Kim, J. Lee, L. Sunwoo, D.W. Kang, Fully Automatic Segmentation of Acute Ischemic Lesions on Diffusion-Weighted Imaging Using Convolutional Neural Networks: Comparison with Conventional Algorithms, Korean J Radiol, 20 (2019) 1275-1284.
K.K. Wong, J.S. Cummock, G. Li, R. Ghosh, P. Xu, J.J. Volpi, S.T.C. Wong, Automatic Segmentation in Acute Ischemic Stroke: Prognostic Significance of Topological Stroke Volumes on Stroke Outcome, Stroke, 53 (2022) 2896-2905.
P. Ashtari, D.M. Sima, L. De Lathauwer, D. Sappey-Marinier, F. Maes, S. Van Huffel, Factorizer: A scalable interpretable approach to context modeling for medical image segmentation, Med Image Anal, 84 (2023) 102706.
L. Liu, L. Kurgan, F.X. Wu, J. Wang, Attention convolutional neural network for accurate segmentation and quantification of lesions in ischemic stroke disease, Med Image Anal, 65 (2020) 101791.
Y.C. Kim, J.E. Lee, I. Yu, H.N. Song, I.Y. Baek, J.K. Seong, H.G. Jeong, B.J. Kim, H.S. Nam, J.W. Chung, O.Y. Bang, G.M. Kim, W.K. Seo, Evaluation of Diffusion Lesion Volume Measurements in Acute Ischemic Stroke Using Encoder-Decoder Convolutional Network, Stroke, 50 (2019) 1444-1451.
O. Maier, B.H. Menze, J. von der Gablentz, L. Hani, M.P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, L. Chen, D. Christiaens, F. Dutil, K. Egger, C.L. Feng, B. Glocker, M. Gotz, T. Haeck, H.L. Halme, M. Havaei, K.M. Iftekharuddin, P.M. Jodoin, K. Kamnitsas, E. Kellner, A. Korvenoja, H. Larochelle, C. Ledig, J.H. Lee, F. Maes, Q. Mahmood, K.H. Maier-Hein, R. McKinley, J. Muschelli, C. Pal, L.M. Pei, J.R. Rangarajan, S.M.S. Reza, D. Robben, D. Rueckert, E. Salli, P. Suetens, C.W. Wang, M. Wilms, J.S. Kirschke, U.M. Kramer, T.F. Munte, P. Schramme, R. Wiest, H. Handels, M. Reyes, ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI, Medical Image Analysis, 35 (2017) 250-269.
K. Kamnitsas, C. Ledig, V.F.J. Newcombe, J.P. Simpson, A.D. Kane, D.K. Menon, D. Rueckert, B. Glocker, Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation, Med Image Anal, 36 (2017) 61-78.
O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention, Pt Iii, 9351 (2015) 234-241.
R. Karthik, U. Gupta, A. Jha, R. Rajalakshmi, R. Menaka, A deep supervised approach for ischemic lesion segmentation from multimodal MRI using Fully Convolutional Network, Appl Soft Comput, 84 (2019).
A. Olivier, O. Moal, B. Moal, F. Munsch, G. Okubo, I. Sibon, V. Dousset, T. Tourdias, Active learning strategy and hybrid training for infarct segmentation on diffusion MRI with a U-shaped network, J Med Imaging (Bellingham), 6 (2019) 044001.
A. Clerigues, S. Valverde, J. Bernal, J. Freixenet, A. Oliver, X. Llado, Acute and sub-acute stroke lesion segmentation from multimodal MRI, Comput Methods Programs Biomed, 194 (2020) 105521.
G. Huang, Z. Liu, K.Q. Weinberger, Densely Connected Convolutional Networks, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016) 2261–2269.
A. Kumar, N. Upadhyay, P. Ghosal, T. Chowdhury, D. Das, A. Mukherjee, D. Nandi, CSNet: A new DeepNet framework for ischemic stroke lesion segmentation, Comput Methods Programs Biomed, 193 (2020) 105524.
Y. Xue, F.G. Farhat, O. Boukrina, A.M. Barrett, J.R. Binder, U.W. Roshan, W.W. Graves, A multi-path 2.5 dimensional convolutional neural network system for segmenting stroke lesions in brain MRI images, Neuroimage Clin, 25 (2020) 102118.
M.R. Hernandez Petzsche, E. de la Rosa, U. Hanning, R. Wiest, W. Valenzuela, M. Reyes, M. Meyer, S.L. Liew, F. Kofler, I. Ezhov, D. Robben, A. Hutton, T. Friedrich, T. Zarth, J. Burkle, T.A. Baran, B. Menze, G. Broocks, L. Meyer, C. Zimmer, T. Boeckh-Behrens, M. Berndt, B. Ikenberg, B. Wiestler, J.S. Kirschke, ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset, Sci Data, 9 (2022) 762.
J. Schroeter, C. Myers-Colet, D.L. Arnold, T. Arbel, Segmentation-Consistent Probabilistic Lesion Counting, in: K. Ender, M. Bjoern, V. Archana, B. Christian, D. Qi, A. Shadi (Eds.) Proceedings of The 5th International Conference on Medical Imaging with Deep Learning, PMLR, Proceedings of Machine Learning Research, 2022, pp. 1034--1056.
B. Bozsik, E. Toth, I. Polyak, F. Kerekes, N. Szabo, K. Bencsik, P. Klivenyi, Z.T. Kincses, Reproducibility of Lesion Count in Various Subregions on MRI Scans in Multiple Sclerosis, Front Neurol, 13 (2022).
O. Commowick, F. Cervenansky, R. Ameli, MSSEG Challenge Proceedings: Multiple Sclerosis Lesions Segmentation Challenge Using a Data Management and Processing Infrastructure, International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016.
H. Zhang, J.W. Zhang, C. Li, E.M. Sweeney, P. Spincemaille, T.D. Nguyen, S.A. Gauthier, Y. Wang, ALL-Net: Anatomical information lesion-wise loss function integrated into neural network for multiple sclerosis lesion segmentation, Neuroimage-Clin, 32 (2021).
F. Isensee, P.F. Jaeger, S.A.A. Kohl, J. Petersen, K.H. Maier-Hein, nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nat Methods, 18 (2021) 203-211.
Y.L. Liu, W.H. Cui, Q. Ha, X.L. Xiong, X.Z. Zeng, C.Y. Ye, Knowledge transfer between brain lesion segmentation tasks with increased model capacity, Comput Med Imag Grap, 88 (2021).
U. Baid, S. Ghodasara, M. Bilello, S. Mohan, E. Calabrese, E. Colak, K. Farahani, J. Kalpathy-Cramer, F.C. Kitamura, S. Pati, L.M. Prevedello, J.D. Rudie, C. Sako, R.T. Shinohara, T. Bergquist, R. Chai, J.A. Eddy, J. Elliott, W.C. Reade, T. Schaffter, T. Yu, J. Zheng, B. Annotators, C. Davatzikos, J.T. Mongan, C.P. Hess, S. Cha, J.E. Villanueva-Meyer, J.B. Freymann, J.S. Kirby, B. Wiestler, P. Crivellaro, R. R.Colen, A. Kotrotsou, D. Marcus, M. Milchenko, A. Nazeri, H.M. Fathallah-Shaykh, R. Wiest, A. Jakab, M.-A. Weber, A. Mahajan, B.H. Menze, A.E. Flanders, S. Bakas, The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification, ArXiv, abs/2107.02314 (2021).
A.-R. Lee, I. Woo, D.-W. Kang, S.C. Jung, H. Lee, N. Kim, Fully automated segmentation on brain ischemic and white matter hyperintensities lesions using semantic segmentation networks with squeeze-and-excitation blocks in MRI, Informatics in Medicine Unlocked, 21 (2020) 100440.
M.D. Schirmer, A.K. Giese, P. Fotiadis, M.R. Etherton, L. Cloonan, A. Viswanathan, S.M. Greenberg, O. Wu, N.S. Rost, Spatial Signature of White Matter Hyperintensities in Stroke Patients, Front Neurol, 10 (2019) 208.
C. Rorden, M. Brett, Stereotaxic display of brain lesions, Behav Neurol, 12 (2000) 191-200.
M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B.A. Landman, ... & M.J. Cardoso, The medical segmentation decathlon. Nature communications, 13(1), 4128, (2022).
V. M. Campello, P. Gkontra, C. Izquierdo, C. Martin-Isla, A. Sojoudi, P. M. Full, ... & K. Lekadir, Multi-centre, multi-vendor and multi-disease cardiac segmentation: the M&Ms challenge. IEEE Transactions on Medical Imaging, 40(12), 3543–3554, (2021).
J. Wasserthal, H. C. Breit, M. T. Meyer, M. Pradella, D. Hinck, A. W. Sauter , ... & M. Segeroth, Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence, 5(5), (2023).
N. Tomita, S. Jiang, M. E. Maeder, S. Hassanpour, Automatic post-stroke lesion segmentation on MR images using 3D residual convolutional neural network, NeuroImage: clinical, 27, (2020)
Z. Huang, H. Wang, Z. Deng, J. Ye, Y. Su, H. Sun, J. He, Y. Gu, L. Gu, S. Zhang, Y. Qiao, STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training, arXiv preprint arXiv:2304.06716, (2023)
Y. He, D. Yang, H. Roth, C. Zhao, D. Xu, DiNTS:Differentiable Neural Network Topology Search for 3D Medical Image Segmentation, Proceedings of the IEEE international conference on computer vision, 2021, pp. 5837–5846.
A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, H.R. Roth, D. Xu, UNETR:Transformers for 3D Medical Image Segmentation, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp.574–584
A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H.R. Roth, D. Xu, Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images, International MICCAI Brainlesion Workshop, 2021
Y. Gao, M. Zhou, D.N. Metaxas, UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention, 2021
H.Y. Zhou, J. Guo, Y. Zhang, X. Han, L. Yu, L. Wang, Y. Yu, nnFormer: volumetric medical image segmentation via a 3D transformer, IEEE Transactions on Image Processing, 2023
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A.L. Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, IEEE transactions on pattern analysis and machine intelligence, 40 (2017) 834-848.
M.M.R. Siddique, D. Yang, Y. He, D. Xu, A. Myronenko, Automated ischemic stroke lesion segmentation from 3D MRI, arXiv preprint arXiv:2209.09546, (2022).
A. Sasagawa, T. Mikami, Y. Kimura, Y. Akiyama, S. Sugita, T. Hasegawa, M. Wanibuchi, N. Mikuni, Stroke Mimics and Chameleons from the Radiological Viewpoint of Glioma Diagnosis, Neurol Med Chir (Tokyo), 61 (2021) 134-143.
L.M. Ballestar, V. Vilaplana, MRI Brain Tumor Segmentation and Uncertainty Estimation Using 3D-UNet Architectures, Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2021, 376–390.
Y. Shi, C. Micklisch, E. Mushtaq, S. Avestimehr, Y. Yan, X. Zhang, An ensemble approach to automatic brain tumor segmentation, In International MICCAI Brainlesion Workshop, 2021, 138–148
53. T. Henry, A. Carre, M. Lerousseau, T. Estienne, C. Robert, N. Paragios, E. Deutsch, Brain Tumor Segmentation with Self-ensembled, Deeply-Supervised 3D U-Net Neural Networks: A BraTS 2020 Challenge Solution, Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2021, 327-339.
T. Wiltgen, J. McGinnis, S. Schlaeger, C. Voon, A. Berthele, D. Bischl, L. Grundl, N. Will, M. Metz, D. Schinz, D. Sepp, P. Prucker, B. Schmitz-Koep, C. Zimmer, B. Menze, D. Rueckert, B. Hemmer, J. Kirschke, M. Muhlau, B. Wiestler, LST-AI: a Deep Learning Ensemble for Accurate MS Lesion Segmentation, medRxiv, 2023
S. Misra, C. Yoon, K. Kim, R. Managuli, R.G. Barr, J. Baek, and C. Kim, Deep learning-based Multimodal Fusion Network for Segmentation and Classification of Breast Cancers using B-mode and Elastography Ultrasound Images, Bioengineering and Translational Medicine, 2023
Funding
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (2023R1A2C3004880) and the Ministry of Education (2020R1A6A1A03047902); by the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (1711195277, RS-2020-KD000008); by National R&D program through the NRF funded by the Ministry of Science and ICT (2021M3C1C3097624); by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019–0-01906, Artificial Intelligence Graduate School Program (POSTECH)) and Korea Evaluation Institute of Industrial Technology (KEIT) grant funded by the Korea government (MOTIE); by the BK21 FOUR project.
Author information
Authors and Affiliations
Contributions
Hyunsu Jeong: conceptualization, methodology, software, investigation, validation, formal analysis, writing—original draft, writing—review and editing. Hyunseok Lim: software, visualization, writing—review and editing. Chiho Yoon: software, writing—review and editing. Jongjun Won: supervision, writing—review and editing. Grace Yoojin Lee: supervision, writing—review and editing. Ezequiel de la Rosa: formal analysis. Jan S. Kirschke: formal analysis. Bumjoon Kim: data curation, supervision, writing—review and editing. Namkug Kim: data curation, supervision, writing—review and editing. Chulhong Kim: supervision, writing—review and editing, funding acquisition.
Corresponding authors
Ethics declarations
Ethics Approval
The AMC I and II datasets were collected according to the principles of the Declaration of Helsinki, and the data collection was performed in accordance with current scientific guidelines. The study protocol was approved by the Institutional Review Board (IRB) Committee of AMC, University of Ulsan College of Medicine, Seoul, Republic of Korea. The requirement for informed patient consent was waived by the IRB Committee of AMC.
Consent to Participate
Informed consent was obtained from all individual participants included in the study.
Consent for Publication
The authors affirm that human research participants provided informed consent for publication of the images in Figs. 1, 2, 3 and 4.
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jeong, H., Lim, H., Yoon, C. et al. Robust Ensemble of Two Different Multimodal Approaches to Segment 3D Ischemic Stroke Segmentation Using Brain Tumor Representation Among Multiple Center Datasets. J Digit Imaging. Inform. med. (2024). https://doi.org/10.1007/s10278-024-01099-6
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10278-024-01099-6