Skip to main content
Log in

Robust Ensemble of Two Different Multimodal Approaches to Segment 3D Ischemic Stroke Segmentation Using Brain Tumor Representation Among Multiple Center Datasets

  • Published:
Journal of Imaging Informatics in Medicine Aims and scope Submit manuscript

Abstract

Ischemic stroke segmentation at an acute stage is vital in assessing the severity of patients’ impairment and guiding therapeutic decision-making for reperfusion. Although many deep learning studies have shown attractive performance in medical segmentation, it is difficult to use these models trained on public data with private hospitals’ datasets. Here, we demonstrate an ensemble model that employs two different multimodal approaches for generalization, a more effective way to perform on external datasets. First, after we jointly train a segmentation model on diffusion-weighted imaging (DWI) and apparent diffusion coefficient (ADC) MR modalities, the model is inferred on the DWI images. Second, a channel-wise segmentation model is trained by concatenating the DWI and ADC images as input, and then is inferred using both MR modalities. Before training with ischemic stroke data, we utilized BraTS 2021, a public brain tumor dataset, for transfer learning. An extensive ablation study evaluates which strategy learns better representations for ischemic stroke segmentation. In our study, nnU-Net well-known for robustness is selected as our baseline model. Our proposed method is evaluated on three different datasets: the Asan Medical Center (AMC) I and II, and the 2022 Ischemic Stroke Lesion Segmentation (ISLES). Our experiments are widely validated over a large, multi-center, and multi-scanner dataset with a huge amount of 846 scans. Not only stroke lesion models can benefit from transfer learning using brain tumor data, but combining the MR modalities using different training schemes also highly improves segmentation performance. The method achieved a top-1 ranking in the ongoing ISLES’22 challenge and performed particularly well on lesion-wise metrics of interest to neuroradiologists, achieving a Dice coefficient of 78.69% and a lesion-wise F1 score of 82.46%. Also, the method was relatively robust on the AMC I (Dice, 60.35%; lesion-wise F1, 68.30%) and II (Dice; 74.12%; lesion-wise F1, 67.53%) datasets in different settings. The high segmentation accuracy of our proposed method could improve radiologists’ ability to detect ischemic stroke lesions in MRI images. Our model weights and inference code are available on https://github.com/MDOpx/ISLES22-model-inference.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

Available upon request.

References

  1. R.L. Sacco, S.E. Kasner, J.P. Broderick, L.R. Caplan, J.J. Connors, A. Culebras, M.S. Elkind, M.G. George, A.D. Hamdan, R.T. Higashida, B.L. Hoh, L.S. Janis, C.S. Kase, D.O. Kleindorfer, J.M. Lee, M.E. Moseley, E.D. Peterson, T.N. Turan, A.L. Valderrama, H.V. Vinters, C.o.C.S. American Heart Association Stroke Council, Anesthesia, R. Council on Cardiovascular, Intervention, C. Council on, N. Stroke, E. Council on, Prevention, D. Council on Peripheral Vascular, P.A. Council on Nutrition, Metabolism, An updated definition of stroke for the 21st century: a statement for healthcare professionals from the American Heart Association/American Stroke Association, Stroke, 44 (2013) 2064–2089.

  2. J.L. Saver, Time is brain--quantified, Stroke, 37 (2006) 263-266.

    Article  PubMed  Google Scholar 

  3. G. Zaharchuk, I.S. El Mogy, N.J. Fischbein, G.W. Albers, Comparison of arterial spin labeling and bolus perfusion-weighted imaging for detecting mismatch in acute stroke, Stroke, 43 (2012) 1843-1848.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. C. Yoon, S. Misra, K.-J. Kim, C. Kim, B.J. Kim, Collaborative multi-modal deep learning and radiomic features for classification of strokes within 6 h, Expert Systems with Applications, 228 (2023) 120473.

    Article  Google Scholar 

  5. J. Vymazal, A.M. Rulseh, J. Keller, L. Janouskova, Comparison of CT and MR imaging in ischemic stroke, Insights Imaging, 3 (2012) 619-627.

    Google Scholar 

  6. B.L. Edlow, S. Hurwitz, J.A. Edlow, Diagnosis of DWI-negative acute ischemic stroke: A meta-analysis, Neurology, 89 (2017) 256-262.

    Article  PubMed  PubMed Central  Google Scholar 

  7. C.Z. Simonsen, M.H. Madsen, M.L. Schmitz, I.K. Mikkelsen, M. Fisher, G. Andersen, Sensitivity of diffusion- and perfusion-weighted imaging for diagnosing acute ischemic stroke is 97.5%, Stroke, 46 (2015) 98–101.

  8. E.C. Jauch, J.L. Saver, H.P. Adams, Jr., A. Bruno, J.J. Connors, B.M. Demaerschalk, P. Khatri, P.W. McMullan, Jr., A.I. Qureshi, K. Rosenfield, P.A. Scott, D.R. Summers, D.Z. Wang, M. Wintermark, H. Yonas, C. American Heart Association Stroke, N. Council on Cardiovascular, D. Council on Peripheral Vascular, C. Council on Clinical, Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association, Stroke, 44 (2013) 870–947.

  9. A.K. Rana, J.M. Wardlaw, P.A. Armitage, M.E. Bastin, Apparent diffusion coefficient (ADC) measurements may be more reliable and reproducible than lesion volume on diffusion-weighted images from patients with acute ischaemic stroke-implications for study design, Magn Reson Imaging, 21 (2003) 617-624.

    Article  PubMed  Google Scholar 

  10. B.T. Bratane, B. Bastan, M. Fisher, J. Bouley, N. Henninger, Ischemic lesion volume determination on diffusion weighted images vs. apparent diffusion coefficient maps, Brain Res, 1279 (2009) 182–188.

  11. R. Zhang, L. Zhao, W. Lou, J.M. Abrigo, V.C.T. Mok, W.C.W. Chu, D. Wang, L. Shi, Automatic Segmentation of Acute Ischemic Stroke From DWI Using 3-D Fully Convolutional DenseNets, IEEE Trans Med Imaging, 37 (2018) 2149-2160.

    Article  PubMed  Google Scholar 

  12. I. Woo, A. Lee, S.C. Jung, H. Lee, N. Kim, S.J. Cho, D. Kim, J. Lee, L. Sunwoo, D.W. Kang, Fully Automatic Segmentation of Acute Ischemic Lesions on Diffusion-Weighted Imaging Using Convolutional Neural Networks: Comparison with Conventional Algorithms, Korean J Radiol, 20 (2019) 1275-1284.

    Article  PubMed  PubMed Central  Google Scholar 

  13. K.K. Wong, J.S. Cummock, G. Li, R. Ghosh, P. Xu, J.J. Volpi, S.T.C. Wong, Automatic Segmentation in Acute Ischemic Stroke: Prognostic Significance of Topological Stroke Volumes on Stroke Outcome, Stroke, 53 (2022) 2896-2905.

    Article  CAS  PubMed  Google Scholar 

  14. P. Ashtari, D.M. Sima, L. De Lathauwer, D. Sappey-Marinier, F. Maes, S. Van Huffel, Factorizer: A scalable interpretable approach to context modeling for medical image segmentation, Med Image Anal, 84 (2023) 102706.

    Article  PubMed  Google Scholar 

  15. L. Liu, L. Kurgan, F.X. Wu, J. Wang, Attention convolutional neural network for accurate segmentation and quantification of lesions in ischemic stroke disease, Med Image Anal, 65 (2020) 101791.

    Article  PubMed  Google Scholar 

  16. Y.C. Kim, J.E. Lee, I. Yu, H.N. Song, I.Y. Baek, J.K. Seong, H.G. Jeong, B.J. Kim, H.S. Nam, J.W. Chung, O.Y. Bang, G.M. Kim, W.K. Seo, Evaluation of Diffusion Lesion Volume Measurements in Acute Ischemic Stroke Using Encoder-Decoder Convolutional Network, Stroke, 50 (2019) 1444-1451.

    Article  PubMed  Google Scholar 

  17. O. Maier, B.H. Menze, J. von der Gablentz, L. Hani, M.P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, L. Chen, D. Christiaens, F. Dutil, K. Egger, C.L. Feng, B. Glocker, M. Gotz, T. Haeck, H.L. Halme, M. Havaei, K.M. Iftekharuddin, P.M. Jodoin, K. Kamnitsas, E. Kellner, A. Korvenoja, H. Larochelle, C. Ledig, J.H. Lee, F. Maes, Q. Mahmood, K.H. Maier-Hein, R. McKinley, J. Muschelli, C. Pal, L.M. Pei, J.R. Rangarajan, S.M.S. Reza, D. Robben, D. Rueckert, E. Salli, P. Suetens, C.W. Wang, M. Wilms, J.S. Kirschke, U.M. Kramer, T.F. Munte, P. Schramme, R. Wiest, H. Handels, M. Reyes, ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI, Medical Image Analysis, 35 (2017) 250-269.

    Article  PubMed  Google Scholar 

  18. K. Kamnitsas, C. Ledig, V.F.J. Newcombe, J.P. Simpson, A.D. Kane, D.K. Menon, D. Rueckert, B. Glocker, Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation, Med Image Anal, 36 (2017) 61-78.

    Article  PubMed  Google Scholar 

  19. O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention, Pt Iii, 9351 (2015) 234-241.

    Google Scholar 

  20. R. Karthik, U. Gupta, A. Jha, R. Rajalakshmi, R. Menaka, A deep supervised approach for ischemic lesion segmentation from multimodal MRI using Fully Convolutional Network, Appl Soft Comput, 84 (2019).

  21. A. Olivier, O. Moal, B. Moal, F. Munsch, G. Okubo, I. Sibon, V. Dousset, T. Tourdias, Active learning strategy and hybrid training for infarct segmentation on diffusion MRI with a U-shaped network, J Med Imaging (Bellingham), 6 (2019) 044001.

    PubMed  Google Scholar 

  22. A. Clerigues, S. Valverde, J. Bernal, J. Freixenet, A. Oliver, X. Llado, Acute and sub-acute stroke lesion segmentation from multimodal MRI, Comput Methods Programs Biomed, 194 (2020) 105521.

    Article  PubMed  Google Scholar 

  23. G. Huang, Z. Liu, K.Q. Weinberger, Densely Connected Convolutional Networks, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016) 2261–2269.

  24. A. Kumar, N. Upadhyay, P. Ghosal, T. Chowdhury, D. Das, A. Mukherjee, D. Nandi, CSNet: A new DeepNet framework for ischemic stroke lesion segmentation, Comput Methods Programs Biomed, 193 (2020) 105524.

    Article  PubMed  Google Scholar 

  25. Y. Xue, F.G. Farhat, O. Boukrina, A.M. Barrett, J.R. Binder, U.W. Roshan, W.W. Graves, A multi-path 2.5 dimensional convolutional neural network system for segmenting stroke lesions in brain MRI images, Neuroimage Clin, 25 (2020) 102118.

  26. M.R. Hernandez Petzsche, E. de la Rosa, U. Hanning, R. Wiest, W. Valenzuela, M. Reyes, M. Meyer, S.L. Liew, F. Kofler, I. Ezhov, D. Robben, A. Hutton, T. Friedrich, T. Zarth, J. Burkle, T.A. Baran, B. Menze, G. Broocks, L. Meyer, C. Zimmer, T. Boeckh-Behrens, M. Berndt, B. Ikenberg, B. Wiestler, J.S. Kirschke, ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset, Sci Data, 9 (2022) 762.

    Article  PubMed  PubMed Central  Google Scholar 

  27. J. Schroeter, C. Myers-Colet, D.L. Arnold, T. Arbel, Segmentation-Consistent Probabilistic Lesion Counting, in: K. Ender, M. Bjoern, V. Archana, B. Christian, D. Qi, A. Shadi (Eds.) Proceedings of The 5th International Conference on Medical Imaging with Deep Learning, PMLR, Proceedings of Machine Learning Research, 2022, pp. 1034--1056.

  28. B. Bozsik, E. Toth, I. Polyak, F. Kerekes, N. Szabo, K. Bencsik, P. Klivenyi, Z.T. Kincses, Reproducibility of Lesion Count in Various Subregions on MRI Scans in Multiple Sclerosis, Front Neurol, 13 (2022).

  29. O. Commowick, F. Cervenansky, R. Ameli, MSSEG Challenge Proceedings: Multiple Sclerosis Lesions Segmentation Challenge Using a Data Management and Processing Infrastructure, International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016.

  30. H. Zhang, J.W. Zhang, C. Li, E.M. Sweeney, P. Spincemaille, T.D. Nguyen, S.A. Gauthier, Y. Wang, ALL-Net: Anatomical information lesion-wise loss function integrated into neural network for multiple sclerosis lesion segmentation, Neuroimage-Clin, 32 (2021).

  31. F. Isensee, P.F. Jaeger, S.A.A. Kohl, J. Petersen, K.H. Maier-Hein, nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nat Methods, 18 (2021) 203-211.

    Article  CAS  PubMed  Google Scholar 

  32. Y.L. Liu, W.H. Cui, Q. Ha, X.L. Xiong, X.Z. Zeng, C.Y. Ye, Knowledge transfer between brain lesion segmentation tasks with increased model capacity, Comput Med Imag Grap, 88 (2021).

  33. U. Baid, S. Ghodasara, M. Bilello, S. Mohan, E. Calabrese, E. Colak, K. Farahani, J. Kalpathy-Cramer, F.C. Kitamura, S. Pati, L.M. Prevedello, J.D. Rudie, C. Sako, R.T. Shinohara, T. Bergquist, R. Chai, J.A. Eddy, J. Elliott, W.C. Reade, T. Schaffter, T. Yu, J. Zheng, B. Annotators, C. Davatzikos, J.T. Mongan, C.P. Hess, S. Cha, J.E. Villanueva-Meyer, J.B. Freymann, J.S. Kirby, B. Wiestler, P. Crivellaro, R. R.Colen, A. Kotrotsou, D. Marcus, M. Milchenko, A. Nazeri, H.M. Fathallah-Shaykh, R. Wiest, A. Jakab, M.-A. Weber, A. Mahajan, B.H. Menze, A.E. Flanders, S. Bakas, The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification, ArXiv, abs/2107.02314 (2021).

  34. A.-R. Lee, I. Woo, D.-W. Kang, S.C. Jung, H. Lee, N. Kim, Fully automated segmentation on brain ischemic and white matter hyperintensities lesions using semantic segmentation networks with squeeze-and-excitation blocks in MRI, Informatics in Medicine Unlocked, 21 (2020) 100440.

    Article  Google Scholar 

  35. M.D. Schirmer, A.K. Giese, P. Fotiadis, M.R. Etherton, L. Cloonan, A. Viswanathan, S.M. Greenberg, O. Wu, N.S. Rost, Spatial Signature of White Matter Hyperintensities in Stroke Patients, Front Neurol, 10 (2019) 208.

    Article  PubMed  PubMed Central  Google Scholar 

  36. C. Rorden, M. Brett, Stereotaxic display of brain lesions, Behav Neurol, 12 (2000) 191-200.

    Article  PubMed  Google Scholar 

  37. M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B.A. Landman, ... & M.J. Cardoso, The medical segmentation decathlon. Nature communications, 13(1), 4128, (2022).

  38. V. M. Campello, P. Gkontra, C. Izquierdo, C. Martin-Isla, A. Sojoudi, P. M. Full, ... & K. Lekadir, Multi-centre, multi-vendor and multi-disease cardiac segmentation: the M&Ms challenge. IEEE Transactions on Medical Imaging, 40(12), 3543–3554, (2021).

  39. J. Wasserthal, H. C. Breit, M. T. Meyer, M. Pradella, D. Hinck, A. W. Sauter , ... & M. Segeroth, Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence, 5(5), (2023).

  40. N. Tomita, S. Jiang, M. E. Maeder, S. Hassanpour, Automatic post-stroke lesion segmentation on MR images using 3D residual convolutional neural network, NeuroImage: clinical, 27, (2020)

  41. Z. Huang, H. Wang, Z. Deng, J. Ye, Y. Su, H. Sun, J. He, Y. Gu, L. Gu, S. Zhang, Y. Qiao, STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training, arXiv preprint arXiv:2304.06716, (2023)

  42. Y. He, D. Yang, H. Roth, C. Zhao, D. Xu, DiNTS:Differentiable Neural Network Topology Search for 3D Medical Image Segmentation, Proceedings of the IEEE international conference on computer vision, 2021, pp. 5837–5846.

  43. A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, H.R. Roth, D. Xu, UNETR:Transformers for 3D Medical Image Segmentation, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp.574–584

  44. A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H.R. Roth, D. Xu, Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images, International MICCAI Brainlesion Workshop, 2021

  45. Y. Gao, M. Zhou, D.N. Metaxas, UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention, 2021

  46. H.Y. Zhou, J. Guo, Y. Zhang, X. Han, L. Yu, L. Wang, Y. Yu, nnFormer: volumetric medical image segmentation via a 3D transformer, IEEE Transactions on Image Processing, 2023

  47. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A.L. Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, IEEE transactions on pattern analysis and machine intelligence, 40 (2017) 834-848.

    Article  PubMed  Google Scholar 

  48. M.M.R. Siddique, D. Yang, Y. He, D. Xu, A. Myronenko, Automated ischemic stroke lesion segmentation from 3D MRI, arXiv preprint arXiv:2209.09546, (2022).

  49. A. Sasagawa, T. Mikami, Y. Kimura, Y. Akiyama, S. Sugita, T. Hasegawa, M. Wanibuchi, N. Mikuni, Stroke Mimics and Chameleons from the Radiological Viewpoint of Glioma Diagnosis, Neurol Med Chir (Tokyo), 61 (2021) 134-143.

    Article  PubMed  Google Scholar 

  50. L.M. Ballestar, V. Vilaplana, MRI Brain Tumor Segmentation and Uncertainty Estimation Using 3D-UNet Architectures, Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2021, 376–390.

  51. Y. Shi, C. Micklisch, E. Mushtaq, S. Avestimehr, Y. Yan, X. Zhang, An ensemble approach to automatic brain tumor segmentation, In International MICCAI Brainlesion Workshop, 2021, 138–148

  52. 53. T. Henry, A. Carre, M. Lerousseau, T. Estienne, C. Robert, N. Paragios, E. Deutsch, Brain Tumor Segmentation with Self-ensembled, Deeply-Supervised 3D U-Net Neural Networks: A BraTS 2020 Challenge Solution, Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2021, 327-339.

    Google Scholar 

  53. T. Wiltgen, J. McGinnis, S. Schlaeger, C. Voon, A. Berthele, D. Bischl, L. Grundl, N. Will, M. Metz, D. Schinz, D. Sepp, P. Prucker, B. Schmitz-Koep, C. Zimmer, B. Menze, D. Rueckert, B. Hemmer, J. Kirschke, M. Muhlau, B. Wiestler, LST-AI: a Deep Learning Ensemble for Accurate MS Lesion Segmentation, medRxiv, 2023

  54. S. Misra, C. Yoon, K. Kim, R. Managuli, R.G. Barr, J. Baek, and C. Kim, Deep learning-based Multimodal Fusion Network for Segmentation and Classification of Breast Cancers using B-mode and Elastography Ultrasound Images, Bioengineering and Translational Medicine, 2023

Download references

Funding

This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (2023R1A2C3004880) and the Ministry of Education (2020R1A6A1A03047902); by the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (1711195277, RS-2020-KD000008); by National R&D program through the NRF funded by the Ministry of Science and ICT (2021M3C1C3097624); by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019–0-01906, Artificial Intelligence Graduate School Program (POSTECH)) and Korea Evaluation Institute of Industrial Technology (KEIT) grant funded by the Korea government (MOTIE); by the BK21 FOUR project.

Author information

Authors and Affiliations

Authors

Contributions

Hyunsu Jeong: conceptualization, methodology, software, investigation, validation, formal analysis, writing—original draft, writing—review and editing. Hyunseok Lim: software, visualization, writing—review and editing. Chiho Yoon: software, writing—review and editing. Jongjun Won: supervision, writing—review and editing. Grace Yoojin Lee: supervision, writing—review and editing. Ezequiel de la Rosa: formal analysis. Jan S. Kirschke: formal analysis. Bumjoon Kim: data curation, supervision, writing—review and editing. Namkug Kim: data curation, supervision, writing—review and editing. Chulhong Kim: supervision, writing—review and editing, funding acquisition.

Corresponding authors

Correspondence to Bumjoon Kim, Namkug Kim or Chulhong Kim.

Ethics declarations

Ethics Approval

The AMC I and II datasets were collected according to the principles of the Declaration of Helsinki, and the data collection was performed in accordance with current scientific guidelines. The study protocol was approved by the Institutional Review Board (IRB) Committee of AMC, University of Ulsan College of Medicine, Seoul, Republic of Korea. The requirement for informed patient consent was waived by the IRB Committee of AMC.

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Consent for Publication

The authors affirm that human research participants provided informed consent for publication of the images in Figs. 1, 2, 3 and 4.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jeong, H., Lim, H., Yoon, C. et al. Robust Ensemble of Two Different Multimodal Approaches to Segment 3D Ischemic Stroke Segmentation Using Brain Tumor Representation Among Multiple Center Datasets. J Digit Imaging. Inform. med. (2024). https://doi.org/10.1007/s10278-024-01099-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10278-024-01099-6

Keywords

Navigation