Prediction of Thrombectomy Functional Outcomes Using Multimodal Data

Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1248)


Recent randomised clinical trials have shown that patients with ischaemic stroke due to occlusion of a large intracranial blood vessel benefit from endovascular thrombectomy. However, predicting outcome of treatment in an individual patient remains a challenge. We propose a novel deep learning approach to directly exploit multimodal data (clinical metadata information, imaging data, and imaging biomarkers extracted from images) to estimate the success of endovascular treatment. We incorporate an attention mechanism in our architecture to model global feature inter-dependencies, both channel-wise and spatially. We perform comparative experiments using unimodal and multimodal data, to predict functional outcome (modified Rankin Scale score, mRS) and achieve 0.75 AUC for dichotomised mRS scores and 0.35 classification accuracy for individual mRS scores.


Stroke Deep learning Thrombectomy CNN NCCT Prognosis 



The authors would like to thank the MR CLEAN Registry team: Prof. Aad van der Lugt, Prof. Diederik W.J. Dippel, Prof. Charles B.L.M. Majoie, Prof. Wim H. van Zwam and Prof. Robert J. van Oostenbrugge for providing the data. Zeynel Samak gratefully acknowledges funding from Ministry of Education (1416/YLSY), the Republic of Turkey. The Titan V used for this research was donated by the NVIDIA Corporation.


  1. 1.
    Abraham, N., Khan, N.M.: A novel focal Tversky loss function with improved attention U-Net for lesion segmentation. In: ISBI, pp. 683–687. IEEE (2019)Google Scholar
  2. 2.
    Albers, G.W., et al.: Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. NEJM 378(8), 708–718 (2018)CrossRefGoogle Scholar
  3. 3.
    Asadi, H., et al.: Machine learning for outcome prediction of acute ischemic stroke post intra-arterial therapy. PLoS ONE 9(2), e88225 (2014)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bacchi, S., et al.: Deep learning in the prediction of ischaemic stroke thrombolysis functional outcomes: a pilot study. Acad. Radiol. 27(2), e19–e23 (2019)CrossRefGoogle Scholar
  5. 5.
    Bentley, P., et al.: Prediction of stroke thrombolysis outcome using CT brain machine learning. NeuroImage Clin. 4, 635–640 (2014)CrossRefGoogle Scholar
  6. 6.
    Berkhemer, O.A., et al.: A randomized trial of intraarterial treatment for acute ischemic stroke. NEJM 372(1), 11–20 (2015)CrossRefGoogle Scholar
  7. 7.
    Boers, A., et al.: Automated cerebral infarct volume measurement in follow-up noncontrast CT scans of patients with acute ischemic stroke. AJN 34(8), 1522–1527 (2013)Google Scholar
  8. 8.
    Böhme, L., Madesta, F., Sentker, T., Werner, R.: Combining good old random forest and DeepLabv3+ for ISLES 2018 CT-based stroke segmentation. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11383, pp. 335–342. Springer, Cham (2019). Scholar
  9. 9.
    Chawla, M., et al.: A method for automatic detection and classification of stroke from brain CT images. In: IEEEMBS, pp. 3581–3584. IEEE (2009)Google Scholar
  10. 10.
    Chen, S., et al.: Med3D: transfer learning for 3D medical image analysis. arXiv preprint arXiv:1904.00625 (2019)
  11. 11.
    Choi, Y., et al.: Ensemble of deep convolutional neural networks for prognosis of ischemic stroke. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Winzeck, S., Handels, H. (eds.) BrainLes 2016. LNCS, pp. 231–243. Springer, Cham (2016). Scholar
  12. 12.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). Scholar
  13. 13.
    Fahed, R., et al.: Dwi-aspects (diffusion-weighted imaging-alberta stroke program early computed tomography scores) and dwi-flair (diffusion-weighted imaging-fluid attenuated inversion recovery) mismatch in thrombectomy candidates: An intrarater and interrater agreement study. Stroke 49(1), 223–227 (2018)CrossRefGoogle Scholar
  14. 14.
    Forkert, N.D., et al.: Multiclass support vector machine-based lesion mapping predicts functional outcome in ischemic stroke patients. PLoS ONE 10(6), e0129569 (2015)CrossRefGoogle Scholar
  15. 15.
    Fransen, P.S., et al.: MR CLEAN, a multicenter randomized clinical trial of endovascular treatment for acute ischemic stroke in the Netherlands: study protocol for a randomized controlled trial. Trials 15(1), 343 (2014)CrossRefGoogle Scholar
  16. 16.
    Goyal, M., et al.: Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. The Lancet 387(10029), 1723–1731 (2016)CrossRefGoogle Scholar
  17. 17.
    Gupta, N., Mittal, A.: Brain ischemic stroke segmentation: a survey. J. Multi Disciplinary Eng. Technol. 8(1), 1 (2014)Google Scholar
  18. 18.
    He, K., et al.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  19. 19.
    Heo, J., et al.: Machine learning-based model can predict stroke outcome. Stroke 49(Suppl 1), A194–A194 (2018)Google Scholar
  20. 20.
    Hilbert, A., et al.: Data-efficient deep learning of radiological image data for outcome prediction after endovascular treatment of patients with acute ischemic stroke. Comput. Biol. Med., 103516 (2019) Google Scholar
  21. 21.
    Hu, J., et al.: Squeeze-and-excitation networks. In: CVPR (2018)Google Scholar
  22. 22.
    Isensee, F., et al.: Brain tumor segmentation and radiomics survival prediction: contribution to BRATS 2017 challenge. In: MICCAIBW, pp. 287–297 (2017)Google Scholar
  23. 23.
    Jansen, I.G., et al.: Endovascular treatment for acute ischaemic stroke in routine clinical practice: prospective, observational cohort study (MR CLEAN Registry). BMJ 360, k949 (2018)CrossRefGoogle Scholar
  24. 24.
    Kamnitsas, K., et al.: Ensembles of multiple models and architectures for robust brain tumour segmentation. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 450–462. Springer, Cham (2018). Scholar
  25. 25.
    Lin, T.Y., et al.: Focal loss for dense object detection. In: CVPR (2017)Google Scholar
  26. 26.
    Lisowska, A., et al.: Context-aware convolutional neural networks for stroke sign detection in non-contrast CT scans. In: Valdés Hernández, M., González-Castro, V. (eds.) MIUA 2017. CCIS, vol. 723, pp. 494–505. Springer, Cham (2017). Scholar
  27. 27.
    Liu, S., et al.: On the design of convolutional neural networks for automatic detection of Alzheimer’s disease. In: NeurIPS ML4H (2019)Google Scholar
  28. 28.
    Maier, O., et al.: Ischemic stroke lesion segmentation in multi-spectral MR images with support vector machine classifiers. In: Medical Imaging 2014: Computer-Aided Diagnosis, vol. 9035, p. 903504. ISOP (2014)Google Scholar
  29. 29.
    Maier, O., et al.: ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. MIA 35, 250–269 (2017)Google Scholar
  30. 30.
    Maier, O., Handels, H.: Predicting Stroke Lesion and Clinical Outcome with Random Forests. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Winzeck, S., Handels, H. (eds.) BrainLes 2016. LNCS, pp. 219–230. Springer, Cham (2016). Scholar
  31. 31.
    Matesin, M., et al.: A rule-based approach to stroke lesion analysis from CT brain images. In: ISPA, pp. 219–223. IEEE (2001)Google Scholar
  32. 32.
    McKinley, R., et al.: Fully automated stroke tissue estimation using random forest classifiers (FASTER). JCBFM 37(8), 2728–2741 (2017)Google Scholar
  33. 33.
    Nishi, H., et al.: Predicting clinical outcomes of large vessel occlusion before mechanical thrombectomy using machine learning. Stroke 50(9), 2379–2388 (2019)CrossRefGoogle Scholar
  34. 34.
    Parisot, S., et al.: Disease prediction using graph convolutional networks: application to autism spectrum disorder and Alzheimer’s disease. MIA 48, 117–130 (2018)Google Scholar
  35. 35.
    Pinto, A., et al.: Enhancing clinical MRI perfusion maps with data-driven maps of complementary nature for lesion outcome prediction. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 107–115. Springer, Cham (2018). Scholar
  36. 36.
    Rekik, I., et al.: Medical image analysis methods in MR/CT-imaged acute-subacute ischemic stroke lesion: Segmentation, prediction and insights into dynamic evolution simulation models. A critical appraisal. NeuroImage Clin. 1(1), 164–178 (2012)CrossRefGoogle Scholar
  37. 37.
    Renowden, S.: Imaging in stroke and vascular disease–part 1: ischaemic stroke. Pract. Neurol. 14(2), 77–87 (2014)CrossRefGoogle Scholar
  38. 38.
    Roy, A.G., Navab, N., Wachinger, C.: Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 421–429. Springer, Cham (2018). Scholar
  39. 39.
    Stroke Association: State of the Nation: stroke statistics (2018). Accessed Nov 2019
  40. 40.
    Van Os, H.J., et al.: Predicting outcome of endovascular treatment for acute ischemic stroke: potential value of machine learning algorithms. Front. Neurol. 9, 784 (2018)CrossRefGoogle Scholar
  41. 41.
    Venema, E., et al.: Selection of patients for intra-arterial treatment for acute ischaemic stroke: development and validation of a clinical decision tool in two randomised trials. BMJ 357, j1710 (2017)CrossRefGoogle Scholar
  42. 42.
    Weimar, C., Ziegler, A., König, I.R., Diener, H.-C.: Predicting functional outcome and survival after acute ischemic stroke. J. Neurol. 249(7), 888–895 (2002). Scholar
  43. 43.
    WHO: The top 10 causes of death (2018). Accessed Nov 2019
  44. 44.
    Winzeck, S., et al.: ISLES 2016 and 2017-benchmarking ischemic stroke lesionoutcome prediction based on multispectral MRI. Front. Neurol. 9, 679 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of BristolBristolUK
  2. 2.Translational Health SciencesUniversity of BristolBristolUK
  3. 3.Stroke Neurology, Southmead HospitalNorth Bristol NHS TrustBristolUK

Personalised recommendations