CNN Prediction of Future Disease Activity for Multiple Sclerosis Patients from Baseline MRI and Lesion Labels

  • Nazanin Mohammadi SepahvandEmail author
  • Tal Hassner
  • Douglas L. Arnold
  • Tal Arbel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11383)


New T2w and gadolineum-enhancing lesions in Magnetic Resonance Images (MRI) are indicators of new disease activity in Multiple Sclerosis (MS) patients. Predicting future disease activity could help predict the progression of the disease as well as efficacy of treatment. We introduce a convolutional neural network (CNN) framework for future MRI disease activity prediction in relapsing-remitting MS (RRMS) patients from multi-modal MR images at baseline and illustrate how the inclusion of T2w lesion labels at baseline can significantly improve prediction accuracy by drawing the attention of the network to the location of lesions. Next, we develop a segmentation network to automatically infer lesion labels when semi-manual expert lesion labels are unavailable. Both prediction and segmentation networks are trained and tested on a large, proprietary, multi-center, multi-modal, clinical trial dataset consisting of 1068 patients. Testing based on a dataset of 95 patients shows that our framework reaches very high performance levels (sensitivities of 80.11% and specificities of 79.16%) when semi-manual expert labels are included as input at baseline in addition to multi-modal MRI. Even with inferred lesion labels replacing semi-manual labels, the method significantly outperforms an identical end-to-end CNN which only includes baseline multi-modal MRI.


Multiple sclerosis Magnetic resonance imaging Disease activity Deep learning 



This work was supported by an award from the International Progressive MS Alliance (PA-1603-08175).


  1. 1.
    Barkhof, F., et al.: Comparison of MRI criteria at first presentation to predict conversion to clinically definite multiple sclerosis. Brain 120(11), 2059–2069 (1997)CrossRefGoogle Scholar
  2. 2.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. TPAMI 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  3. 3.
    Brosch, T., Yoo, Y., Li, D.K.B., Traboulsee, A., Tam, R.: Modeling the variability in brain morphology and lesion distribution in multiple sclerosis by deep learning. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 462–469. Springer, Cham (2014). Scholar
  4. 4.
    Carass, A., et al.: Longitudinal multiple sclerosis lesion segmentation: resource and challenge. Neuroimage 148, 77–102 (2017)CrossRefGoogle Scholar
  5. 5.
    Doyle, A., Precup, D., Arnold, D.L., Arbel, T.: Predicting future disease activity and treatment responders for multiple sclerosis patients using a bag-of-lesions brain representation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 186–194. Springer, Cham (2017). Scholar
  6. 6.
    Elliott, C., et al.: Temporally consistent probabilistic detection of new multiple sclerosis lesions in brain MRI. IEEE TMI 32(8), 1490–1503 (2013)Google Scholar
  7. 7.
    Gold, R., et al.: Placebo-controlled phase 3 study of oral BG-12 for relapsing multiple sclerosis. N. Engl. J. Med. 367(12), 1098–1107 (2012)CrossRefGoogle Scholar
  8. 8.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  9. 9.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)Google Scholar
  10. 10.
    Kaunzner, U., Gauthier, S.: MRI in the assessment and monitoring of multiple sclerosis: an update on best practice. Ther. Adv. Neurol. Disord. 10(6), 247–261 (2017)CrossRefGoogle Scholar
  11. 11.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2014)Google Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  13. 13.
    Menze, B., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE TMI 34(10), 1993 (2015)Google Scholar
  14. 14.
    Moccia, M., de Stefano, N., Barkhof, F.: Imaging outcome measures for progressive multiple sclerosis trials. Mult. Scler. J. 23(12), 1614–1626 (2017)CrossRefGoogle Scholar
  15. 15.
    Nyúl, L., Udupa, J.: On standardizing the MR image intensity scale. Magn. Reson. Med.: Off. J. Int. Soc. Magn. Reson. Med. 42(6), 1072–1081 (1999)CrossRefGoogle Scholar
  16. 16.
    Río, J., et al.: MR imaging in monitoring and predicting treatment response in multiple sclerosis. Neuroimaging Clin. 27(2), 277–287 (2017)CrossRefGoogle Scholar
  17. 17.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  18. 18.
    Sled, J., Zijdenbos, A., Evans, A.: A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE TMI 17(1), 87–97 (1998)Google Scholar
  19. 19.
    Smith, S.: Fast robust automated brain extraction. Hum. Brain Mapp. 17(3), 143–155 (2002)CrossRefGoogle Scholar
  20. 20.
    Sormani, M.P., Bruzzi, P.: MRI lesions as a surrogate for relapses in multiple sclerosis: a meta-analysis of randomised trials. Lancet Neurol. 12(7), 669–676 (2013)CrossRefGoogle Scholar
  21. 21.
    Stangel, M., et al.: Towards the implementation of ‘no evidence of disease activity’ in multiple sclerosis treatment: the multiple sclerosis decision model. Ther. Adv. Neurol. Disord. 8(1), 3–13 (2015)CrossRefGoogle Scholar
  22. 22.
    Windham, B., et al.: Small brain lesions and incident stroke and mortality: a cohort study. Ann. Intern. Med. 163(1), 22–31 (2015)CrossRefGoogle Scholar
  23. 23.
    Yoo, Y., et al.: Deep learning of brain lesion patterns for predicting future disease activity in patients with early symptoms of multiple sclerosis. In: Carneiro, G., et al. (eds.) LABELS/DLMIA-2016. LNCS, vol. 10008, pp. 86–94. Springer, Cham (2016). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nazanin Mohammadi Sepahvand
    • 1
    Email author
  • Tal Hassner
    • 2
  • Douglas L. Arnold
    • 3
    • 4
  • Tal Arbel
    • 1
  1. 1.Centre for Intelligent MachinesMcGill UniversityMontrealCanada
  2. 2.The Open University of IsraelRa’ananaIsrael
  3. 3.Montreal Neurological InstituteMcGill UniversityMontréalCanada
  4. 4.NeuroRx ResearchMontréalCanada

Personalised recommendations