, Volume 16, Issue 1, pp 51–63 | Cite as

A Novel Public MR Image Dataset of Multiple Sclerosis Patients With Lesion Segmentations Based on Multi-rater Consensus

  • Žiga Lesjak
  • Alfiia Galimzianova
  • Aleš Koren
  • Matej Lukin
  • Franjo Pernuš
  • Boštjan Likar
  • Žiga Špiclin
Data Original Article


Quantified volume and count of white-matter lesions based on magnetic resonance (MR) images are important biomarkers in several neurodegenerative diseases. For a routine extraction of these biomarkers an accurate and reliable automated lesion segmentation is required. To objectively and reliably determine a standard automated method, however, creation of standard validation datasets is of extremely high importance. Ideally, these datasets should be publicly available in conjunction with standardized evaluation methodology to enable objective validation of novel and existing methods. For validation purposes, we present a novel MR dataset of 30 multiple sclerosis patients and a novel protocol for creating reference white-matter lesion segmentations based on multi-rater consensus. On these datasets three expert raters individually segmented white-matter lesions, using in-house developed semi-automated lesion contouring tools. Later, the raters revised the segmentations in several joint sessions to reach a consensus on segmentation of lesions. To evaluate the variability, and as quality assurance, the protocol was executed twice on the same MR images, with a six months break. The obtained intra-consensus variability was substantially lower compared to the intra- and inter-rater variabilities, showing improved reliability of lesion segmentation by the proposed protocol. Hence, the obtained reference segmentations may represent a more precise target to evaluate, compare against and also train, the automatic segmentations. To encourage further use and research we will publicly disseminate on our website the tools used to create lesion segmentations, the original and preprocessed MR image datasets and the consensus lesion segmentations.


Clinical image dataset White matter lesion Image segmentation Intra- and inter-rater variability Gold standard 



The authors would like to acknowledge Nuška Pečarič, MD, from the University Medical Centre Ljubljana for performing the lesion segmentations.

Supplementary material

12021_2017_9348_MOESM1_ESM.mp4 (22.9 mb)
(MP4 22.8 MB)
12021_2017_9348_MOESM2_ESM.pdf (834 kb)
(PDF 833 KB)


  1. Akhondi-Asl, A., Hoyte, L., Lockhart, M.E., & Warfield, S.K. (2014). A logarithmic opinion pool based staple algorithm for the fusion of segmentations with associated reliability weights. IEEE Transactions on Medical Imaging, 33(10), 1997–2009.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Anbeek, P., Vincken, K.L., van Osch, M.J.P., Bisschops, R.H.C., & van der Grond, J. (2004). Probabilistic segmentation of white matter lesions in MR, imaging. NeuroImage, 21(3), 1037–1044.CrossRefPubMedGoogle Scholar
  3. Arthur, D., & Vassilvitskii, S. (2007). K-means++: the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM Symposium on Discrete algorithms, SODA ’07 (pp. 1027–1035). Philadelphia: Society for Industrial and Applied Mathematics.Google Scholar
  4. Barillot, C., Commowick, O., Guttmann, C., Styner, M., & Warfield, S. (2016). MS Segmentation challenge. Last accessed: 20 oct, 2016.
  5. Cardoso, M.J., Modat, M., Wolz, R., Melbourne, A., Cash, D., Rueckert, D., & Ourselin, S. (2015). Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion. IEEE Transactions on Medical Imaging, 34(9), 1976–1988. Scholar
  6. CIBC. (2016). Seg3d: Volumetric image segmentation and visualization. Scientific computing and imaging institute (SCI), download from:
  7. Cocosco, C.A., Kollokian, V., Kwan, R.K.S., Pike, G.B., & Evans, A.C. (1997). BrainWeb: online interface to a 3d MRI simulated brain database. NeuroImage, 5, 425.Google Scholar
  8. Commowick, O., & Warfield, S. (2009). A continuous STAPLE for scalar, vector, and tensor images: an application to DTI analysis. IEEE Transactions on Medical Imaging, 28(6), 838–846. Scholar
  9. Debette, S., & Markus, H.S. (2010). The clinical importance of white matter hyperintensities on brain magnetic resonance imaging: systematic review and meta-analysis. BMJ, 341, c3666. Scholar
  10. Di Perri, C., Dwyer, M.G., Wack, D.S., Cox, J.L., Hashmi, K., Saluste, E., Hussein, S., Schirda, C., Stosic, M., Durfee, J., Poloni, G. U., Nayyar, N., Bergamaschi, R., & Zivadinov, R. (2009). Signal abnormalities on 1.5 and 3 Tesla brain mri in multiple sclerosis patients and healthy controls. A morphological and spatial quantitative comparison study. NeuroImage, 47(4), 1352–1362.CrossRefPubMedGoogle Scholar
  11. Dice, L.R. (1945). Measures of the amount of ecologic association between species. Ecology, 26(3), 297–302. ArticleType: research-article / Full publication date: Jul., 1945 / Copyright Ⓒ1945 Ecological Society of America.CrossRefGoogle Scholar
  12. Filippi, M., Horsfield, M.A., Bressi, S., Martinelli, V., Baratti, C., Reganati, P., Campi, A., Miller, D.H., & Comi, G. (1995). Intra- and inter-observer agreement of brain MRI lesion volume measurements in multiple sclerosis. A comparison of techniques. Brain: A Journal of Neurology, 118( Pt 6), 1593–1600.CrossRefGoogle Scholar
  13. Fonov, V., Evans, A., McKinstry, R., Almli, C., & Collins, D. (2009). Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage, 47(Supplement 1), S102.CrossRefGoogle Scholar
  14. Garcia-Lorenzo, D., Francis, S., Narayanan, S., Arnold, D.L., & Collins, D.L. (2013). Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging. Medical Image Analysis, 17(1), 1–18. Scholar
  15. Grimaud, J., Lai, M., Thorpe, J., Adeleine, P., Wang, L., Barker, G.J., Plummer, D.L, Tofts, P.S., McDonald, W.I, & Miller, D.H. (1996). Quantification of MRI lesion load in multiple sclerosis: A comparison of three computer-assisted techniques. Magnetic Resonance Imaging, 14(5), 495–505. Scholar
  16. Iglesias, J., Liu, C.Y., Thompson, P., & Tu, Z. (2011). Robust brain extraction across datasets and comparison with publicly available methods. IEEE Transactions on Medical Imaging, 30(9), 1617–1634. Scholar
  17. Klein, S., Staring, M., Murphy, K., Viergever, M., & Pluim, J.P.W. (2010). Elastix: A toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 29(1), 196–205.CrossRefPubMedGoogle Scholar
  18. Lesjak, Z., Galimzianova, A., Likar, B, Pernuš, F., & Špiclin, Z. (2015). Increased accuracy and reproducibility of MS lesion volume quantification by using publicly available BrainSeg3d image analysis software. In ECTRIMS Online Library, no. 116236.
  19. Llado, X., Oliver, A., Cabezas, M., Freixenet, J., Vilanova, J.C., Quiles, A., Valls, L., Ramio-TorrentA, L., & Rovira, A. (2012). Segmentation of multiple sclerosis lesions in brain MRI: A review of automated approaches. Information Sciences, 186(1), 164–185. Scholar
  20. Patzig, M., Burke, M, Brückmann, H., & Fesl, G. (2014). Comparison of 3d cube FLAIR with 2d FLAIR for multiple sclerosis imaging at 3 Tesla. RoFo: Fortschritte Auf Dem Gebiete Der Rontgenstrahlen Und Der Nuklearmedizin, 186(5), 484–488.
  21. Pearson, K. (1895). Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58, 240–242.CrossRefGoogle Scholar
  22. Pham, D. (2015). Longitudinal MS lesion segmentation challenge. Last accessed: 20 oct, 2016.
  23. Polman, C.H., Reingold, S.C., Banwell, B., Clanet, M., Cohen, J.A., Filippi, M., Fujihara, K., Havrdova, E., Hutchinson, M., Kappos, L., Lublin, F.D., Montalban, X., O’Connor, P., Sandberg-Wollheim, M., Thompson, A.J., Waubant, E., Weinshenker, B., & Wolinsky, J.S. (2011). Diagnostic criteria for multiple sclerosis: 2010 Revisions to the McDonald criteria. Annals of Neurology, 69(2), 292–302. Scholar
  24. Popescu, V., Agosta, F., Hulst, H.E., Sluimer, I.C., Knol, D.L., Sormani, M.P., Enzinger, C., Ropele, S., Alonso, J., Sastre-Garriga, J., Rovira, A., Montalban, X., Bodini, B., Ciccarelli, O., Khaleeli, Z., Chard, D.T., Matthews, L., Palace, J., Giorgio, A., De Stefano, N., Eisele, P., Gass, A., Polman, C.H., Uitdehaag, B.M.J., Messina, M.J., Comi, G., Filippi, M., Barkhof, F., Venken, H., & MAGNIMS Study Group. (2013). Brain atrophy and lesion load predict long term disability in multiple sclerosis. Journal of Neurology, Neurosurgery, and Psychiatry, 84(10), 1082–1091.CrossRefPubMedGoogle Scholar
  25. Rovaris, M., Rocca, M.A., Sormani, M.P., Comi, G., & Filippi, M. (1999). Reproducibility of brain MRI lesion volume measurements in multiple sclerosis using a local thresholding technique: effects of formal operator training. European Neurology, 41(4), 226–230.CrossRefPubMedGoogle Scholar
  26. Rovira, A., Wattjes, M.P, Tintoré, M., Tur, C., Yousry, T.A., Sormani, M.P., De Stefano, N., Filippi, M., Auger, C., Rocca, M.A., Barkhof, F., Fazekas, F., Kappos, L., Polman, C., Miller, D., Montalban, X, & MAGNIMS Study group. (2015). Evidence-based guidelines: MAGNIMS consensus guidelines on the use of MRI in multiple sclerosis-clinical implementation in the diagnostic process. Nature Reviews Neurology, 11(8), 471–482. Scholar
  27. Stangel, M., Penner, I. K., Kallmann, B.A., Lukas, C., & Kieseier, B.C. (2015). Towards the implementation of ’no evidence of disease activity’ in multiple sclerosis treatment: the multiple sclerosis decision model. Therapeutic Advances in Neurological Disorders, 8(1), 3–13.CrossRefPubMedPubMedCentralGoogle Scholar
  28. Styner, M., Lee, J., Chin, B., Chin, M., Commowick, O., Tran, H., Markovic-Plese, S., Jewells, V., & Warfield, S. (2008). 3D segmentation in the clinic: A grand challenge II: MS lesion segmentation. MIDAS Journal, 2008, 1–6.Google Scholar
  29. Tustison, N., Avants, B., Cook, P., Zheng, Y., Egan, A., Yushkevich, P., & Gee, J. (2010). N4ITK: Improved N3 bias correction. IEEE Transactions on Medical Imaging, 29(6), 1310–1320.CrossRefPubMedPubMedCentralGoogle Scholar
  30. Uher, T., Vaneckova, M., Sobisek, L., Tyblova, M., Seidl, Z., Krasensky, J., Ramasamy, D., Zivadinov, R., Havrdova, E., Kalincik, T., & Horakova, D. (2016). Combining clinical and magnetic resonance imaging markers enhances prediction of 12-year disability in multiple sclerosis. Houndmills: Multiple Sclerosis.Google Scholar
  31. Vrenken, H, Jenkinson, M, Horsfield, M.A., Battaglini, M, Schijndel, R.A.V., Rostrup, E., Geurts, J.J.G., Fisher, E., Zijdenbos, A., Ashburner, J., Miller, D.H., Filippi, M., Fazekas, F., Rovaris, M., Rovira, A., Barkhof, F, Stefano, N.D., & Group, M.S. (2013). Recommendations to improve imaging and analysis of brain lesion load and atrophy in longitudinal studies of multiple sclerosis. Journal of Neurology, 260(10), 2458–2471. Scholar
  32. Warfield, S.K., Zou, K.H., & Wells, W.M. (2004). Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging, 23(7), 903–921.CrossRefPubMedPubMedCentralGoogle Scholar
  33. Zijdenbos, A., Dawant, B., Margolin, R., & Palmer, A. (1994). Morphometric analysis of white matter lesions in MR images: method and validation. IEEE Transactions on Medical Imaging, 13(4), 716–724. Scholar
  34. Zijdenbos, A.P., Forghani, R., & Evans, A.C. (2002). Automatic “pipeline” analysis of 3-D MRI data for clinical trials: application to multiple sclerosis. IEEE Transactions on Medical Imaging, 21(10), 1280–1291.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Žiga Lesjak
    • 1
  • Alfiia Galimzianova
    • 1
  • Aleš Koren
    • 2
  • Matej Lukin
    • 2
  • Franjo Pernuš
    • 1
  • Boštjan Likar
    • 3
  • Žiga Špiclin
    • 1
  1. 1.Faculty of Electrical EngineeringUniversity of LjubljanaLjubljanaSlovenia
  2. 2.Institute of RadiologyUniversity Medical Center LjubljanaLjubljanaSlovenia
  3. 3.SensumComputer Vision SystemsLjubljanaSlovenia

Personalised recommendations