Abstract
Quantified volume and count of white-matter lesions based on magnetic resonance (MR) images are important biomarkers in several neurodegenerative diseases. For a routine extraction of these biomarkers an accurate and reliable automated lesion segmentation is required. To objectively and reliably determine a standard automated method, however, creation of standard validation datasets is of extremely high importance. Ideally, these datasets should be publicly available in conjunction with standardized evaluation methodology to enable objective validation of novel and existing methods. For validation purposes, we present a novel MR dataset of 30 multiple sclerosis patients and a novel protocol for creating reference white-matter lesion segmentations based on multi-rater consensus. On these datasets three expert raters individually segmented white-matter lesions, using in-house developed semi-automated lesion contouring tools. Later, the raters revised the segmentations in several joint sessions to reach a consensus on segmentation of lesions. To evaluate the variability, and as quality assurance, the protocol was executed twice on the same MR images, with a six months break. The obtained intra-consensus variability was substantially lower compared to the intra- and inter-rater variabilities, showing improved reliability of lesion segmentation by the proposed protocol. Hence, the obtained reference segmentations may represent a more precise target to evaluate, compare against and also train, the automatic segmentations. To encourage further use and research we will publicly disseminate on our website http://lit.fe.uni-lj.si/tools the tools used to create lesion segmentations, the original and preprocessed MR image datasets and the consensus lesion segmentations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Magnetic resonance imaging in MS www.magnims.eu
References
Akhondi-Asl, A., Hoyte, L., Lockhart, M.E., & Warfield, S.K. (2014). A logarithmic opinion pool based staple algorithm for the fusion of segmentations with associated reliability weights. IEEE Transactions on Medical Imaging, 33(10), 1997–2009.
Anbeek, P., Vincken, K.L., van Osch, M.J.P., Bisschops, R.H.C., & van der Grond, J. (2004). Probabilistic segmentation of white matter lesions in MR, imaging. NeuroImage, 21(3), 1037–1044.
Arthur, D., & Vassilvitskii, S. (2007). K-means++: the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM Symposium on Discrete algorithms, SODA ’07 (pp. 1027–1035). Philadelphia: Society for Industrial and Applied Mathematics.
Barillot, C., Commowick, O., Guttmann, C., Styner, M., & Warfield, S. (2016). MS Segmentation challenge. Last accessed: 20 oct, 2016. https://portal.fli-iam.irisa.fr/msseg-challenge/overview.
Cardoso, M.J., Modat, M., Wolz, R., Melbourne, A., Cash, D., Rueckert, D., & Ourselin, S. (2015). Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion. IEEE Transactions on Medical Imaging, 34(9), 1976–1988. https://doi.org/10.1109/TMI.2015.2418298.
CIBC. (2016). Seg3d: Volumetric image segmentation and visualization. Scientific computing and imaging institute (SCI), download from: http://www.seg3d.org.
Cocosco, C.A., Kollokian, V., Kwan, R.K.S., Pike, G.B., & Evans, A.C. (1997). BrainWeb: online interface to a 3d MRI simulated brain database. NeuroImage, 5, 425.
Commowick, O., & Warfield, S. (2009). A continuous STAPLE for scalar, vector, and tensor images: an application to DTI analysis. IEEE Transactions on Medical Imaging, 28(6), 838–846. https://doi.org/10.1109/TMI.2008.2010438.
Debette, S., & Markus, H.S. (2010). The clinical importance of white matter hyperintensities on brain magnetic resonance imaging: systematic review and meta-analysis. BMJ, 341, c3666. https://doi.org/10.1136/bmj.c3666.
Di Perri, C., Dwyer, M.G., Wack, D.S., Cox, J.L., Hashmi, K., Saluste, E., Hussein, S., Schirda, C., Stosic, M., Durfee, J., Poloni, G. U., Nayyar, N., Bergamaschi, R., & Zivadinov, R. (2009). Signal abnormalities on 1.5 and 3 Tesla brain mri in multiple sclerosis patients and healthy controls. A morphological and spatial quantitative comparison study. NeuroImage, 47(4), 1352–1362.
Dice, L.R. (1945). Measures of the amount of ecologic association between species. Ecology, 26(3), 297–302. https://doi.org/10.2307/1932409. ArticleType: research-article / Full publication date: Jul., 1945 / Copyright Ⓒ1945 Ecological Society of America.
Filippi, M., Horsfield, M.A., Bressi, S., Martinelli, V., Baratti, C., Reganati, P., Campi, A., Miller, D.H., & Comi, G. (1995). Intra- and inter-observer agreement of brain MRI lesion volume measurements in multiple sclerosis. A comparison of techniques. Brain: A Journal of Neurology, 118( Pt 6), 1593–1600.
Fonov, V., Evans, A., McKinstry, R., Almli, C., & Collins, D. (2009). Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage, 47(Supplement 1), S102.
Garcia-Lorenzo, D., Francis, S., Narayanan, S., Arnold, D.L., & Collins, D.L. (2013). Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging. Medical Image Analysis, 17(1), 1–18. https://doi.org/10.1016/j.media.2012.09.004.
Grimaud, J., Lai, M., Thorpe, J., Adeleine, P., Wang, L., Barker, G.J., Plummer, D.L, Tofts, P.S., McDonald, W.I, & Miller, D.H. (1996). Quantification of MRI lesion load in multiple sclerosis: A comparison of three computer-assisted techniques. Magnetic Resonance Imaging, 14(5), 495–505. https://doi.org/10.1016/0730-725X(96)00018-5.
Iglesias, J., Liu, C.Y., Thompson, P., & Tu, Z. (2011). Robust brain extraction across datasets and comparison with publicly available methods. IEEE Transactions on Medical Imaging, 30(9), 1617–1634. https://doi.org/10.1109/TMI.2011.2138152.
Klein, S., Staring, M., Murphy, K., Viergever, M., & Pluim, J.P.W. (2010). Elastix: A toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 29(1), 196–205.
Lesjak, Z., Galimzianova, A., Likar, B, Pernuš, F., & Špiclin, Z. (2015). Increased accuracy and reproducibility of MS lesion volume quantification by using publicly available BrainSeg3d image analysis software. In ECTRIMS Online Library, no. 116236. http://onlinelibrary.ectrims-congress.eu/ectrims/2015/31st/116236/iga.lesjak.increased.accuracy.and.reproducibility.of.ms.lesion.volume.html?f=p6m3e891o11460.
Llado, X., Oliver, A., Cabezas, M., Freixenet, J., Vilanova, J.C., Quiles, A., Valls, L., Ramio-TorrentA, L., & Rovira, A. (2012). Segmentation of multiple sclerosis lesions in brain MRI: A review of automated approaches. Information Sciences, 186(1), 164–185. https://doi.org/10.1016/j.ins.2011.10.011.
Patzig, M., Burke, M, Brückmann, H., & Fesl, G. (2014). Comparison of 3d cube FLAIR with 2d FLAIR for multiple sclerosis imaging at 3 Tesla. RoFo: Fortschritte Auf Dem Gebiete Der Rontgenstrahlen Und Der Nuklearmedizin, 186(5), 484–488. https://doi.org/10.1055/s-0033-1355896.
Pearson, K. (1895). Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58, 240–242.
Pham, D. (2015). Longitudinal MS lesion segmentation challenge. Last accessed: 20 oct, 2016. http://iacl.ece.jhu.edu/index.php/MSChallenge.
Polman, C.H., Reingold, S.C., Banwell, B., Clanet, M., Cohen, J.A., Filippi, M., Fujihara, K., Havrdova, E., Hutchinson, M., Kappos, L., Lublin, F.D., Montalban, X., O’Connor, P., Sandberg-Wollheim, M., Thompson, A.J., Waubant, E., Weinshenker, B., & Wolinsky, J.S. (2011). Diagnostic criteria for multiple sclerosis: 2010 Revisions to the McDonald criteria. Annals of Neurology, 69(2), 292–302. https://doi.org/10.1002/ana.22366.
Popescu, V., Agosta, F., Hulst, H.E., Sluimer, I.C., Knol, D.L., Sormani, M.P., Enzinger, C., Ropele, S., Alonso, J., Sastre-Garriga, J., Rovira, A., Montalban, X., Bodini, B., Ciccarelli, O., Khaleeli, Z., Chard, D.T., Matthews, L., Palace, J., Giorgio, A., De Stefano, N., Eisele, P., Gass, A., Polman, C.H., Uitdehaag, B.M.J., Messina, M.J., Comi, G., Filippi, M., Barkhof, F., Venken, H., & MAGNIMS Study Group. (2013). Brain atrophy and lesion load predict long term disability in multiple sclerosis. Journal of Neurology, Neurosurgery, and Psychiatry, 84(10), 1082–1091.
Rovaris, M., Rocca, M.A., Sormani, M.P., Comi, G., & Filippi, M. (1999). Reproducibility of brain MRI lesion volume measurements in multiple sclerosis using a local thresholding technique: effects of formal operator training. European Neurology, 41(4), 226–230.
Rovira, A., Wattjes, M.P, Tintoré, M., Tur, C., Yousry, T.A., Sormani, M.P., De Stefano, N., Filippi, M., Auger, C., Rocca, M.A., Barkhof, F., Fazekas, F., Kappos, L., Polman, C., Miller, D., Montalban, X, & MAGNIMS Study group. (2015). Evidence-based guidelines: MAGNIMS consensus guidelines on the use of MRI in multiple sclerosis-clinical implementation in the diagnostic process. Nature Reviews Neurology, 11(8), 471–482. https://doi.org/10.1038/nrneurol.2015.106.
Stangel, M., Penner, I. K., Kallmann, B.A., Lukas, C., & Kieseier, B.C. (2015). Towards the implementation of ’no evidence of disease activity’ in multiple sclerosis treatment: the multiple sclerosis decision model. Therapeutic Advances in Neurological Disorders, 8(1), 3–13.
Styner, M., Lee, J., Chin, B., Chin, M., Commowick, O., Tran, H., Markovic-Plese, S., Jewells, V., & Warfield, S. (2008). 3D segmentation in the clinic: A grand challenge II: MS lesion segmentation. MIDAS Journal, 2008, 1–6.
Tustison, N., Avants, B., Cook, P., Zheng, Y., Egan, A., Yushkevich, P., & Gee, J. (2010). N4ITK: Improved N3 bias correction. IEEE Transactions on Medical Imaging, 29(6), 1310–1320.
Uher, T., Vaneckova, M., Sobisek, L., Tyblova, M., Seidl, Z., Krasensky, J., Ramasamy, D., Zivadinov, R., Havrdova, E., Kalincik, T., & Horakova, D. (2016). Combining clinical and magnetic resonance imaging markers enhances prediction of 12-year disability in multiple sclerosis. Houndmills: Multiple Sclerosis.
Vrenken, H, Jenkinson, M, Horsfield, M.A., Battaglini, M, Schijndel, R.A.V., Rostrup, E., Geurts, J.J.G., Fisher, E., Zijdenbos, A., Ashburner, J., Miller, D.H., Filippi, M., Fazekas, F., Rovaris, M., Rovira, A., Barkhof, F, Stefano, N.D., & Group, M.S. (2013). Recommendations to improve imaging and analysis of brain lesion load and atrophy in longitudinal studies of multiple sclerosis. Journal of Neurology, 260(10), 2458–2471. https://doi.org/10.1007/s00415-012-6762-5.
Warfield, S.K., Zou, K.H., & Wells, W.M. (2004). Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging, 23(7), 903–921.
Zijdenbos, A., Dawant, B., Margolin, R., & Palmer, A. (1994). Morphometric analysis of white matter lesions in MR images: method and validation. IEEE Transactions on Medical Imaging, 13(4), 716–724. https://doi.org/10.1109/42.363096.
Zijdenbos, A.P., Forghani, R., & Evans, A.C. (2002). Automatic “pipeline” analysis of 3-D MRI data for clinical trials: application to multiple sclerosis. IEEE Transactions on Medical Imaging, 21(10), 1280–1291.
Acknowledgements
The authors would like to acknowledge Nuška Pečarič, MD, from the University Medical Centre Ljubljana for performing the lesion segmentations.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported by the Slovenian Research Agency under grants J2-5473, L2-5472, J7-6781 and J2-8173.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Lesjak, Ž., Galimzianova, A., Koren, A. et al. A Novel Public MR Image Dataset of Multiple Sclerosis Patients With Lesion Segmentations Based on Multi-rater Consensus. Neuroinform 16, 51–63 (2018). https://doi.org/10.1007/s12021-017-9348-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12021-017-9348-7