Dice Overlap Measures for Objects of Unknown Number: Application to Lesion Segmentation

  • Ipek Oguz
  • Aaron Carass
  • Dzung L. Pham
  • Snehashis Roy
  • Nagesh Subbana
  • Peter A. Calabresi
  • Paul A. Yushkevich
  • Russell T. Shinohara
  • Jerry L. Prince
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10670)


The Dice overlap ratio is commonly used to evaluate the performance of image segmentation algorithms. While Dice overlap is very useful as a standardized quantitative measure of segmentation accuracy in many applications, it offers a very limited picture of segmentation quality in complex segmentation tasks where the number of target objects is not known a priori, such as the segmentation of white matter lesions or lung nodules. While Dice overlap can still be used in these applications, segmentation algorithms may perform quite differently in ways not reflected by differences in their Dice score. Here we propose a new set of evaluation techniques that offer new insights into the behavior of segmentation algorithms. We illustrate these techniques with a case study comparing two popular multiple sclerosis (MS) lesion segmentation algorithms: OASIS and LesionTOADS.


Segmentation Evaluation MS Lesion 



This work was supported, in part, by NIH grants NINDS R01-NS094456, NINDS R01-NS085211, NINDS R21-NS093349, NIBIB R01-EB017255, NINDS R01-NS082347, NINDS R01-NS070906, as well as National MS Society grant RG-1507-05243.


  1. 1.
    Birenbaum, A., Greenspan, H.: Multi-view longitudinal CNN for multiple sclerosis lesion segmentation. Eng. Appl. Artif. Intell. 65, 111–118 (2017)CrossRefGoogle Scholar
  2. 2.
    Carass, A., Roy, S., Jog, A., Cuzzocreo, J.L., Magrath, E., Gherman, A., Button, J., Nguyen, J., Prados, F., Sudre, C.H., Cardoso, M.J., Cawley, N., Ciccarelli, O., Wheeler-Kingshott, C.A.M., Ourselin, S., Catanese, L., Deshpande, H., Maurel, P., Commowick, O., Barillot, C., Tomas-Fernandez, X., Warfield, S.K., Vaidya, S., Chunduru, A., Muthuganapathy, R., Krishnamurthi, G., Jesson, A., Arbel, T., Maier, O., Handels, H., Iheme, L.O., Unay, D., Jain, S., Sima, D.M., Smeets, D., Ghafoorian, M., Platel, B., Birenbaum, A., Greenspan, H., Bazin, P.L., Calabresi, P.A., Crainiceanu, C., Ellingsen, L.M., Reich, D.S., Prince, J.L., Pham, D.L.: Longitudinal multiple sclerosis lesion segmentation: resource & challenge. NeuroImage 148, 77–102 (2017)CrossRefGoogle Scholar
  3. 3.
    Cleveland, W.S.: Robust locally weighted regression and smoothing scatterplots. J. Am. Stat. Assoc. 74(368), 829–836 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Crimi, A.: Brain lesions, introduction. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 1–5. Springer, Cham (2016). CrossRefGoogle Scholar
  5. 5.
    Crum, W.R., Camara, O., Hill, D.L.G.: Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans. Med. Imag. 25(11), 1451–1461 (2006)CrossRefGoogle Scholar
  6. 6.
    Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)CrossRefGoogle Scholar
  7. 7.
    Elliott, C., Arnold, D.L., Collins, D.L., Arbel, T.: Temporally consistent probabilistic detection of new multiple sclerosis lesions in brain MRI. IEEE Trans. Med. Imag. 32(8), 1490–1503 (2013)CrossRefGoogle Scholar
  8. 8.
    García-Lorenzo, D., Lecoeur, J., Arnold, D.L., Collins, D.L., Barillot, C.: Multiple sclerosis lesion segmentation using an automatic multimodal graph cuts. In: Yang, G.-Z., Hawkes, D., Rueckert, D., Noble, A., Taylor, C. (eds.) MICCAI 2009. LNCS, vol. 5762, pp. 584–591. Springer, Heidelberg (2009). CrossRefGoogle Scholar
  9. 9.
    Gerig, G., Jomier, M., Chakos, M.: Valmet: a new validation tool for assessing and improving 3D object segmentation. In: Niessen, W.J., Viergever, M.A. (eds.) MICCAI 2001. LNCS, vol. 2208, pp. 516–523. Springer, Heidelberg (2001). CrossRefGoogle Scholar
  10. 10.
    Goldberg-Zimring, D., Achiron, A., Miron, S., Faibel, M., Azhari, H.: Automated detection and characterization of multiple sclerosis lesions in brain MR images. Mag. Reson. Imaging 16(3), 311–318 (1998)CrossRefGoogle Scholar
  11. 11.
    Jaccard, P.: The distribution of the flora in the alpine zone. New Phytol. 11(2), 37–50 (1912)CrossRefGoogle Scholar
  12. 12.
    Jog, A., Carass, A., Pham, D.L., Prince, J.L.: Multi-output decision trees for lesion segmentation in multiple sclerosis. In: Proceedings of SPIE Medical Imaging (SPIE-MI 2015), Orlando, FL, 21–26 February 2015, vol. 9413, pp. 94131C–94131C-6 (2015)Google Scholar
  13. 13.
    Maier, O., Menze, B.H., von der Gablentz, J., Häni, L., Heinrich, M.P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., Christiaens, D., Dutil, F., Egger, K., Feng, C., Glocker, B., Götz, M., Haeck, T., Halme, H.L., Havaei, M., Iftekharuddin, K.M., Jodoin, P.M., Kamnitsas, K., Kellner, E., Korvenoja, A., Larochelle, H., Ledig, C., Lee, J.H., Maes, F., Mahmood, Q., Maier-Hein, K.H., McKinley, R., Muschelli, J., Pal, C., Pei, L., Rangarajan, J.R., Reza, S.M.S., Robben, D., Rueckert, D., Salli, E., Suetens, P., Wang, C.W., Wilms, M., Kirschke, J.S., Krämer, U.M., Münte, T.F., Schramm, P., Wiest, R., Handels, H., Reyes, M.: ISLES 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Med. Image Anal. 35, 250–269 (2017)CrossRefGoogle Scholar
  14. 14.
    Meier, D.S., Guttmann, C.R.G.: MRI time series modeling of MS lesion development. NeuroImage 32(2), 531–537 (2006)CrossRefGoogle Scholar
  15. 15.
    Nascimento, J.C., Marques, J.S.: Performance evaluation of object detection algorithms for video surveillance. IEEE Trans. Multimed. 8(4), 761–774 (2006)CrossRefGoogle Scholar
  16. 16.
    Rohlfing, T.: Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable. IEEE Trans. Med. Imaging 31(2), 153–163 (2012)CrossRefGoogle Scholar
  17. 17.
    Shiee, N., Bazin, P.L., Ozturk, A., Reich, D.S., Calabresi, P.A., Pham, D.L.: A topology-preserving approach to the segmentation of brain images with multiple sclerosis lesions. NeuroImage 49(2), 1524–1535 (2010)CrossRefGoogle Scholar
  18. 18.
    Shiee, N., Bazin, P.L., Zackowski, K., Farrell, S.K., Harrison, D.M., Newsome, S.D., Ratchford, J.N., Caffo, B.S., Calabresi, P.A., Pham, D.L., Reich, D.S.: Revisiting brain atrophy and its relationship to disability in multiple sclerosis. PLoS ONE 7(5), e37049 (2012)CrossRefGoogle Scholar
  19. 19.
    Styner, M., Lee, J., Chin, B., Chin, M.S., Commowick, O., Tran, H.H., Markovic-Plese, S., Jewells, V., Warfield, S.: 3D segmentation in the clinic: a grand challenge II: MS lesion segmentation. In: 11th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2008) 3D Segmentation in the Clinic: A Grand Challenge II, pp. 1–6 (2008)Google Scholar
  20. 20.
    Sweeney, E.M., Shinohara, R.T., Dewey, B.E., Schindler, M.K., Muschelli, J., Reich, D.S., Crainiceanu, C.M., Eloyan, A.: Relating multi-sequence longitudinal intensity profiles and clinical covariates in incident multiple sclerosis lesions. NeuroImage Clin. 10, 1–17 (2016)CrossRefGoogle Scholar
  21. 21.
    Sweeney, E.M., Shinohara, R.T., Shiee, N., Mateen, F.J., Chudgar, A.A., Cuzzocreo, J.L., Calabresi, P.A., Pham, D.L., Reich, D.S., Crainiceanu, C.M.: OASIS is automated statistical inference for segmentation, with applications to multiple sclerosis lesion segmentation in MRI. NeuroImage Clin. 2, 402–413 (2013)CrossRefGoogle Scholar
  22. 22.
    Taha, A.A., Hanbury, A.: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med. Imaging 15(1), 29 (2015)CrossRefGoogle Scholar
  23. 23.
    Tomas-Fernandez, X., Warfield, S.K.: A model of population and subject (MOPS) intensities with application to multiple sclerosis lesion segmentation. IEEE Trans. Med. Imaging 34(6), 1349–1361 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Ipek Oguz
    • 1
  • Aaron Carass
    • 2
    • 3
  • Dzung L. Pham
    • 4
  • Snehashis Roy
    • 4
  • Nagesh Subbana
    • 1
  • Peter A. Calabresi
    • 5
  • Paul A. Yushkevich
    • 1
  • Russell T. Shinohara
    • 6
  • Jerry L. Prince
    • 2
    • 3
  1. 1.Department of RadiologyUniversity of PennsylvaniaPhiladelphiaUSA
  2. 2.Department of Electrical and Computer EngineeringThe Johns Hopkins UniversityBaltimoreUSA
  3. 3.Department of Computer ScienceThe Johns Hopkins UniversityBaltimoreUSA
  4. 4.CNRMThe Henry M. Jackson Foundation for the Advancement of Military MedicineBethesdaUSA
  5. 5.Department of NeurologyThe Johns Hopkins University School of MedicineBaltimoreUSA
  6. 6.Department of Biostatistics and EpidemiologyUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations