The Visual Computer

, Volume 31, Issue 6–8, pp 989–999 | Cite as

Extended surface distance for local evaluation of 3D medical image segmentations

  • Roman Getto
  • Arjan Kuijper
  • Tatiana von Landesberger
Original Article


The evaluation of 3D medical image segmentation quality requires a reliable detailed comparison of a reference segmentation with an automatic segmentation. It should be able to measure the quality accurately and, thus, to reveal problematic regions. While several (global) measures, providing a single quality value, are available, the only widely used local measure is the Surface Distance (i.e., point-to-surface distance). This measure, however, has significant drawbacks such as asymmetry and underestimation in distant and differently formed regions. Other available measures have limited suitability for 3D medical segmentation evaluation. We present a more reliable distance measure for assessing and analyzing local differences between automatic and reference (i.e., ground truth) 3D segmentations. We identify and overcome Surface Distance drawbacks, esp. in regions with larger dissimilarities. We evaluated our approach on four real medical image datasets. The results indicate that our measure provides more accurate local distance values.


Evaluation Distance measure  3D medical image segmentation Segmentation quality Local distance Mesh distance 



The work has been partially supported by DFG within an SPP 1335 project. The authors are grateful to Prof. Georg Sakas and Dr. Meike Becker for the data provision, support with the project and helpful comments on the paper draft.


  1. 1.
    Aspert, N., Santa-Cruz, D., Ebrahimi, T.: Mesh: measuring errors between surfaces using the hausdorff distance. In: Proceedings of Multimedia and Expo, ICME, vol. 1, pp. 705–708. IEEE, New York (2002)Google Scholar
  2. 2.
    Becker, M., Kirschner, M., Sakas, G.: Segmentation of risk structures for otologic surgery using the probabilistic active shape model (pasm). In: Proceedings of SPIE Medical Imaging, pp. 90,360O–90,360O. International Society for Optics and Photonics (2014)Google Scholar
  3. 3.
    Belongie, S., Malik, J., Puzicha, J.: Matching shapes. In: Proceedings of Computer Vision, ICCV, vol. 1, pp. 454–461. IEEE, New York (2001)Google Scholar
  4. 4.
    Brown, B.J., Rusinkiewicz, S.: Global non-rigid alignment of 3-d scans. In: Proceedings of ACM T. Graphics (TOG), vol. 26, p. 21. ACM, New York (2007)Google Scholar
  5. 5.
    Cates, J., Meyer, M., Fletcher, T., Whitaker, R., et al.: Entropy-based particle systems for shape correspondence. In: Proceedings of 1st MICCAI Workshop on Mathematical Foundations of Computational Anatomy: Geometrical, Statistical and Registration Methods for Modeling Biological Shape Variability, pp. 90–99 (2006)Google Scholar
  6. 6.
    Chalana, V., Kim, Y.: A methodology for evaluation of boundary detection algorithms on medical images. IEEE T. Med. Imaging 16(5), 642–652 (1997)CrossRefGoogle Scholar
  7. 7.
    Cohen-Steiner, D., Alliez, P., Desbrun, M.: Variational shape approximation. ACM T. Gr. 23(3), 905–914 (2004)CrossRefGoogle Scholar
  8. 8.
    Gerig, G., Jomier, M., Chakos, M.: Valmet: a new validation tool for assessing and improving 3D object segmentation. In: Proceedings of Medical Image Computing and Computer-Assisted Intervention, MICCAI, pp. 516–523. Springer, Berlin, Germany (2001)Google Scholar
  9. 9.
    Gueziec, A.: Meshsweeper: dynamic point-to-polygonal mesh distance and applications. IEEE T. Vis. Comput. Gr. 7(1), 47–61 (2001)CrossRefGoogle Scholar
  10. 10.
    Haehnel, D., Thrun, S., Burgard, W.: An extension of the icp algorithm for modeling nonrigid objects with mobile robots. In: Proceedings of IJCAI, pp. 915–920 (2003)Google Scholar
  11. 11.
    Heimann, T., van Ginneken, B., Styner, M.A., Arzhaeva, Y., Aurich, V., Bauer, C., Beck, A., Becker, C., Beichel, R., Bekes, G., et al.: Comparison and evaluation of methods for liver segmentation from ct datasets. IEEE T. Med. Imaging 28(8), 1251–1265 (2009)CrossRefGoogle Scholar
  12. 12.
    Heimann, T., Meinzer, H.P.: Statistical shape models for 3d medical image segmentation: a review. Med. Image Anal. 13(4), 543–563 (2009)CrossRefGoogle Scholar
  13. 13.
    Hoppe, H.: Progressive meshes. In: Proceedings of Computer Graphics and Interactive Techniques, pp. 99–108. ACM, New York (1996)Google Scholar
  14. 14.
    Johnson, A.E.: Spin-images: a representation for 3-D surface matching. Ph.D. thesis, Carnegie Mellon University (1997)Google Scholar
  15. 15.
    Kirschner, M., Becker, M., Wesarg, S.: 3D active shape model segmentation with nonlinear shape priors. In: Proceedings of Medical Image Computing and Computer-Assisted Intervention, MICCAI, LNCS, vol. 6892, pp. 492–499 (2011)Google Scholar
  16. 16.
    Kraevoy, V., Sheffer, A.: Cross-parameterization and compatible remeshing of 3d models. In: Proceedings of ACM T. Graphics (TOG), vol. 23, pp. 861–869. ACM, New York (2004)Google Scholar
  17. 17.
    Landesberger, T.V., Andrienko, G., Andrienko, N., Bremm, S., Kirschner, M., Wesarg, S., Kuijper, A.: Opening up the “black box” of medical image segmentation with statistical shape models. Vis. Comput. 29(9), 893–905 (2013)CrossRefGoogle Scholar
  18. 18.
    Li, H., Hartley, R.: The 3D–3D registration problem revisited. In: Proceedings of Computer Vision, ICCV, pp. 1–8. IEEE, New York (2007)Google Scholar
  19. 19.
    Li, H., Sumner, R.W., Pauly, M.: Global correspondence optimization for non-rigid registration of depth scans. In: Proceedings of Computer Graphics Forum, vol. 27, pp. 1421–1430. Wiley, New Jersey (2008)Google Scholar
  20. 20.
    Lipman, Y., Rustamov, R.M., Funkhouser, T.A.: Biharmonic distance. ACM T. Gr. 29(3), 27 (2010)Google Scholar
  21. 21.
    Lorenz, C., Krahnstover, N.: 3D statistical shape models for medical image segmentation. In: Proceedings of 3-D Digital Imaging and Modeling, pp. 414–423. IEEE, New York (1999)Google Scholar
  22. 22.
    Martinek, M., Grosso, R., Greiner, G.: Interactive partial 3d shape matching with geometric distance optimization. Vis. Comput. 31(2), 223–233 (2015)CrossRefGoogle Scholar
  23. 23.
    Pizer, S.M., Fletcher, P.T., Joshi, S., Thall, A., Chen, J.Z., Fridman, Y., Fritsch, D.S., Gash, A.G., Glotzer, J.M., Jiroutek, M.R., et al.: Deformable m-reps for 3d medical image segmentation. Int. J. Comput. Vis. 55(2–3), 85–106 (2003)CrossRefGoogle Scholar
  24. 24.
    Sahillioglu, Y., Yemez, Y.: Coarse-to-fine combinatorial matching for dense isometric shape correspondence. In: Proceedings of Computer Graphics Forum, vol. 30, pp. 1461–1470. Wiley, New Jersey (2011)Google Scholar
  25. 25.
    Strecha, C., von Hansen, W., Van Gool, L., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: Proceedings of Computer Vision and Pattern Recognition, CVPR, pp. 1–8. IEEE, New York (2008)Google Scholar
  26. 26.
    Udupa, J.K., Leblanc, V.R., Zhuge, Y., Imielinska, C., Schmidt, H., Currie, L.M., Hirsch, B.E., Woodburn, J.: A framework for evaluating image segmentation algorithms. Comput. Med. Imaging Gr. 30(2), 75–87 (2006)CrossRefGoogle Scholar
  27. 27.
    Van Ginneken, B., Heimann, T., Styner, M.: 3D segmentation in the clinic: a grand challenge. 3D segmentation in the clinic: a grand challenge, pp. 7–15 (2007)Google Scholar
  28. 28.
    Van Kaick, O., Zhang, H., Hamarneh, G., Cohen-Or, D.: A survey on shape correspondence. In: Proceedings of Computer Graphics Forum, vol. 30, pp. 1681–1707. Wiley, New Jersey (2011)Google Scholar
  29. 29.
    Veltkamp, R.C., Hagedoorn, M.: State-of-the-art in shape matching. In: Proceedings of Technical Report, Principles of Visual Information Retrieval (1999)Google Scholar
  30. 30.
    Wu, H., Miao, Z., Wang, Y., Lin, M.: Optimized recognition with few instances based on semantic distance. Vis. Comput. 31(4), 367–375 (2015)CrossRefGoogle Scholar
  31. 31.
    Zhang, H., Sheffer, A., Cohen-Or, D., Zhou, Q., Van Kaick, O., Tagliasacchi, A.: Deformation-driven shape correspondence. In: Proceedings of Computer Graphics Forum, vol. 27, pp. 1431–1439. Wiley, New York (2008)Google Scholar
  32. 32.
    Zou, K.H., Warfield, S.K., Bharatha, A., Tempany, C.M., Kaus, M.R., Haker, S.J., Wells, W.M., Jolesz, F.A., Kikinis, R.: Statistical validation of image segmentation quality based on a spatial overlap index 1: scientific reports. Acad. Radiol. 11(2), 178–189 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Roman Getto
    • 1
  • Arjan Kuijper
    • 2
  • Tatiana von Landesberger
    • 1
  1. 1.TU DarmstadtDarmstadtGermany
  2. 2.TU Darmstadt and Fraunhofer IGDDarmstadtGermany

Personalised recommendations