Journal of Mathematical Imaging and Vision

, Volume 28, Issue 3, pp 225–241 | Cite as

Measures for Benchmarking of Automatic Correspondence Algorithms

  • Anders Ericsson
  • Johan Karlsson


Automatic localisation of correspondences for the construction of Statistical Shape Models from examples has been the focus of intense research during the last decade. Several algorithms are available and benchmarking is needed to rank the different algorithms. Prior work has argued that the quality of the models produced by the algorithms can be evaluated by measuring compactness, generality and specificity. In this paper severe problems with these standard measures are analysed both theoretically and experimentally both on natural and synthetic datasets. We also propose that a Ground Truth Correspondence Measure (GCM) is used for benchmarking and in this paper benchmarking is performed on several state of the art algorithms using seven real and one synthetic dataset.


Active shape Shape modelling Correspondence Benchmarking GCM Specificity Generality Compactness Evaluation Verification Measure 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baumberg, A., Hogg, D.: Learning flexible models from image sequences. In: Proc. European Conf. on Computer Vision, ECCV’94, pp. 299–308 (1994) Google Scholar
  2. 2.
    Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(24), 509–522 (2002) CrossRefGoogle Scholar
  3. 3.
    Benayoun, A., Ayache, N., Cohen, I.: Adaptive meshes and nonrigid motion computation. In: Proc. International Conference on Pattern Recognition, Jerusalem, Israel, pp. 730–732 (1994) Google Scholar
  4. 4.
    Bookstein, F.: Landmark methods for forms without landmarks: morphometrics of group differences in outline shape. Med. Image Anal. 3, 225–243 (1999) Google Scholar
  5. 5.
    Chui, H., Rangarajan, A.: A feature registration framework using mixture models. In: IEEE Workshop on Mathematical Methods in Biomedical Image Analysis (MMBIA), pp. 190–197 (2000) Google Scholar
  6. 6.
    Chui, H., Rangarajan, A.: A new algorithm for non-rigid point matching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. II, pp. 44–51 (2000) Google Scholar
  7. 7.
    Davies, R.: Learning shape: optimal models for analysing natural variability. PhD thesis, University of Manchester (2002) Google Scholar
  8. 8.
    Davies, R., Twining, C., Cootes, T., Waterton, J., Taylor, C.: A minimum description length approach to statistical shape modeling. IEEE Trans. Med. Imaging 21(5), 525–537 (2002) CrossRefGoogle Scholar
  9. 9.
    Davies, R.H., Cootes, T.F., Waterton, J.C., Taylor, C.J.: An efficient method for constructing optimal statistical shape models. In: Medical Image Computing and Computer-Assisted Intervention MICCAI’2001, pp. 57–65 (2001) Google Scholar
  10. 10.
    Davies, R.H., Twining, C.J., Allen, P.D., Cootes, T.F., Taylor, C.J.: Shape discrimination in the hippocampus using an MDL model. In: Information Processing in Medical Imaging (2003) Google Scholar
  11. 11.
    Dryden, I.L., Mardia, K.V.: Statistical Shape Analysis. Wiley, New York (1998) zbMATHGoogle Scholar
  12. 12.
    Ericsson, A.: Automatic shape modelling and applications in medical imaging. Technical report, Mathematics LTH, Centre for Mathematical Sciences, Box 118, 22100 Lund, Sweden, November 2003 Google Scholar
  13. 13.
    Ericsson, A.: Automatic shape modelling with applications in medical imaging. PhD thesis, Lund University, Centre for Mathematical Sciences, Box 118, 22100 Lund, Sweden, September 2006 Google Scholar
  14. 14.
    Ericsson, A., Åström, K.: An affine invariant deformable shape representation for general curves. In: Proc. 9th Int. Conf. on Computer Vision, Nice, France, pp. 1142–1149 (2003) Google Scholar
  15. 15.
    Ericsson, A., Åström, K.: Minimizing the description length using steepest descent. In: Proc. British Machine Vision Conference, Norwich, United Kingdom, vol. 2, pp. 93–102 (2003) Google Scholar
  16. 16.
    Ericsson, A., Karlsson, J.: Aligning shapes by minimising the description length. In: Proc. Scandinavian Conf. on Image Analysis, SCIA’05, Joensuu, Finland, vol. 3540/2005, pp. 709–718 (2005) Google Scholar
  17. 17.
    Fisher, R.: Caviar project. Ground truth labelled video sequences (2005), available at
  18. 18.
    Gower, J.: Generalized procrustes analysis. Psychometrica 40, 33–50 (1975) zbMATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Hill, A., Taylor, C.: Automatic landmark generation for point distribution models. In: Proc. British Machine Vision Conference, pp. 429–438 (1994) Google Scholar
  20. 20.
    Hill, A., Taylor, C.: A framework for automatic landmark indentification using a new method of nonrigid correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 22, 241–251 (2000) CrossRefGoogle Scholar
  21. 21.
    Kambhamettu, C., Goldgof, D.: Points correspondences recovery in non-rigid motion. In: Proc. Conf. Computer Vision and Pattern Recognition, CVPR’92, pp. 222–237 (1992) Google Scholar
  22. 22.
    Karlsson, J., Ericsson, A., Åström, K.: Parameterisation invariant statistical shape models. In: Proc. International Conference on Pattern Recognition, Cambridge, UK (2004) Google Scholar
  23. 23.
    Kelemen, A., Szekely, G., Gerig, G.: Elastic model-based segmentation of 3D neuroradiological data sets. IEEE Trans. Med. Imaging 18(10), 828–839 (1999) CrossRefGoogle Scholar
  24. 24.
    Kotcheff, A., Taylor, C.: Automatic construction of eigenshape models by direct optimization. Med. Image Anal. 2, 303–314 (1998) CrossRefGoogle Scholar
  25. 25.
    Mao, Z., Ju, X., Siebert, J., Cockshott, W., Ayoub, A.: Constructing dense correspondences for the analysis of 3D facial morphology. Pattern Recognit. Lett. 27(6), 597–608 (2006) CrossRefGoogle Scholar
  26. 26.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5), 530–549 (2004) CrossRefGoogle Scholar
  27. 27.
    Richter, J., Ericsson, A., Åström, K., Kahl, F., Edenbrant, L.: Automated interpretation of cardiac scintigrams. In: Proc. 13th Scandinavian Conf. on Image Analysis, Gothenburg, Sweden (2003) Google Scholar
  28. 28.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1/2/3), 7–42 (2002) zbMATHCrossRefGoogle Scholar
  29. 29.
    Schestowitz, R., Twining, C., Cootes, T., Petrović, V., Taylor, C., Crum, B.: Assessing the accuracy of non-rigid registration with and without ground truth. In: Proc. IEEE International Symposium on Biomedical Imaging (2006) Google Scholar
  30. 30.
    Sebastian, T., Klein, P., Kimia, B.: Constructing 2D curve atlases. In: IEEE Workshop on Mathematical Methods in Biomedical Image Analysis, pp. 70–77 (2000) Google Scholar
  31. 31.
    Sebastian, T., Klein, P., Kimia, B.: On aligning curves. IEEE Trans. Pattern Anal. Mach. Intell. 25(1), 116–125 (2003) CrossRefGoogle Scholar
  32. 32.
    Sharvit, D., Chan, J., Tek, H., Kimia, B.: Symmetry-based indexing of image databases. J. Vis. Commun. Image Represent. 4(4), 366–380 (1998) CrossRefGoogle Scholar
  33. 33.
    Sternby, J., Ericsson, A.: Core points—a framework for structural parameterization. In: Proc. Internation Conference on Document Analysis and Recognition, ICDAR’05, Seoul, Korea (2005) Google Scholar
  34. 34.
    Styner, M., Rajamani, K., Nolte, L., Zsemlye, G., Szekely, G., Taylor, C., Davies, R.H.: Evaluation of 3D correspondence methods for model building. In: Information Processing in Medical Imaging (IPMI), pp. 63–75 (2003) Google Scholar
  35. 35.
    Tagare, H.: Shape-based nonrigid correspondence with application to heart motion analysis. IEEE Trans. Med. Imaging 18, 570–579 (1999) CrossRefGoogle Scholar
  36. 36.
    Thodberg, H.H.: Minimum description length shape and appearance models. In: Image Processing Medical Imaging, IPMI (2003) Google Scholar
  37. 37.
    Thodberg, H.H., Olafsdottir, H.: Adding curvature to minimum description length shape models. In: Proc. British Machine Vision Conference (2003) Google Scholar
  38. 38.
    Twining, C., Taylor, C.: Specificity as a graph-based estimator of cross-entropy. In: Proc. British Machine Vision Conference, Edinburgh, United Kingdom, vol. 2, pp. 459–468 (2006) Google Scholar
  39. 39.
    Wang, Y., Peterson, B., Staib, L.: Shape-based 3D surface correspondence using geodesics and local geometry. In: Proc. Conf. Computer Vision and Pattern Recognition, CVPR’00, pp. 644–651 (2000) Google Scholar
  40. 40.
    Zheng, Y., Doermann, D.: Robust point matching for non-rigid shapes: a relaxation labeling based approach. Technical Report: Lamp-tr-117, University of Maryland, College Park (2004) Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Centre for Mathematical SciencesLund UniversityLundSweden

Personalised recommendations