Abstract
While the development of particular video segmentation algorithms has attracted considerable research interest, relatively little effort has been devoted to provide a methodology for evaluating their performance. In this paper, we propose a methodology to objectively evaluate video segmentation algorithm with ground-truth, which is based on computing the deviation of segmentation results from the reference segmentation. Four different metrics based on classification pixels, edges, relative foreground area and relative position respectively are combined to address the spatial accuracy. Temporal coherency is evaluated by utilizing the difference of spatial accuracy between successive frames. The experimental results show the feasibility of our approach. Moreover, it is computationally more efficient than previous methods. It can be applied to provide an offline ranking among different segmentation algorithms and to optimally set the parameters for a given algorithm.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Erdem C E, Sankur B, Tekalp A M. Non-rigid object tracking using performance evaluation measures as feedback [A]. Proc. of IEEE Conference on Computer Vision and Pattern Recognition, CVPR’2001 [C]. Hawaii, 11–13 Dec. 2001: 323–330.
Villegas P, Marichal X, et al. Objective evaluation of segmentation masks in video sequences [A]. Proc. of WIAMIS 99 workshop[C]. Berlin, May 1999: 85–88.
Wollborn M, Mech R. Refined Procedure for Objective Evaluation of VOP Generation Algorithms, Doc, ISO/IEC JTC1/SC29/WG11 MPEG98/3448 [R]. Tokyo, March 1998.
Erdem C E, Sandur B. Performance evaluation of segmentation masks in video sequences [A]. EUSIPCO ’2000: 10th European Signal Processing Conference [C]. Tampere, Finland, 5–8 September 2000: 917–920.
Correia P, Pereira F. Objective evaluation of relative segmentation quality [A]. Proc. of IEEE Conference on Image Processing (ICIP2000)[C]. Canada, 10–13, Sept. 2000: 308–311.
Correia P, Pereira F. Objective evaluation of standalone segmentation quality [A]. Proc. of WIAMIS 2001 workshop[C]. Tampere, 2001.
Erdem C E, Tekalp A M, Sandur B. Metrics for performance evaluation of video object segmentation and tracking without ground-truth [A]. Proc. of IEEE Conference on Image Processing (ICIP2001)[C]. Thessaloniki, Greece, 2001.
Cavallaro A, Drelie E, Ebrahimi T. Objective evaluation of segmentation quality using spatio-temporal context [A]. Proc. of IEEE International Conference on Image Processing[C]. Rochester, New York, Sept. 2002: 301–304.
Working site for sequences and algorithms exchange[J/OL]. http: //www. tele. ucl. ac. be/exchange
Pratt WK. Digital Image Processing[M]. Wiley, New York, 1978.
Cavallaro A, Ebrahhimi T. Change detection based on color edges [A]. Proc. of IEEE International Symposium on Circuits and Systems [C]. Sydney, Australia, 2001.
Experimental results [J /OL]. http: //www .csd.uoc.gr/ ∼ tziritas
Author information
Authors and Affiliations
Additional information
Project supported by the National Nature Science Foundation of China (Grant No. 60172020)
About this article
Cite this article
Yang, Gb., Zhang, Zy. Objective performance evaluation of video segmentation algorithms with ground-truth. J. of Shanghai Univ. 8, 70–74 (2004). https://doi.org/10.1007/s11741-004-0015-5
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/s11741-004-0015-5