Abstract
The evaluation of the quality of segmentations of an image, and the assessment of intra- and inter-expert variability in segmentation performance, has long been recognized as a difficult task. Recently an Expectation Maximization (EM) algorithm for Simultaneous Truth and Performance Level Estimation (Staple), was developed to compute both an estimate of the reference standard segmentation and performance parameters from a set of segmentations of an image. The performance is characterized by the rate of detection of each segmentation label by each expert in comparison to the estimated reference standard.
This previous work provides estimates of performance parameters, but does not provide any information regarding their uncertainty. An estimate of this inferential uncertainty, if available, would allow estimation of confidence intervals for the values of the parameters, aid in the interpretation of the performance of segmentation generators, and help determine if sufficient data size and number of segmentations have been obtained to accurately characterize the performance parameters.
We present a new algorithm to estimate the inferential uncertainty of the performance parameters for binary segmentations. It is derived for the special case of the Staple algorithm based on established theory for general purpose covariance matrix estimation for EM algorithms. The bounds on performance estimates are estimated by the computation of the observed Information Matrix. We use this algorithm to study the bounds on performance estimates from simulated images with specified performance parameters, and from interactive segmentations of neonatal brain MRIs. We demonstrate that confidence intervals for expert segmentation performance parameters can be estimated with our algorithm. We investigate the influence of the number of experts and of the image size on these bounds, showing that it is possible to determine the number of image segmentations and the size of images necessary to achieve a chosen level of accuracy in segmentation performance assessment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Huttenlocher, D., Klanderman, D., Rucklige, A.: Comparing images using the Hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(9), 850–863 (1993)
Chalana, V., Kim, Y.: A methodology for evaluation of boundary detection algorithms on medical images. IEEE Transactions on Medical Imaging 16(5), 642–652 (1997)
Dice, L.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)
Jaccard, P.: The distribution of flora in the alpine zone. New Phytologist 11, 37–50 (1912)
Zou, K.H., Warfield, S.K., Bharatha, A., Tempany, C.M.C., Tempany, C., Kaus, M.R., Haker, S.J., Wells, W.M., Jolesz, F.A., Kikinis, R.: Statistical validation of image segmentation quality based on a spatial overlap index. Acad. Radiol. 11(2), 178–189 (2004)
Gerig, G., Jomier, M., Chakos, M.: Valmet: A new validation tool for assessing and improving 3D object segmentation. In: Niessen, W.J., Viergever, M.A. (eds.) MICCAI 2001. LNCS, vol. 2208, pp. 516–523. Springer, Heidelberg (2001)
Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging 23(7), 903–921 (2004)
McLachlan, G., Krishnan, T.: The EM Algorithm and Extensions. John Wiley and Sons, Chichester (1997)
Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society (Series B) 39 (1977)
Meng, X., Rubin, D.: Using EM to obtain asymptotic variance-covariance matrices: the SEM algorithm. Journal of the American Statistical Association 86, 899–909 (1991)
Oakes, D.: Direct calculation of the information matrix via the EM algorithm. J. R. Statistical Society 61(2), 479–482 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Commowick, O., Warfield, S.K. (2009). Estimation of Inferential Uncertainty in Assessing Expert Segmentation Performance from Staple . In: Prince, J.L., Pham, D.L., Myers, K.J. (eds) Information Processing in Medical Imaging. IPMI 2009. Lecture Notes in Computer Science, vol 5636. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02498-6_58
Download citation
DOI: https://doi.org/10.1007/978-3-642-02498-6_58
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02497-9
Online ISBN: 978-3-642-02498-6
eBook Packages: Computer ScienceComputer Science (R0)