Model for defining and reporting reference-based validation protocols in medical image processing
- 292 Downloads
Objectives Image processing tools are often embedded in larger systems. Validation of image processing methods is important because the performance of such methods can have an impact on the performance of the larger systems and consequently on decisions and actions based on the use of these systems. Most validation studies compare the direct or indirect results of a method with a reference that is assumed to be very close or equal to the correct solution. In this paper, we propose a model for defining and reporting reference-based validation protocols in medical image processing.
Materials and methods The model was built using an ontological approach. Its components were identified from the analysis of initial publications (mainly reviews) on medical image processing, especially registration and segmentation, and from discussions with experts from the medical imaging community during international conferences and workshops. The model was validated by its instantiation for 38 selected papers that include a validation study, mainly for medical image registration and segmentation.
Results The model includes the main components of a validation procedure and their inter-relationships. A checklist for reporting reference-based validation studies for medical image processing was also developed.
Conclusion The proposed model and associated checklist may be used in formal reference-based validation studies of registration and segmentation and for the complete and accurate reporting of such studies. The model facilitates the standardization of validation terminology and methodology, improves the comparison of validation studies and results, provides insight into the validation process, and, finally, may lead to better quality image management and decision making.
KeywordsReference-based validation Medical image processing Image registration Segmentation Gold standard Ground truth Guidelines
Unable to display preview. Download preview PDF.
- 2.Balci O (2003) Verification, validation and certification of modeling and simulation applications. In: Proccedings of the 2003 winter simulation conference, pp 150–158Google Scholar
- 4.Bowyer KW, Loew MH, Stiehl HS, Viergever MA (2001) Methodology of evaluation in medical image computing. Report of Dagstuhl workshop, March 2001. http://www.dagstuhl.de/DATA/Reports/01111/ (Accessed in January 2006)Google Scholar
- 5.Bromiley PA, Pokric M, Thacker NA (2004) Empirical evaluation of covariance estimates for mutual information coregistration. In: Barillot C, Haynor DR, Hellier P. (eds). MICCAI 2004—Part I. Lecture notes in computer sciences vol LNCS 3216. Springer, Berlin Heidelberg New York, pp 607–614Google Scholar
- 7.Buvat I, Chameroy V, Aubry F et al (1999) The need to develop guidelines for evaluations of medical image processing procedures. In: Proccedings of SPIE medical imaging, Vol 3661, pp 1466–1477Google Scholar
- 9.Fitzpatrick JM, Hill DLG, Maurer CR Jr (2000) Image registration. In: Sonka M, Fitzpatrick JM (eds) Handbook of medical imaging.Medical image processing and analysis. vol 2 SPIE Press, Bellingham, pp 447–513Google Scholar
- 10.Fitzpatrick JM (2001) Detecting failure, assessing success. In: Hajnal JV, Hill DLG, and Hawkes DJ (eds) Medical image registration. CRC Press, Boca Raten, pp 117–139Google Scholar
- 13.Gee J (2000) Performance evaluation of medical image processing algorithms. In: Hanson K (eds) Proccedings. of SPIE medical imaging, image processing, vol 3979, pp 19–27Google Scholar
- 14.General principles of software validation; Final for industry and FDA staff v2.0 (2002) http://www.fda.gov/ cdrh/comp/guidance/938.html (Accessed in January 2006)Google Scholar
- 18.http://idm.univ-rennes1.fr/VMIP/model (Accessed in May 2006)Google Scholar
- 19.Jannin P, Grova C, Gibaud B (2001) Medical applications of NDT data fusion. In: Gros XE (ed), Applications of NDT data fusion. Kluwer, Dordrecht, pp 227–267Google Scholar
- 23.Maurer CR Jr, Rohlfing T, Dean D, West JB, Rueckert D, Mori K, Shahidi R, Martin DP, Heilbrun MP, Maciunas RJ (2002) Sources of error in image registration for cranial image-guided surgery. In: Germano IM. (eds). Advanced techniques in image-guided brain and spine surgery. Thieme, New York, pp 10–36Google Scholar
- 27.Udupa J, Leblanc V, Schmidt H, Imielinska C, Saha P, Grevera G, Zhuge Y, Currie L, Molholt P, Jin Y (2002) Methodology for evaluating image-segmentation algorithms. In: Proccedings of SPIE medical imaging, vol 4684, pp 266–277Google Scholar
- 30.Woods RP, Grafton ST, Holmes CJ et al (1998) Automated image registration: I. General methods and intrasubject, intramodality validation. J Comput Assist Tomogr 22(1):139–152Google Scholar
- 31.Woods RP (2000) Validation of registration accuracy.In: Bankman IN (ed), Handbook of medical imaging, processing and analysis,vol 30. Academic, pp 491–497Google Scholar
- 32.Yoo TS, Ackerman MJ, Vannier M (2000) Toward a common validation methodology for segmentation and registration algorithms. In: Delp LD, DiGioia AM, Jaramaz B (eds) MICCAI 2000. Lecture notes in computer sciences, vol. LNCS-1935. Springer, Berlin Heidelberg New York, pp 422–431Google Scholar