Model for defining and reporting reference-based validation protocols in medical image processing

  • Pierre Jannin
  • Christophe Grova
  • Calvin R. MaurerJr.
Original Article


Objectives Image processing tools are often embedded in larger systems. Validation of image processing methods is important because the performance of such methods can have an impact on the performance of the larger systems and consequently on decisions and actions based on the use of these systems. Most validation studies compare the direct or indirect results of a method with a reference that is assumed to be very close or equal to the correct solution. In this paper, we propose a model for defining and reporting reference-based validation protocols in medical image processing.

Materials and methods The model was built using an ontological approach. Its components were identified from the analysis of initial publications (mainly reviews) on medical image processing, especially registration and segmentation, and from discussions with experts from the medical imaging community during international conferences and workshops. The model was validated by its instantiation for 38 selected papers that include a validation study, mainly for medical image registration and segmentation.

Results The model includes the main components of a validation procedure and their inter-relationships. A checklist for reporting reference-based validation studies for medical image processing was also developed.

Conclusion The proposed model and associated checklist may be used in formal reference-based validation studies of registration and segmentation and for the complete and accurate reporting of such studies. The model facilitates the standardization of validation terminology and methodology, improves the comparison of validation studies and results, provides insight into the validation process, and, finally, may lead to better quality image management and decision making.


Reference-based validation Medical image processing Image registration Segmentation Gold standard Ground truth Guidelines 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aubert-Broche B, Grova C, Jannin P, Buvat I, Benali H, Gibaud B (2003) Detection of inter- hemispheric asymmetries of brain perfusion in SPECT. Phys Med Biol 48(11):1505–1517PubMedCrossRefGoogle Scholar
  2. 2.
    Balci O (2003) Verification, validation and certification of modeling and simulation applications. In: Proccedings of the 2003 winter simulation conference, pp 150–158Google Scholar
  3. 3.
    Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW (2003) Toward complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Acad Radiol 10(6):664–669PubMedCrossRefGoogle Scholar
  4. 4.
    Bowyer KW, Loew MH, Stiehl HS, Viergever MA (2001) Methodology of evaluation in medical image computing. Report of Dagstuhl workshop, March 2001. (Accessed in January 2006)Google Scholar
  5. 5.
    Bromiley PA, Pokric M, Thacker NA (2004) Empirical evaluation of covariance estimates for mutual information coregistration. In: Barillot C, Haynor DR, Hellier P. (eds). MICCAI 2004—Part I. Lecture notes in computer sciences vol LNCS 3216. Springer, Berlin Heidelberg New York, pp 607–614Google Scholar
  6. 6.
    Brown LG (1992) A survey of image registration techniques. ACM Comput Surv 24(4):325–376CrossRefGoogle Scholar
  7. 7.
    Buvat I, Chameroy V, Aubry F et al (1999) The need to develop guidelines for evaluations of medical image processing procedures. In: Proccedings of SPIE medical imaging, Vol 3661, pp 1466–1477Google Scholar
  8. 8.
    Fitzpatrick JM, West JB, Maurer CR Jr. (1998) Predicting error in rigid-body, point-based registration. IEEE Trans Med Imaging 17(5):694–702PubMedCrossRefGoogle Scholar
  9. 9.
    Fitzpatrick JM, Hill DLG, Maurer CR Jr (2000) Image registration. In: Sonka M, Fitzpatrick JM (eds) Handbook of medical imaging.Medical image processing and analysis. vol 2 SPIE Press, Bellingham, pp 447–513Google Scholar
  10. 10.
    Fitzpatrick JM (2001) Detecting failure, assessing success. In: Hajnal JV, Hill DLG, and Hawkes DJ (eds) Medical image registration. CRC Press, Boca Raten, pp 117–139Google Scholar
  11. 11.
    Fryback DG, Thornbury JR (1991) The efficacy of diagnostic imaging. Med Decis Mak 11(2):88–94CrossRefGoogle Scholar
  12. 12.
    Gazelle GS, Seltzer SE, Judy PF (2003) Assessment and validation of imaging methods and technologies. Acad Radiol 10(8):894–896PubMedCrossRefGoogle Scholar
  13. 13.
    Gee J (2000) Performance evaluation of medical image processing algorithms. In: Hanson K (eds) Proccedings. of SPIE medical imaging, image processing, vol 3979, pp 19–27Google Scholar
  14. 14.
    General principles of software validation; Final for industry and FDA staff v2.0 (2002) cdrh/comp/guidance/938.html (Accessed in January 2006)Google Scholar
  15. 15.
    Grova C, Daunizeau J, Lina JM, Benar CG, Benali H, Gotman J (2005) Evaluation of EEG localization methods using realistic simulations of interictal spikes. Neuroimage 29(3):734–753PubMedCrossRefGoogle Scholar
  16. 16.
    Hawkes DJ (1998) Algorithms for radiological image registration and their clinical application. J Anat 193:347–361PubMedCrossRefGoogle Scholar
  17. 17.
    Hill DLG, Batchelor PG, Holden M, Hawkes DJ (2001) Medical image registration. Phys Med Biol 46(3):1–45CrossRefGoogle Scholar
  18. 18. (Accessed in May 2006)Google Scholar
  19. 19.
    Jannin P, Grova C, Gibaud B (2001) Medical applications of NDT data fusion. In: Gros XE (ed), Applications of NDT data fusion. Kluwer, Dordrecht, pp 227–267Google Scholar
  20. 20.
    Jannin P, Fitzpatrick JM, Hawkes DJ, Pennec X, Shahidi R, Vannier MW (2002) Validation of medical image processing in image-guided therapy. IEEE Trans Med Imaging 21(12):1445–1449PubMedCrossRefGoogle Scholar
  21. 21.
    Maintz JBA, Viergever MA (1998) A survey of medical image registration. Med Image Anal 2:1–36PubMedGoogle Scholar
  22. 22.
    Maurer CR Jr, Maciunas RJ, Fitzpatrick JM (1998) Registration of head CT images to physical space using a weighted combination of points and surfaces. IEEE Trans Med Imaging 17:753–761PubMedCrossRefGoogle Scholar
  23. 23.
    Maurer CR Jr, Rohlfing T, Dean D, West JB, Rueckert D, Mori K, Shahidi R, Martin DP, Heilbrun MP, Maciunas RJ (2002) Sources of error in image registration for cranial image-guided surgery. In: Germano IM. (eds). Advanced techniques in image-guided brain and spine surgery. Thieme, New York, pp 10–36Google Scholar
  24. 24.
    McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM et al (2005) Reporting recommendations for tumour MARKer prognostic studies (REMARK). Br J Cancer 93(4):387–91PubMedCrossRefGoogle Scholar
  25. 25.
    Paul P, Fleig O, Jannin P (2005) Augmented virtuality based on stereoscopic reconstruction in multimodal image-guided neurosurgery: methods and performance evaluation. IEEE Trans Med Imaging 24(11):1500–1511PubMedCrossRefGoogle Scholar
  26. 26.
    Pennec X, Thirion JP (1997) A framework for uncertainty and validation of 3D registration methods based on points and frames. Int J Comput Vis 25(3):203–229CrossRefGoogle Scholar
  27. 27.
    Udupa J, Leblanc V, Schmidt H, Imielinska C, Saha P, Grevera G, Zhuge Y, Currie L, Molholt P, Jin Y (2002) Methodology for evaluating image-segmentation algorithms. In: Proccedings of SPIE medical imaging, vol 4684, pp 266–277Google Scholar
  28. 28.
    Van DenElsen PA, Pol EJD, Viergever MA (1993) Medical image matching—a review with classification. IEEE Eng Med Biol Mag 12(1):26–39CrossRefGoogle Scholar
  29. 29.
    West J, Fitzpatrick JM, Wang MY et al (1997) Comparison and evaluation of retrospective intermodality brain image registration techniques. J Comput Assist Tomogr 21(4):554–566PubMedCrossRefGoogle Scholar
  30. 30.
    Woods RP, Grafton ST, Holmes CJ et al (1998) Automated image registration: I. General methods and intrasubject, intramodality validation. J Comput Assist Tomogr 22(1):139–152Google Scholar
  31. 31.
    Woods RP (2000) Validation of registration accuracy.In: Bankman IN (ed), Handbook of medical imaging, processing and analysis,vol 30. Academic, pp 491–497Google Scholar
  32. 32.
    Yoo TS, Ackerman MJ, Vannier M (2000) Toward a common validation methodology for segmentation and registration algorithms. In: Delp LD, DiGioia AM, Jaramaz B (eds) MICCAI 2000. Lecture notes in computer sciences, vol. LNCS-1935. Springer, Berlin Heidelberg New York, pp 422–431Google Scholar
  33. 33.
    Zhang YJ (1996) A survey on evaluation methods for image segmentation. Pattern Recognit 29(8):1335–1346CrossRefGoogle Scholar
  34. 34.
    Zijdenbos A, Dawant B, Marjolin R (1994) Morphometric analysis of white matter lesions in mr images: methods and validation. IEEE Trans Med Imaging 13(4):716–724PubMedCrossRefGoogle Scholar
  35. 35.
    Zou KH, Warfield SK, Bharatha A, Tempany CMC, Kaus MR, Haker SJ, Wells III WM, Jolesz FA, Kikinis R (2004) Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol 11(2):178–189PubMedCrossRefGoogle Scholar

Copyright information

© CARS 2006

Authors and Affiliations

  • Pierre Jannin
    • 1
  • Christophe Grova
    • 1
    • 2
  • Calvin R. MaurerJr.
    • 3
  1. 1.Visages, U 746, INSERM-INRIA-CNRS, Medical SchoolUniversity of RennesRennes CedexFrance
  2. 2.Montreal Neurological InstituteMcGill UniversityMontrealCanada
  3. 3.Department of NeurosurgeryStanford UniversityStanfordUSA

Personalised recommendations