Skip to main content
SpringerLink
Log in

Model for defining and reporting reference-based validation protocols in medical image processing

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Objectives Image processing tools are often embedded in larger systems. Validation of image processing methods is important because the performance of such methods can have an impact on the performance of the larger systems and consequently on decisions and actions based on the use of these systems. Most validation studies compare the direct or indirect results of a method with a reference that is assumed to be very close or equal to the correct solution. In this paper, we propose a model for defining and reporting reference-based validation protocols in medical image processing.

Materials and methods The model was built using an ontological approach. Its components were identified from the analysis of initial publications (mainly reviews) on medical image processing, especially registration and segmentation, and from discussions with experts from the medical imaging community during international conferences and workshops. The model was validated by its instantiation for 38 selected papers that include a validation study, mainly for medical image registration and segmentation.

Results The model includes the main components of a validation procedure and their inter-relationships. A checklist for reporting reference-based validation studies for medical image processing was also developed.

Conclusion The proposed model and associated checklist may be used in formal reference-based validation studies of registration and segmentation and for the complete and accurate reporting of such studies. The model facilitates the standardization of validation terminology and methodology, improves the comparison of validation studies and results, provides insight into the validation process, and, finally, may lead to better quality image management and decision making.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aubert-Broche B, Grova C, Jannin P, Buvat I, Benali H, Gibaud B (2003) Detection of inter- hemispheric asymmetries of brain perfusion in SPECT. Phys Med Biol 48(11):1505–1517

    Article  PubMed  CAS  Google Scholar 

  2. Balci O (2003) Verification, validation and certification of modeling and simulation applications. In: Proccedings of the 2003 winter simulation conference, pp 150–158

  3. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW (2003) Toward complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Acad Radiol 10(6):664–669

    Article  PubMed  Google Scholar 

  4. Bowyer KW, Loew MH, Stiehl HS, Viergever MA (2001) Methodology of evaluation in medical image computing. Report of Dagstuhl workshop, March 2001. http://www.dagstuhl.de/DATA/Reports/01111/ (Accessed in January 2006)

  5. Bromiley PA, Pokric M, Thacker NA (2004) Empirical evaluation of covariance estimates for mutual information coregistration. In: Barillot C, Haynor DR, Hellier P. (eds). MICCAI 2004—Part I. Lecture notes in computer sciences vol LNCS 3216. Springer, Berlin Heidelberg New York, pp 607–614

    Google Scholar 

  6. Brown LG (1992) A survey of image registration techniques. ACM Comput Surv 24(4):325–376

    Article  Google Scholar 

  7. Buvat I, Chameroy V, Aubry F et al (1999) The need to develop guidelines for evaluations of medical image processing procedures. In: Proccedings of SPIE medical imaging, Vol 3661, pp 1466–1477

  8. Fitzpatrick JM, West JB, Maurer CR Jr. (1998) Predicting error in rigid-body, point-based registration. IEEE Trans Med Imaging 17(5):694–702

    Article  PubMed  CAS  Google Scholar 

  9. Fitzpatrick JM, Hill DLG, Maurer CR Jr (2000) Image registration. In: Sonka M, Fitzpatrick JM (eds) Handbook of medical imaging.Medical image processing and analysis. vol 2 SPIE Press, Bellingham, pp 447–513

    Google Scholar 

  10. Fitzpatrick JM (2001) Detecting failure, assessing success. In: Hajnal JV, Hill DLG, and Hawkes DJ (eds) Medical image registration. CRC Press, Boca Raten, pp 117–139

    Google Scholar 

  11. Fryback DG, Thornbury JR (1991) The efficacy of diagnostic imaging. Med Decis Mak 11(2):88–94

    Article  CAS  Google Scholar 

  12. Gazelle GS, Seltzer SE, Judy PF (2003) Assessment and validation of imaging methods and technologies. Acad Radiol 10(8):894–896

    Article  PubMed  Google Scholar 

  13. Gee J (2000) Performance evaluation of medical image processing algorithms. In: Hanson K (eds) Proccedings. of SPIE medical imaging, image processing, vol 3979, pp 19–27

  14. General principles of software validation; Final for industry and FDA staff v2.0 (2002) http://www.fda.gov/ cdrh/comp/guidance/938.html (Accessed in January 2006)

  15. Grova C, Daunizeau J, Lina JM, Benar CG, Benali H, Gotman J (2005) Evaluation of EEG localization methods using realistic simulations of interictal spikes. Neuroimage 29(3):734–753

    Article  PubMed  Google Scholar 

  16. Hawkes DJ (1998) Algorithms for radiological image registration and their clinical application. J Anat 193:347–361

    Article  PubMed  Google Scholar 

  17. Hill DLG, Batchelor PG, Holden M, Hawkes DJ (2001) Medical image registration. Phys Med Biol 46(3):1–45

    Article  Google Scholar 

  18. http://idm.univ-rennes1.fr/VMIP/model (Accessed in May 2006)

  19. Jannin P, Grova C, Gibaud B (2001) Medical applications of NDT data fusion. In: Gros XE (ed), Applications of NDT data fusion. Kluwer, Dordrecht, pp 227–267

    Google Scholar 

  20. Jannin P, Fitzpatrick JM, Hawkes DJ, Pennec X, Shahidi R, Vannier MW (2002) Validation of medical image processing in image-guided therapy. IEEE Trans Med Imaging 21(12):1445–1449

    Article  PubMed  Google Scholar 

  21. Maintz JBA, Viergever MA (1998) A survey of medical image registration. Med Image Anal 2:1–36

    PubMed  CAS  Google Scholar 

  22. Maurer CR Jr, Maciunas RJ, Fitzpatrick JM (1998) Registration of head CT images to physical space using a weighted combination of points and surfaces. IEEE Trans Med Imaging 17:753–761

    Article  PubMed  Google Scholar 

  23. Maurer CR Jr, Rohlfing T, Dean D, West JB, Rueckert D, Mori K, Shahidi R, Martin DP, Heilbrun MP, Maciunas RJ (2002) Sources of error in image registration for cranial image-guided surgery. In: Germano IM. (eds). Advanced techniques in image-guided brain and spine surgery. Thieme, New York, pp 10–36

    Google Scholar 

  24. McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM et al (2005) Reporting recommendations for tumour MARKer prognostic studies (REMARK). Br J Cancer 93(4):387–91

    Article  PubMed  CAS  Google Scholar 

  25. Paul P, Fleig O, Jannin P (2005) Augmented virtuality based on stereoscopic reconstruction in multimodal image-guided neurosurgery: methods and performance evaluation. IEEE Trans Med Imaging 24(11):1500–1511

    Article  PubMed  Google Scholar 

  26. Pennec X, Thirion JP (1997) A framework for uncertainty and validation of 3D registration methods based on points and frames. Int J Comput Vis 25(3):203–229

    Article  Google Scholar 

  27. Udupa J, Leblanc V, Schmidt H, Imielinska C, Saha P, Grevera G, Zhuge Y, Currie L, Molholt P, Jin Y (2002) Methodology for evaluating image-segmentation algorithms. In: Proccedings of SPIE medical imaging, vol 4684, pp 266–277

  28. Van DenElsen PA, Pol EJD, Viergever MA (1993) Medical image matching—a review with classification. IEEE Eng Med Biol Mag 12(1):26–39

    Article  Google Scholar 

  29. West J, Fitzpatrick JM, Wang MY et al (1997) Comparison and evaluation of retrospective intermodality brain image registration techniques. J Comput Assist Tomogr 21(4):554–566

    Article  PubMed  CAS  Google Scholar 

  30. Woods RP, Grafton ST, Holmes CJ et al (1998) Automated image registration: I. General methods and intrasubject, intramodality validation. J Comput Assist Tomogr 22(1):139–152

    CAS  Google Scholar 

  31. Woods RP (2000) Validation of registration accuracy.In: Bankman IN (ed), Handbook of medical imaging, processing and analysis,vol 30. Academic, pp 491–497

  32. Yoo TS, Ackerman MJ, Vannier M (2000) Toward a common validation methodology for segmentation and registration algorithms. In: Delp LD, DiGioia AM, Jaramaz B (eds) MICCAI 2000. Lecture notes in computer sciences, vol. LNCS-1935. Springer, Berlin Heidelberg New York, pp 422–431

    Google Scholar 

  33. Zhang YJ (1996) A survey on evaluation methods for image segmentation. Pattern Recognit 29(8):1335–1346

    Article  Google Scholar 

  34. Zijdenbos A, Dawant B, Marjolin R (1994) Morphometric analysis of white matter lesions in mr images: methods and validation. IEEE Trans Med Imaging 13(4):716–724

    Article  PubMed  CAS  Google Scholar 

  35. Zou KH, Warfield SK, Bharatha A, Tempany CMC, Kaus MR, Haker SJ, Wells III WM, Jolesz FA, Kikinis R (2004) Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol 11(2):178–189

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Visages, U 746, INSERM-INRIA-CNRS, Medical School, University of Rennes, CS 34317, 35043, Rennes Cedex, France

    Pierre Jannin & Christophe Grova

  2. Montreal Neurological Institute, McGill University, Montreal, Canada

    Christophe Grova

  3. Department of Neurosurgery, Stanford University, Stanford, CA, 94305-5327, USA

    Calvin R. Maurer Jr.

Authors
  1. Pierre Jannin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christophe Grova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Calvin R. Maurer Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pierre Jannin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jannin, P., Grova, C. & Maurer, C.R. Model for defining and reporting reference-based validation protocols in medical image processing. Int J CARS 1, 63–73 (2006). https://doi.org/10.1007/s11548-006-0044-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-006-0044-6

Keywords

Search

Navigation