Strahlentherapie und Onkologie

, Volume 188, Issue 2, pp 160–167 | Cite as

Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy

  • I. Fotina
  • C. Lütgendorf-Caucig
  • M. Stock
  • R. Pötter
  • D. Georg
Original article


Background and purpose

Inter-observer studies represent a valid method for the evaluation of target definition uncertainties and contouring guidelines. However, data from the literature do not yet give clear guidelines for reporting contouring variability. Thus, the purpose of this work was to compare and discuss various methods to determine variability on the basis of clinical cases and a literature review.

Patients and methods

In this study, 7 prostate and 8 lung cases were contoured on CT images by 8 experienced observers. Analysis of variability included descriptive statistics, calculation of overlap measures, and statistical measures of agreement. Cross tables with ratios and correlations were established for overlap parameters.


It was shown that the minimal set of parameters to be reported should include at least one of three volume overlap measures (i.e., generalized conformity index, Jaccard coefficient, or conformation number). High correlation between these parameters and scatter of the results was observed.


A combination of descriptive statistics, overlap measure, and statistical measure of agreement or reliability analysis is required to fully report the interrater variability in delineation.


Inter-observer variability Target volume delineation Conformity index Similarity metrics Radiotherapy 

Kritische Diskussion von Evaluierungsparametern der Inter-Beobachter-Variabilität bei der Konturierung von Zielvolumina in der Strahlentherapie



Inter-Beobachter-Studien sind eine sehr beliebte und auch adäquate Methode, um Unsicherheiten in der Zielvolumendefinition zu erfassen, Zielvolumenkonzepte zu beschreiben und folglich Qualitätsmaßstäbe in der Strahlentherapie zu setzen. In der Literatur finden sich bis dato keine klaren Richtlinien zur Beschreibung von Konturierungsvariabilität in Inter-Beobachter-Studien. Am Beispiel von klinischen Fallbeispielen sowie einer Literaturübersicht wurden verschiedene Ansätze zur Variabilitätsbestimmung verglichen.

Material und Methode

7 Prostata- und 8 Lungenfälle wurden je von 8 erfahrenen Strahlentherapeuten konturiert. Die Variabilitätsanalyse beinhaltete eine deskriptive Statistik, eine Überlappungsberechnung sowie eine statistische Methode zur Erfassung von Übereinstimmungen. Außerdem wurden Kreuztabellen mit Verhältnis und Korrelation zwischen den Überlappungsparametern erstellt.


Es konnte gezeigt werden, dass zur adäquaten Beschreibung der Inter-Beobachter-Variabilität einer der 3 Parameter (generalisierter Konformitätsindex, Jaccard-Koeffizient oder Konformationsnummer) ausreicht. Es wurde eine stark positive Korrelation zwischen diesen Parametern und der Streuung der Ergebnisse beobachtet.


Die Kombination aus deskriptiver Statistik, Überlappungsparametern und statistischem Ähnlichkeitsmaß oder Reliabilitätsanalyse ist für die vollständige Beschreibung der Inter-Beobachter-Variabilität erforderlich.


Inter-Beobachter-Variationen Konturierung des klinischen Zielvolumens Konformitätsindex Ähnlichkeitsmaße Strahlentherapie 



Authors would like to gratefully acknowledge the observers for donating their time for contouring and making this study possible. Irina Fotina acknowledges the financial support from Austrian National Bank (OeNB, project number 12972).

Conflict of interest

The corresponding author states that there are no conflicts of interest.


  1. 1.
    Allozi R, Li XA, White J et al (2010) Tools for consensus analysis of experts’ contours for radiotherapy structure definitions. Radiother Oncol 97:572–578PubMedCrossRefGoogle Scholar
  2. 2.
    Altorjai G, Fotina I, Lütgendorf-Caucig C et al (2011) Cone-beam ct-based delineation of stereotactic lung targets: the influence of image modality and target size on interobserver variability. Int J Radiat Oncol Biol Phys [Epub ahead of print]Google Scholar
  3. 3.
    Berthelet E, Liu MC, Agranovich A et al (2002) Computed tomography determination of prostate volume and maximum dimensions: a study of interobserver variability. Radiother Oncol 63:37–40PubMedCrossRefGoogle Scholar
  4. 4.
    Breen SL, Publicover J, De Silva S et al (2007) Intraobserver and inter-observer variability in GTV delineation on FDG-PET-CT images of head and neck cancers. Int J Radiat Oncol Biol Phys 68:763–770PubMedCrossRefGoogle Scholar
  5. 5.
    Castro Pena P, Kirova YM, Campana F et al (2009) Anatomical, clinical and radiological delineation of target volumes in breast cancer radiotherapy planning: individual variability, questions and answers. Br J Radiol 82:595–599CrossRefGoogle Scholar
  6. 6.
    Dimopoulos JC, De Vos V, Berger D et al (2009) Inter-observer comparison of target delineation for MRI-assisted cervical cancer brachytherapy: application of the GYN GEC-ESTRO recommendations. Radiother Oncol 91:166–172PubMedCrossRefGoogle Scholar
  7. 7.
    Eliasziw M, Young SL, Woodbury MG et al (1994) Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther 74:777–788PubMedGoogle Scholar
  8. 8.
    Fuller CD, Nijkamp J, Duppen JC et al (2011) Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys 79:481–489PubMedCrossRefGoogle Scholar
  9. 9.
    Genovesi D, Cèfaro GA, Vinciguerra A et al (2011) Interobserver variability of clinical target volume delineation in supra-diaphragmatic Hodgkin’s disease: a multi-institutional experience. Strahlenther Onkol 187:357–366PubMedCrossRefGoogle Scholar
  10. 10.
    Giezen M, Kouwenhoven E, Scholten AN et al (2011) Magnetic resonance imaging- versus computed tomography-based target volume delineation of the glandular breast tissue in breast-conserving therapy: an exploratory study. Int J Radiat Oncol Biol Phys 81:804–811PubMedCrossRefGoogle Scholar
  11. 11.
    Giraud P, Elles S, Helfre S et al (2002) Conformal radiotherapy for lung cancer: different delineation of the gross tumour volume (GTV) by radiologists and radiation oncologists. Radiother Oncol 62:27–36PubMedCrossRefGoogle Scholar
  12. 12.
    Goldner G, Dimopoulos J, Kirisits C, Pötter R (2009) Moderate dose escalation in three-dimensional conformal localized prostate cancer radiotherapy: single-institutional experience in 398 patients comparing 66 Gy versus 70 Gy versus 74 Gy. Strahlenther Onkol 185:438–445PubMedCrossRefGoogle Scholar
  13. 13.
    Grabarz D, Panzarella T, Bezjak A et al (2011) Quantifying interobserver variation in target definition in Palliative Radiotherapy. Int J Radiat Oncol Biol Phys 80:1498–1504PubMedCrossRefGoogle Scholar
  14. 14.
    Guckenberger M, Ok S, Polat B et al (2010) Toxicity after intensity-modulated, image-guided radiotherapy for prostate cancer. Strahlenther Onkol 186:535–43 (Erratum: Strahlenther Onkol 2010;186:705)Google Scholar
  15. 15.
    International Commission on Radiation Units and Measurements. ICRU Report 62 (1999) Prescribing, recording, and reporting photon beam therapy (Supplement to ICRU Report 50). ICRU, Bethesda, MD, USAGoogle Scholar
  16. 16.
    Hentschel B, Oehler W, Strauss D et al (2011) Definition of the CTV prostate in CT and MRI by using CT-MRI image fusion in IMRT planning for prostate cancer. Strahlenther Onkol 187:183–190PubMedCrossRefGoogle Scholar
  17. 17.
    Jansen EP, Nijkamp J, Gubanski M et al (2010) Interobserver variation of clinical target volume delineation in gastric cancer. Int J Radiat Oncol Biol Phys 77:1166–1170PubMedCrossRefGoogle Scholar
  18. 18.
    Kouwenhoven E, Giezen M, Struikmans H (2009) Measuring the similarity of target volume delineations independent of the number of observers. Phys Med Biol 54:2863–2873PubMedCrossRefGoogle Scholar
  19. 19.
    Krengli M, Cannillo B, Turri L et al (2010) Target volume delineation for preoperative radiotherapy of rectal cancer: inter-observer variability and potential impact of FDG-PET/CT imaging. Technol Cancer Res Treat 9:393–398PubMedGoogle Scholar
  20. 20.
    Lawton CA, Michalski J, El-Naqa I et al (2009) Variation in the definition of clinical target volumes for pelvic nodal conformal radiation therapy for prostate cancer. Int J Radiat Oncol Biol Phys 74:377–382PubMedCrossRefGoogle Scholar
  21. 21.
    Leunens G, Menten J, Weltens C et al (1993) Quality assessment of medical decision making in radiation oncology: variability in target volume delineation for brain tumours. Radiother Oncol 29:169–175PubMedCrossRefGoogle Scholar
  22. 22.
    Li XA, Tai A, Arthur DW, Buchholz TA et al (2009) Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG Multi-Institutional and Multiobserver Study. Int J Radiat Oncol Biol Phys 73:944–951PubMedCrossRefGoogle Scholar
  23. 23.
    Logue JP, Sharrock CL, Cowan RA et al (1998) Clinical variability of target volume description in conformal radiotherapy planning. Int J Radiat Oncol Biol Phys 41:929–9231PubMedCrossRefGoogle Scholar
  24. 24.
    Louie AV, Rodrigues G, Olsthoorn J et al (2010) Inter-observer and intra-observer reliability for lung cancer target volume delineation in the 4D-CT era. Radiother Oncol 95:166–171PubMedCrossRefGoogle Scholar
  25. 25.
    Lütgendorf-Caucig C, Fotina I, Stock M et al (2011) Feasibility of CBCT-based target and normal structure delineation in prostate cancer radiotherapy: multi-observer and image multi-modality study. Radiother Oncol 98:154–161PubMedCrossRefGoogle Scholar
  26. 26.
    Metwally H, Courbon F, David I et al (2011) Coregistration of prechemotherapy PET-CT for planning pediatric Hodgkin’s disease radiotherapy significantly diminishes interobserver variability of clinical target volume definition. Int J Radiat Oncol Biol Phys 80:793–799PubMedCrossRefGoogle Scholar
  27. 27.
    Mitchell JR, Karlik SJ, Lee DH et al (1996) The variability of manual and computer assisted quantification of multiple sclerosis lesion volumes. Med Phys 23:85–97PubMedCrossRefGoogle Scholar
  28. 28.
    Njeh CF (2008) Tumor delineation: The weakest link in the search for accuracy in radiotherapy. J Med Phys 33:136–140PubMedCrossRefGoogle Scholar
  29. 29.
    Petersen RP, Truong PT, Kader HA et al (2007) Target volume delineation for partial breast radiotherapy planning: Clinical characteristics associated with low interobserver concordance. Int J Radiat Oncol Biol Phys 69:41–48PubMedCrossRefGoogle Scholar
  30. 30.
    Pinkawa M, Holy R, Piroth MD et al (2010) Intensity-modulated radiotherapy for prostate cancer implementing molecular imaging with 18 F-choline PET-CT to define a simultaneous integrated boost. Strahlenther Onkol 186:600–606PubMedCrossRefGoogle Scholar
  31. 31.
    Pinkawa M, Piroth MD, Holy R et al (2011) Combination of dose escalation with technological advances (intensity-modulated and image-guided radiotherapy) is not associated with increased morbidity for patients with prostate cancer. Strahlenther Onkol 187:479–484PubMedCrossRefGoogle Scholar
  32. 32.
    Rasch C, Barillot I, Remeijer P et al (1999) Definition of the prostate in CT and MRI: a multi-observer study. Int J Radiat Oncol Biol Phys 43:57–66PubMedCrossRefGoogle Scholar
  33. 33.
    Rasch C, Steenbakkers R, Herk M van (2005) Target definition in prostate, head, and neck. Semin Radiat Oncol 15:136–145PubMedCrossRefGoogle Scholar
  34. 34.
    Rasch CR, Steenbakkers RJ, Fitton I et al (2010) Decreased 3D observer variation with matched CT-MRI, for target delineation in nasopharynx cancer. Radiat Oncol 5:21PubMedCrossRefGoogle Scholar
  35. 35.
    Senan S, Koste J de, Samson M et al (1999) Evaluation of a target contouring protocol for 3D conformal radiotherapy in non-small cell lung cancer. Radiother Oncol 53:247–255PubMedCrossRefGoogle Scholar
  36. 36.
    Smith WL, Lewis C, Bauman G et al (2007) Prostate volume contouring: a 3D analysis of segmentation using 3DTRUS, CT, and MR. Int J Radiat Oncol Biol Phys 67:1238–1247PubMedCrossRefGoogle Scholar
  37. 37.
    Song W, Chiu B, Bauman G et al (2006) Prostate contouring uncertainty in megavoltage computed tomography images acquired with a helical tomotherapy unit during image-guided radiation therapy. Int J Radiat Oncol Biol Phys 65:595–607PubMedCrossRefGoogle Scholar
  38. 38.
    Steenbakkers RJ, Duppen JC, Fitton I et al (2006) Reduction of observer variation using matched CT-PET for lung cancer delineation: a three-dimensional analysis. Int J Radiat Oncol Biol Phys 64:435–434PubMedCrossRefGoogle Scholar
  39. 39.
    Stock M, Dörr W, Stromberger C et al (2010) Investigations on parotid gland recovery after IMRT in head and neck tumor patients. Strahlenther Onkol 186:665–671PubMedCrossRefGoogle Scholar
  40. 40.
    Stroom JC, Heijmen BJ (2002) Geometrical uncertainties, radiotherapy planning margins, and the ICRU-62 report. Radiother Oncol 64:75–83PubMedCrossRefGoogle Scholar
  41. 41.
    Struikmans H, Wárlám-Rodenhuis C, Stam T et al (2005) Interobserver variability of clinical target volume delineation of glandular breast tissue and of boost volume in tangential breast irradiation. Radiother Oncol 76:293–299PubMedCrossRefGoogle Scholar
  42. 42.
    Tong S, Cardinal HN, McLoughlin RF et al (1998) Intra- and interobserver variability and reliability of prostate volume measurement via two-dimensional and three-dimensional ultrasound imaging. Ultrasound Med Biol 24:673–681PubMedCrossRefGoogle Scholar
  43. 43.
    Tyng CJ, Chojniak R, Pinto PN et al (2009) Conformal radiotherapy for lung cancer: interobservers’ variability in the definition of gross tumor volume between radiologists and radiotherapists. Radiat Oncol 4:28PubMedCrossRefGoogle Scholar
  44. 44.
    Usmani N, Sloboda R, Kamal W et al (2011) Can images obtained with high field strength magnetic resonance imaging reduce contouring variability of the prostate? Int J Radiat Oncol Biol Phys 80:728–734PubMedCrossRefGoogle Scholar
  45. 45.
    Baardwijk A van, Bosmans G, Boersma et al (2007) PET-CT-based auto-contouring in non-small-cell lung cancer correlates with pathology and reduces interobserver variability in the delineation of the primary tumor and involved nodal volumes. Int J Radiat Oncol Biol Phys 68:771–778PubMedCrossRefGoogle Scholar
  46. 46.
    Mourik AM van, Elkhuizen PH, Minkema D et al (2010) Multiinstitutional study on target volume delineation variation in breast radiotherapy in the presence of guidelines. Radiother Oncol 94:286–291PubMedCrossRefGoogle Scholar
  47. 47.
    Villeirs GM, Van Vaerenbergh K, Vakaet L et al (2005) Interobserver delineation variation using CT versus combined CT + MRI in intensity-modulated radiotherapy for prostate cancer. Strahlenther Onkol 181:424–430PubMedCrossRefGoogle Scholar
  48. 48.
    Vorwerk H, Beckmann G, Bremer M et al (2009) The delineation of target volumes for radiotherapy of lung cancer patients. Radiother Oncol 91:455–460PubMedCrossRefGoogle Scholar
  49. 49.
    Weiss E, Hess CF (2003) The impact of gross tumor volume (GTV) and clinical target volume (CTV) definition on the total accuracy in radiotherapy theoretical aspects and practical experiences. Strahlenther Onkol 179:21–30PubMedCrossRefGoogle Scholar
  50. 50.
    Weiss E, Richter S, Krauss T et al (2003) Conformal radiotherapy planning of cervix carcinoma: differences in the delineation of the clinical target volume. A comparison between gynaecologic and radiation oncologists. Radiother Oncol 67:87–95PubMedCrossRefGoogle Scholar
  51. 51.
    Weiss E, Wu J, Sleeman W et al (2010) Clinical evaluation of soft tissue organ boundary visualization on cone-beam computed tomographic imaging. Int J Radiat Oncol Biol Phys 78:929–936PubMedCrossRefGoogle Scholar
  52. 52.
    Weltens C, Menten J, Feron M et al (2001) Interobserver variations in gross tumor volume delineation of brain tumors on computed tomography and impact of magnetic resonance imaging. Radiother Oncol 60:49–59PubMedCrossRefGoogle Scholar
  53. 53.
    White EA, Brock KK, Jaffray DA et al (2009) Inter-observer variability of prostate delineation on cone beam computerised tomography images. Clin Oncol 21:32–38CrossRefGoogle Scholar

Copyright information

© Springer Heidelberg Berlin 2012

Authors and Affiliations

  • I. Fotina
    • 1
  • C. Lütgendorf-Caucig
    • 1
  • M. Stock
    • 1
  • R. Pötter
    • 1
  • D. Georg
    • 1
  1. 1.Div. Medical Radiation Physics, Department of RadiotherapyMedical University Vienna/AKH ViennaViennaAustria

Personalised recommendations