Skip to main content
SpringerLink
Log in

Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy

Kritische Diskussion von Evaluierungsparametern der Inter-Beobachter-Variabilität bei der Konturierung von Zielvolumina in der Strahlentherapie

  • Original article
  • Published:
Strahlentherapie und Onkologie Aims and scope Submit manuscript

Abstract

Background and purpose

Inter-observer studies represent a valid method for the evaluation of target definition uncertainties and contouring guidelines. However, data from the literature do not yet give clear guidelines for reporting contouring variability. Thus, the purpose of this work was to compare and discuss various methods to determine variability on the basis of clinical cases and a literature review.

Patients and methods

In this study, 7 prostate and 8 lung cases were contoured on CT images by 8 experienced observers. Analysis of variability included descriptive statistics, calculation of overlap measures, and statistical measures of agreement. Cross tables with ratios and correlations were established for overlap parameters.

Results

It was shown that the minimal set of parameters to be reported should include at least one of three volume overlap measures (i.e., generalized conformity index, Jaccard coefficient, or conformation number). High correlation between these parameters and scatter of the results was observed.

Conclusion

A combination of descriptive statistics, overlap measure, and statistical measure of agreement or reliability analysis is required to fully report the interrater variability in delineation.

Zusammenfassung

Hintergrund

Inter-Beobachter-Studien sind eine sehr beliebte und auch adäquate Methode, um Unsicherheiten in der Zielvolumendefinition zu erfassen, Zielvolumenkonzepte zu beschreiben und folglich Qualitätsmaßstäbe in der Strahlentherapie zu setzen. In der Literatur finden sich bis dato keine klaren Richtlinien zur Beschreibung von Konturierungsvariabilität in Inter-Beobachter-Studien. Am Beispiel von klinischen Fallbeispielen sowie einer Literaturübersicht wurden verschiedene Ansätze zur Variabilitätsbestimmung verglichen.

Material und Methode

7 Prostata- und 8 Lungenfälle wurden je von 8 erfahrenen Strahlentherapeuten konturiert. Die Variabilitätsanalyse beinhaltete eine deskriptive Statistik, eine Überlappungsberechnung sowie eine statistische Methode zur Erfassung von Übereinstimmungen. Außerdem wurden Kreuztabellen mit Verhältnis und Korrelation zwischen den Überlappungsparametern erstellt.

Ergebnisse

Es konnte gezeigt werden, dass zur adäquaten Beschreibung der Inter-Beobachter-Variabilität einer der 3 Parameter (generalisierter Konformitätsindex, Jaccard-Koeffizient oder Konformationsnummer) ausreicht. Es wurde eine stark positive Korrelation zwischen diesen Parametern und der Streuung der Ergebnisse beobachtet.

Schlussfolgerung

Die Kombination aus deskriptiver Statistik, Überlappungsparametern und statistischem Ähnlichkeitsmaß oder Reliabilitätsanalyse ist für die vollständige Beschreibung der Inter-Beobachter-Variabilität erforderlich.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

References

  1. Allozi R, Li XA, White J et al (2010) Tools for consensus analysis of experts’ contours for radiotherapy structure definitions. Radiother Oncol 97:572–578

    Article  PubMed  Google Scholar 

  2. Altorjai G, Fotina I, Lütgendorf-Caucig C et al (2011) Cone-beam ct-based delineation of stereotactic lung targets: the influence of image modality and target size on interobserver variability. Int J Radiat Oncol Biol Phys [Epub ahead of print]

  3. Berthelet E, Liu MC, Agranovich A et al (2002) Computed tomography determination of prostate volume and maximum dimensions: a study of interobserver variability. Radiother Oncol 63:37–40

    Article  PubMed  Google Scholar 

  4. Breen SL, Publicover J, De Silva S et al (2007) Intraobserver and inter-observer variability in GTV delineation on FDG-PET-CT images of head and neck cancers. Int J Radiat Oncol Biol Phys 68:763–770

    Article  PubMed  Google Scholar 

  5. Castro Pena P, Kirova YM, Campana F et al (2009) Anatomical, clinical and radiological delineation of target volumes in breast cancer radiotherapy planning: individual variability, questions and answers. Br J Radiol 82:595–599

    Article  Google Scholar 

  6. Dimopoulos JC, De Vos V, Berger D et al (2009) Inter-observer comparison of target delineation for MRI-assisted cervical cancer brachytherapy: application of the GYN GEC-ESTRO recommendations. Radiother Oncol 91:166–172

    Article  PubMed  Google Scholar 

  7. Eliasziw M, Young SL, Woodbury MG et al (1994) Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther 74:777–788

    PubMed  CAS  Google Scholar 

  8. Fuller CD, Nijkamp J, Duppen JC et al (2011) Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys 79:481–489

    Article  PubMed  Google Scholar 

  9. Genovesi D, Cèfaro GA, Vinciguerra A et al (2011) Interobserver variability of clinical target volume delineation in supra-diaphragmatic Hodgkin’s disease: a multi-institutional experience. Strahlenther Onkol 187:357–366

    Article  PubMed  Google Scholar 

  10. Giezen M, Kouwenhoven E, Scholten AN et al (2011) Magnetic resonance imaging- versus computed tomography-based target volume delineation of the glandular breast tissue in breast-conserving therapy: an exploratory study. Int J Radiat Oncol Biol Phys 81:804–811

    Article  PubMed  Google Scholar 

  11. Giraud P, Elles S, Helfre S et al (2002) Conformal radiotherapy for lung cancer: different delineation of the gross tumour volume (GTV) by radiologists and radiation oncologists. Radiother Oncol 62:27–36

    Article  PubMed  Google Scholar 

  12. Goldner G, Dimopoulos J, Kirisits C, Pötter R (2009) Moderate dose escalation in three-dimensional conformal localized prostate cancer radiotherapy: single-institutional experience in 398 patients comparing 66 Gy versus 70 Gy versus 74 Gy. Strahlenther Onkol 185:438–445

    Article  PubMed  Google Scholar 

  13. Grabarz D, Panzarella T, Bezjak A et al (2011) Quantifying interobserver variation in target definition in Palliative Radiotherapy. Int J Radiat Oncol Biol Phys 80:1498–1504

    Article  PubMed  Google Scholar 

  14. Guckenberger M, Ok S, Polat B et al (2010) Toxicity after intensity-modulated, image-guided radiotherapy for prostate cancer. Strahlenther Onkol 186:535–43 (Erratum: Strahlenther Onkol 2010;186:705)

    Google Scholar 

  15. International Commission on Radiation Units and Measurements. ICRU Report 62 (1999) Prescribing, recording, and reporting photon beam therapy (Supplement to ICRU Report 50). ICRU, Bethesda, MD, USA

  16. Hentschel B, Oehler W, Strauss D et al (2011) Definition of the CTV prostate in CT and MRI by using CT-MRI image fusion in IMRT planning for prostate cancer. Strahlenther Onkol 187:183–190

    Article  PubMed  Google Scholar 

  17. Jansen EP, Nijkamp J, Gubanski M et al (2010) Interobserver variation of clinical target volume delineation in gastric cancer. Int J Radiat Oncol Biol Phys 77:1166–1170

    Article  PubMed  Google Scholar 

  18. Kouwenhoven E, Giezen M, Struikmans H (2009) Measuring the similarity of target volume delineations independent of the number of observers. Phys Med Biol 54:2863–2873

    Article  PubMed  Google Scholar 

  19. Krengli M, Cannillo B, Turri L et al (2010) Target volume delineation for preoperative radiotherapy of rectal cancer: inter-observer variability and potential impact of FDG-PET/CT imaging. Technol Cancer Res Treat 9:393–398

    PubMed  Google Scholar 

  20. Lawton CA, Michalski J, El-Naqa I et al (2009) Variation in the definition of clinical target volumes for pelvic nodal conformal radiation therapy for prostate cancer. Int J Radiat Oncol Biol Phys 74:377–382

    Article  PubMed  Google Scholar 

  21. Leunens G, Menten J, Weltens C et al (1993) Quality assessment of medical decision making in radiation oncology: variability in target volume delineation for brain tumours. Radiother Oncol 29:169–175

    Article  PubMed  CAS  Google Scholar 

  22. Li XA, Tai A, Arthur DW, Buchholz TA et al (2009) Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG Multi-Institutional and Multiobserver Study. Int J Radiat Oncol Biol Phys 73:944–951

    Article  PubMed  Google Scholar 

  23. Logue JP, Sharrock CL, Cowan RA et al (1998) Clinical variability of target volume description in conformal radiotherapy planning. Int J Radiat Oncol Biol Phys 41:929–9231

    Article  PubMed  CAS  Google Scholar 

  24. Louie AV, Rodrigues G, Olsthoorn J et al (2010) Inter-observer and intra-observer reliability for lung cancer target volume delineation in the 4D-CT era. Radiother Oncol 95:166–171

    Article  PubMed  Google Scholar 

  25. Lütgendorf-Caucig C, Fotina I, Stock M et al (2011) Feasibility of CBCT-based target and normal structure delineation in prostate cancer radiotherapy: multi-observer and image multi-modality study. Radiother Oncol 98:154–161

    Article  PubMed  Google Scholar 

  26. Metwally H, Courbon F, David I et al (2011) Coregistration of prechemotherapy PET-CT for planning pediatric Hodgkin’s disease radiotherapy significantly diminishes interobserver variability of clinical target volume definition. Int J Radiat Oncol Biol Phys 80:793–799

    Article  PubMed  Google Scholar 

  27. Mitchell JR, Karlik SJ, Lee DH et al (1996) The variability of manual and computer assisted quantification of multiple sclerosis lesion volumes. Med Phys 23:85–97

    Article  PubMed  CAS  Google Scholar 

  28. Njeh CF (2008) Tumor delineation: The weakest link in the search for accuracy in radiotherapy. J Med Phys 33:136–140

    Article  PubMed  CAS  Google Scholar 

  29. Petersen RP, Truong PT, Kader HA et al (2007) Target volume delineation for partial breast radiotherapy planning: Clinical characteristics associated with low interobserver concordance. Int J Radiat Oncol Biol Phys 69:41–48

    Article  PubMed  Google Scholar 

  30. Pinkawa M, Holy R, Piroth MD et al (2010) Intensity-modulated radiotherapy for prostate cancer implementing molecular imaging with 18 F-choline PET-CT to define a simultaneous integrated boost. Strahlenther Onkol 186:600–606

    Article  PubMed  Google Scholar 

  31. Pinkawa M, Piroth MD, Holy R et al (2011) Combination of dose escalation with technological advances (intensity-modulated and image-guided radiotherapy) is not associated with increased morbidity for patients with prostate cancer. Strahlenther Onkol 187:479–484

    Article  PubMed  Google Scholar 

  32. Rasch C, Barillot I, Remeijer P et al (1999) Definition of the prostate in CT and MRI: a multi-observer study. Int J Radiat Oncol Biol Phys 43:57–66

    Article  PubMed  CAS  Google Scholar 

  33. Rasch C, Steenbakkers R, Herk M van (2005) Target definition in prostate, head, and neck. Semin Radiat Oncol 15:136–145

    Article  PubMed  Google Scholar 

  34. Rasch CR, Steenbakkers RJ, Fitton I et al (2010) Decreased 3D observer variation with matched CT-MRI, for target delineation in nasopharynx cancer. Radiat Oncol 5:21

    Article  PubMed  Google Scholar 

  35. Senan S, Koste J de, Samson M et al (1999) Evaluation of a target contouring protocol for 3D conformal radiotherapy in non-small cell lung cancer. Radiother Oncol 53:247–255

    Article  PubMed  CAS  Google Scholar 

  36. Smith WL, Lewis C, Bauman G et al (2007) Prostate volume contouring: a 3D analysis of segmentation using 3DTRUS, CT, and MR. Int J Radiat Oncol Biol Phys 67:1238–1247

    Article  PubMed  Google Scholar 

  37. Song W, Chiu B, Bauman G et al (2006) Prostate contouring uncertainty in megavoltage computed tomography images acquired with a helical tomotherapy unit during image-guided radiation therapy. Int J Radiat Oncol Biol Phys 65:595–607

    Article  PubMed  Google Scholar 

  38. Steenbakkers RJ, Duppen JC, Fitton I et al (2006) Reduction of observer variation using matched CT-PET for lung cancer delineation: a three-dimensional analysis. Int J Radiat Oncol Biol Phys 64:435–434

    Article  PubMed  Google Scholar 

  39. Stock M, Dörr W, Stromberger C et al (2010) Investigations on parotid gland recovery after IMRT in head and neck tumor patients. Strahlenther Onkol 186:665–671

    Article  PubMed  Google Scholar 

  40. Stroom JC, Heijmen BJ (2002) Geometrical uncertainties, radiotherapy planning margins, and the ICRU-62 report. Radiother Oncol 64:75–83

    Article  PubMed  Google Scholar 

  41. Struikmans H, Wárlám-Rodenhuis C, Stam T et al (2005) Interobserver variability of clinical target volume delineation of glandular breast tissue and of boost volume in tangential breast irradiation. Radiother Oncol 76:293–299

    Article  PubMed  Google Scholar 

  42. Tong S, Cardinal HN, McLoughlin RF et al (1998) Intra- and interobserver variability and reliability of prostate volume measurement via two-dimensional and three-dimensional ultrasound imaging. Ultrasound Med Biol 24:673–681

    Article  PubMed  CAS  Google Scholar 

  43. Tyng CJ, Chojniak R, Pinto PN et al (2009) Conformal radiotherapy for lung cancer: interobservers’ variability in the definition of gross tumor volume between radiologists and radiotherapists. Radiat Oncol 4:28

    Article  PubMed  Google Scholar 

  44. Usmani N, Sloboda R, Kamal W et al (2011) Can images obtained with high field strength magnetic resonance imaging reduce contouring variability of the prostate? Int J Radiat Oncol Biol Phys 80:728–734

    Article  PubMed  Google Scholar 

  45. Baardwijk A van, Bosmans G, Boersma et al (2007) PET-CT-based auto-contouring in non-small-cell lung cancer correlates with pathology and reduces interobserver variability in the delineation of the primary tumor and involved nodal volumes. Int J Radiat Oncol Biol Phys 68:771–778

    Article  PubMed  Google Scholar 

  46. Mourik AM van, Elkhuizen PH, Minkema D et al (2010) Multiinstitutional study on target volume delineation variation in breast radiotherapy in the presence of guidelines. Radiother Oncol 94:286–291

    Article  PubMed  Google Scholar 

  47. Villeirs GM, Van Vaerenbergh K, Vakaet L et al (2005) Interobserver delineation variation using CT versus combined CT + MRI in intensity-modulated radiotherapy for prostate cancer. Strahlenther Onkol 181:424–430

    Article  PubMed  Google Scholar 

  48. Vorwerk H, Beckmann G, Bremer M et al (2009) The delineation of target volumes for radiotherapy of lung cancer patients. Radiother Oncol 91:455–460

    Article  PubMed  Google Scholar 

  49. Weiss E, Hess CF (2003) The impact of gross tumor volume (GTV) and clinical target volume (CTV) definition on the total accuracy in radiotherapy theoretical aspects and practical experiences. Strahlenther Onkol 179:21–30

    Article  PubMed  Google Scholar 

  50. Weiss E, Richter S, Krauss T et al (2003) Conformal radiotherapy planning of cervix carcinoma: differences in the delineation of the clinical target volume. A comparison between gynaecologic and radiation oncologists. Radiother Oncol 67:87–95

    Article  PubMed  Google Scholar 

  51. Weiss E, Wu J, Sleeman W et al (2010) Clinical evaluation of soft tissue organ boundary visualization on cone-beam computed tomographic imaging. Int J Radiat Oncol Biol Phys 78:929–936

    Article  PubMed  Google Scholar 

  52. Weltens C, Menten J, Feron M et al (2001) Interobserver variations in gross tumor volume delineation of brain tumors on computed tomography and impact of magnetic resonance imaging. Radiother Oncol 60:49–59

    Article  PubMed  CAS  Google Scholar 

  53. White EA, Brock KK, Jaffray DA et al (2009) Inter-observer variability of prostate delineation on cone beam computerised tomography images. Clin Oncol 21:32–38

    Article  CAS  Google Scholar 

Download references

Acknowledgments

Authors would like to gratefully acknowledge the observers for donating their time for contouring and making this study possible. Irina Fotina acknowledges the financial support from Austrian National Bank (OeNB, project number 12972).

Conflict of interest

The corresponding author states that there are no conflicts of interest.

Author information

Authors and Affiliations

  1. Div. Medical Radiation Physics, Department of Radiotherapy, Medical University Vienna/AKH Vienna, Währinger Gürtel 18–20, 1090, Vienna, Austria

    I. Fotina, C. Lütgendorf-Caucig, M. Stock, R. Pötter & D. Georg

Authors
  1. I. Fotina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. Lütgendorf-Caucig
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Stock
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Pötter
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. D. Georg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to I. Fotina.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fotina, I., Lütgendorf-Caucig, C., Stock, M. et al. Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy. Strahlenther Onkol 188, 160–167 (2012). https://doi.org/10.1007/s00066-011-0027-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00066-011-0027-6

Keywords

Schlüsselwörter

Search

Navigation