Advertisement

Strahlentherapie und Onkologie

, Volume 191, Issue 6, pp 525–533 | Cite as

A teaching intervention in a contouring dummy run improved target volume delineation in locally advanced non-small cell lung cancer

Reducing the interobserver variability in multicentre clinical studies
  • Tanja Schimek-JaschEmail author
  • Esther G. C. Troost
  • Gerta Rücker
  • Vesna Prokic
  • Melanie Avlar
  • Viola Duncker-Rohr
  • Michael Mix
  • Christian Doll
  • Anca-Ligia Grosu
  • Ursula Nestle
Original Article

Abstract

Introduction

Interobserver variability in the definition of target volumes (TVs) is a well-known confounding factor in (multicentre) clinical studies employing radiotherapy. Therefore, detailed contouring guidelines are provided in the prospective randomised multicentre PET-Plan (NCT00697333) clinical trial protocol. This trial compares strictly FDG-PET-based TV delineation with conventional TV delineation in patients with locally advanced non-small cell lung cancer (NSCLC). Despite detailed contouring guidelines, their interpretation by different radiation oncologists can vary considerably, leading to undesirable discrepancies in TV delineation. Considering this, as part of the PET-Plan study quality assurance (QA), a contouring dummy run (DR) consisting of two phases was performed to analyse the interobserver variability before and after teaching.

Materials and methods

In the first phase of the DR (DR1), radiation oncologists from 14 study centres were asked to delineate TVs as defined by the study protocol (gross TV, GTV; and two clinical TVs, CTV-A and CTV-B) in a test patient. A teaching session was held at a study group meeting, including a discussion of the results focussing on discordances in comparison to the per-protocol solution. Subsequently, the second phase of the DR (DR2) was performed in order to evaluate the impact of teaching.

Results

Teaching after DR1 resulted in a reduction of absolute TVs in DR2, as well as in better concordance of TVs. The Overall Kappa(κ) indices increased from 0.63 to 0.71 (GTV), 0.60 to 0.65 (CTV-A) and from 0.59 to 0.63 (CTV-B), demonstrating improvements in overall interobserver agreement.

Conclusion

Contouring DRs and study group meetings as part of QA in multicentre clinical trials help to identify misinterpretations of per-protocol TV delineation. Teaching the correct interpretation of protocol contouring guidelines leads to a reduction in interobserver variability and to more consistent contouring, which should consequently improve the validity of the overall study results.

Keywords

Target volume definition Radiochemotherapy Positron-emission tomography Dummy Run nodal irradiation 

Verbesserung der Zielvolumendefinition durch einen Dummy Run mit interventionellem Konturierungstraining beim lokal fortgeschrittenen nicht-kleinzelligen Lungenkarzinom

Reduktion der Interobserver-Variabilität in klinischen Multizenterstudien

Zusammenfassung

Einleitung

Die Interobserver-Variabilität bei der Definition der Zielvolumina („target volumes“, TVs) ist ein anerkanntes Problem in strahlentherapeutischen (multizentrischen) klinischen Studien. In der prospektiven randomisierten multizentrischen PET-Plan-Studie (NCT00697333) wird eine strikt FDG-PET-basierte mit einer konventionellen TV-Definition bei Patienten mit einem lokal fortgeschrittenen nicht-kleinzelligem Lungenkarzinom (NSCLC) verglichen. Im Studienprotokoll ist eine detaillierte Konturierungsanleitung implementiert. Trotz genauer Vorgaben zur TV-Definition kann deren Interpretation durch verschiedene Strahlentherapeuten sehr differieren und zu unerwünschten Diskrepanzen bei den TVs führen. Unter diesem Gesichtspunkt wurde als ein Aspekt der Qualitätssicherungsmaßnahmen der PET-Plan-Studie ein Konturierungs-„Dummy-Run“ in zwei Abschnitten durchgeführt. Analysiert wurde die Interobserver-Variabilität vor und nach einem Konturierungstraining.

Material und Methoden

Im ersten Teil des „Dummy Run“ (DR1) wurden Strahlentherapeuten aus 14 Studienzentren gebeten, eine protokollgemäße Definition der TVs (GTV, CTVA, CTV-B) an einem Beispielpatienten durchzuführen. Bei einem Studientreffen wurden die Ergebnisse mit Fokus auf Diskrepanzen im Vergleich zur protokollgemäßen Konturierung diskutiert. Mit dem Ziel, den Einfluss dieses Konturierungstrainings zu evaluieren, wurde der zweite Teil des „Dummy Run“ (DR2) durchgeführt.

Ergebnisse

Das Training der protokollgemäßen Konturierung nach dem DR1 führte zu einer Reduktion der absoluten Volumina sowie zu einer besseren Übereinstimmung der TVs im DR2. Der „Overall“-Kappa erhöhte sich von 0,63 auf 0,71 (GTV), von 0,60 auf 0,65 (CTV-A) und von 0,59 auf 0,63 (CTV-B).

Schlussfolgerung

Durch Qualitätssicherungsmaßnahmen wie „Dummy Runs“ und Studientreffen können Fehlinterpretationen bei der protokollgemäßen TV-Definition in multizentrischen klinischen Studien in der Strahlenheilkunde identifiziert und durch entsprechendes Training korrigiert werden. Dies sollte sich positiv auf die Validität der Studienergebnisse auswirken.

Schlüsselwörter

Zielvolumendefinition Radiochemotherapie Positronenemissionstomographie Dummy Run Nodalbestrahlung 

Notes

Compliance with ethics guidelines

Conflict of interest

T. Schimek-Jasch, E.G.C. Troost, G. Rücker, V. Prokic, M. Avlar, V. Duncker-Rohr, M. Mix, C. Doll, A.-L. Grosu and U. Nestle state that there are no conflicts of interest.

This study was conducted with the approval of the Ethics Committee of the Albert-Ludwigs-University Freiburg.

All studies on humans described in the present manuscript were carried out with the approval of the responsible ethics committee and in accordance with national law and the Helsinki Declaration of 1975 (in its current, revised form). Informed consent was obtained from all patients included in studies.

Consent was obtained from all patients identifiable from images or other information within the manuscript. In the case of underage patients, consent was obtained from a parent or legal guardian.

Supplementary material

66_2015_812_MOESM1_ESM.docx (14 kb)
(DOCX 14 kb)

References

  1. 1.
    WHO (2014) World Health Organization. World cancer factsheet January 2014: world cancer burden (2012). Accessed 6 Nov. 2014Google Scholar
  2. 2.
    Auperin A, Le Pechoux C, Rolland E, Curran WJ, Furuse K, Fournel P, Belderbos J, Clamon G, Ulutin HC, Paulus R, Yamanaka T, Bozonnat MC, Uitterhoeve A, Wang X, Stewart L, Arriagada R, Burdett S, Pignon JP (2010) Meta-analysis of concomitant versus sequential radiochemotherapy in locally advanced non-small-cell lung cancer. J Clin Oncol 28:2181–2190. doi:10.1200/JCO.2009.26.2543CrossRefPubMedGoogle Scholar
  3. 3.
    Goeckenjan G, Sitter H, Thomas M, Branscheid D, Flentje M, Griesinger F, Niederle N, Stuschke M, Blum T, Deppermann KM, Ficker JH, Freitag L, Lubbe AS, Reinhold T, Spath-Schwalbe E, Ukena D, Wickert M, Wolf M, Andreas S, Auberger T, Baum RP, Baysal B, Beuth J, Bickeboller H, Bocking A, Bohle RM, Bruske I, Burghuber O, Dickgreber N, Diederich S, Dienemann H, Eberhardt W, Eggeling S, Fink T, Fischer B, Franke M, Friedel G, Gauler T, Gutz S, Hautmann H, Hellmann A, Hellwig D, Herth F, Heussel CP, Hilbe W, Hoffmeyer F, Horneber M, Huber RM, Hubner J, Kauczor HU, Kirchbacher K, Kirsten D, Kraus T, Lang SM, Martens U, Mohn-Staudner A, Muller KM, Muller-Nordhorn J, Nowak D, Ochmann U, Passlick B, Petersen I, Pirker R, Pokrajac B, Reck M, Riha S, Rube C, Schmittel A, Schonfeld N, Schutte W, Serke M, Stamatis G, Steingraber M, Steins M, Stoelben E, Swoboda L, Teschler H, Tessen HW, Weber M, Werner A, Wichmann HE, Irlinger Wimmer E, Witt C, Worth H, German Respiratory S, German Cancer S (2011) Prevention, diagnosis, therapy, and follow-up of lung cancer: interdisciplinary guideline of the German Respiratory Society and the German Cancer Society. Pneumologie 65:39–59. doi:10.1055/s-0030-1255961CrossRefPubMedGoogle Scholar
  4. 4.
    Fleckenstein J, Hellwig D, Kremp S, Grgic A, Groschel A, Kirsch CM, Nestle U, Rube C (2011) F-18-FDG-PET confined radiotherapy of locally advanced NSCLC with concomitant chemotherapy: results of the PET-PLAN pilot trial. International journal of radiation oncology, biology, physics. 81:e283–e289. doi:10.1016/j.ijrobp.2011.01.020Google Scholar
  5. 5.
    van Baardwijk A, Wanders S, Boersma L, Borger J, Ollers M, Dingemans AM, Bootsma G, Geraedts W, Pitz C, Lunde R, Lambin P, De Ruysscher D (2010) Mature results of an individualized radiation dose prescription study based on normal tissue constraints in stages I to III non-small-cell lung cancer. J Clin Oncol 28:1380–1386. doi:10.1200/JCO.2009.24.7221CrossRefPubMedGoogle Scholar
  6. 6.
    Hoffmann AL, Troost EG, Huizenga H, Kaanders JH, Bussink J (2012) Individualized dose prescription for hypofractionation in advanced non-small-cell lung cancer radiotherapy: an in silico trial. International journal of radiation oncology, biology, physics. Int J Radiat Oncol Biol Phys 83:1596–1602. doi:10.1016/j.ijrobp.2011.10.032CrossRefPubMedGoogle Scholar
  7. 7.
    van Elmpt W, De Ruysscher D, van der Salm A, Lakeman A, van der Stoep J, Emans D, Damen E, Ollers M, Sonke JJ, Belderbos J (2012) The PET-boost randomised phase II dose-escalation trial in non-small cell lung cancer. Radiother Oncol 104:67–71. doi:10.1016/j.radonc.2012.03.005CrossRefPubMedGoogle Scholar
  8. 8.
    Nestle U, Kremp S, Grosu AL (2006) Practical integration of [18F]-FDG-PET and PET-CT in the planning of radiotherapy for non-small cell lung cancer (NSCLC): the technical basis, ICRU-target volumes, problems, perspectives. Radiother Oncol 81:209–225. doi:10.1016/j.radonc.2006.09.011CrossRefPubMedGoogle Scholar
  9. 9.
    MacManus M, Nestle U, Rosenzweig KE, Carrio I, Messa C, Belohlavek O, Danna M, Inoue T, Deniaud-Alexandre E, Schipani S, Watanabe N, Dondi M, Jeremic B (2009) Use of PET and PET/CT for radiation therapy planning: IAEA expert report 2006–2007. Radiother Oncol 91:85–94. doi:10.1016/j.radonc.2008.11.008CrossRefPubMedGoogle Scholar
  10. 10.
    Belderbos JS, Heemsbergen WD, De Jaeger K, Baas P, Lebesque JV (2006) Final results of a Phase I/II dose escalation trial in non-small-cell lung cancer using three-dimensional conformal radiotherapy. Int J Radiat Oncol Biol Phys 66:126–134. doi:10.1016/j.ijrobp.2006.04.034CrossRefPubMedGoogle Scholar
  11. 11.
    De Ruysscher D, Wanders S, van Haren E, Hochstenbag M, Geeraedts W, Utama I, Simons J, Dohmen J, Rhami A, Buell U, Thimister P, Snoep G, Boersma L, Verschueren T, van Baardwijk A, Minken A, Bentzen SM, Lambin P (2005) Selective mediastinal node irradiation based on FDG-PET scan data in patients with non-small-cell lung cancer: a prospective clinical study. Int J Radiat Oncol Biol Phys 62:988–994. doi:10.1016/j.ijrobp.2004.12.019CrossRefPubMedGoogle Scholar
  12. 12.
    Caldwell CB, Mah K, Ung YC, Danjoux CE, Balogh JM, Ganguli SN, Ehrlich LE (2001) Observer variation in contouring gross tumor volume in patients with poorly defined non-small-cell lung tumors on CT: the impact of 18FDG-hybrid PET fusion. Int J Radiat Oncol Biol Phys 51:923–931CrossRefPubMedGoogle Scholar
  13. 13.
    Nestle U, Weber W, Hentschel M, Grosu AL (2009) Biological imaging in radiation therapy: role of positron emission tomography. Phys Med Biol 54:R1–R25. doi:10.1088/0031-9155/54/1/R01CrossRefPubMedGoogle Scholar
  14. 14.
    Steenbakkers RJ, Duppen JC, Fitton I, Deurloo KE, Zijp LJ, Comans EF, Uitterhoeve AL, Rodrigus PT, Kramer GW, Bussink J, De Jaeger K, Belderbos JS, Nowak PJ, van Herk M, Rasch CR (2006) Reduction of observer variation using matched CT-PET for lung cancer delineation: a three-dimensional analysis. Int J Radiat Oncol Biol Phys 64:435–448. doi:10.1016/j.ijrobp.2005.06.034CrossRefPubMedGoogle Scholar
  15. 15.
    Nestle U, Mix M, Weber W, Grosu AL (2011) Klinische Studien zum Einsatz der PET in der Bestrahlungsplanung in Deutschland: Ein Update. Nuklearmediziner 34:130–132. doi:10.1055/s-0031-1280790CrossRefGoogle Scholar
  16. 16.
    Schimek-Jasch T, Küsters A, Hoffmans H, Nestle U (2014) Optimierung der Strahlentherapieplanung bei lokal fortgeschrittenen NSCLC durch 18F-FDG PET-CT: Stand der PET-Plan-Studie. Nuklearmediziner 37:175–180. doi:10.1055/s-0034-1376972CrossRefGoogle Scholar
  17. 17.
    Djarv E, Nyman J, Baumann P, Ekberg L, Hoyer M, Lax I, Lewensohn R, Levin N, Lund JA, Morhed E, Ericsson SR, Traberg A, Wittgren L, Johansson KA (2006) Dummy run for a phase II study of stereotactic body radiotherapy of T1-T2 N0M0 medical inoperable non-small cell lung cancer. Acta Oncol 45:973–977. doi:10.1080/02841860600919241CrossRefPubMedGoogle Scholar
  18. 18.
    Foppiano F, Fiorino C, Frezza G, Greco C, Valdagni R, Radiotherapy ANWGoP (2003) The impact of contouring uncertainty on rectal 3D dose-volume data: results of a dummy run in a multicenter trial (AIROPROS01-02). Int J Radiat Oncol Biol Phys 57:573–579CrossRefPubMedGoogle Scholar
  19. 19.
    Giraud P, Elles S, Helfre S, De Rycke Y, Servois V, Carette MF, Alzieu C, Bondiau PY, Dubray B, Touboul E, Housset M, Rosenwald JC, Cosset JM (2002) Conformal radiotherapy for lung cancer: different delineation of the gross tumor volume (GTV) by radiologists and radiation oncologists. Radiother Oncol 62:27–36CrossRefPubMedGoogle Scholar
  20. 20.
    Matzinger O, Poortmans P, Giraud JY, Maingon P, Budiharto T, van den Bergh AC, Davis JB, Musat E, Ataman F, Huyskens DP, Gulyban A, Bolla M, Group ERO (2009) Quality assurance in the 22991 EORTC ROG trial in localized prostate cancer: dummy run and individual case review. Radiother Oncol 90:285–290. doi:10.1016/j.radonc.2008.10.022CrossRefPubMedGoogle Scholar
  21. 21.
    Poortmans PM, Venselaar JL, Struikmans H, Hurkmans CW, Davis JB, Huyskens D, van Tienhoven G, Vlaun V, Lagendijk JJ, Mijnheer BJ, De Winter KA, Van der Hulst MH, Van den Bogaert WF (2001) The potential impact of treatment variations on the results of radiotherapy of the internal mammary lymph node chain: a quality-assurance report on the dummy run of EORTC Phase III randomized trial 22922/10925 in Stage I-III breast cancer(1). Int J Radiat Oncol Biol Phys 49:1399–1408CrossRefPubMedGoogle Scholar
  22. 22.
    Spoelstra FO, Senan S, Le Pechoux C, Ishikura S, Casas F, Ball D, Price A, De Ruysscher D, van Sornsen de Koste JR, Lung Adjuvant Radiotherapy Trial Investigators G (2010) Variations in target volume definition for postoperative radiotherapy in stage III non-small-cell lung cancer: analysis of an international contouring study. Int J Radiat Oncol Biol Phys 76:1106–1113. doi:10.1016/j.ijrobp.2009.02.072Google Scholar
  23. 23.
    Senan S, van Sornsen de Koste J, Samson M, Tankink H, Jansen P, Nowak PJ, Krol AD, Schmitz P, Lagerwaard FJ (1999) Evaluation of a target contouring protocol for 3D conformal radiotherapy in non-small cell lung cancer. Radiother Oncol 53:247–255CrossRefPubMedGoogle Scholar
  24. 24.
    Senan S, Le Pechoux C, Spoelstra FO, Ishikura S, Casas F, Ball D, Price A, De Ruysscher D, van Sörnsen de Koste JR (2007) A need to standardize post-operative radiotherapy (PORT) fields used for non-small cell lung cancer (NSCLC): analysis of an international dummy-run study: A1–02. J Thorac Oncol 2:S308–S309CrossRefGoogle Scholar
  25. 25.
    Valley JF, Bernier J, Tercier PA, Fogliata-Cozzi A, Rosset A, Garavaglia G, Mirimanoff RO (1998) Quality assurance of the EORTC radiotherapy trial 22931 for head and neck carcinomas: the dummy run. Radiother Oncol 47:37–44CrossRefPubMedGoogle Scholar
  26. 26.
    Van de Steene J, Linthout N, de Mey J, Vinh-Hung V, Claassens C, Noppen M, Bel A, Storme G (2002) Definition of gross tumor volume in lung cancer: inter-observer variability. Radiother Oncol 62:37–49CrossRefGoogle Scholar
  27. 27.
    Vorwerk H, Beckmann G, Bremer M, Degen M, Dietl B, Fietkau R, Gsanger T, Hermann RM, Alfred Herrmann MK, Holler U, van Kampen M, Korber W, Maier B, Martin T, Metz M, Richter R, Siekmeyer B, Steder M, Wagner D, Hess CF, Weiss E, Christiansen H (2009) The delineation of target volumes for radiotherapy of lung cancer patients. Radiother Oncol 91:455–460. doi:10.1016/j.radonc.2009.03.014CrossRefPubMedGoogle Scholar
  28. 28.
    Schaefer A, Nestle U, Kremp S, Hellwig D, Grgic A, Buchholz HG, Mischke W, Gromoll C, Dennert P, Plotkin M, Senftleben S, Thorwarth D, Tosch M, Wahl A, Wengenmair H, Rube C, Kirsch CM (2012) Multi-centre calibration of an adaptive thresholding method for PET-based delineation of tumour volumes in radiotherapy planning of lung cancer. Nuklearmedizin 51:101–110. doi:10.3413/Nukmed-0452-11-12CrossRefPubMedGoogle Scholar
  29. 29.
    Schaefer A, Kremp S, Hellwig D, Rube C, Kirsch CM, Nestle U (2008) A contrast-oriented algorithm for FDG-PET-based delineation of tumour volumes for the radiotherapy of lung cancer: derivation from phantom measurements and validation in patient data. Eur J Nucl Med Mol Imaging 35:1989–1999. doi:10.1007/s00259-008-0875-1CrossRefPubMedGoogle Scholar
  30. 30.
    Chapet O, Kong FM, Quint LE, Chang AC, Ten Haken RK, Eisbruch A, Hayman JA (2005) CT-based definition of thoracic lymph node stations: an atlas from the University of Michigan. Int J Radiat Oncol Biol Phys 63:170–178. doi:10.1016/j.ijrobp.2004.12.060CrossRefPubMedGoogle Scholar
  31. 31.
    Giraud P, De Rycke Y, Lavole A, Milleron B, Cosset JM, Rosenzweig KE (2006) Probability of mediastinal involvement in non-small-cell lung cancer: a statistical definition of the clinical target volume for 3-dimensional conformal radiotherapy? Int J Radiat Oncol Biol Phys 64:127–135. doi:10.1016/j.ijrobp.2005.06.043CrossRefPubMedGoogle Scholar
  32. 32.
    Rucker G, Schimek-Jasch T, Nestle U (2012) Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using Fleissʼ kappa. Methods Inf Med 51:489–494. doi:10.3414/ME12-01-0005CrossRefPubMedGoogle Scholar
  33. 33.
    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Tanja Schimek-Jasch
    • 1
    • 6
    Email author
  • Esther G. C. Troost
    • 2
  • Gerta Rücker
    • 3
  • Vesna Prokic
    • 1
    • 6
  • Melanie Avlar
    • 6
  • Viola Duncker-Rohr
    • 4
  • Michael Mix
    • 5
    • 6
  • Christian Doll
    • 1
    • 6
  • Anca-Ligia Grosu
    • 1
    • 6
  • Ursula Nestle
    • 1
    • 6
  1. 1.Department of Radiation OncologyUniversity Medical Center FreiburgFreiburgGermany
  2. 2.Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental BiologyMaastricht University Medical CentreMaastrichtThe Netherlands
  3. 3.Institute for Medical Biometry and Statistics, Centre for Medical Biometry and Medical InformaticsUniversity Medical Center FreiburgFreiburgGermany
  4. 4.Department of Radiation OncologyOrtenau-Klinikum Offenburg-GengenbachGengenbachGermany
  5. 5.Department of Nuclear MedicineUniversity Medical Center FreiburgFreiburgGermany
  6. 6.German Cancer Research Center (DKFZ), Heidelberg, Germany and German Cancer Consortium (DKTK) partner site: FreiburgHeidelbergGermany

Personalised recommendations