Impact of consensus contours from multiple PET segmentation methods on the accuracy of functional volume delineation

  • A. Schaefer
  • M. VermandelEmail author
  • C. Baillet
  • A. S. Dewalle-Vignion
  • R. Modzelewski
  • P. Vera
  • L. Massoptier
  • C. Parcq
  • D. Gibon
  • T. Fechter
  • U. Nemer
  • I. Gardin
  • U. Nestle
Original Article



The aim of this study was to evaluate the impact of consensus algorithms on segmentation results when applied to clinical PET images. In particular, whether the use of the majority vote or STAPLE algorithm could improve the accuracy and reproducibility of the segmentation provided by the combination of three semiautomatic segmentation algorithms was investigated.


Three published segmentation methods (contrast-oriented, possibility theory and adaptive thresholding) and two consensus algorithms (majority vote and STAPLE) were implemented in a single software platform (Artiview®). Four clinical datasets including different locations (thorax, breast, abdomen) or pathologies (primary NSCLC tumours, metastasis, lymphoma) were used to evaluate accuracy and reproducibility of the consensus approach in comparison with pathology as the ground truth or CT as a ground truth surrogate.


Variability in the performance of the individual segmentation algorithms for lesions of different tumour entities reflected the variability in PET images in terms of resolution, contrast and noise. Independent of location and pathology of the lesion, however, the consensus method resulted in improved accuracy in volume segmentation compared with the worst-performing individual method in the majority of cases and was close to the best-performing method in many cases. In addition, the implementation revealed high reproducibility in the segmentation results with small changes in the respective starting conditions. There were no significant differences in the results with the STAPLE algorithm and the majority vote algorithm.


This study showed that combining different PET segmentation methods by the use of a consensus algorithm offers robustness against the variable performance of individual segmentation methods and this approach would therefore be useful in radiation oncology. It might also be relevant for other scenarios such as the merging of expert recommendations in clinical routine and trials or the multiobserver generation of contours for standardization of automatic contouring.


PET image segmentation Consensus algorithms STAPLE Radiation oncology 18F-FDG PET Image segmentation 



A. Schaefer is very grateful for the valuable support of Y.-J. Kim PhD, Department of Pathology, Saarland University Medical Centre, in preparing the pathological reference database of centre 1. U. Nestle thanks Christian Doll, MD, Department for Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany, and Alin Chirindel, MD, Department of Nuclear Medicine, St. Claraspital, Basel, Switzerland, for assistance in analysing the data of centre 2.

Compliance with ethical standards


This work was partially supported by EU project E5949 SALOME under the Eurostars Program, which is supported by EUREKA and the European Community.

Conflicts of interest


Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the principles of the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Informed consent

Formal consent is not required for this type of retrospective study.


  1. 1.
    Miller TR, Grigsby PW. Measurement of tumor volume by PET to evaluate prognosis in patients with advanced cervical cancer treated by radiation therapy. Int J Radiat Oncol Biol Phys. 2002;53:353–9.CrossRefPubMedGoogle Scholar
  2. 2.
    Fernando S, Kong F, Kessler M, Chetty I, Narayan S, Tatro D, et al. Using FDG-PET to delineate gross tumor and internal target volumes. Int J Radiat Oncol Biol Phys. 2005;63 Suppl 1:S400-1.CrossRefGoogle Scholar
  3. 3.
    Zasadny KR, Kison PV, Francis IR, Wahl RL. FDG-PET determination of metabolically active tumor volume and comparison with CT. Clin Positron Imaging. 1998;1:123–9.Google Scholar
  4. 4.
    Bryant AS, Cerfolio RJ. The maximum standardized uptake values on integrated FDG-PET/CT is useful in differentiating benign from malignant pulmonary nodules. Ann Thorac Surg. 2006;81:1016–20.CrossRefGoogle Scholar
  5. 5.
    Black QC, Grills IS, Kestin LL, Wong CY, Wong JW, Martinez AA, et al. Defining a radiotherapy target with positron emission tomography. Int J Radiat Oncol Biol Phys. 2004;60:1272–82.CrossRefPubMedGoogle Scholar
  6. 6.
    Daisne JF, Duprez T, Weynand B, Lonneux M, Hamoir M, Reychler H, et al. Tumor volume in pharyngolaryngeal squamous cell carcinoma: Comparison between CT, MR imaging, and FDG PET and validation with surgical specimen. Radiology. 2004;233:93–100.CrossRefPubMedGoogle Scholar
  7. 7.
    Daisne JF, Sibomana M, Bol A, Doumont T, Lonneux M, Grégoire V. Tri-dimensional automatic segmentation of PET volumes based on measured source-to-background ratios: influence of the reconstruction algorithms. Radiother Oncol. 2003;69:247–50.CrossRefPubMedGoogle Scholar
  8. 8.
    Erdi YE, Mawlawi O, Larson SM, Imbriaco M, Yeung H, Finn R, et al. Segmentation of lung lesion volume by adaptive positron emission tomography image thresholding. Cancer. 1997;80(12 Suppl):2505–9.CrossRefPubMedGoogle Scholar
  9. 9.
    Erdi YE, Wessels BW, Loew MH, Erdi AK. Threshold estimation in single photon emission computed tomography and planar imaging for clinical radioimmunotherapy. Cancer Res. 1995;55(23 Suppl):S5823–6.Google Scholar
  10. 10.
    Jentzen W, Freudenberg L, Eising EG, Heinze M, Brandau W, Bockisch A. Segmentation of PET volumes by iterative image thresholding. J Nucl Med. 2007;48:108–14.PubMedGoogle Scholar
  11. 11.
    Nestle U, Kremp S, Schaefer-Schuler A, Sebastian-Welsch C, Hellwig D, Rube C, et al. Comparison of different methods for delineation of 18F-FDG PET-positive tissue for target volume definition in radiotherapy of patients with non-small cell lung cancer. J Nucl Med. 2005;46:1342–8.PubMedGoogle Scholar
  12. 12.
    Nestle U, Schaefer-Schuler A, Kremp S, Groeschel A, Hellwig D, Rube C, et al. Target volume definition for 18 F-FDG PET-positive lymph nodes in radiotherapy of patients with non-small cell lung cancer. Eur J Nucl Med Mol Imaging. 2007;34:453–62.CrossRefPubMedGoogle Scholar
  13. 13.
    Schaefer A, Kremp S, Hellwig D, Rube C, Kirsch CM, Nestle U. A contrast-oriented algorithm for FDG-PET-based delineation of tumour volumes for the radiotherapy of lung cancer: derivation from phantom measurements and validation in patient data. Eur J Nucl Med Mol Imaging. 2008;35:1989–99.CrossRefPubMedGoogle Scholar
  14. 14.
    Riddell C, Brigger P, Carson RE, Bacharach SL. The watershed algorithm: a method to segment noisy PET transmission images. IEEE Trans Nucl Sci. 1999;46:713–9.CrossRefGoogle Scholar
  15. 15.
    Tylski P, Bonniaud G, Decenciere E, Stawiaski J, Coulot J, Lefkopoulos D, et al. 18F-FDG PET images segmentation using morphological watershed: a phantom study. IEEE Nucl Sci Symp Conf Rec. 2006;4:2063–7.Google Scholar
  16. 16.
    Nissen I, Yaqub M, Lammertsma A, Lee J, Geets X, Boellaard R. A novel supervised watershed method for segmentation of tumors with heterogeneous tracer uptake in PET. Proceedings of the Society of Nuclear Medicine Annual Meeting, St. Louis, Mo, 7–11 June 2014. Abstract 260.Google Scholar
  17. 17.
    Geets X, Lee JA, Bol A, Lonneux M, Grégoire V. A gradient-based method for segmenting FDG-PET images: methodology and validation. Eur J Nucl Med Mol Imaging. 2007;34:1427–38.CrossRefPubMedGoogle Scholar
  18. 18.
    Zhu W, Jiang T. Automation segmentation of PET image for brain tumors. IEEE Nucl Sci Symp Conf Rec. 2003;4:2627–9.Google Scholar
  19. 19.
    Belhassen S, Zaidi H. A novel fuzzy C-means algorithm for unsupervised heterogeneous tumor quantification in PET. Med Phys. 2010;37:1309–24.CrossRefPubMedGoogle Scholar
  20. 20.
    Dewalle-Vignion AS, Betrouni N, Lopes R, Huglo D, Stute S, Vermandel M. A new method for volume segmentation of PET images, based on possibility theory. IEEE Trans Med Imaging. 2011;30:409–23. doi: 10.1109/TMI.2010.2083681.CrossRefPubMedGoogle Scholar
  21. 21.
    Dewalle-Vignion AS, Betrouni N, Makni N, Huglo D, Rousseau J, Vermandel M. A new method based on both fuzzy set and possibility theories for tumor volume segmentation on PET images. Conf Proc IEEE Eng Med Biol Soc. 2008;2008:3122–5.Google Scholar
  22. 22.
    Hatt M, Cheze le Rest C, Turzo A, Roux C, Visvikis D. A fuzzy locally adaptive Bayesian segmentation approach for volume determination in PET. IEEE Trans Med Imaging. 2009;28:881–93. doi: 10.1109/TMI.2008.2012036.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Hatt M, Lamare F, Boussion N, Turzo A, Collet C, Salzenstein F, et al. Fuzzy hidden Markov chains segmentation for volume determination and quantitation in PET. Phys Med Biol. 2007;52:3467–91. doi: 10.1088/0031-9155/52/12/010.CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Prieto E, Lecumberri P, Pagola M, Gomez M, Bilbao I, Ecay M, et al. Twelve automated thresholding methods for segmentation of PET images: a phantom study. Phys Med Biol. 2012;57:3963–80. doi: 10.1088/0031-9155/57/12/3963.CrossRefPubMedGoogle Scholar
  25. 25.
    Shepherd T, Teras M, Beichel R, Boellaard R, Bruynooghe M, Dicken V, et al. Comparative study with new accuracy metrics for target volume contouring in PET image guided radiation therapy. IEEE Trans Med Imaging. 2012;31:2006–24. doi: 10.1109/TMI.2012.2202322.
  26. 26.
    Østergaard LR, Larsen OV. Applying voting to segmentation of MR images. In: Amin A, Dori D, Pudil P, Freeman H, editors. Advances in pattern recognition. Berlin: Springer; 1998. p. 795–804.Google Scholar
  27. 27.
    Artaechevarria X, Munoz-Barrutia A, Ortiz-de-Solorzano C. Combination strategies in multi-atlas image segmentation: application to brain MR data. IEEE Trans Med Imaging. 2009;28:1266–77. doi: 10.1109/TMI.2009.2014372.CrossRefPubMedGoogle Scholar
  28. 28.
    Lam L, Suen CY. Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE Trans Syst Man Cybern A Syst Humans. 1997;27:553–68.CrossRefGoogle Scholar
  29. 29.
    Warfield SK, Zou KH, Wells WM. Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging. 2004;23:903–21. doi: 10.1109/TMI.2004.828354.CrossRefPubMedPubMedCentralGoogle Scholar
  30. 30.
    Commowick O, Akhondi-Asl A, Warfield SK. Estimating a reference standard segmentation with spatially varying performance parameters: local MAP STAPLE. IEEE Trans Med Imaging. 2012;31:1593–606. doi: 10.1109/TMI.2012.2197406.CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    McGurk RJ, Bowsher J, Lee JA, Das SK. Combining multiple FDG-PET radiotherapy target segmentation methods to reduce the effect of variable performance of individual segmentation methods. Med Phys. 2013;40:042501. doi: 10.1118/1.4793721.CrossRefPubMedPubMedCentralGoogle Scholar
  32. 32.
    Vauclin S, Doyeux K, Hapdey S, Edet-Sanson A, Vera P, Gardin I. Development of a generic thresholding algorithm for the delineation of 18 FDG-PET-positive tissue: application to the comparison of three thresholding models. Phys Med Biol. 2009;54:6901–16.CrossRefPubMedGoogle Scholar
  33. 33.
    Doll C, Parcq C, Modzelewski R, Dewalle-Vignion AS, Christ U, Loquin K, et al. PET-based target volume delineation in radiation therapy planning: are different implementations of the same automatic delineation method really equal? Proceedings of the Annual Congress of the European Association of Nuclear Medicine (EANM'13), October 2013, Lyon, France. Vol. 40 (2 suppl), p. S244 Google Scholar
  34. 34.
    Schaefer A, Nestle U, Kremp S, Hellwig D, Grgic A, Buchholz HG, et al. Multi-centre calibration of an adaptive thresholding method for PET-based delineation of tumour volumes in radiotherapy planning of lung cancer. Nuklearmedizin. 2012;51:101–10. doi: 10.3413/Nukmed-0452-11-12.CrossRefPubMedGoogle Scholar
  35. 35.
    Zadeh LA. Fuzzy sets as the basis for a theory of possibility. Fuzzy Sets Systems. 1978;1:3–28.CrossRefGoogle Scholar
  36. 36.
    Kittler J, Alkoot FM. Sum versus vote fusion in multiple classifier systems. IEEE Trans Pattern Anal Mach Intell. 2003;25:110–5.CrossRefGoogle Scholar
  37. 37.
    Dewalle-Vignion AS, Betrouni N, Baillet C, Vermandel M. Is STAPLE algorithm confident to assess segmentation methods in PET imaging? Phys Med Biol. 2015. In press.Google Scholar
  38. 38.
    Schaefer A, Kim YJ, Kremp S, Mai S, Fleckenstein J, Bohnenberger H, et al. PET-based delineation of tumour volumes in lung cancer: comparison with pathological findings. Eur J Nucl Med Mol Imaging. 2013;40:1233–44. doi: 10.1007/s00259-013-2407-x.CrossRefPubMedGoogle Scholar
  39. 39.
    Hapdey S, Edet-Sanson A, Gouel P, Martin B, Modzelewski R, Baron M, et al. Delineation of small mobile tumours with FDG-PET/CT in comparison to pathology in breast cancer patients. Radiother Oncol. 2014;112:407–12. doi: 10.1016/j.radonc.2014.08.005.CrossRefPubMedGoogle Scholar
  40. 40.
    Zou KH, Warfield SK, Bharatha A, Tempany CM, Kaus MR, Haker SJ, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol. 2004;11:178–89.CrossRefPubMedPubMedCentralGoogle Scholar
  41. 41.
    Bartlett JW, Frost C. Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables. Ultrasound Obstet Gynecol. 2008;31:466–75. doi: 10.1002/uog.5256.CrossRefPubMedGoogle Scholar
  42. 42.
    Aristophanous M, Berbeco RI, Killoran JH, Yap JT, Sher DJ, Allen AM, et al. Clinical utility of 4D FDG-PET/CT scans in radiation treatment planning. Int J Radiat Oncol Biol Phys. 2012;82:e99–105. doi: 10.1016/j.ijrobp.2010.12.060.CrossRefPubMedGoogle Scholar
  43. 43.
    Hatt M, Cheze-le Rest C, van Baardwijk A, Lambin P, Pradier O, Visvikis D. Impact of tumor size and tracer uptake heterogeneity in (18)F-FDG PET and CT non-small cell lung cancer tumor delineation. J Nucl Med. 2011;52:1690–7. doi: 10.2967/jnumed.111.092767.CrossRefPubMedPubMedCentralGoogle Scholar
  44. 44.
    Steenbakkers RJ, Duppen JC, Fitton I, Deurloo KE, Zijp L, Uitterhoeve AL, et al. Observer variation in target volume delineation of lung cancer related to radiation oncologist-computer interaction: a ‘Big Brother’ evaluation. Radiother Oncol. 2005;77:182–90. doi: 10.1016/j.radonc.2005.09.017.CrossRefPubMedGoogle Scholar
  45. 45.
    Erasmus JJ, Gladish GW, Broemeling L, Sabloff BS, Truong MT, Herbst RS, et al. Interobserver and intraobserver variability in measurement of non-small-cell carcinoma lung lesions: implications for assessment of tumor response. J Clin Oncol. 2003;21:2574–82. doi: 10.1200/JCO.2003.01.144.CrossRefPubMedGoogle Scholar
  46. 46.
    McGurk RJ. Consensus segmentation for positron emission tomography: development and applications in radiation therapy. Duke University. 2013.Google Scholar
  47. 47.
    Zaidi H, El Naqa I. PET-guided delineation of radiation therapy treatment volumes: a survey of image segmentation techniques. Eur J Nucl Med Mol Imaging. 2010;37:2165–87. doi: 10.1007/s00259-010-1423-3.CrossRefPubMedGoogle Scholar
  48. 48.
    Kuncheva LI. Combining pattern classifiers: methods and algorithms. Hoboken, NJ: Wiley. 2004.CrossRefGoogle Scholar
  49. 49.
    Azuma A, Tozaki M, Ito K, Fukuma E, Tanaka T, O’Uchi T. Ductal carcinoma in situ: correlation between FDG-PET/CT and histopathology. Radiat Med. 2008;26:488–93. doi: 10.1007/s11604-008-0263-6.CrossRefPubMedGoogle Scholar
  50. 50.
    Hatt M, Cheze le Rest C, Descourt P, Dekker A, De Ruysscher D, Oellers M, et al. Accurate automatic delineation of heterogeneous functional volumes in positron emission tomography for oncology applications. Int J Radiat Oncol Biol Phys. 2010;77:301–8. doi: 10.1016/j.ijrobp.2009.08.018.CrossRefPubMedGoogle Scholar
  51. 51.
    Papadimitroulas P, Loudos G, Le Maitre A, Hatt M, Tixier F, Efthimiou N, et al. Investigation of realistic PET simulations incorporating tumor patient’s specificity using anthropomorphic models: creation of an oncology database. Med Phys. 2013;40:112506. doi: 10.1118/1.4826162.CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • A. Schaefer
    • 1
  • M. Vermandel
    • 2
    • 3
    Email author
  • C. Baillet
    • 3
  • A. S. Dewalle-Vignion
    • 2
  • R. Modzelewski
    • 4
  • P. Vera
    • 4
  • L. Massoptier
    • 5
  • C. Parcq
    • 5
  • D. Gibon
    • 5
  • T. Fechter
    • 6
    • 7
  • U. Nemer
    • 8
  • I. Gardin
    • 4
  • U. Nestle
    • 6
    • 7
  1. 1.Department of Nuclear MedicineSaarland University Medical CentreHomburgGermany
  2. 2.University of Lille, Inserm, CHU LilleU1189 - ONCO-THAI - Image Assisted Laser Therapy for OncologyLilleFrance
  3. 3.Nuclear Medicine DepartmentCHU LilleLilleFrance
  4. 4.Centre Henri-Becquerel and LITIS EA4108RouenFrance
  5. 5.Research and Innovation DepartmentAQUILABLoos Les LilleFrance
  6. 6.Department for Radiation OncologyUniversity Medical Center FreiburgFreiburgGermany
  7. 7.German Cancer Consortium (DKTK) Freiburg and German Cancer Research Center (DKFZ)HeidelbergGermany
  8. 8.Department of Nuclear MedicineUniversity Medical Center FreiburgFreiburgGermany

Personalised recommendations