Skip to main content

Overview of the HECKTOR Challenge at MICCAI 2020: Automatic Head and Neck Tumor Segmentation in PET/CT

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12603)

Abstract

This paper presents an overview of the first HEad and neCK TumOR (HECKTOR) challenge, organized as a satellite event of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2020. The task of the challenge is the automatic segmentation of head and neck primary Gross Tumor Volume in FDG-PET/CT images, focusing on the oropharynx region. The data were collected from five centers for a total of 254 images, split into 201 training and 53 testing cases. The interest in the task was shown by the important participation with 64 teams registered and 18 team submissions. The best method obtained a Dice Similarity Coefficient (DSC) of 0.7591, showing a large improvement over our proposed baseline method with a DSC of 0.6610 as well as inter-observer DSC agreement reported in the literature (0.69).

Keywords

  • Automatic segmentation
  • Challenge
  • Medical imaging
  • Head and neck cancer
  • Oropharynx

V. Andrearczyk and V. Oreiller—Equal contribution.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-67194-5_1
  • Chapter length: 21 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-67194-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   74.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.

Notes

  1. 1.

    www.aicrowd.com/challenges/hecktor, as of October 2020.

  2. 2.

    https://portal.fli-iam.irisa.fr/petseg-challenge/overview#_ftn1, as of October 2020.

  3. 3.

    The target cohort refers to the subjects from whom the data would be acquired in the final biomedical application. It is mentioned for additional information as suggested in BIAS, although all data provided for the challenge are part of the challenge cohort.

  4. 4.

    The challenge cohort refers to the subjects from whom the challenge data were acquired.

  5. 5.

    For simplicity, these centers were renamed CHGJ and CHMR during the challenge.

  6. 6.

    github.com/voreille/hecktor, as of October 2020.

  7. 7.

    github.com/voreille/hecktor/tree/master/src/evaluation, as of October 2020.

  8. 8.

    These values are reported only to give an idea of inter-observer variability on a similar task reported in the literature. The datasets are different and the comparison is limited. In future work, we will compute the inter-observer agreement on the challenge data.

  9. 9.

    www.aicrowd.com/challenges/miccai-2020-hecktor/leaderboards.

  10. 10.

    www.aicrowd.com/challenges/hecktor.

  11. 11.

    www.aicrowd.com/challenges/hecktor#results-submission%20format.

  12. 12.

    github.com/voreille/hecktor/tree/master/src/evaluation.

References

  1. Andrearczyk, V., Oreiller, V., Depeursinge, A.: Oropharynx detection in PET-CT for tumor segmentation. In: Irish Machine Vision and Image Processing (2020)

    Google Scholar 

  2. Andrearczyk, V., et al.: Automatic segmentation of head and neck tumors and nodal metastases in PET-CT scans. In: International Conference on Medical Imaging with Deep Learning (MIDL) (2020)

    Google Scholar 

  3. Bogowicz, M., Tanadini-Lang, S., Guckenberger, M., Riesterer, O.: Combined CT radiomics of primary tumor and metastatic lymph nodes improves prediction of loco-regional control in head and neck cancer. Sci. Rep. 9(1), 1–7 (2019)

    Google Scholar 

  4. Chen, H., Chen, H., Wang, L.: Iteratively refine the segmentation of head and neck tumor in FDG-PET and CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 53–58. Springer, Cham (2021)

    Google Scholar 

  5. Foster, B., Bagci, U., Mansoor, A., Xu, Z., Mollura, D.J.: A review on segmentation of positron emission tomography images. Comput. Biol. Med. 50, 76–96 (2014)

    Google Scholar 

  6. Ghimire, K., Chen, Q., Feng, X.: Patch-based 3D UNet for head and neck tumor segmentation with an ensemble of conventional and dilated convolutions. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 78–84. Springer, Cham (2021)

    Google Scholar 

  7. Gudi, S., et al.: Interobserver variability in the delineation of gross tumour volume and specified organs-at-risk during IMRT for head and neck cancers and the impact of FDG-PET/CT on such variability at the primary site. J. Med. Imaging Radiat. Sci. 48(2), 184–192 (2017)

    Google Scholar 

  8. Hatt, M., et al.: The first MICCAI challenge on PET tumor segmentation. Med. Image Anal. 44, 177–195 (2018)

    Google Scholar 

  9. Hatt, M., et al.: Classification and evaluation strategies of auto-segmentation approaches for PET: report of AAPM task group no. 211. Med. Phys. 44(6), e1–e42 (2017)

    Google Scholar 

  10. Iantsen, A., Visvikis, D., Hatt, M.: Squeeze-and-excitation normalization for automated delineation of head and neck primary tumors in combined PET and CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 37–43. Springer, Cham (2021)

    Google Scholar 

  11. Kumar, A., Fulham, M., Feng, D., Kim, J.: Co-learning feature fusion maps from PET-CT images of lung cancer. IEEE Trans. Med. Imaging 39(1), 204–217 (2019)

    Google Scholar 

  12. Li, L., Zhao, X., Lu, W., Tan, S.: Deep learning for variational multimodality tumor segmentation in PET/CT. Neurocomputing 392, 277–295 (2019)

    Google Scholar 

  13. Ma, J., Yang, X.: Combining CNN and hybrid active contours for head and neck tumor segmentation in CT and PET Images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 59–64. Springer, Cham (2021)

    Google Scholar 

  14. Maier-Hein, L., et al.: Why rankings of biomedical image analysis competitions should be interpreted with care. Nat. Commun. 9(1), 1–13 (2018)

    Google Scholar 

  15. Maier-Hein, L., et al.: BIAS: transparent reporting of biomedical image analysis challenges. Med. Image Anal. 66, 101796 (2020)

    Google Scholar 

  16. Moe, Y.M., et al.: Deep learning for automatic tumour segmentation in PET/CT images of patients with head and neck cancers. In: Medical Imaging with Deep Learning (2019)

    Google Scholar 

  17. Naser, M.A., van Dijk, L.V., He, R., Wahid, K.A., Fuller, C.D.: Tumor segmentation in patients with head and neck cancers using deep learning based-on multi-modality PET/CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 85–98. Springer, Cham (2021)

    Google Scholar 

  18. Rao, C., et al.: Oropharyngeal Tumour Segmentation using Ensemble 3D PET-CT Fusion Networks for the HECKTOR Challenge. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 65–77. Springer, Cham (2021)

    Google Scholar 

  19. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    CrossRef  Google Scholar 

  20. Vallieres, M., et al.: Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci. Rep. 7(1), 1–14 (2017)

    Google Scholar 

  21. Xie, J., Peng, Y.: The head and neck tumor segmentation using nnU-Net with spatial and channel ‘squeeze & excitation’ blocks. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 28–36. Springer, Cham (2021)

    Google Scholar 

  22. Xu, L., et al.: Automated whole-body bone lesion detection for multiple myeloma on 68Ga-pentixafor PET/CT imaging using deep learning methods. Contrast Media Mol. Imaging 2018, 11 (2018). https://doi.org/10.1155/2018/2391925

  23. Yousefirizi, F., Rahmim, A.: GAN-based bi-modal segmentation using mumford-shah loss: Application to head and neck tumors in PET-CT images. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 99–108. Springer, Cham (2021)

    Google Scholar 

  24. Yuan, Y.: Automatic head and neck tumor segmentation in PET/CT with scale attention network. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 44–52. Springer, Cham (2021)

    Google Scholar 

  25. Zhao, X., Li, L., Lu, W., Tan, S.: Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys. Med. Biol. 64(1), 015011 (2018)

    Google Scholar 

  26. Zhong, Z., et al.: 3D fully convolutional networks for co-segmentation of tumors on PET-CT images. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 228–231. IEEE (2018)

    Google Scholar 

  27. Zhu, S., Dai, Z., Ning, W.: Two-stage approach for segmenting gross tumor volume in head and neck cancer with CT and PET imaging. In: Andrearczyk, V., et al. (eds.) HECKTOR 2020. LNCS, vol. 12603, pp. 22–27. Springer, Cham (2021)

    Google Scholar 

Download references

Acknowledgments

The organizers thank all the teams for their participation and valuable work. This challenge and the winner prize are sponsored by Siemens Healthineers Switzerland. This work was also partially supported by the Swiss National Science Foundation (SNSF, grant 205320_179069) and the Swiss Personalized Health Network (SPHN, via the IMAGINE and QA4IQI projects).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Vincent Andrearczyk .

Editor information

Editors and Affiliations

Appendices

Appendix 1: Challenge Information

In this appendix, we list important information about the challenge as suggested in the BIAS guidelines [15].

Challenge Name

HEad and neCK TumOR segmentation challenge (HECKTOR) 2020.

Organizing Team

(Authors of this paper) Vincent Andrearczyk, Valentin Oreiller, Martin Vallières, Joel Castelli, Mario Jreige, John O. Prior and Adrien Depeursinge.

Life Cycle Type

A fixed submission deadline was set for the challenge results. Open online leaderboard following the conference.

Challenge Venue and Platform

The challenge is associated with MICCAI 2020. Information on the challenge is available on the website, together with the link to download the data, the submission platform and the leaderboardFootnote 10.

Participation Policies

  1. (a)

    Algorithms producing fully-automatic segmentation of the test cases were allowed.

  2. (b)

    The data used to train algorithms was not restricted. If using external data (private or public), participants were asked to also report results using only the HECKTOR data.

  3. (c)

    Members of the organizers’ institutes could participate in the challenge but were not eligible for awards.

  4. (d)

    The award was 500 euros, sponsored by Siemens Healthineers Switzerland.

  5. (e)

    Policy for results announcement: The results were made available on the AIcrowd leaderboard and the best three results were announced publicly. Once participants submitted their results on the test set to the challenge organizers via the challenge website, they were considered fully vested in the challenge, so that their performance results (without identifying the participant unless permission is granted) became part of any presentations, publications, or subsequent analyses derived from the challenge at the discretion of the organizers.

  6. (f)

    Publication policy: This overview paper was written by the organizing team’s members. The participating teams were encouraged to submit a paper describing their method. The participants can publish their results separately elsewhere when citing the overview paper, and (if so) no embargo will be applied.

Submission Method

Submission instructions are available on the websiteFootnote 11 and are reported in the following. Results should be provided as a single binary mask (1 in the predicted GTVt) .nii.gz file per patient in the CT original resolution and cropped using the provided bounding boxes. The participants should pay attention to saving NIfTI volumes with the correct pixel spacing and origin with respect to the original reference frame. The .nii files should be named [PatientID].nii.gz, matching the patients’ file names, e.g.. CHUV001.nii.gz and placed in a folder. This folder should be zipped before submission. If results were submitted without cropping and/or resampling, we employed nearest-neighbor interpolation given that the coordinate system is provided. Participants were allowed five valid submissions. The best result was reported for each team.

Challenge Schedule

The schedule of the challenge, including modifications, is reported in the following.

  • the release date of the training cases: June 01 2020 June 10 2020

  • the release date of the test cases: Aug. 01 2020

  • the results submission date(s): opens Sept. 01 2020 closes Sept. 10 2020

  • paper submission deadline: Sept. 18 2020 Sept. 15 2020

  • the release date of the results: Sept. 15 2020

  • associated workshop days: Oct. 04 2020, 9:00-13:00 UTC

Ethics Approval

Training dataset: The ethics approval was granted by the Research Ethics Committee of McGill University Health Center (Protocol Number: MM-JGH-CR15-50). Test dataset: The ethics approval was obtained from the Commission cantonale (VD) d’éthique de la recherche sur l’étre humain (CER-VD) with protocol number: 2018-01513.

Data Usage Agreement

The participants had to fill out and sign an end-user-agreement in order to be granted access to the data. The form can be found under the Resources tab of the HECKTOR website.

Code Availability

The evaluation software was made available on our github pageFootnote 12. The participating teams decided whether they wanted to disclose their code (they were encouraged to do so).

Conflict of Interest

No conflict of interest applies. Fundings are specified in the acknowledgments. Only the organizers had access to the test cases ground truth contours.

Author Contributions

Vincent Andrearczyk

Design of the task and of the challenge, writing of the proposal, development of baseline algorithms, development of the AIcrowd website, writing of the overview paper, organization of the challenge event, organization of the submission and reviewing process of the participants’ papers.

Valentin Oreiller

Design of the task and of the challenge, writing of the proposal, development of the AIcrowd website, development of the evaluation code, writing of the overview paper, organization of the challenge event, organization of the submission and reviewing process of the papers.

Mario Jreige

Design of the task and of the challenge, quality control/annotations, annotations for inter-annotator agreement, revision of the paper and accepted the last version of the submitted paper.

Martin Vallières

Design of the task and of the challenge, provided the initial data and annotations for the training set [20], revision of the paper and accepted the last version of the submitted paper.

Joel Castelli

Design of the task and of the challenge, annotations for inter-annotator agreement.

Hesham Elhalawani

Design of the task and of the challenge, annotations for inter-annotator agreement.

Sarah Boughdad

Design of the task and of the challenge, annotations for inter-annotator agreement.

John O. Prior

Design of the task and of the challenge, revision of the paper and accepted the last version of the submitted paper.

Adrien Depeursinge

Design of the task and of the challenge, writing of the proposal, writing of the overview paper, organization of the challenge event.

Appendix 2: Image Acquisition Details

HGJ: All patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (Discovery ST, GE Healthcare) within 37 days before treatment (median: 14 days). For the PET portion of the FDG-PET/CT scan, a median of 584 MBq (range: 368–715) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 300 s (range: 180–420) per bed position. Attenuation corrected images were reconstructed using an ordered subset expectation maximization (OSEM) iterative algorithm and a span (axial mash) of 5. The FDG-PET slice thickness resolution was 3.27 mm for all patients and the median in-plane resolution was 3.52 \(\times \) 3.52 mm\(^2\) (range: 3.52–4.69). For the CT portion of the FDG-PET/CT scan, an energy of 140 kVp with an exposure of 12 mAs was used. The CT slice thickness resolution was 3.75 mm and the median in-plane resolution was 0.98 \(\times \) 0.98 mm\(^2\) for all patients.

CHUS: All 102 eligible patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (GeminiGXL 16, Philips) within 54 days before treatment (median: 19 days). For the PET portion of the FDG-PET/CT scan, a median of 325 MBq (range: 165–517) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 150 s (range: 120–151) per bed position. Attenuation corrected images were reconstructed using a LOR-RAMLA iterative algorithm. The FDG-PET slice thickness resolution was 4 mm and the median in-plane resolution was 4 \(\times \) 4 mm\(^2\) for all patients. For the CT portion of the FDG-PET/CT scan, a median energy of 140 kVp (range: 12–140) with a median exposure of 210 mAs (range: 43–250) was used. The median CT slice thickness resolution was 3 mm (range: 2–5) and the median in-plane resolution was 1.17 \(\times \) 1.17 mm\(^2\) (range: 0.68–1.17).

HMR: All patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (Discovery STE, GE Healthcare) within 60 days before treatment (median: 34 days). For the PET portion of the FDG-PET/CT scan, a median of 475 MBq (range: 227–859) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 360 s (range: 120–360) per bed position. Attenuation corrected images were reconstructed using an ordered subset expectation maximization (OSEM) iterative algorithm and a median span (axial mash) of 5 (range: 3–5). The FDG-PET slice thickness resolution was 3.27 mm for all patients and the median in-plane resolution was 3.52 \(\times \) 3.52 mm\(^2\) (range: 3.52–5.47). For the CT portion of the FDG-PET/CT scan, a median energy of 140 kVp (range: 120–140) with a median exposure of 11 mAs (range: 5–16) was used. The CT slice thickness resolution was 3.75 mm for all patients and the median in-plane resolution was 0.98 \(\times \) 0.98 mm\(^2\) (range: 0.98–1.37).

CHUM: All patients had FDG-PET and CT scans done on a hybrid PET/CT scanner (Discovery STE, GE Healthcare) within 66 days before treatment (median: 12 days). For the PET portion of the FDG-PET/CT scan, a median of 315 MBq (range: 199–3182) was injected intravenously. Imaging acquisition of the head and neck was performed using multiple bed positions with a median of 300 s (range: 120–420) per bed position. Attenuation corrected images were reconstructed using an ordered subset expectation maximization (OSEM) iterative algorithm and a medianspan (axial mash) of 3 (range: 3–5). The median FDG-PET slice thickness resolution was 4 mm (range: 3.27–4) and the median in-plane resolution was 4 \(\times \) 4 mm\(^2\) (range: 3.52–5.47). For the CT portion of the FDG-PET/CT scan, a median energy of 120 kVp (range: 120–140) with a median exposure of 350 mAs (range: 5–350) was used. The median CT slice thickness resolution was 1.5 mm (range: 1.5–3.75) and the median in-plane resolution was 0.98 \(\times \) 0.98 mm\(^2\) (range: 0.98–1.37). All patients received their FDG-PET/CT scan dedicated to the head and neck area right before their planning CT scan, in the same position with the immobilization device.

CHUV (test): All patients underwent FDG PET/CT for staging before treatment. Blood glucose levels were checked before the injection of (18F)-FDG. After a 60-min uptake period of rest, patients were imaged with the Discovery D690 TOF PET/CT (General Electric Healthcare, Milwaukee, WI, USA). First, a CT (120 kV, 80 mA, 0.8-s rotation time, slice thickness 3.75 mm) was performed from the base of the skull to the mid-thigh. PET scanning was performed immediately after acquisition of the CT. Images were acquired from the base of the skull to the mid-thigh (2 min/bed position). PET images were reconstructed after time-of-flight and point-spread-function recovery corrections by using an ordered-subset expectation maximization iterative reconstruction (OSEM) (two iterations, 28 subsets) and an iterative fully 3D (Discovery ST). CT data were used for attenuation calculation.

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Andrearczyk, V. et al. (2021). Overview of the HECKTOR Challenge at MICCAI 2020: Automatic Head and Neck Tumor Segmentation in PET/CT. In: Andrearczyk, V., Oreiller, V., Depeursinge, A. (eds) Head and Neck Tumor Segmentation. HECKTOR 2020. Lecture Notes in Computer Science(), vol 12603. Springer, Cham. https://doi.org/10.1007/978-3-030-67194-5_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67194-5_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67193-8

  • Online ISBN: 978-3-030-67194-5

  • eBook Packages: Computer ScienceComputer Science (R0)