Advertisement

How to Exploit Weaknesses in Biomedical Challenge Design and Organization

  • Annika ReinkeEmail author
  • Matthias Eisenmann
  • Sinan Onogur
  • Marko Stankovic
  • Patrick Scholz
  • Peter M. Full
  • Hrvoje Bogunovic
  • Bennett A. Landman
  • Oskar Maier
  • Bjoern Menze
  • Gregory C. Sharp
  • Korsuk Sirinukunwattana
  • Stefanie Speidel
  • Fons van der Sommen
  • Guoyan Zheng
  • Henning Müller
  • Michal Kozubek
  • Tal Arbel
  • Andrew P. Bradley
  • Pierre Jannin
  • Annette Kopp-Schneider
  • Lena Maier-HeinEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11073)

Abstract

Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

Notes

Acknowledgments

We thank all of the organizers of the 2015 segmentation challenges who are not co-authoring this paper. We further thank A. Laha, D. Mindroc-Filimon, B. Pekdemir and J. Yoganathan (DKFZ, Germany) for helping with the comprehensive challenge capturing. Finally, we acknowledge support from the European Union through the ERC starting grant COMBIOSCOPY under the New Horizon Framework Programme under grant agreement ERC-2015-StG-37960.

References

  1. 1.
    Boettiger, C.: An introduction to docker for reproducible research. ACM SIGOPS Oper. Syst. Rev. 49(1), 71–79 (2015).  https://doi.org/10.1145/2723872.2723882CrossRefGoogle Scholar
  2. 2.
    Carass, A., et al.: Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage 148, 77–102 (2017).  https://doi.org/10.1016/j.neuroimage.2016.12.064CrossRefGoogle Scholar
  3. 3.
    Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A.: The reusable holdout: preserving validity in adaptive data analysis. Science 349(6248), 636–638 (2015).  https://doi.org/10.1126/science.aaa9375MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    van Ginneken, B., Heimann, T., Styner, M.: 3D Segmentation in the Clinic: A Grand Challenge, pp. 7–15 (2007)Google Scholar
  5. 5.
    Ioannidis, J.P.: Why most published research findings are false. PLoS Med. 2(8), e124 (2005).  https://doi.org/10.1371/journal.pmed.0020124CrossRefGoogle Scholar
  6. 6.
    Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)CrossRefGoogle Scholar
  7. 7.
    Maier, O., et al.: ISLES 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Med. Image Anal. 35, 250–269 (2017).  https://doi.org/10.1016/j.media.2016.07.009CrossRefGoogle Scholar
  8. 8.
    Maška, M., et al.: A benchmark for comparison of cell tracking algorithms. Bioinformatics 30(11), 1609–1617 (2014).  https://doi.org/10.1093/bioinformatics/btu080CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Annika Reinke
    • 1
    Email author
  • Matthias Eisenmann
    • 1
  • Sinan Onogur
    • 1
  • Marko Stankovic
    • 1
  • Patrick Scholz
    • 1
  • Peter M. Full
    • 1
  • Hrvoje Bogunovic
    • 2
  • Bennett A. Landman
    • 3
  • Oskar Maier
    • 4
  • Bjoern Menze
    • 5
  • Gregory C. Sharp
    • 6
  • Korsuk Sirinukunwattana
    • 7
  • Stefanie Speidel
    • 8
  • Fons van der Sommen
    • 9
  • Guoyan Zheng
    • 10
  • Henning Müller
    • 11
  • Michal Kozubek
    • 12
  • Tal Arbel
    • 13
  • Andrew P. Bradley
    • 14
  • Pierre Jannin
    • 15
  • Annette Kopp-Schneider
    • 16
  • Lena Maier-Hein
    • 1
    Email author
  1. 1.Division Computer Assisted Medical Interventions (CAMI)German Cancer Research Center (DKFZ)HeidelbergGermany
  2. 2.Christian Doppler Laboratory for Ophthalmic Image Analysis, Department of OphthalmologyMedical University ViennaViennaAustria
  3. 3.Electrical EngineeringVanderbilt UniversityNashvilleUSA
  4. 4.Institute Medical InformaticsUniversity of LübeckLübeckGermany
  5. 5.Institute Advanced Studies, Department of InformaticsTechnical University of MunichMunichGermany
  6. 6.Department Radiation OncologyMassachusetts General HospitalBostonUSA
  7. 7.Institute Biomedical EngineeringUniversity of OxfordOxfordUK
  8. 8.Division Translational Surgical Oncology (TCO)National Center for Tumor Diseases DresdenDresdenGermany
  9. 9.Department Electrical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
  10. 10.Institute Surgical Technology and BiomechanicsUniversity of BernBernSwitzerland
  11. 11.Information System Institute, HES-SOSierreSwitzerland
  12. 12.Centre for Biomedical Image AnalysisMasaryk UniversityBrnoCzech Republic
  13. 13.Department of Electrical and Computer EngineeringMcGill UniversityMontrealCanada
  14. 14.Science and Engineering FacultyQueensland University of TechnologyBrisbaneAustralia
  15. 15.Laboratoire du Traitement du Signal et de l’Image, INSERMUniversity of Rennes 1RennesFrance
  16. 16.Division BiostatisticsGerman Cancer Research Center (DKFZ)HeidelbergGermany

Personalised recommendations