How to Exploit Weaknesses in Biomedical Challenge Design and Organization
Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.
KeywordsSegmentation Challenge Challenge Organizers MICCAI Challenge MICCAI Conference Rank Aggregation Methods
We thank all of the organizers of the 2015 segmentation challenges who are not co-authoring this paper. We further thank A. Laha, D. Mindroc-Filimon, B. Pekdemir and J. Yoganathan (DKFZ, Germany) for helping with the comprehensive challenge capturing. Finally, we acknowledge support from the European Union through the ERC starting grant COMBIOSCOPY under the New Horizon Framework Programme under grant agreement ERC-2015-StG-37960.
- 4.van Ginneken, B., Heimann, T., Styner, M.: 3D Segmentation in the Clinic: A Grand Challenge, pp. 7–15 (2007)Google Scholar