Abstract
Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high interpathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work we evaluated the benefit of multi-expert consensus (n = 3, 5, 7, 9, 11) on algorithmic performance. While training with individual databases resulted in highly variable F1 scores, performance was notably increased and more consistent when using the consensus of three annotators. Adding more annotators only resulted in minor improvements. We conclude that databases by few pathologists and high label precision may be the best compromise between high algorithmic performance and time investment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Aubreville M, Bertram CA, Marzahl C, et al. Deep learning algorithms out-perform veterinary pathologists in detecting the mitotically most active tumor region. Sci Rep. 2020;10(16447):1–11.
Bertram CA, Veta M, Marzahl C, et al. Are pathologist-defined labels reproducible? Comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels. In: Interpretable and Annotation-Efficient Learning for Medical Image Computing. IMIMIC 2020, MIL3ID 2020, LABELS 2020. Lecture Notes in Computer Science. Springer; 2020. p. 204–213.
Veta M, Van Diest PJ, Willems SM, et al. Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal. 2015;20(1):237–248.
Veta M, Heng YJ, Stathonikos N, et al. Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge. Med Image Anal. 2019;54:111–121.
Roux L, Racoceanu D, Capron F, et al. MITOS & ATYPIA-detection of mitosis and evaluation of nuclear atypia score in breast cancer histological images. IPAL, Agency Sci. Technol Res Inst Infocom Res, Singapore, Tech Rep. 2014;.
Bertram CA, Aubreville M, Marzahl C, et al. A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Scientific Data. 2019;6(1):19.
Veta M, Diest PJV, Jiwa M, et al. Mitosis counting in breast cancer: objectlevel interobserver agreement and comparison to an automatic method. PloS one. 2016;11(8).
Tellez D, Balkenhol M, Otte-Höller I, et al. Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans Med Imaging. 2018;37(9):2126–2136.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Der/die Autor(en), exklusiv lizenziert durch Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature
About this paper
Cite this paper
Wilm, F. et al. (2021). Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment. In: Palm, C., Deserno, T.M., Handels, H., Maier, A., Maier-Hein, K., Tolxdorff, T. (eds) Bildverarbeitung für die Medizin 2021. Informatik aktuell. Springer Vieweg, Wiesbaden. https://doi.org/10.1007/978-3-658-33198-6_56
Download citation
DOI: https://doi.org/10.1007/978-3-658-33198-6_56
Published:
Publisher Name: Springer Vieweg, Wiesbaden
Print ISBN: 978-3-658-33197-9
Online ISBN: 978-3-658-33198-6
eBook Packages: Computer Science and Engineering (German Language)