Abstract
Reproducibility of AI models on biomedical data still stays as a major concern for their acceptance into the clinical practice. Initiatives for reproducibility in the development of predictive biomarkers as the MAQC Consortium already underlined the importance of appropriate Data Analysis Plans (DAPs) to control for different types of bias, including data leakage from the training to the test set. In the context of digital pathology, the leakage typically lurks in weakly designed experiments not accounting for the subjects in their data partitioning schemes. This issue is then exacerbated when fractions or subregions of slides (i.e. “tiles”) are considered. Despite this aspect is largely recognized by the community, we argue that it is often overlooked. In this study, we assess the impact of data leakage on the performance of machine learning models trained and validated on multiple histology data collection. We prove that, even with a properly designed DAP (\(10 \times 5 \) repeated cross-validation), predictive scores can be inflated up to \(41\%\) when tiles from the same subject are used both in training and validation sets by deep learning models. We replicate the experiments for 4 classification tasks on 3 histopathological datasets, for a total of 374 subjects, 556 slides and more than 27, 000 tiles. Also, we discuss the effects of data leakage on transfer learning strategies with models pre-trained on general-purpose datasets or off-task digital pathology collections. Finally, we propose a solution that automates the creation of leakage-free deep learning pipelines for digital pathology based on histolab, a novel Python package for histology data preprocessing. We validate the solution on two public datasets (TCGA and GTEx).
N. Bussola and A. Marcolini—Joint first author.
G. Jurman and C. Furlanello—Joint last author.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
DenseNet-201: \(\sim \)12M parameters; ResNet-152: \(\sim \)6M parameters.
References
Ioannidis, J.P.A., et al.: Repeatability of published microarray gene expression analyses. Nat. Genet. 41(2), 149 (2009)
Iqbal, S.A., et al.: Reproducible research practices and transparency across the biomedical literature. PLoS Biol. 14(1), e1002333 (2016)
National Academies of Sciences, Engineering, and Medicine, Policy and Global Affairs. Reproducibility and Replicability in Science. National Academies Press (2019)
Leek, J.T., et al.: Tackling the widespread and critical impact of batch effects in high-throughput data. Nat. Rev. Genet. 11(10), 733 (2010)
Moossavi, S., et al.: Repeatability and reproducibility assessment in a large-scale population-based microbiota study: case study on human milk microbiota. bioRxiv:2020.04.20.052035 (2020)
Turner, B.O., et al.: Small sample sizes reduce the replicability of task-based fMRI studies. Commun. Biol. 1(1), 1–10 (2018)
Barla, A., et al.: Machine learning methods for predictive proteomics. Briefings Bioinform. 9(2), 119–128 (2008)
Peixoto, L., et al.: How data analysis affects power, reproducibility and biological insight of RNA-seq studies in complex datasets. Nucleic Acids Res. 43(16), 7664–7674 (2015)
The MAQC Consortium: The MAQC-II project: a comprehensive study of common practices for the development and validation of microarray-based predictive models. Nat. Biotechnol. 28(8), 827–838 (2010)
Ching, T., et al.: Opportunities and obstacles for deep learning in biology and medicine. J. Roy. Soc. Interface 15(141), 20170387 (2018)
Saravanan, N., et al.: Data wrangling and data leakage in machine learning for healthcare. Int. J. Emerg. Technol. Innov. Res. 5(8), 553–557 (2018)
Spanhol, F.A., et al.: A dataset for breast cancer histopathological image classification. IEEE Trans. Biomed. Eng. 63(7), 1455–1462 (2016)
Shahidi, F., et al.: Breast cancer classification using deep learning approaches and histopathology image: a comparison study. IEEE Access 8, 187531–187552 (2020)
Cohen, S.: Artificial Intelligence and Deep Learning in Pathology. Elsevier, Amsterdam (2020)
Komura, D., et al.: Machine learning methods for histopathological image analysis. Comput. Struct. Biotechnol. J. 16, 34–42 (2018)
Mormont, R., et al.: Comparison of deep transfer learning strategies for digital pathology. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2343–234309. IEEE (2018)
Marée, R.: The need for careful data collection for pattern recognition in digital pathology. J. Pathol. Inform. 8(1), 19 (2017)
Bizzego, A., et al.: Evaluating reproducibility of AI algorithms in digital pathology with DAPPER. PLOS Comput. Biol. 15(3), 1–24 (2019)
Huang, G., et al.: Densely connected convolutional networks. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269. IEEE (2018)
He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE (2016)
Barisoni, L., et al.: Digital pathology and computational image analysis in nephropathology. Nat. Rev. Nephrol. 16, 669–685 (2020)
The GTEx Consortium: The genotype-tissue expression (GTEx) project. Nat. Genet. 45(6), 580–585 (2013)
Tomczak, K., et al.: The cancer genome atlas (TCGA): an immeasurable source of knowledge. Contemp. Oncol. 19(1A), A68 (2015)
Nirschl, J.J., et al.: A deep-learning classifier identifies patients with clinical heart failure using whole-slide images of H&E tissue. PLOS ONE 13(4), e0192726 (2018)
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)
Raschka, S.: Model evaluation, model selection, and algorithm selection in machine learning. arXiv:1811.12808v3 (2020)
Kingma, D.P., et al.: Adam: a method for stochastic optimization. In: Published as a conference paper at ICLR 2015. arXiv:1412.6980 (2014)
Jurman, G., et al.: A comparison of MCC and CEN error measures in multi-class prediction. PLOS ONE 7(8), 1–8 (2012)
Alom, M.Z., Yakopcic, C., Nasrin, M.S., Taha, T.M., Asari, V.K.: Breast cancer classification from histopathological images with inception recurrent residual convolutional neural network. J. Digital Imaging 32(4), 605–617 (2019). https://doi.org/10.1007/s10278-019-00182-7
Jiang, Y., et al.: Breast cancer histopathological image classification using convolutional neural networks with small SE-ResNet module. PLOS ONE 14(3), e0214587 (2019)
Jannesari, M., et al.: Breast cancer histopathological image classification: a deep learning approach. In: Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 2405–2412 (2018)
Xie, J., et al.: Deep learning based analysis of histopathological images of breast cancer. Front. Genet. 10, 80 (2019)
Nawaz, M., et al.: Multi-class breast cancer classification using deep learning convolutional neural network. Int. J. Adv. Comput. Sci. Appl. 9(6), 316–332 (2018)
Han, Z., et al.: Breast cancer multi-classification from histopathological images with structured deep learning model. Sci. Rep. 7(1), 4172 (2017)
Alom, M.J., et al.: Advanced deep convolutional neural network approaches for digital pathology image analysis: a comprehensive evaluation with different use cases. arXiv:1904.09075 (2019)
Nguyen, P.T., et al.: Multiclass breast cancer classification using convolutional neural network. In: Proceedings of the 2019 International Symposium on Electrical and Electronics Engineering (ISEE), pp. 130–134. IEEE (2019)
Deniz, E., Şengür, A., Kadiroğlu, Z., Guo, Y., Bajaj, V., Budak, Ü.: Transfer learning based histopathologic image classification for breast cancer detection. Health Inf. Sci. Syst. 6(1), 1–7 (2018). https://doi.org/10.1007/s13755-018-0057-x
Myung, J.L., et al.: Deep convolution neural networks for medical image analysis. Int. J. Eng. Technol. 7(3), 115–119 (2018)
Pan, X., et al.: Multi-task deep learning for fine-grained classification/grading in breast cancer histopathological images. In: Lu, H. (ed.) ISAIR 2018. SCI, vol. 810, pp. 85–95. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-04946-1_10
Shallu, R.M.: Breast cancer histology images classification: training from scratch or transfer learning? ICT Exp. 4(4), 247–254 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Bussola, N., Marcolini, A., Maggio, V., Jurman, G., Furlanello, C. (2021). AI Slipping on Tiles: Data Leakage in Digital Pathology. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12661. Springer, Cham. https://doi.org/10.1007/978-3-030-68763-2_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-68763-2_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68762-5
Online ISBN: 978-3-030-68763-2
eBook Packages: Computer ScienceComputer Science (R0)