Abstract
Machine learning techniques, especially deep learning techniques such as convolutional neural networks, have been successfully applied to general image recognitions since their overwhelming performance at the 2012 ImageNet Large Scale Visual Recognition Challenge. Recently, such techniques have also been applied to various medical, including histopathological, images to assist the process of medical diagnosis. In some cases, deep learning–based algorithms have already outperformed experienced pathologists for recognition of histopathological images. However, pathological images differ from general images in some aspects, and thus, machine learning of histopathological images requires specialized learning methods. Moreover, many pathologists are skeptical about the ability of deep learning technology to accurately recognize histopathological images because what the learned neural network recognizes is often indecipherable to humans. In this review, we first introduce various applications incorporating machine learning developed to assist the process of pathologic diagnosis, and then describe machine learning problems related to histopathological image analysis, and review potential ways to solve these problems.
Similar content being viewed by others
References
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115:211–252. https://doi.org/10.1007/s11263-015-0816-y
Bejnordi BE, Veta M, van DPJ et al (2017) Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318:2199–2210. https://doi.org/10.1001/jama.2017.14585
An augmented reality microscope for cancer detection. In: Google AI Blog. http://ai.googleblog.com/2018/04/an-augmented-reality-microscope.html. Accessed 13 Aug 2018
Ciompi F, Geessink O, Bejnordi BE, et al (2017) The importance of stain normalization in colorectal tissue classification with convolutional networks. arXiv:170205931 [cs]
Bejnordi BE, Litjens G, Timofeeva N et al (2016) Stain specific standardization of whole-slide histopathological images. IEEE Trans Med Imaging 35:404–415. https://doi.org/10.1109/TMI.2015.2476509
Khan AM, Rajpoot N, Treanor D, Magee D (2014) A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans Biomed Eng 61:1729–1738. https://doi.org/10.1109/TBME.2014.2303294
Li X, Plataniotis KN (2015) A complete color normalization approach to histopathology images using color cues computed from saturation-weighted statistics. IEEE Trans Biomed Eng 62:1862–1873. https://doi.org/10.1109/TBME.2015.2405791
Sethi A, Sha L, Vahadane AR, Deaton RJ, Kumar N, Macias V, Gann PH (2016) Empirical comparison of color normalization methods for epithelial-stromal classification in H and E images. J Pathol Inform 7:17. https://doi.org/10.4103/2153-3539.179984
Selvaraju RR, Cogswell M, Das A, et al (2016) Grad-CAM: visual explanations from deep networks via gradient-based localization. arXiv:161002391 [cs]
Koh PW, Liang P (2017) Understanding black-box predictions via influence functions. arXiv:170304730 [cs, stat]
Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: the all convolutional net. arXiv:14126806 [cs]
Bayramoglu N, Kannala J, Heikkilä J (2016) Deep learning for magnification independent breast cancer histopathology image classification. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp 2440–2445
Hou L, Nguyen V, Samaras D, et al (2017) Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images. arXiv:170400406 [cs]
Transfer learning for cell nuclei classification in histopathology images | SpringerLink. https://link.springer.com/chapter/10.1007/978-3-319-49409-8_46. Accessed 22 Nov 2017
Xing F, Yang L (2016) Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review. IEEE Rev Biomed Eng 9:234–263. https://doi.org/10.1109/RBME.2016.2515127
Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Naour GL, Gurcan MN (2013) Mitosis detection in breast cancer histological images an ICPR 2012 contest. J Pathol Inform 4:8. https://doi.org/10.4103/2153-3539.112693
Mitosis detection in breast cancer histological images. http://ludo17.free.fr/mitos_2012/index.html. Accessed 29 Nov 2017
Veta M, van Diest PJ, Willems SM, Wang H, Madabhushi A, Cruz-Roa A, Gonzalez F, Larsen ABL, Vestergaard JS, Dahl AB, Cireşan DC, Schmidhuber J, Giusti A, Gambardella LM, Tek FB, Walter T, Wang CW, Kondo S, Matuszewski BJ, Precioso F, Snell V, Kittler J, de Campos TE, Khan AM, Rajpoot NM, Arkoumani E, Lacle MM, Viergever MA, Pluim JPW (2015) Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 20:237–248. https://doi.org/10.1016/j.media.2014.11.010
Chen H, Qi X, Yu L, Heng PA (2016) DCAN: deep contour-aware networks for accurate gland segmentation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2487–2496
Sirinukunwattana K, Pluim JPW, Chen H, Qi X, Heng PA, Guo YB, Wang LY, Matuszewski BJ, Bruni E, Sanchez U, Böhm A, Ronneberger O, Cheikh BB, Racoceanu D, Kainz P, Pfeiffer M, Urschler M, Snead DRJ, Rajpoot NM (2017) Gland segmentation in colon histology images: the glas challenge contest. Med Image Anal 35:489–502. https://doi.org/10.1016/j.media.2016.08.008
Kather JN, Weis C-A (2016) Validation data set for automatic blood vessel segmentation in colorectal cancer histology (IHC)
Gupta V, Bhavsar A (2017) Breast cancer histopathological image classification: is magnification important? In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 769–776
Araújo T, Aresta G, Castro E, Rouco J, Aguiar P, Eloy C, Polónia A, Campilho A (2017) Classification of breast cancer histology images using convolutional neural networks. PLoS One 12:e0177544. https://doi.org/10.1371/journal.pone.0177544
CAMELYON17. https://camelyon17.grand-challenge.org/. Accessed 21 Aug 2017
Liu J, Xu B, Zheng C, et al (2018) An end-to-end deep learning histochemical scoring system for breast cancer tissue microarray. arXiv:180106288 [cs]
HALO AI – Indica Labs. http://www.indicalab.com/halo-ai/. Accessed 20 Dec 2018
PAIGE. https://www.paigeai.com/. Accessed 20 Dec 2018
PathAI. https://www.pathai.com/. Accessed 20 Dec 2018
Proscia. https://proscia.com/. Accessed 20 Dec 2018
Contextvision. http://www.contextvision.com/pathology/. Accessed 20 Dec 2018
Luigi: large-scale histopathological image retrieval system using deep texture representations | bioRxiv. https://www.biorxiv.org/content/early/2018/07/19/345785. Accessed 13 Aug 2018
Caicedo JC, González FA, Romero E (2011) Content-based histopathology image retrieval using a kernel-based semantic annotation framework. J Biomed Inform 44:519–528. https://doi.org/10.1016/j.jbi.2011.01.011
Mehta N, Raja’S A, Chaudhary V (2009) Content based sub-image retrieval system for high resolution pathology images using salient interest points. In: Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE. IEEE, pp 3719–3722
Qi X, Wang D, Rodero I, Diaz-Montes J, Gensure RH, Xing F, Zhong H, Goodell L, Parashar M, Foran DJ, Yang L (2014) Content-based histopathology image retrieval using CometCloud. BMC Bioinform 15:287
Sridhar A, Doyle S, Madabhushi A (2015) Content-based image retrieval of digitized histopathology in boosted spectrally embedded spaces. J Pathol Inform 6:41. https://doi.org/10.4103/2153-3539.159441
Lafarge MW, Pluim JPW, Eppenhof KAJ, et al (2017) Domain-adversarial neural networks to address the appearance variability of histopathology images. arXiv:170706183 [cs]
ScanNet: a fast and dense scanning framework for metastatic breast cancer detection from whole-slide images - semantic scholar. /paper/ScanNet-A-Fast-and-Dense-Scanning-Framework-for-Me-Lin-Chen/9484287f4d5d52d10b5d362c462d4d6955655f8e. Accessed 22 Nov 2017
Vahadane A, Peng T, Sethi A, Albarqouni S, Wang L, Baust M, Steiger K, Schlitter AM, Esposito I, Navab N (2016) Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans Med Imaging 35:1962–1971. https://doi.org/10.1109/TMI.2016.2529665
Goodfellow IJ, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial networks
Shaban MT, Baur C, Navab N, Albarqouni S (2018) StainGAN: stain style transfer for digital histological images. arXiv:180401601 [cs]
Zanjani FG, Zinger S, Bejnordi BE, et al (2018) Histopathology stain-color normalization using deep generative models
Mariani G, Scheidegger F, Istrate R, et al (2018) BAGAN: data augmentation with balancing GAN
Komura D, Ishikawa S (2018) Machine learning methods for histopathological image analysis. Comput Struct Biotechnol J 16:34–42. https://doi.org/10.1016/j.csbj.2018.01.001
grand-challenges - Home. https://grand-challenge.org/. Accessed 17 Dec 2018
Peikari M, Zubovits J, Clarke G, Martel AL (2015) Clustering analysis for semi-supervised learning improves classification performance of digital pathology. In: Machine learning in medical imaging. Springer, Cham, pp 263–270
Doyle S, Monaco J, Feldman M, Tomaszewski J, Madabhushi A (2011) An active learning based classification strategy for the minority class problem: application to histopathology annotation. BMC Bioinform 12(424). https://doi.org/10.1186/1471-2105-12-424
Padmanabhan RK, Somasundar VH, Griffith SD, Zhu J, Samoyedny D, Tan KS, Hu J, Liao X, Carin L, Yoon SS, Flaherty KT, DiPaola RS, Heitjan DF, Lal P, Feldman MD, Roysam B, Lee WMF (2014) An active learning approach for rapid characterization of endothelial cells in human tumors. PLoS One 9:e90495. https://doi.org/10.1371/journal.pone.0090495
[1805.06983] Terabyte-scale deep multiple instance learning for classification and localization in pathology. https://arxiv.org/abs/1805.06983. Accessed 13 Aug 2018
Li Z, Wang C, Han M, et al (2017) Thoracic disease identification and localization with limited supervision. arXiv:171106373 [cs, stat]
Gal Y, Ghahramani Z (2015) Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv:150602158 [cs, stat]
Novak R, Xiao L, Lee J, et al (2018) Bayesian convolutional neural networks with many channels are Gaussian processes. arXiv:181005148 [cs, stat]
Shridhar K, Laumann F, Maurin AL, et al (2018) Bayesian convolutional neural networks with variational inference. arXiv:180605978 [cs, stat]
Zhao G, Liu F, Oler JA, Meyerand ME, Kalin NH, Birn RM (2018) Bayesian convolutional neural network based MRI brain extraction on nonhuman primates. NeuroImage 175:32–44. https://doi.org/10.1016/j.neuroimage.2018.03.065
Buda M, Maki A, Mazurowski MA (2017) A systematic study of the class imbalance problem in convolutional neural networks. https://doi.org/10.1016/j.neunet.2018.07.011
Haixiang G, Yijing L, Shang J, Mingyun G, Yuanyue H, Bing G (2017) Learning from class-imbalanced data: review of methods and applications. Expert Syst Appl 73:220–239. https://doi.org/10.1016/j.eswa.2016.12.035
Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velázquez Vega JE, Brat DJ, Cooper LAD (2018) Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A 115:E2970–E2979. https://doi.org/10.1073/pnas.1717139115
Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, Peng L, Webster DR (2018) Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125:1264–1272. https://doi.org/10.1016/j.ophtha.2018.01.034
Hou L, Agarwal A, Samaras D, et al (2017) Unsupervised histopathology image synthesis. arXiv:171205021 [cs]
Sayres R, Taly A, Rahimy E, Blumer K, Coz D, Hammel N, Krause J, Narayanaswamy A, Rastegar Z, Wu D, Xu S, Barb S, Joseph A, Shumski M, Smith J, Sood AB, Corrado GS, Peng L, Webster DR (2018) Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy. Ophthalmology 0:552–564. https://doi.org/10.1016/j.ophtha.2018.11.016
Acknowledgments
This study was supported by the Practical Research for Innovative Cancer Control from the Japan Agency for Medical Research and Development (AMED) (S.I.).
Contributions
Ishikawa S and Komura D wrote and reviewed the manuscript.
Funding
This research was supported by AMED under the Practical Research for Innovative Cancer Control, grant number JP19ck0106400 (S.I.).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
ESM 1
(DOCX 34402 kb)
Rights and permissions
About this article
Cite this article
Komura, D., Ishikawa, S. Machine learning approaches for pathologic diagnosis. Virchows Arch 475, 131–138 (2019). https://doi.org/10.1007/s00428-019-02594-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00428-019-02594-w