Abstract
The identification of tissue regions within histopathological images represents a fundamental step for diagnosis, patient stratification and follow-up. However, the huge amount of image data made available by the ever improving whole-slide imaging devices gives rise to a bottleneck in manual, microscopy-based evaluation. Furthermore, manual procedures generally show a significant intra- and/or inter-observer variability. In this scenario the objective of this chapter is to investigate the effectiveness of image features from last-generation, pre-trained convolutional networks against variants of Local Binary Patterns for classifying tissue sub-regions into meaningful classes such as epithelium, stroma, lymphocytes and necrosis. Experimenting with seven datasets of histopathological images we show that both classes of methods can be quite effective for the task, but with a noticeable superiority of descriptors based on convolutional neural networks. In particular, we show that these can be seamlessly integrated with standard classifiers (e.g. Support Vector Machines) to attain overall discrimination accuracy between 95 and 99%.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Underline indicates colour descriptors. For a detailed description of each method please refer to the given references.
References
Huijbers, A., Tollenaar, R.A.E.M., Pelt, G.W.V., Zeestraten, E.C.M., Dutton, S., McConkey, C.C., Domingo, E., Smit, V.T.H.B.M. , Midgley, R., Warren, B.F., Johnstone, E.C., Kerr, D.J., Mesker, W.E.: The proportion of tumor-stroma as a strong prognosticator for stage II and III colon cancer patients: validation in the VICTOR trial. Ann. Oncol. 24(1), 179–185 (2013)
Park, J.H., Richards, C.H., McMillan, D.C., Horgan, P.G., Roxburgh, C.S.D.: The relationship between tumour stroma percentage, the tumour microenvironment and survival in patients with primary operable colorectal cancer. Ann. Oncol. 25(3), 644–651 (2014)
van Pelt, G.W., Sandberg, T.P., Morreau, H., Gelderblom, H., van Krieken, J.H.J.M., Tollenaar, R.A.E.M., Mesker, W.E.: The tumour-stroma ratio in colon cancer: the biological role and its prognostic impact. Histopathology 73(2), 197–206 (2018). August
Chen, Y., Zhang, L., Liu, W., Liu, X.: Prognostic significance of the tumor-stroma ratio in epithelial ovarian cancer. BioMed Res. Int. 2015 (2015)
Lv, Z., Cai, X., Weng, X., Xiao, H., Du, C., Cheng, J., Zhou, L., Xie, H., Sun, K., Wu, J., Zheng, S.: Tumor-stroma ratio is a prognostic factor for survival in hepatocellular carcinoma patients after liver resection or transplantation. Surgery 158(1), 142–150 (2015)
Zhang, X.-L., Jiang, C., Zhang, Z.-X., Liu, F., Zhang, F., Cheng, Y.-F.: The tumor-stroma ratio is an independent predictor for survival in nasopharyngeal cancer. Oncol. Res. Treat. 37(9), 480–484 (2014)
De Kruijf, E.M., van Nes, J.G.H., van De Velde,C.J.H., Putter, H., Smit, V.T.H.B.M., Liefers, G.J., Kuppen, P.J.K., Tollenaar, R.A.E.M., Mesker, W.E.: Tumor-stroma ratio in the primary tumor is a prognostic factor in early breast cancer patients, especially in triple-negative carcinoma patients. Breast Cancer Res. Treat. 125(3), 687–696 (2011)
Dekker, T.J.A., van De Velde, C.J.H., van Pelt, G.W., Kroep, J.R., Julien, J.-P., Smit, V.T.H.B.M., Tollenaar, R.A.E.M., Mesker, W.E.: Prognostic significance of the tumor-stroma ratio: validation study in node-negative premenopausal breast cancer patients from the eortc perioperative chemotherapy (pop) trial (10854). BBreast Cancer Res. Treat. 139(2):371–379 (2013)
Vangangelt, K.M.H., van Pelt, G.W., Engels, C.C., Putter, H., Liefers, G.J., Smit, V.T.H.B.M., Tollenaar, R.A.E.M., Kuppen, P.J.K., Mesker, W.E.: Prognostic value of tumor–stroma ratio combined with the immune status of tumors in invasive breast carcinoma. Breast Cancer Res. Treat. 168(3), 601–612 (2018)
Mouawad, R., Spano, J.-P., Khayat, D.: Lymphocyte infiltration in breast cancer: a key prognostic factor that should not be ignored. J. Clin. Oncol. 29(33), 4471 (2011)
Correale, P., Rotundo, M.S., Botta, C., Vecchio, M.T.D., Tassone, P., Tagliaferri, P.: Tumor infiltration by chemokine receptor 7 (ccr7)+ t-lymphocytes is a favorable prognostic factor in metastatic colorectal cancer. OncoImmunology 1(4), 531–532 (2012)
Galon, J., Costes, A., Sanchez-Cabo, F., Kirilovsky, A., Mlecnik, B., Lagorce-Pagès, C., Tosolini, M., Camus, M., Berger, A., Wind, P., Zinzindohoué, F., Bruneval, P., Cugnenc, P.H., Trajanoski, Z., Fridman, W.H., Pagès, F.: Type, density, and location of immune cells within human colorectal tumors predict clinical outcome. Science 313(5795), 1960–1964 (2006). September
Halama, N., Michel, S., Kloor, M., Zoernig, I., Benner, A., Spille, A., Pommerencke, T., von Knebel, D.M., Folprecht, G., Luber, B., Feyen, N., Martens, U.M., Beckhove, P., Gnjatic, S., Schirmacher, P., Herpel, E., Weitz, J., Grabe, N., Jaeger, D.: Localization and density of immune cells in the invasive margin of human colorectal cancer liver metastases are prognostic for response to chemotherapy. Cancer Res. 71(17), 5670–5677 (2011)
Ness, N., Andersen, S., Valkov, A., Nordby, Y., Donnem, T., Al-Saad, S., Busund, L.-T., Bremnes, R.M., Richardsen, E.: Infiltration of cd8+ lymphocytes is an independent prognostic factor of biochemical failure-free survival in prostate cancer. Prostate 74(14), 1452–1461 (2014)
Caruso, R., Parisi, A., Bonanno, A., Paparo, D., Emilia, Q., Branca, G., Scardigno, M., Fedele, F.: Histologic coagulative tumour necrosis as a prognostic indicator of aggressiveness in renal, lung, thyroid and colorectal carcinomas: a brief review. Oncol. Lett. 3(1), 16–18 (2012)
Hynes, S.O., Coleman, H.G., Kelly, P.J., Irwin, S., O’Neill, R.F., Gray, R.T., Mcgready, C., Dunne, P.D., Mcquaid, S., James, J.A., Salto-Tellez, M., Loughrey, M.B.: Back to the future: routine morphological assessment of the tumour microenvironment is prognostic in stage ii/iii colon cancer in a large population-based study. Histopathology 71(1), 12–26 (2017). In press
Linder, N., Konsti, J., Turkki, R., Rahtu, E., Lundin, M., Nordling, S., Haglund, C., Ahonen, T., Pietikäinen, M., Lundin, J.: Identification of tumor epithelium and stroma in tissue microarrays using texture analysis. Diagn. Pathol. 7(22), 1–11 (2012)
Courrech Staal, E.F.W., Smit, V.T.H.B.M., van Velthuysen, M.-L.F., Spitzer-Naaykens, J.M.J., Wouters, M.W.J.M., Mesker,W.E., Tollenaar, R.A.E.M., van Sandick, J.W.: Reproducibility and validation of tumour stroma ratio scoring on oesophageal adenocarcinoma biopsies. Eur. J. Cancer 47(3), 375–382 (2011)
Fuchs, T.J., Buhmann, J.M.: Computational pathology: challenges and promises for tissue analysis. Comput. Med. Imaging Graph. 35(7–8), 515–530 (2011)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 26th Annual Conference on Neural Information Processing Systems, Lake Tahoe, USA (December 2012)
Sang, H., Zhou, Z.: Automatic detection of human faces in color images via convolutional neural networks. ICIC Express Lett., Part B: Appl. 7(4) (2016)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep Fisher networks for large-scale image classification. In:Proceedings of the 27th Annual Conference on Neural Information Processing Systems, Lake Tahoe, USA (December 2013)
Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). February
Cimpoi, M., Maji, S., Kokkinos, I., Vedaldi, A.: Deep filter banks for texture recognition, description, and segmentation. Int. J. Comput. Vis. 118(1), 65–94 (2016)
Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2014, pp. 512–519, Columbus, USA (June 2014)
Fernández, A., Álvarez, M.X., Bianconi, F.: Texture description through histograms of equivalent patterns. J. Math. Imaging Vis. 45(1), 76–102 (2013)
Liu, L., Fieguth, P., Guo, Y., Wang, X., Pietikäinen, M.: Local binary features for texture classification: taxonomy and experimental study. Pattern Recognit. 62, 135–160 (2017)
Meijer, G.A., Beliën, J.A.M., Van Diest, P.J., Baak, J.P.A.: Image analysis in clinical pathology. J. Clin. Pathol. 50(5), 365–370 (1997)
Al-Janabi, S., Huisman, A., Van Diest, P.J.: Digital pathology: current status and future perspectives. Histopathology 61(1), 1–9 (2012)
Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2, 147–171 (2009)
Janowczyk, A., Madabhushi, A.: Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J. Pathol. Inform. 7(1) (2016). Art. no. 29
Madabhushi, A., Lee, G.: Image analysis and machine learning in digital pathology: challenges and opportunities. Med. Image Anal. 33, 170–175 (2016)
Veta, M., Pluim, J.P.W., Van Diest, P.J., Viergever, M.A.: Breast cancer histopathology image analysis: a review. IEEE Trans. Biomed. Eng. 61(5), 1400–1411 (2014)
Watanabe, K., Kobayashi, T., Wada,T.: Semi-supervised feature transformation for tissue image classification. PLoS ONE 11(12) (2016). Article number e0166413
Diamond, J., Anderson, N.H., Bartels, P.H., Montironi, R., Hamilton, P.W.: The use of morphological characteristics and texture analysis in the identification of tissue composition in prostatic neoplasia. Hum. Pathol. 35(9), 1121–1131 (2004)
Bianconi, F., Fernández, A., Álvarez Larrán, A.: Discrimination between tumour epithelium and stroma via perception-based features. Neurocomputing 154, 119–126 (2015)
Nava, R., González, G., Kybic, J., Escalante-Ramírez, B.: Classification of tumor epithelium and stroma in colorectal cancer based on discrete Tchebichef moments. Lect. Notes Comput. Sci. (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9401, 79–87 (2016)
Kather, J.N., Weis, C.-A., Bianconi, F., Melchers, S.M., Schad, L.R., Gaiser, T., Marx, A., Zöllner, F.G.: Multi-class texture analysis in colorectal cancer histology. Sci. Rep. 6 (2016). Art. no. 27988
Badejo, J.A., Adetiba, E., Akinrinmade, A., Akanle, M.B.: Medical image classification with hand-designed or machine-designed texture descriptors: a performance evaluation. In: Rojas, I., Ortuño, F. (eds.) Proceedings of the International Conference on Bioinformatics and Biomedical Engineering (IWBBIO), Lecture Notes in Computer Science, vol. 10814, Granada, Spain, April 2018, pp. 266–275. Springer
Greenspan, H., van Ginneken, B., Summers, R.M.: Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35(5), 1153–1159 (2016)
Ciompi, F., Geessink, O., Bejnordi, B.E., de Souza, G.S., Baidoshvili, A., Litjens, G., van Ginneken, B., Nagtegaal, I., van der Laak, J.: The importance of stain normalization in colorectal tissue classification with convolutional networks. In: Proceedings of the IEEE International Symposium on Biomedical Imaging, Melbourne, Australia, April 2017 (2017). To appear
Spanhol, F.A., Oliveira, L.S., Petitjean, C., Heutte, L.: Breast cancer histopathological image classification using convolutional neural networks. In: Proceedings of the International Joint Conference on Neural Networks, pp. 2560–2567, Vancouver, Canada (July 2016)
Xu, J., Luo, X., Wang, G., Gilmore, H., Madabhushi, A.: A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191, 214–223 (2016)
Gao, Z., Wang, L., Zhou, L., Zhang, J.: HEp-2 cell image classification with deep convolutional neural networks. IEEE J. Biomed. Health Inform. 21(2), 416–428 (2017). March
Veta, M., van Diest, P.J., Willems, S.M., Wang, H., Madabhushi, A., Cruz-Roa, A., Gonzalez, F., Larsen, A.B.L., Vestergaard, J.S., Dahl, A.B., Cireşan, D.C., Schmidhuber, J., Giusti, A., Gambardella, L.M., Tek, F.B., Walter, T., Wang, C.-W., Kondo, S., Matuszewski, B.J., Precioso, F., Snell, V., Kittler, J., de Campos, T.E., Khan, A.M., Rajpoot, N.M., Arkoumani, E., Viergever, M.A, Lacle, M.M., Pluim, J.P.W.: Assessment of algorithms for mitosis detection in breast cancer histopathology images. Medical Image Analysis 20(1):237–248 (2015)
Tajbakhsh, N., Shin, J.Y., Gurudu, S.R., Hurst, R.T., Kendall, C.B., Gotway, M.B., Liang, J.: Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35(5), 1299–1312 (2016)
Van Ginneken, B., Setio, A.A.A., Jacobs, C., Ciompi,F.: Off-the-shelf convolutional neural network features for pulmonary nodule detection in computed tomography scans. In: Proceedings of the 12th IEEE International Symposium on Biomedical Imaging, ISBI 2015, pp. 286–289, Brooklyn, USA (April 2015)
Arevalo, J., Gonzalez, F.A., Ramos-Pollan, R., Oliveira, J.L., Lopez, M.A.G.: Convolutional neural networks for mammography mass lesion classification. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), Milan, Italy, November 2015, pp. 797–800
Vedaldi, A., Lenc, K.: MatConvNet: convolutional neural networks for MATLAB. In: Proceedings of the 23rd ACM International Conference on Multimedia (MM 2015), pp. 689–692, Brisbane, Australia (October 2015)
Jia, J., Shelhamer, E.: Caffe deep learning framework. http://caffe.berkeleyvision.org/. Last accessed 19 Apr 2017
Webmicroscope: EGFR colon TMA stroma LBP classification (2012). http://fimm.webmicroscope.net/Research/Supplements/epistroma. Accessed 17 Aug 2018
Linder, N., Martelin, E., Lundin, M., Louhimo, J., Nordling, S., Haglund, C., Lundin, J.: Xanthine oxidoreductase—clinical significance in colorectal cancer and in vitro expression of the protein in human colon cancer cells. Eur. J. Cancer 45(4), 648–655 (2009). March
Beck, A.H., Sangoi, A.R., Leung, S., Marinelli, R.J., Nielsen, T.O., van de Vijver, M.J., West, R.B., van de Rijn, M., Koller, D.: Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3(108), 1–11 (2011)
Beck, A., Sangoi, A., Leung, S., Marinelli, R., Nielsen, T., van de Vijver, M., West, R., van de Rijn, M., Koller, D.: Systematic analysis of breast cancer morphology uncovers stromal features associated with survival (2011). https://tma.im/tma-portal/C-Path/images.html. Accessed 2 Mar 2017
Kather, J.N., Marx, A., Reyes-Aldasoro, C.C., Schad, L.R., Zöllner, F.G., Weis, C.A.: Continuous representation of tumor microvessel density and detection of angiogenic hotspots in histological whole-slide images. Oncotarget 6(22), 19163–19176 (2015). August
Kather, J. N., Zöllner, F. G., Bianconi, F., Melchers, S. M., Schad, L. R., Gaiser, T., Marx, A., Weis, C.-A.: Collection of textures in colorectal cancer histology (May 2016)
Mesker, W.E., Junggeburt, J.M.C., Szuhai, K., De Heer, P., Morreau, H., Tanke, H.J., Tollenaar, R.A.E.M.: The carcinoma-stromal ratio of colon carcinoma is an independent factor for survival compared to lymph node status and tumor stage. Cell. Oncol. 29(5), 387–398 (2007)
Cusano, C., Napoletano, P., Schettini, R.: Combining multiple features for color texture classification. J. Electron. Imaging 25(6), (2016). Article number 061410
Parkhi, O., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of the British Machine Vision Conference 2015, Swansea, UK (2015)
ImageNet. http://www.image-net.org. Accessed 23 Feb 2018
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR2015), Boston, USA (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (2016)
Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. In: Proceedings of the British Machine Vision Conference 2014, Nottingham, United Kingdom (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In:Proceedings of the 5th International Conference on Learning Representations, San Diego, USA (2015)
Fernández, A., Ghita, O., González, E., Bianconi, F., Whelan, P.F.: Evaluation of robustness against rotation of lbp, ccr and ilbp features in granite texture classification. Mach. Vis. Appl. 22(6), 913–926 (2011)
Guo, Z., Zhang, L.: A completed modeling of local binary patterns operator for texture classification. IEEE Trans. Image Process. 19(6), 1657–1663 (2010)
Liu, L., Zhao, L., Long, Y., Kuang, G., Fieguth, P.: Extended local binary patterns for texture classification. Image Vis. Comput. 30, 86–99 (2012)
He, Y., Sang, N., Robust illumination invariant texture classification using gradient local binary patterns. In Proceedings of, : International Workshop on Multi-Platform/Multi-Sensor Remote Sensing and Mapping, p. 2011. Xiamen, China (2011)
Jin, H., Liu, Q., Lu, H., Tong, X.: Face detection using improved LBP under Bayesian framework. In: Proceedings of the 3rd International Conference on Image and Graphics, pp. 306–309, Hong Kong, China (December 2004)
Bianconi, F., Bello-Cerezo, R., Napoletano, P.: Improved opponent color local binary patterns: an effective local image descriptor for color texture classification. J. Electron. Imaging 27(1) (2018). Art. No. 011002
Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)
Lee, S., Choi, J., Ro, Y., Plataniotis, K.: Local color vector binary patterns from multichannel face images for face recognition. IEEE Trans. Image Process. 21(4), 2347–2353 (2012)
Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. In: Analysis and Modelling of Faces and Gestures, vol. 4778. Lecture Notes in Computer Science. Springer (2007)
Mäenpää, T., Pietikäinen, M.: Texture analysis with local binary patterns. In: Chen, C.H., Wang, P.S.P. (eds) Handbook of Pattern Recognition and Computer Vision, 3rd edn, pp. 197–216. World Scientific Publishing (2005)
Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Proceedings of the 3rd European Conference on Computer Vision (ECCV 1994) (1994)
He, D.-C., Wang, L.: Texture unit, texture spectrum, and texture analysis. IEEE Trans. Geosci. Remote. Sens. 28(4), 509–512 (1990)
Bianconi, F., Bello-Cerezo, R., Napoletano, P., Di Maria, F.: Improved opponent colour local binary patterns for colour texture classification. In: Bianco, S., Schettini, R., Tominaga, S., Tremeau, A. (eds.) Proceedings of the 6th Computational Color Imaging Workshop (CCIW 2017). Lecture Notes in Computer Science, vol. 10213, Milan, Italy, March 2017, pp. 272–281. Springer
Cernadas, E., Fernández-Delgado, M., González-Rufino, E., Carrión, P.: Influence of normalization and color space to color texture classification. Pattern Recognit. 61, 120–138 (2017)
Hsu, C.-W., Chang, C.-C., Lin, C.-J.: A practical guide to support vector classification, 2016. http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf. Last accessed 22 Mar 2017
Kanji, G.K.: 100 Statistical Tests, 3rd edn. Society for Industrial and Applied Mathematics (2006)
Cusano, C., Napoletano, P., Schettini, R.: Evaluating color texture descriptors under large variations of controlled lighting conditions. J. Opt. Soc. Am. A: Opt. Image Sci. Vis. 33(1), 17–30 (2016)
Huang, Y., Zheng, H., Liui, C., Ding, X., Rohde, G.K.: Epithelium-stroma classification via convolutional neural networks and unsupervised domain adaptation in histopathological images. IEEE J. Biomed. Health Inform. 21(6), 1625–1632 (2017)
Acknowledgements
This work was partially supported by the Italian Ministry of University and Research (MIUR) under the Individual Funding Scheme for Fundamental Research ‘FFABR’ 2017 (F. Bianconi) and by the Department of Engineering at the Università degli Studi di Perugia, Italy, within the Fundamental Research Grants Scheme 2018 (F. Bianconi).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Kather, J.N. et al. (2020). Classification of Tissue Regions in Histopathological Images: Comparison Between Pre-trained Convolutional Neural Networks and Local Binary Patterns Variants. In: Nanni, L., Brahnam, S., Brattin, R., Ghidoni, S., Jain, L. (eds) Deep Learners and Deep Learner Descriptors for Medical Applications. Intelligent Systems Reference Library, vol 186. Springer, Cham. https://doi.org/10.1007/978-3-030-42750-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-42750-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-42748-1
Online ISBN: 978-3-030-42750-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)