Virchows Archiv

, Volume 474, Issue 4, pp 511–522 | Cite as

Precision immunoprofiling by image analysis and artificial intelligence

  • Viktor H. KoelzerEmail author
  • Korsuk Sirinukunwattana
  • Jens Rittscher
  • Kirsten D. Mertz
Open Access
Review Article


Clinical success of immunotherapy is driving the need for new prognostic and predictive assays to inform patient selection and stratification. This requirement can be met by a combination of computational pathology and artificial intelligence. Here, we critically assess computational approaches supporting the development of a standardized methodology in the assessment of immune-oncology biomarkers, such as PD-L1 and immune cell infiltrates. We examine immunoprofiling through spatial analysis of tumor-immune cell interactions and multiplexing technologies as a predictor of patient response to cancer treatment. Further, we discuss how integrated bioinformatics can enable the amalgamation of complex morphological phenotypes with the multiomics datasets that drive precision medicine. We provide an outline to machine learning (ML) and artificial intelligence tools and illustrate fields of application in immune-oncology, such as pattern-recognition in large and complex datasets and deep learning approaches for survival analysis. Synergies of surgical pathology and computational analyses are expected to improve patient stratification in immuno-oncology. We propose that future clinical demands will be best met by (1) dedicated research at the interface of pathology and bioinformatics, supported by professional societies, and (2) the integration of data sciences and digital image analysis in the professional education of pathologists.


Personalized medicine Immuno-oncology Immunotherapy Digital pathology Image analysis Machine learning Artificial intelligence 


A picture is worth a thousand words. This is the essence of the technological transition from macropathology to micropathology as captured in Virchow’s core principle “omnis cellula et cellulae” first popularized in 1858 [81]. One hundred sixty years later, pathologists are in a technological transition phase of similar importance [61]: We are beginning to recognize that images of cells contain more information than what can be extracted by the human eye [20, 67, 83]. Computer-aided image analysis has the potential to make complex morphological information more accessible in daily diagnostic practice, improving prognostic and predictive patient stratification. Artificial intelligence has already been successfully employed in the setting of computational pathology to categorize diseases based on their molecular features [152159]. In combination, it seems likely that image-based digital pathology in combination with artificial intelligence will become part of a pathologist’s tool repertoire in the near future.

Immuno-oncology requires a detailed understanding of the tumor microenvironment, including the identification and quantification of different immune cell subsets, their spatial context, and the expression of immune checkpoint markers. Changes in immune cell infiltration and biomarker expression before and after therapeutic intervention are critical parameters for clinical development [80]. Image analysis tools can carry out complex and repetitive biomarker analyses with high precision and excellent reproducibility and can thus greatly assist pathologists in their key role to integrate clinical, morphologic, and molecular information for personalized treatment [20, 83, 84]. Actively engaging in image analysis methods is therefore becoming increasingly important for pathologists to maintain their role as leaders in precision medicine and diagnostics.

In recent decades, electronic data processing has transformed medical radiology, molecular diagnostics, and genetic testing. At the same time, the practice of pathology which forms a crucial link between these clinical disciplines has not changed significantly. The current convergence of new imaging technologies with tissue-based multiplexed immunohistochemistry (IHC) (reviewed in “Multiplexing”) and molecular phenotyping, our ability to digitize and process large collections of histology slides, and the promise of supporting human interpretation through automated analysis and artificial intelligence will have a dramatic impact on the field [15, 59].

While traditional medical device companies, such as Philips, GE, and Leica advance new platforms for digital pathology and commercialize new imaging technologies, the major IT companies, including Google, IBM, and Microsoft, as well as numerous start-up companies (e.g., PathAI) enter the space by applying their expertise in big data science and artificial intelligence to data analysis and integrated decision making. Pharmaceutical companies have also recognized the importance of advanced pathology to their own work. Roche has already received FDA clearance for the VENTANA MMR IHC Panel for patients diagnosed with colorectal cancer (CRC) [2]. Developing and maintaining an understanding of digital imaging and data mining will therefore be beneficial skills for pathologists practicing in the twenty-first century [67, 83].

This paper provides a comprehensive outline of image analysis and machine learning (ML) applications for precision immunoprofiling. We will demonstrate how digital tools can facilitate pathology workflows in the assessment of established immune biomarkers and enable the deep characterization of the tumor microenvironment through spatial analysis and multiplexing. Further, we will identify computational methods driving morpho-molecular integration and deep learning methodologies for the discovery of novel therapeutic targets.

Computational pathology to assist in the assessment of established biomarkers

Immuno-oncology has been revolutionized by the introduction of immune checkpoint inhibitors (ICI). ICI are monoclonal antibodies targeting immuno-regulatory molecules on the surface of T cells, antigen-presenting cells, and neoplastic cell populations [39]. Clinical success of reagents blocking the CTLA-4 (cytotoxic T lymphocyte-associated protein 4, CD152) and PD-1/PD-L1 checkpoints (programmed cell death protein 1, CD279; programmed death-ligand 1, CD274) has driven rapid regulatory approval for treatment of patients with both solid and hematologic malignancies [33]. Assessment of PD-L1 expression by IHC has emerged as an important predictive biomarker for patients with non-small cell lung cancer (NSCLC) [12, 13], urothelial carcinoma [34], and renal cell cancer [60]. However, assessment of PD-L1 is inherently difficult due to expression in both neoplastic and non-neoplastic cell populations, considerable marker heterogeneity and non-intuitive cutoffs. To complicate matters, PD-L1 positivity thresholds vary [80]. Inter-observer variability of PD-L1 assessment by pathologists is a known problem [17, 77]. This may contribute to inaccurate patient stratification and the misinterpretation of the impact of PD-L1 expression on clinical outcome: Subjective decisions can lead to radically different therapeutic stratification when scoring around cutoffs.

Digital scoring of PD-L1 can assist the pathologist to overcome these barriers by providing standardized metrics for biomarker assessment at single cell resolution across whole tissue sections [38, 44, 55] (Fig. 1a, b). A single tumor section can contain up to 106–107 cells, which can be rapidly and reproducibly scored using purpose-built image analysis algorithms [55]. Assessment of full tissue sections is a robust approach to account for tissue heterogeneity and to reliably measure PD-L1 expression at low levels [44, 55]. Fine-scale differences in biomarker compartmentalization can be captured at single cell resolution. ML technologies are beginning to be utilized in digital image analysis [40]. Pathologist input is critical for the creation of homogenous artificial intelligence training datasets that incorporate a sufficiently large number of true positive and negative staining examples as well as cases with unspecific staining and common technical artifacts. Expertly designed ML applications then have the potential to aid the human observer in assigning biomarker scores to defined cell populations based on morphological criteria and staining characteristics. For PD-L1 scoring, this approach can be particularly helpful to include or exclude PD-L1 expression in tumor infiltrating immune cells and tumor regions with non-specific, e.g., cytoplasmic staining (Fig. 1c–f)[44].
Fig. 1

Assessment of PD-L1 expression by computational image analysis. a Malignant melanoma of the superficial spreading type stained for PD-L1 by IHC. Digital annotation of the tumor tissue is shown in yellow. Substantial marker heterogeneity and expression of PD-L1 in tumor-infiltrating inflammatory cells complicates conventional histopathological assessment of PD-L1 positivity. b Digital scoring of PD-L1 implemented on the HALO™ platform (Indica labs, Corrales, NM, USA). A total of 34.882 cells were detected by nuclear segmentation using the hematoxylin counterstain for nuclear seeding followed by cell/nuclear boundary detection and postprocessing according to pathologist-controlled cellular parameters, such as nuclear size, roundness, and optical density. Membranous reactivity for 3,3′-diaminobenzidine (DAB) is detected and analyzed according to pathologist-set positivity thresholds. PD-L1 negative stromal cells and normal squamous epithelium serve as on-slide negative controls. In this case, 11.802 PD-L1 positive cells were detected for a total of 33.8% positive cells within the annotation region. cd PD-L1 reactivity in infiltrating immune cells can skew the assessment of PD-L1 expression in solid tumors with intrinsically low expression levels of PD-L1. ef Machine learning algorithms trained on large sample sets to differentiate PD-L1 positive immune cells (green) from tumor cell populations (red) represent a powerful approach for tissue classification. Tissue classification is followed by cell-level analysis of DAB expression for the precise assessment of PD-L1 expression in tumor cells only

Standardized biomarker assessment protocols are essential to accelerate clinical development of immuno-oncology therapeutics [27]. Co-development of purpose-built computational pathology solutions in parallel to clinical trials may help to harmonize immunotherapy companion diagnostics, since image analysis algorithms can be easily shared and standardized across diagnostic labs. Alternatively, standardized biomarker testing may be conducted via telepathology at a central location. In addition, computational pathology applications can also increase the quality of biomarker assessment within an institution by using standardized tissue blocks and scanning procedures. While computational pathology will become a crucial tool in extracting quantitative information from digitized slides, digitization will also enable new applications. It will be possible to compare current cases with annotated phenotype libraries, hence effectively integrating a vast amount of knowledge into clinical decision-making [52]. Taken together, computational pathology methods are likely to become important tools for making the assessment of immune biomarkers more reproducible, robust, and reliable.

Spatial analysis of tumor immune cell infiltration

The identification, localization, and spatial relationships of specific immune cells—before, during, or after therapy—have significant prognostic and predictive potential [10]. Both commercial and freely available open-source image analysis solutions are available to perform area-based quantification of immune cells by IHC or immunofluorescence (IF) for prognostic and predictive analyses [6, 25, 54]. This includes the enumeration of peri- and intratumoral CD3+ and CD8+ T cell populations per mm2 of tumor tissue to form the Immunoscore® (IS) as marketed by HalioDx [63]. This approach has shown independent prognostic value in CRC beyond usual risk factors and has strong potential to aid patient stratification in other solid tumors [41]. A worldwide consortium-based validation study in stage I–III CRC reported significantly longer recurrence free intervals in patients with high numbers of tumor-infiltrating T cells [63]. Recent data from stage III and stage IV CRC patients treated with adjuvant chemotherapy also indicate that the quantification of IS markers by image analysis has the potential to guide decisions on treatment duration and follow-up in advanced disease [57]. However, the predictive potential of the Immunoscore® for immunotherapy response beyond well-known predictors, such as DNA mismatch-repair (MMR) deficiency remains to be investigated [62].

Assessing immune cell infiltration is an important component of the “Cancer Immunogram” for patient stratification in future immunotherapy trials [10]. Initial studies have identified a broad association of T cell location at the tumor invasive margin and in nodal metastasis with response to ICI in melanoma patients [19, 79]. Modern digital image analysis techniques will enable a more detailed analysis: The exact coding of the x-y location of each individual marker positive cell on a histological slide for precise nearest neighbor and infiltration analyses is already feasible (Fig. 2a–c). This will allow to better understand the pathophysiology of tumor-host interaction, checkpoint molecule expression, and therapy effects in archival samples and preclinical models [8, 31]. Using these modern digital tools to record immune cell infiltrates over the course of therapy will empower data mining with patient characteristics, clinical response profiles, and genomic markers to build novel predictive indicators. Spatial profiling and compartmentalization studies can also be used to monitor and better understand immunotherapy associated adverse events which are commonly observed with ICI therapy [47].
Fig. 2

Spatial analysis of tumor immune cell infiltration. a Colorectal adenocarcinoma tissue microarray (TMA) spot stained for cytokeratin (Fast Red) and CD8+ T cells (DAB) with hematoxylin as a nuclear counterstain (left). Computational color deconvolution is performed for separate detection of cell nuclei, Fast Red, and DAB reaction products (middle), followed by nuclear segmentation and scoring of all cell populations (tumor cells: red; CD-8+ T cells: brown; marker negative cell nuclei: blue). A total of 1.623 cells were detected in this sample including 867 tumor cells, 330 CD8+ T cells, and 644 marker-negative cells. b Spatial plotting implemented on the HALO™ platform showing the localization of 867 cytokeratin positive tumor cells and 330 CD8+ T cells in this TMA spot. This allows to extract precise data on the relative distribution of T cells to the intraepithelial and stromal compartment in the tumor microenvironment. In this sample, 112 CD8+ T cells (or 33.9%) are localized to the intraepithelial compartment, while 218 CD8+ T cells (or 66.1%) localize to the tumor stroma. c Recording of the x-y coordinates in the tissue sample allows to define cell-cell relations by spatial analysis, such as the definition of nearest neighbor relationships between the tumor and CD8+ cell population (left) as well as the extraction of precise cell-cell distance measures (right)


To empower more accurate patient stratification for immunotherapy, histological analysis should aim for a simultaneous characterization of both immune and tumor-related pathways in a single tissue sample [10, 29, 73]. Multiplexed immunoprofiling is key to generate comprehensive biomarker datasets for correlation with clinical parameters [11, 80]. Digital image analysis is an effective tool to extract comprehensive information on biomarker expression levels, co-localization, and compartmentalization. Technical approaches include virtual multistaining by computational methods, simultaneous multimarker staining of single slides, and sequential staining and quenching protocols.

Image registration is a powerful approach to generate virtual multistains from serial sections using computational methods: Following sectioning, staining, and scanning, serial sections can be computationally aligned by automated or user guided approaches [82]. Digital image analysis is carried out and the results are used to generate virtual multimarker profiles. Standard manual or automated staining procedures can be used for visualization. Virtual multistaining also offers the unique opportunity of combining distinct histological methods, such as RNA in situ hybridization and IHC for multiparametric tissue profiling. This approach has been recently used to spatially correlate the expression of specific microRNAs, interferon-stimulated target genes, T cell infiltration, and the expression of cytotoxic effector molecules in the microenvironment of CRC [45]. Limitations include the requirement for multiple tissue sections, inevitable sectioning artifacts, and the need to perform complex image transformation procedures for precise image alignment.

Highly optimized multistaining panels in combination with purpose-built digital image analysis software can overcome these drawbacks. However, substantial investment into a multimodal staining platform and analysis software is required to tap into this technology. Examples of this approach include the Vectra® Polaris™ Automated Quantitative Pathology Imaging System introduced by PerkinElmer [35]. Using a single slide, this workflow allows to generate highly multiplexed IF panels to investigate six or more antigens in a strictly quantitative manner. Standardization is facilitated by commercially available marker panels, which may assist future clinical applications. A recent study by Mezheyeuski and colleagues elegantly illustrates the combination of the Vectra® approach with tissue microarray technology for a high-throughput profiling of the immune-environment in NSCLC samples [56]. By combining a tumor-specific marker with a panel of immune-cell associated antigens, the authors analyze both quantitative and spatial information of specific lymphocyte subpopulations with RNA-expression levels and prognosis.

Interesting alternatives to simultaneous multiplexed IF are sequential workflows consistent of staining, digitalization, inactivation of fluorescent dyes, and re-staining. Profiling of up to 100 IF biomarkers on a single tissue section is technically feasible [29, 70]. Exemplary applications include multiplexed fluorescence microscopy method (MxIF) [29, 73] and multiepitope-ligand cartography (MELC) [70]. Similar approaches have been tested and successfully applied for IHC protocols [66, 78]. The single channel outputs from each staining cycle can be easily merged into comprehensive expression maps for characterization of the tumor microenvironment using computational methods [78]. However, individual optimization of the staining protocols, tissue degradation with iterative staining cycles, process time, and standardization may be hurdles in applying this technology to large clinical trial sample sets.

CO-Detection by indEXing (CODEX) is among the most recent and innovative approaches for high-dimensional imaging of antibody-tagged epitopes in FFPE tissue [30]. CODEX staining uses antibodies labeled with unique DNA barcodes that are iteratively detected by in-situ polymerization with fluorescently labeled dNTP analogues. Tissue slides are stained in a single incubation step, followed by imaging cycles for the spatially resolved visualization of all antibody binding events. Pilot studies for have demonstrated the technical feasibility and power of this approach for deep profiling of immune tissue architecture [30]. Application to immunoprofiling of cancer tissues is a logical next step for detailed analysis of the microenvironment in correlation with clinical and molecular parameters. A decisive advantage of this approach is the possibility to image CODEX-labels on a standard three-color fluorescence microscope or scanner.

Combining spatial imaging approaches with mass spectrometry may serve to reach yet another dimension in precision profiling of the tumor microenvironment. Multiplexed ion beam imaging (MIBI) uses antibodies labeled with isotopically pure elemental metal reporters (mass tags) to simultaneously detect up to 100 targets on FFPE tissue sections [5]. Conventional histologic stains can be included in this panel, allowing precise reconstruction of virtual histology images with antibody expression data. Decisive technological advantages of MIBI include the stability of mass tag labels, a sensitivity exceeding chromogenic IHC by up to three log levels, and no spectral overlap between individual labels [5]. Recent studies have demonstrated the technical feasibility to integrate RNAscope-based metal in situ hybridization with MIBI for multiparametric profiling [71]. Purpose-built digital image analysis methods are available to reconstruct virtual histology images and explore cell phenotypes, spatial interaction, and morphological structures [69]. Further technological advances using dissociated tissue samples allow to perform single cell proteomic analysis using a mass spectrometry-flow cytometry hybrid device, the so-called “CyTOF” [9]. However, substantial investment and expertise in mass spectrometry are necessary to access this technology.

Morpho-molecular integration

The recently proposed “Cancer Immunogram” suggests seven categories including tumor foreignness, general immune status, immune cell infiltration, absence of checkpoints, absence of soluble inhibitors, absence of inhibitory tumor metabolism, and tumor sensitivity to immune effectors as the most important predictors of immunotherapy response [10]. As such, the “Cancer Immunogram” is highly integrative and includes both tumor- and immune-related parameters assessed with both molecular and image-based methods for individualized prediction of immunotherapy response. By providing continuous data on tissue-based parameters, such as immune cell infiltration and expression of immune checkpoints, computational pathology methods are ideally suited for data integration with molecular parameters.

Tumors with a high mutational load frequently exhibit an immunologically activated phenotype [4]. Recent study has linked defects in DNA mismatch repair to high frequencies of neoantigens and immunotherapy response in advanced solid tumors [49, 62]. However, the correlation between genomic parameters and immunogenicity is far from linear [53]. In some solid tumors, such as melanoma, a lack of association of neoantigen type and frequency with baseline immune activation has been elegantly demonstrated [74]. In these tumors, strong regulatory T cell infiltration may curb the anti-tumoral immune response [75]. Other known mechanisms that may impact tumor antigenicity are defects in antigen-presentation and overexpression of immunosuppressive molecules, such as CD47, PD-L1, and indolamine-2,3-dioxygenase [42, 43, 75]. These image-based features can be reliably captured by computational image analysis methods for in-depth profiling of the tumor microenvironment.

Incorporating digital information on immune cell infiltration with molecular data is a powerful approach to inform the “Cancer Immunogram.” Conde and colleagues recently provided a proof of principle for the association of specific genetic alterations with defined immunophenotypes by combining digital assessment of CD8+ T cell infiltration in lung squamous-cell carcinoma with manual scoring of PD-L1 and targeted next generation sequencing [16]. Joining digital and molecular pathology also facilitates the development of novel assays to predict immunotherapy response. We have recently developed and validated a computational pathology assay that identifies specific PD1-positive subpopulations in NSCLC as a powerful predictive indicator of response to ICI treatment [76]. Methodologically, this assay was driven by morpho-molecular integration of RNA sequencing data with tissue-based methods. Computational image analysis was a key tool in this process, driving the development of a standardized algorithm for the detection of tumor-infiltrating lymphocyte populations with uniquely high expression levels of PD1 protein (PD1T lymphocytes). PD1T cells were reproducibly detected and quantified in pretreatment biopsies of lung cancer patients and correlated strongly with treatment response to ICI in two independent clinical cohorts. This translational approach highlights how digital image analysis can represent a powerful companion diagnostic for cancer immunotherapy applications.

Machine learning and artificial intelligence

ML will transform the field of immuno-oncology. We expect that ML will drive a paradigm shift in the way data are collected and analyzed to discover new prognostic markers or to construct more rigorous risk classification to empower stratified medicine. ML is essentially a set of computer algorithms that learn generic rules to perform any given task directly from data, without requirements of predefined knowledge or domain expertise. This, in many ways, is similar to how pathologists have previously gained knowledge and expertise through continuous practice that has led to new diagnostic classifications or prognostic factors in the clinical routine.

Since ML is heavily data-driven, it enables a means to derive unbiased statistics from data. Accurate and continuous variables are more informative and can provide more biologically relevant information than the semi-quantitative scores presently implemented in diagnostic practice. Although several classical regression models exist to facilitate the discovery process of predictive or prognostic factors, the power of ML surpasses that of traditional tools when data is large and complex. ML can crunch through a vast amount of data and learn complex relationships between parameters and outcomes without the need to prespecify the relationships as normally required in traditional regression models. Nonetheless, data-driven approaches have critical downsides if not used with caution. ML can learn meaningless, biologically uninterpretable features that correlate to outcomes if the data has not been carefully preprocessed to remove any spurious features. Training datasets must have been generated in a highly standardized manner; otherwise, AI algorithms are likely to “misinterpret” sampling differences and artifacts between sets as distinctive biological characteristics. This phenomenon is known as “garbage in, garbage out.” Expert pathologist input in the training of image analysis algorithms is therefore of critical importance. Furthermore, ML tends to “overfit” to discovery data which results in an over-optimistic estimate of the performance of a model, while, in fact, it cannot be generalized well to new unseen datasets. It is, therefore, necessary that ML models are rigorously validated on new datasets that are independent of those used during the model development. Originating from the field of computer vision, deep convolutional neural network (DCNN) is a ML approach that is specialized in image analysis tasks [46, 50, 51]. In recent years, DCNN has been successfully applied to key applications in diagnostic pathology, including cell classification [72], cell enumeration [85], tumor grading [23], cancer diagnosis [22], and cancer prognostication [58].

Application of machine learning in immuno-oncology: pattern recognition

In immuno-oncology, ML as a pattern recognition tool enables an accurate and reproducible means for the unbiased assessment of regularities in the expression of immunohistochemical markers, tumor morphology, and the spatial distribution of tumor infiltrating lymphocytes (TILs). The ability of ML tools to detect key features in complex immunophenotypic datasets underlines their potential importance for the development of novel predictive models in cancer research. Initial studies underline the potential of ML methods for clinical translation:

Yuan et al. computationally profiled cells of triple-negative breast cancer cohorts on H&E-stained sections and were able to uncover three categories of lymphocytes (intra-tumor, adjacent-tumor, and distal-tumor) with an unsupervised clustering method based on the proximities of lymphocytes to tumor cells [87]. Interestingly, the ratio between the total number of intratumoural lymphocytes and the number of cancer cells was strongly associated with disease-specific survival and strongly correlated with the expression of CTLA-4, a known immunotherapy target. Saltz et al. employed a DCNN model to determine the probability of TIL infiltration for every small region of H&E stained sections [68]. This resulted in a map which indicates the degree of local lymphocyte infiltration. Unsupervised clustering methods were employed to group small TIL regions into spatially coherent structures [28]. The resulting clustering patterns were characterized using parameters related to cluster size and shape. Importantly, the covariates summarizing complex characteristics of clusters were found to be associated with overall survival in various cancer groups, illustrating the potential of ML methodologies applied to standard H&E slides for the development of biomarkers in immuno-oncology.

Heindl et al. demonstrated that quantifying the number of spatial clusters or hotspots of immune cells and cancer cells is prognostic in ER+ breast cancer using a fully automated H&E stained image analysis algorithm [32]. The spatial clustering is identified as an area in which the number of cells of interest is greater than expected by chance given the distribution of the cells on the whole tissue section. The increased immune spatial clustering is associated with poor prognosis and has a level of prognostic significance on par with the IHC test [18] and the Oncotype DX 21-gene recurrence score [64]. Interestingly, immune scores which are related to the ratios of distinct categories of lymphocytes to the total number of cancer cells did not provide prognostic information in this study. This illustrates the importance of ML tools to detect novel morphologic features that provide prognostic information beyond established classifications.

Application of machine learning in Immuno-oncology: survival analysis

In medical research, survival analysis is a traditional means to assess the prognostic significance of each candidate covariate. The most widely used survival model is the Cox proportional hazards model. It assumes that the risk of an event of interest to occur (failure) is not time-dependent, is determined by a linear combination of all covariates, and might also include interaction terms between the covariates. This somewhat too simplistic assumption may not be able to capture the precise effects of the observed medical covariates on the risk of failure. This is particularly true in immuno-oncology where variable effects of immune infiltrates and immune-related gene signatures are observed depending on the tumor type and host immune status [7]. DNN applied to survival analysis could provide a better way to model complex nonlinear relationships among prognostic factors that better fit to the survival data.

It has been shown that DNN performs better than the Cox proportional hazard model in various medical applications [24, 58, 86]. Katzman et al. investigated the risk associated with treatment choices and demonstrated that their DeepSurv system could provide treatment recommendations that increase the median survival time of patients [37]. Yousefi et al. used their SurvivalNet to analyze large-scale genomic profiles obtained from the TCGA database [14, 86]. They have demonstrated the robustness of the model across different cancer types even if the number of input covariates is sometimes considerably larger than the sample size. Moreover, the model also allows the interpretation of the prognostic significance of individual covariates based on their contributions to overall risk. The same model has been extended to allow integration between histology image and genomic data in a unified framework [58]. Likewise, specific methylome signatures queried by ML were shown to be suitable for prediction of response to immune-checkpoint inhibitors. In analogy to image analysis, this modality takes both neoplastic and reactive cells into account [21]. These results suggest the potential of deep learning as a discovery tool to provide insight into the biology of immuno-oncology.


Regulatory issues in the use of digital pathology devices have been broadly recognized by national and international bodies. Pioneering companies have recently received CE certification for routine pathology applications in the European Union under the In vitro diagnostic medical devices directive [1]. Comprehensive, non-binding recommendations have been issued by the Food and Drug Administration (FDA) and the College of American pathologists (CAP) [3, 65]. These guidelines highlight two important aspects of testing emerging biomarkers for immunotherapy by computational pathology methods: First, the quality and reliability of the imaging system as recommended by the FDA and mandated by the CE certification process, and second, the consistency of diagnoses made by pathologists using digital systems as advised by the CAP [3, 65]. Both are of essential importance in the design of reliable computational pathology workflows to inform clinical decision making.

Concerns have been raised that inaccurate study design, statistical analysis, and reporting of research lead to a significant waste of research funding and inaccurate scientific conclusions [36]. A major challenge inherent to the use of digital image analysis is the substantial expansion in the number of variables. Although it may seem attractive from a scientific perspective to extract as much information as possible from a limited amount of tissue, it is important to recognize that big data mining inherently increases the level of statistical noise. Robust statistical considerations are required to avoid multiple testing problems and misleading conclusions. As with any diagnostic tool, digital image analysis is therefore critically dependent on scientific rigor, reliable documentation, consequent quality control, and adequate training of the pathology workforce [48]. Unsolved problems also concern the standardization of the preanalytical steps before the imaging and analysis of tissue slides. A perfect example is Ki-67 staining, which has proven difficult to harmonize between labs despite being one of the most frequently performed assay of its kind [26]. The same applies to the majority of immunohistochemical stains, in particular those where expression levels matter.


Advanced multiparametric imaging applications and ML have the potential to translate our evolving understanding of tumor-host interaction into better patient stratification and new treatment strategies. Computational pathology will help to derive complete, standardized, and reproducible datasets to facilitate the individualized prediction of immunotherapy response. Upskilling the pathology workforce through professional education and the recognition of computational pathology by professional societies will be essential to meet future clinical demands for optimal patient care.


Author contributions

VHK, KS, JR, and KDM jointly conceptualized this work, drafted the manuscript, and approved the final version to be published.

Funding information

Viktor H Koelzer is supported by a research fellowship of the Swiss National Science Foundation (P2SKP3_168322). Kirsten D Mertz is supported by a grant of the Krebsliga beider Basel (KLbB-4182-03-2017) and by a grant of the “Freiwillige Akademische Gesellschaft Basel.” KS and JR were supported by the Oxford NIHR Oxford Biomedical Research Centre.

Compliance with ethical standards

No research involving human participants and/or animals was performed for this study.

Conflict of interest

KS and JR have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, royalties, or patents received or pending. V.H.K. and K.D.M. are participants of a patent application co-owned by the Netherlands Cancer Institute (NKI-AVL) and the University of Basel on the assessment of cancer immunotherapy biomarkers by digital pathology. Based on NKI-AVL and the University of Basel policy on management of intellectual property, V.H.K. and K.D.M. would be entitled to a portion of received royalty income. V.H.K. has served as an invited speaker on behalf of Indica Labs.


  1. 1.
    Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on in vitro diagnostic medical devices. Accessed March 6th, 2018
  2. 2.
    Roche receives FDA clearance for the VENTANA MMR IHC panel for patients diagnosed with colorectal cancer. Accessed May 14th, 2018
  3. 3.
    U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, Office of In Vitro Diagnostics and Radiological Health, Division of Molecular Genetics and Pathology MP and CB. [Internet]. Technical performance assessment of digital pathology whole slide imagingdevices; 2016. Available from: Accessed March 7th, 2018
  4. 4.
    Alexandrov LB, Nik-Zainal S, Wedge DC, Aparicio SA, Behjati S, Biankin AV, Bignell GR, Bolli N, Borg A, Borresen-Dale AL, Boyault S, Burkhardt B, Butler AP, Caldas C, Davies HR, Desmedt C, Eils R, Eyfjord JE, Foekens JA, Greaves M, Hosoda F, Hutter B, Ilicic T, Imbeaud S, Imielinski M, Jager N, Jones DT, Jones D, Knappskog S, Kool M, Lakhani SR, Lopez-Otin C, Martin S, Munshi NC, Nakamura H, Northcott PA, Pajic M, Papaemmanuil E, Paradiso A, Pearson JV, Puente XS, Raine K, Ramakrishna M, Richardson AL, Richter J, Rosenstiel P, Schlesner M, Schumacher TN, Span PN, Teague JW, Totoki Y, Tutt AN, Valdes-Mas R, van Buuren MM, van ‘t Veer L, Vincent-Salomon A, Waddell N, Yates LR, Australian Pancreatic Cancer Genome I, Consortium IBC, Consortium IM-S, PedBrain I, Zucman-Rossi J, Futreal PA, McDermott U, Lichter P, Meyerson M, Grimmond SM, Siebert R, Campo E, Shibata T, Pfister SM, Campbell PJ, Stratton MR (2013) Signatures of mutational processes in human cancer. Nature 500:415–421. CrossRefGoogle Scholar
  5. 5.
    Angelo M, Bendall SC, Finck R, Hale MB, Hitzman C, Borowsky AD, Levenson RM, Lowe JB, Liu SD, Zhao S, Natkunam Y, Nolan GP (2014) Multiplexed ion beam imaging of human breast tumors. Nat Med 20:436–442. CrossRefGoogle Scholar
  6. 6.
    Bankhead P, Loughrey MB, Fernandez JA, Dombrowski Y, McArt DG, Dunne PD, McQuaid S, Gray RT, Murray LJ, Coleman HG, James JA, Salto-Tellez M, Hamilton PW (2017) QuPath: open source software for digital pathology image analysis. Sci Rep 7:16878. CrossRefGoogle Scholar
  7. 7.
    Barnes TA, Amir E (2017) HYPE or HOPE: the prognostic value of infiltrating immune cells in cancer. Br J Cancer 117:451–460. CrossRefGoogle Scholar
  8. 8.
    Barua S, Fang P, Sharma A, Fujimoto J, Wistuba I, Rao AUK, Lin SH (2018) Spatial interaction of tumor cells and regulatory T cells correlates with survival in non-small cell lung cancer. Lung Cancer.
  9. 9.
    Bendall SC, Nolan GP, Roederer M, Chattopadhyay PK (2012) A deep profiler’s guide to cytometry. Trends Immunol 33:323–332. CrossRefGoogle Scholar
  10. 10.
    Blank CU, Haanen JB, Ribas A, Schumacher TN (2016) CANCER IMMUNOLOGY. The “Cancer Immunogram”. Science 352:658–660. CrossRefGoogle Scholar
  11. 11.
    Blom S, Paavolainen L, Bychkov D, Turkki R, Maki-Teeri P, Hemmes A, Valimaki K, Lundin J, Kallioniemi O, Pellinen T (2017) Systems pathology by multiplexed immunohistochemistry and whole-slide digital image analysis. Sci Rep 7:15580. CrossRefGoogle Scholar
  12. 12.
    Borghaei H, Paz-Ares L, Horn L, Spigel DR, Steins M, Ready NE, Chow LQ, Vokes EE, Felip E, Holgado E, Barlesi F, Kohlhaufl M, Arrieta O, Burgio MA, Fayette J, Lena H, Poddubskaya E, Gerber DE, Gettinger SN, Rudin CM, Rizvi N, Crino L, Blumenschein GR Jr, Antonia SJ, Dorange C, Harbison CT, Graf Finckenstein F, Brahmer JR (2015) Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancer. N Engl J Med 373:1627–1639. CrossRefGoogle Scholar
  13. 13.
    Brahmer J, Reckamp KL, Baas P, Crino L, Eberhardt WE, Poddubskaya E, Antonia S, Pluzanski A, Vokes EE, Holgado E, Waterhouse D, Ready N, Gainor J, Aren Frontera O, Havel L, Steins M, Garassino MC, Aerts JG, Domine M, Paz-Ares L, Reck M, Baudelet C, Harbison CT, Lestini B, Spigel DR (2015) Nivolumab versus docetaxel in advanced squamous-cell non-small-cell lung cancer. N Engl J Med 373:123–135. CrossRefGoogle Scholar
  14. 14.
    Cancer Genome Atlas Research N, Weinstein JN, Collisson EA, Mills GB, Shaw KR, Ozenberger BA, Ellrott K, Shmulevich I, Sander C, Stuart JM (2013) The cancer genome atlas pan-cancer analysis project. Nat Genet 45:1113–1120. CrossRefGoogle Scholar
  15. 15.
    Capper D, Jones DTW, Sill M, Hovestadt V, Schrimpf D, Sturm D, Koelsche C, Sahm F, Chavez L, Reuss DE, Kratz A, Wefers AK, Huang K, Pajtler KW, Schweizer L, Stichel D, Olar A, Engel NW, Lindenberg K, Harter PN, Braczynski AK, Plate KH, Dohmen H, Garvalov BK, Coras R, Holsken A, Hewer E, Bewerunge-Hudler M, Schick M, Fischer R, Beschorner R, Schittenhelm J, Staszewski O, Wani K, Varlet P, Pages M, Temming P, Lohmann D, Selt F, Witt H, Milde T, Witt O, Aronica E, Giangaspero F, Rushing E, Scheurlen W, Geisenberger C, Rodriguez FJ, Becker A, Preusser M, Haberler C, Bjerkvig R, Cryan J, Farrell M, Deckert M, Hench J, Frank S, Serrano J, Kannan K, Tsirigos A, Bruck W, Hofer S, Brehmer S, Seiz-Rosenhagen M, Hanggi D, Hans V, Rozsnoki S, Hansford JR, Kohlhof P, Kristensen BW, Lechner M, Lopes B, Mawrin C, Ketter R, Kulozik A, Khatib Z, Heppner F, Koch A, Jouvet A, Keohane C, Muhleisen H, Mueller W, Pohl U, Prinz M, Benner A, Zapatka M, Gottardo NG, Driever PH, Kramm CM, Muller HL, Rutkowski S, von Hoff K, Fruhwald MC, Gnekow A, Fleischhack G, Tippelt S, Calaminus G, Monoranu CM, Perry A, Jones C, Jacques TS, Radlwimmer B, Gessi M, Pietsch T, Schramm J, Schackert G, Westphal M, Reifenberger G, Wesseling P, Weller M, Collins VP, Blumcke I, Bendszus M, Debus J, Huang A, Jabado N, Northcott PA, Paulus W, Gajjar A, Robinson GW, Taylor MD, Jaunmuktane Z, Ryzhova M, Platten M, Unterberg A, Wick W, Karajannis MA, Mittelbronn M, Acker T, Hartmann C, Aldape K, Schuller U, Buslei R, Lichter P, Kool M, Herold-Mende C, Ellison DW, Hasselblatt M, Snuderl M, Brandner S, Korshunov A, von Deimling A, Pfister SM (2018) DNA methylation-based classification of central nervous system tumours. Nature 555:469–474. CrossRefGoogle Scholar
  16. 16.
    Conde E, Caminoa A, Dominguez C, Calles A, Walter S, Angulo B, Sanchez E, Alonso M, Jimenez L, Madrigal L, Hernando F, Sanz-Ortega J, Jimenez B, Garrido P, Paz-Ares L, de Castro J, Hernandez S, Lopez-Rios F (2018) Aligning digital CD8(+) scoring and targeted next-generation sequencing with programmed death ligand 1 expression: a pragmatic approach in early-stage squamous cell lung carcinoma. Histopathology 72:270–284. CrossRefGoogle Scholar
  17. 17.
    Cooper WA, Russell PA, Cherian M, Duhig EE, Godbolt D, Jessup PJ, Khoo C, Leslie C, Mahar A, Moffat DF, Sivasubramaniam V, Faure C, Reznichenko A, Grattan A, Fox SB (2017) Intra- and Interobserver reproducibility assessment of PD-L1 biomarker in non-small cell lung cancer. Clin Cancer Res 23:4569–4577. CrossRefGoogle Scholar
  18. 18.
    Cuzick J, Dowsett M, Pineda S, Wale C, Salter J, Quinn E, Zabaglo L, Mallon E, Green AR, Ellis IO, Howell A, Buzdar AU, Forbes JF (2011) Prognostic value of a combined estrogen receptor, progesterone receptor, Ki-67, and human epidermal growth factor receptor 2 immunohistochemical score and comparison with the Genomic Health recurrence score in early breast cancer. J Clin Oncol 29:4273–4278. CrossRefGoogle Scholar
  19. 19.
    Diem S, Hasan Ali O, Ackermann CJ, Bomze D, Koelzer VH, Jochum W, Speiser DE, Mertz KD, Flatz L (2018) Tumor infiltrating lymphocytes in lymph node metastases of stage III melanoma correspond to response and survival in nine patients treated with ipilimumab at the time of stage IV disease. Cancer Immunol Immunother 67:39–45. CrossRefGoogle Scholar
  20. 20.
    Djuric U, Zadeh G, Aldape K, Diamandis P (2017) Precision histology: how deep learning is poised to revitalize histomorphology for personalized cancer care. NPJ Precis. Oncol.
  21. 21.
    Duruisseaux M, Martinez-Cardus A, Calleja-Cervantes ME, Moran S, Castro de Moura M, Davalos V, Pineyro D, Sanchez-Cespedes M, Girard N, Brevet M, Giroux-Leprieur E, Dumenil C, Pradotto M, Bironzo P, Capelletto E, Novello S, Cortot A, Copin MC, Karachaliou N, Gonzalez-Cao M, Peralta S, Montuenga LM, Gil-Bazo I, Baraibar I, Lozano MD, Varela M, Ruffinelli JC, Palmero R, Nadal E, Moran T, Perez L, Ramos I, Xiao Q, Fernandez AF, Fraga MF, Gut M, Gut I, Teixido C, Vilarino N, Prat A, Reguart N, Benito A, Garrido P, Barragan I, Emile JF, Rosell R, Brambilla E, Esteller M (2018) Epigenetic prediction of response to anti-PD-1 treatment in non-small-cell lung cancer: a multicentre, retrospective analysis. Lancet Respir Med 6:771–781. CrossRefGoogle Scholar
  22. 22.
    Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak J, the CC, Hermsen M, Manson QF, Balkenhol M, Geessink O, Stathonikos N, van Dijk MC, Bult P, Beca F, Beck AH, Wang D, Khosla A, Gargeya R, Irshad H, Zhong A, Dou Q, Li Q, Chen H, Lin HJ, Heng PA, Hass C, Bruni E, Wong Q, Halici U, Oner MU, Cetin-Atalay R, Berseth M, Khvatkov V, Vylegzhanin A, Kraus O, Shaban M, Rajpoot N, Awan R, Sirinukunwattana K, Qaiser T, Tsang YW, Tellez D, Annuscheit J, Hufnagl P, Valkonen M, Kartasalo K, Latonen L, Ruusuvuori P, Liimatainen K, Albarqouni S, Mungal B, George A, Demirci S, Navab N, Watanabe S, Seno S, Takenaka Y, Matsuda H, Ahmady Phoulady H, Kovalev V, Kalinovsky A, Liauchuk V, Bueno G, Fernandez-Carrobles MM, Serrano I, Deniz O, Racoceanu D, Venancio R (2017) Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer JAMA 318:2199–2210. doi:
  23. 23.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118. CrossRefGoogle Scholar
  24. 24.
    Faraggi D, Simon R (1995) A neural network model for survival data. Stat Med 14:73–82CrossRefGoogle Scholar
  25. 25.
    Feng Z, Bethmann D, Kappler M, Ballesteros-Merino C, Eckert A, Bell RB, Cheng A, Bui T, Leidner R, Urba WJ, Johnson K, Hoyt C, Bifulco CB, Bukur J, Wickenhauser C, Seliger B, Fox BA (2017) Multiparametric immune profiling in HPV-oral squamous cell cancer. JCI Insight 2.
  26. 26.
    Focke CM, Burger H, van Diest PJ, Finsterbusch K, Glaser D, Korsching E, Decker T, German Breast Screening Pathology I (2017) Interlaboratory variability of Ki67 staining in breast cancer. Eur J Cancer 84:219–227. CrossRefGoogle Scholar
  27. 27.
    Fox BA, Schendel DJ, Butterfield LH, Aamdal S, Allison JP, Ascierto PA, Atkins MB, Bartunkova J, Bergmann L, Berinstein N, Bonorino CC, Borden E, Bramson JL, Britten CM, Cao X, Carson WE, Chang AE, Characiejus D, Choudhury AR, Coukos G, de Gruijl T, Dillman RO, Dolstra H, Dranoff G, Durrant LG, Finke JH, Galon J, Gollob JA, Gouttefangeas C, Grizzi F, Guida M, Hakansson L, Hege K, Herberman RB, Hodi FS, Hoos A, Huber C, Hwu P, Imai K, Jaffee EM, Janetzki S, June CH, Kalinski P, Kaufman HL, Kawakami K, Kawakami Y, Keilholtz U, Khleif SN, Kiessling R, Kotlan B, Kroemer G, Lapointe R, Levitsky HI, Lotze MT, Maccalli C, Maio M, Marschner JP, Mastrangelo MJ, Masucci G, Melero I, Melief C, Murphy WJ, Nelson B, Nicolini A, Nishimura MI, Odunsi K, Ohashi PS, O'Donnell-Tormey J, Old LJ, Ottensmeier C, Papamichail M, Parmiani G, Pawelec G, Proietti E, Qin S, Rees R, Ribas A, Ridolfi R, Ritter G, Rivoltini L, Romero PJ, Salem ML, Scheper RJ, Seliger B, Sharma P, Shiku H, Singh-Jasuja H, Song W, Straten PT, Tahara H, Tian Z, van Der Burg SH, von Hoegen P, Wang E, Welters MJ, Winter H, Withington T, Wolchok JD, Xiao W, Zitvogel L, Zwierzina H, Marincola FM, Gajewski TF, Wigginton JM, Disis ML (2011) Defining the critical hurdles in cancer immunotherapy. J Transl Med 9:214. CrossRefGoogle Scholar
  28. 28.
    Frey BJ, Dueck D (2007) Clustering by passing messages between data points. Science 315:972–976. CrossRefGoogle Scholar
  29. 29.
    Gerdes MJ, Sevinsky CJ, Sood A, Adak S, Bello MO, Bordwell A, Can A, Corwin A, Dinn S, Filkins RJ, Hollman D, Kamath V, Kaanumalle S, Kenny K, Larsen M, Lazare M, Li Q, Lowes C, McCulloch CC, McDonough E, Montalto MC, Pang Z, Rittscher J, Santamaria-Pang A, Sarachan BD, Seel ML, Seppo A, Shaikh K, Sui Y, Zhang J, Ginty F (2013) Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc Natl Acad Sci USA 110:11982–11987. CrossRefGoogle Scholar
  30. 30.
    Goltsev Y, Samusik N, Kennedy-Darling J, Bhate S, Hale M, Vasquez G, Black S, Nolan G (2018) Deep profiling of mouse splenic architecture with CODEX multiplexed imaging. bioRxiv.
  31. 31.
    Heindl A, Nawaz S, Yuan Y (2015) Mapping spatial heterogeneity in the tumor microenvironment: a new era for digital pathology. Lab Investig 95:377–384. CrossRefGoogle Scholar
  32. 32.
    Heindl A, Sestak I, Naidoo K, Cuzick J, Dowsett M, Yuan Y (2018) Relevance of spatial heterogeneity of immune infiltration for predicting risk of recurrence after endocrine therapy of ER+ breast cancer. J Natl Cancer Inst 110.
  33. 33.
    Hoos A (2016) Development of immuno-oncology drugs—from CTLA4 to PD1 to the next generations. Nat Rev Drug Discov 15:235–247. CrossRefGoogle Scholar
  34. 34.
    Hsu FS, Su CH, Huang KH (2017) A comprehensive review of US FDA-approved immune checkpoint inhibitors in urothelial carcinoma. J Immunol Res 2017:6940546. CrossRefGoogle Scholar
  35. 35.
    Huang W, Hennrick K, Drew S (2013) A colorful future of quantitative pathology: validation of Vectra technology using chromogenic multiplexed immunohistochemistry and prostate tissue microarrays. Hum Pathol 44:29–38. CrossRefGoogle Scholar
  36. 36.
    Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R (2014) Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383:166–175. CrossRefGoogle Scholar
  37. 37.
    Katzman JL, Shaham U, Cloninger A, Bates J, Jiang T, Kluger Y (2018) DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med Res Methodol 18:24. CrossRefGoogle Scholar
  38. 38.
    Kearney S, Black J, Aeffner F, Black J, Pratte L, Krueger J (2017) Abstract 4582: evaluating benefits of PD-L1 image analysis for the clinical setting. Cancer Res 77:4582–4582. CrossRefGoogle Scholar
  39. 39.
    Khagi Y, Kurzrock R, Patel SP (2017) Next generation predictive biomarkers for immune checkpoint inhibition. Cancer Metastasis Rev 36:179–190. CrossRefGoogle Scholar
  40. 40.
    Khosravi P, Kazemi E, Imielinski M, Elemento O, Hajirasouliha I (2018) Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine 27:317–328. CrossRefGoogle Scholar
  41. 41.
    Kirilovsky A, Marliot F, El Sissy C, Haicheur N, Galon J, Pages F (2016) Rational bases for the use of the immunoscore in routine clinical settings as a prognostic and predictive biomarker in cancer patients. Int Immunol 28:373–382. CrossRefGoogle Scholar
  42. 42.
    Koelzer VH, Canonica K, Dawson H, Sokol L, Karamitopoulou-Diamantis E, Lugli A, Zlobec I (2016) Phenotyping of tumor-associated macrophages in colorectal cancer: impact on single cell invasion (tumor budding) and clinicopathological outcome. Oncoimmunology 5:e1106677. CrossRefGoogle Scholar
  43. 43.
    Koelzer VH, Dawson H, Andersson E, Karamitopoulou E, Masucci GV, Lugli A, Zlobec I (2015) Active immunosurveillance in the tumor microenvironment of colorectal cancer is associated with low frequency tumor budding and improved outcome. Transl Res 166:207–217. CrossRefGoogle Scholar
  44. 44.
    Koelzer VH, Gisler A, Hanhart JC, Griss J, Wagner SN, Willi N, Cathomas G, Sachs M, Kempf W, Thommen DS, Mertz KD (2018) Digital image analysis improves precision of programmed death ligand 1 (PD-L1) scoring in cutaneous melanoma. Histopathology.
  45. 45.
    Koelzer VH, Sokol L, Zahnd S, Christe L, Dawson H, Berger MD, Inderbitzin D, Zlobec I, Lugli A (2017) Digital analysis and epigenetic regulation of the signature of rejection in colorectal cancer. Oncoimmunology 6:e1288330. CrossRefGoogle Scholar
  46. 46.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst:1097–1105Google Scholar
  47. 47.
    Laubli H, Koelzer VH, Matter MS, Herzig P, Dolder Schlienger B, Wiese MN, Lardinois D, Mertz KD, Zippelius A (2018) The T cell repertoire in tumors overlaps with pulmonary inflammatory lesions in patients treated with checkpoint inhibitors. Oncoimmunology 7:e1386362. CrossRefGoogle Scholar
  48. 48.
    Lawler M, Alsina D, Adams RA, Anderson AS, Brown G, Fearnhead NS, Fenwick SW, Halloran SP, Hochhauser D, Hull MA, Koelzer VH, McNair AGK, Monahan KJ, Nathke I, Norton C, Novelli MR, Steele RJC, Thomas AL, Wilde LM, Wilson RH, Tomlinson I, Bowel Cancer UKCRGiCCI (2018) Critical research gaps and recommendations to inform research prioritisation for more effective prevention and improved outcomes in colorectal cancer. Gut 67:179–193. CrossRefGoogle Scholar
  49. 49.
    Le DT, Durham JN, Smith KN, Wang H, Bartlett BR, Aulakh LK, Lu S, Kemberling H, Wilt C, Luber BS, Wong F, Azad NS, Rucki AA, Laheru D, Donehower R, Zaheer A, Fisher GA, Crocenzi TS, Lee JJ, Greten TF, Duffy AG, Ciombor KK, Eyring AD, Lam BH, Joe A, Kang SP, Holdhoff M, Danilova L, Cope L, Meyer C, Zhou S, Goldberg RM, Armstrong DK, Bever KM, Fader AN, Taube J, Housseau F, Spetzler D, Xiao N, Pardoll DM, Papadopoulos N, Kinzler KW, Eshleman JR, Vogelstein B, Anders RA, Diaz LA Jr (2017) Mismatch repair deficiency predicts response of solid tumors to PD-1 blockade. Science 357:409–413. CrossRefGoogle Scholar
  50. 50.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444. CrossRefGoogle Scholar
  51. 51.
    LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE:2278–2324Google Scholar
  52. 52.
    Louis DN, Gerber GK, Baron JM, Bry L, Dighe AS, Getz G, Higgins JM, Kuo FC, Lane WJ, Michaelson JS, Le LP, Mermel CH, Gilbertson JR, Golden JA (2014) Computational pathology: an emerging definition. Arch Pathol Lab Med 138:1133–1138. CrossRefGoogle Scholar
  53. 53.
    Luksza M, Riaz N, Makarov V, Balachandran VP, Hellmann MD, Solovyov A, Rizvi NA, Merghoub T, Levine AJ, Chan TA, Wolchok JD, Greenbaum BD (2017) A neoantigen fitness model predicts tumour response to checkpoint blockade immunotherapy. Nature 551:517–520. CrossRefGoogle Scholar
  54. 54.
    Ma Z, Shiao SL, Yoshida EJ, Swartwood S, Huang F, Doche ME, Chung AP, Knudsen BS, Gertych A (2017) Data integration from pathology slides for quantitative imaging of multiple cell types within the tumor immune cell infiltrate. Diagn Pathol 12:69. CrossRefGoogle Scholar
  55. 55.
    Martin NT, Black JC, Pollack Z, Aeffner F, Krueger J (2017) Abstract 661: evaluating “harmonization” of PD-L1 assays using image analysis. Cancer Res 77:661–661. CrossRefGoogle Scholar
  56. 56.
    Mezheyeuski A, Bergsland CH, Backman M, Djureinovic D, Sjoblom T, Bruun J, Micke P (2017) Multispectral imaging for quantitative and compartment-specific immune infiltrates reveals distinct immune profiles that classify lung cancer patients. J Pathol.
  57. 57.
    Mlecnik B, Van den Eynde M, Bindea G, Church SE, Vasaturo A, Fredriksen T, Lafontaine L, Haicheur N, Marliot F, Debetancourt D, Pairet G, Jouret-Mourin A, Gigot JF, Hubert C, Danse E, Dragean C, Carrasco J, Humblet Y, Valge-Archer V, Berger A, Pages F, Machiels JP, Galon J (2018) Comprehensive Intrametastatic immune quantification and major impact of immunoscore on survival. J Natl Cancer Inst 110.
  58. 58.
    Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velazquez Vega JE, Brat DJ, Cooper LAD (2018) Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A 115:E2970–E2979. CrossRefGoogle Scholar
  59. 59.
    Moran S, Martinez-Cardus A, Sayols S, Musulen E, Balana C, Estival-Gonzalez A, Moutinho C, Heyn H, Diaz-Lagares A, de Moura MC, Stella GM, Comoglio PM, Ruiz-Miro M, Matias-Guiu X, Pazo-Cid R, Anton A, Lopez-Lopez R, Soler G, Longo F, Guerra I, Fernandez S, Assenov Y, Plass C, Morales R, Carles J, Bowtell D, Mileshkin L, Sia D, Tothill R, Tabernero J, Llovet JM, Esteller M (2016) Epigenetic profiling to classify cancer of unknown primary: a multicentre, retrospective analysis. Lancet Oncol 17:1386–1395. CrossRefGoogle Scholar
  60. 60.
    Motzer RJ, Escudier B, McDermott DF, George S, Hammers HJ, Srinivas S, Tykodi SS, Sosman JA, Procopio G, Plimack ER, Castellano D, Choueiri TK, Gurney H, Donskov F, Bono P, Wagstaff J, Gauler TC, Ueda T, Tomita Y, Schutz FA, Kollmannsberger C, Larkin J, Ravaud A, Simon JS, Xu LA, Waxman IM, Sharma P, CheckMate I (2015) Nivolumab versus everolimus in advanced renal-cell carcinoma. N Engl J Med 373:1803–1813. CrossRefGoogle Scholar
  61. 61.
    Obermeyer Z, Emanuel EJ (2016) Predicting the future—big data, machine learning, and clinical medicine. N Engl J Med 375:1216–1219. CrossRefGoogle Scholar
  62. 62.
    Overman MJ, McDermott R, Leach JL, Lonardi S, Lenz HJ, Morse MA, Desai J, Hill A, Axelson M, Moss RA, Goldberg MV, Cao ZA, Ledeine JM, Maglinte GA, Kopetz S, Andre T (2017) Nivolumab in patients with metastatic DNA mismatch repair-deficient or microsatellite instability-high colorectal cancer (CheckMate 142): an open-label, multicentre, phase 2 study. Lancet Oncol 18:1182–1191. CrossRefGoogle Scholar
  63. 63.
    Pagès F, Mlecnik B, Marliot F, Bindea G, Ou F-S, Bifulco C, Lugli A, Zlobec I, Rau TT, Berger MD, Nagtegaal ID, Vink-Börger E, Hartmann A, Geppert C, Kolwelter J, Merkel S, Grützmann R, Van den Eynde M, Jouret-Mourin A, Kartheuser A, Léonard D, Remue C, Wang JY, Bavi P, Roehrl MHA, Ohashi PS, Nguyen LT, Han S, MacGregor HL, Hafezi-Bakhtiari S, Wouters BG, Masucci GV, Andersson EK, Zavadova E, Vocka M, Spacek J, Petruzelka L, Konopasek B, Dundr P, Skalova H, Nemejcova K, Botti G, Tatangelo F, Delrio P, Ciliberto G, Maio M, Laghi L, Grizzi F, Fredriksen T, Buttard B, Angelova M, Vasaturo A, Maby P, Church SE, Angell HK, Lafontaine L, Bruni D, El Sissy C, Haicheur N, Kirilovsky A, Berger A, Lagorce C, Meyers JP, Paustian C, Feng Z, Ballesteros-Merino C, Dijkstra J, van de Water C, van Lent-van Vliet S, Knijn N, Mușină A-M, Scripcariu D-V, Popivanova B, Xu M, Fujita T, Hazama S, Suzuki N, Nagano H, Okuno K, Torigoe T, Sato N, Furuhata T, Takemasa I, Itoh K, Patel PS, Vora HH, Shah B, Patel JB, Rajvik KN, Pandya SJ, Shukla SN, Wang Y, Zhang G, Kawakami Y, Marincola FM, Ascierto PA, Sargent DJ, Fox BA, Galon J International validation of the consensus Immunoscore for the classification of colon cancer: a prognostic and accuracy study Lancet.
  64. 64.
    Paik S, Shak S, Tang G, Kim C, Baker J, Cronin M, Baehner FL, Walker MG, Watson D, Park T, Hiller W, Fisher ER, Wickerham DL, Bryant J, Wolmark N (2004) A multigene assay to predict recurrence of tamoxifen-treated, node-negative breast cancer. N Engl J Med 351:2817–2826. CrossRefGoogle Scholar
  65. 65.
    Pantanowitz L, Sinard JH, Henricks WH, Fatheree LA, Carter AB, Contis L, Beckwith BA, Evans AJ, Lal A, Parwani AV, College of American Pathologists P, Laboratory Quality C (2013) Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch Pathol Lab Med 137:1710–1722. CrossRefGoogle Scholar
  66. 66.
    Remark R, Merghoub T, Grabe N, Litjens G, Damotte D, Wolchok JD, Merad M, Gnjatic S (2016) In-depth tissue profiling using multiplexed immunohistochemical consecutive staining on single slide. Sci Immunol 1:aaf6925. CrossRefGoogle Scholar
  67. 67.
    Salto-Tellez M, Maxwell P, Hamilton P (2018) Artificial intelligence—the third revolution in pathology. Histopathology.
  68. 68.
    Saltz J, Gupta R, Hou L, Kurc T, Singh P, Nguyen V, Samaras D, Shroyer KR, Zhao T, Batiste R, Van Arnam J, Cancer Genome Atlas Research N, Shmulevich I, AUK R, Lazar AJ, Sharma A, Thorsson V (2018) Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep 23:181–193 e187. CrossRefGoogle Scholar
  69. 69.
    Schapiro D, Jackson HW, Raghuraman S, Fischer JR, Zanotelli VRT, Schulz D, Giesen C, Catena R, Varga Z, Bodenmiller B (2017) histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. Nat Methods 14:873–876. CrossRefGoogle Scholar
  70. 70.
    Schubert W, Bonnekoh B, Pommer AJ, Philipsen L, Bockelmann R, Malykh Y, Gollnick H, Friedenberger M, Bode M, Dress AW (2006) Analyzing proteome topology and function by automated multidimensional fluorescence microscopy. Nat Biotechnol 24:1270–1278. CrossRefGoogle Scholar
  71. 71.
    Schulz D, Zanotelli VRT, Fischer JR, Schapiro D, Engler S, Lun XK, Jackson HW, Bodenmiller B (2018) Simultaneous multiplexed imaging of mrna and proteins with subcellular resolution in breast cancer tissue samples by mass cytometry. Cell Syst 6:25–36 e25. CrossRefGoogle Scholar
  72. 72.
    Sirinukunwattana K, Ahmed Raza SE, Yee-Wah T, Snead DR, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 35:1196–1206. CrossRefGoogle Scholar
  73. 73.
    Sood A, Miller AM, Brogi E, Sui Y, Armenia J, McDonough E, Santamaria-Pang A, Carlin S, Stamper A, Campos C, Pang Z, Li Q, Port E, Graeber TG, Schultz N, Ginty F, Larson SM, Mellinghoff IK (2016) Multiplexed immunofluorescence delineates proteomic cancer cell states associated with metabolism. JCI Insight 1.
  74. 74.
    Spranger S, Luke JJ, Bao R, Zha Y, Hernandez KM, Li Y, Gajewski AP, Andrade J, Gajewski TF (2016) Density of immunogenic antigens does not explain the presence or absence of the T cell-inflamed tumor microenvironment in melanoma. Proc Natl Acad Sci USA 113:E7759–E7768. CrossRefGoogle Scholar
  75. 75.
    Spranger S, Spaapen RM, Zha Y, Williams J, Meng Y, Ha TT, Gajewski TF (2013) Up-regulation of PD-L1, IDO, and T(regs) in the melanoma tumor microenvironment is driven by CD8(+) T cells. Sci Transl Med 5:200ra116. CrossRefGoogle Scholar
  76. 76.
    Thommen DS, Koelzer VH, Herzig P, Roller A, Trefny M, Dimeloe S, Kiialainen A, Hanhart J, Schill C, Hess C, Savic Prince S, Wiese M, Lardinois D, Ho PC, Klein C, Karanikas V, Mertz KD, Schumacher TN, Zippelius A (2018) A transcriptionally and functionally distinct PD-1(+) CD8(+) T cell pool with predictive potential in non-small-cell lung cancer treated with PD-1 blockade. Nat Med 24:994–1004. CrossRefGoogle Scholar
  77. 77.
    Troncone G, Gridelli C (2017) The reproducibility of PD-L1 scoring in lung cancer: can the pathologists do better? Transl Lung Cancer Res 6:S74–S77. CrossRefGoogle Scholar
  78. 78.
    Tsujikawa T, Kumar S, Borkar RN, Azimi V, Thibault G, Chang YH, Balter A, Kawashima R, Choe G, Sauer D, El Rassi E, Clayburgh DR, Kulesz-Martin MF, Lutz ER, Zheng L, Jaffee EM, Leyshock P, Margolin AA, Mori M, Gray JW, Flint PW, Coussens LM (2017) Quantitative multiplex immunohistochemistry reveals myeloid-inflamed tumor-immune complexity associated with poor prognosis. Cell Rep 19:203–217. CrossRefGoogle Scholar
  79. 79.
    Tumeh PC, Harview CL, Yearley JH, Shintaku IP, Taylor EJ, Robert L, Chmielowski B, Spasic M, Henry G, Ciobanu V, West AN, Carmona M, Kivork C, Seja E, Cherry G, Gutierrez AJ, Grogan TR, Mateus C, Tomasic G, Glaspy JA, Emerson RO, Robins H, Pierce RH, Elashoff DA, Robert C, Ribas A (2014) PD-1 blockade induces responses by inhibiting adaptive immune resistance. Nature 515:568–571. CrossRefGoogle Scholar
  80. 80.
    Ung C, Kockx M, Waumans Y (2017) Digital pathology in immuno-oncology—a roadmap for clinical development. Expert Rev Precis Med Drug Dev 2:9–19. CrossRefGoogle Scholar
  81. 81.
    Virchow R (1858) Die Cellularpathologie in ihrer Begründung auf physiologische und pathologische Gewebelehre. August Hirschwald, BerlinGoogle Scholar
  82. 82.
    Wang CW, Ka SM, Chen A (2014) Robust image registration of biological microscopic images. Sci Rep 4:6050. CrossRefGoogle Scholar
  83. 83.
    Webb S (2018) Deep learning for biology. Nature 554:555–557. CrossRefGoogle Scholar
  84. 84.
    Williams BJ, Bottoms D, Treanor D (2017) Future-proofing pathology: the case for clinical adoption of digital pathology. J Clin Pathol 70:1010–1018. CrossRefGoogle Scholar
  85. 85.
    Xie W, Noble JA, Zisserman A (2018) Microscopy cell counting and detection with fully convolutional regression networks. Comput Meth Biomech Biomed Eng Imaging Visualization 6:283–292. CrossRefGoogle Scholar
  86. 86.
    Yousefi S, Amrollahi F, Amgad M, Dong C, Lewis JE, Song C, Gutman DA, Halani SH, Velazquez Vega JE, Brat DJ, Cooper LAD (2017) Predicting clinical outcomes from large scale cancer genomic profiles with deep survival models. Sci Rep 7:11707. CrossRefGoogle Scholar
  87. 87.
    Yuan Y (2015) Modelling the spatial heterogeneity and molecular correlates of lymphocytic infiltration in triple-negative breast cancer. J R Soc Interface 12.

Copyright information

© The Author(s) 2018

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Institute of Cancer and Genomic ScienceUniversity of BirminghamBirminghamUK
  2. 2.Molecular and Population Genetics Laboratory, Wellcome Centre for Human GeneticsUniversity of OxfordOxfordUK
  3. 3.Institute of Biomedical Engineering, Department of Engineering ScienceUniversity of OxfordOxfordUK
  4. 4.Ludwig Institute for Cancer Research, Nuffield Department of MedicineUniversity of OxfordOxfordUK
  5. 5.Target Discovery Institute, NDM Research BuildingUniversity of OxfordHeadingtonUK
  6. 6.Institute of PathologyCantonal Hospital BasellandLiestalSwitzerland

Personalised recommendations