The CIMT-monitoring panel: a two-step approach to harmonize the enumeration of antigen-specific CD8+ T lymphocytes by structural and functional assays
- First Online:
- Received:
- Accepted:
DOI: 10.1007/s00262-007-0378-0
- Cite this article as:
- Britten, C.M., Gouttefangeas, C., Welters, M.J.P. et al. Cancer Immunol Immunother (2008) 57: 289. doi:10.1007/s00262-007-0378-0
Abstract
The interpretation of the results obtained from immunomonitoring of clinical trials is a difficult task due to the variety of methods and protocols available to detect vaccine-specific T-cell responses. This heterogeneity as well as the lack of standards has led to significant scepticism towards published results. In February 2005, a working group was therefore founded under the aegis of the Association for Immunotherapy of Cancer (“CIMT”) in order to compare techniques and protocols applied for the enumeration of antigen-specific T-cell responses. Here we present the results from two consecutive phases of an international inter-laboratory testing project referred to as the “CIMT monitoring panel”. A total of 13 centers from six European countries participated in the study in which pre-tested PBMC samples, synthetic peptides and PE-conjugated HLA-tetramers were prepared centrally and distributed to participants. All were asked to determine the number of antigen-specific T-cells in each sample using tetramer staining and one functional assay. The results of the first testing round revealed that the total number of cells analyzed was the most important determinant for the sensitive detection of antigen-specific CD8+ T-cells by tetramer staining. Analysis by ELISPOT was influenced by a combination of cell number and a resting phase after thawing of peripheral blood mononuclear cells. Therefore, the experiments were repeated in a second phase but now the participants were asked to change their protocols according to the new guidelines distilled from the results of the first phase. The recommendations improved the number of antigen-specific T-cell responses that were detected and decreased the variability between the laboratories. We conclude that a two-step approach in inter-laboratory testing allows the identification of distinct variables that influence the sensitivity of different T-cell assays and to formally show that a defined correction to the protocols successfully increases the sensitivity and reduces the inter-center variability. Such “two-step” inter-laboratory projects could define rational bases for accepted international guidelines and thereby lead to the harmonization of the techniques used for immune monitoring.
Keywords
ELISPOT Tetramer Standardization Interlaboratory testingAbbreviations
- APC
Antigen presenting cell
- ELISPOT
Enzyme linked immuno spot
- PBMC
Peripheral blood mononuclear cells
- IFNγ
Interferon gamma
- ICS
Intracellular cytokine staining
- ND
Not determined
Introduction
In the last two decades, more than 200 clinical trials of different anti-tumor vaccines aiming to induce tumor-specific immunity in cancer patients have been described [1]. Most of these trials primarily assessed safety and immunogenicity while reporting partial or complete clinical responses in a minority of patients [2, 3]. Despite the fact that the low fraction of clinical responders still precludes the establishment of a direct correlation between clinical efficacy and T-cell reactivity, it has become clear from animal models and clinical observations that naturally-occurring or vaccine-induced CD8+ or CD4+ T-cells play an important role in the control and regression of tumors [4, 5, 6, 7, 8, 9]. Therefore, the number of subjects that mount a vaccine-induced T-cell response as well as the strength of a detected T-cell response represent important surrogate markers for vaccine efficacy. The enzyme-linked immunospot (ELISPOT) assay [10, 11], staining with HLA-peptide multimers [12] and intracellular cytokine staining (ICS) [13, 14] are technologies used commonly for the monitoring of antigen-specific immune responses. For these three assays, a huge variety of different protocols are available worldwide. This heterogeneity, together with the fact that the sensitivity of the individual protocols can vary significantly, makes a comparison of the results obtained in different trials a difficult task. Moreover, an increasing number of new technologies are constantly being introduced to the field, which makes interpretations even more complex [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Current data and opinion support the use of a functional assay like the ELISPOT or ICS in combination with a phenotyping assay like HLA-multimers [26, 27], but recognized international standards for all these methodologies are still lacking.
The main aim of the “CIMT monitoring panel” is to harmonize and optimize the monitoring of antigen-specific T-cells among the participating laboratories, based on objective rationales with respect to the testing procedure, the analysis and the interpretation of results. Important requirements for an immunological test are sensitivity, applicability to large amounts of clinical material and feasibility at reasonable cost. The results generated by the tests should be reproducible and sensitive, independently of the place where they have been performed. After the first meeting of the working group, a series of inter-laboratory testing projects was initiated, in which individual laboratories could compare their performance, express their needs and exchange experience in order to improve their local assays. Here we report the results of the first two phases of the CIMT monitoring panel, with 13 participating centers from six European countries.
Materials and methods
Preparation and screening of PBMC samples
Buffy coats from HLA-typed healthy volunteers were kindly provided from the Blood Bank of the University Mainz. HCMV sero-status was known. PBMC were isolated by Ficoll density gradient separation (Pharmacia, Uppsala, Sweden), washed two times in RPMI 1640 (GIBCO BRL, Grand Island, NY, USA) containing 10 mM Hepes buffer, l-arginine (116 mg/ml), l-glutamine (216 mg/ml), penicillin (10 IU/ml), streptomycin (100 mg/ml) and 10% FCS (GIBCO BRL), counted and frozen at 10 to 20 × 106 cells per cryovial in 1 ml of FCS 90% + DMSO 10% at −80°C in freezing-boxes filled with iso-propanol. After 20 h, all cryovials were transferred to liquid nitrogen and stored until distribution to the participating laboratories.
Pre-screening and selection of the PBMC donors for influenza- and CMV- T-cell reactivities were performed by a central lab using the IFNγ ELISPOT assay following a local protocol as described previously [38]. Five donors were selected for the first phase of the panel and eight for the second phase. One HLA-A*0201-negative donor was included in each phase (negative control), all other samples were HLA-A*0201-positive.
Synthetic peptides and HLA-tetramers
Peptides were synthesized using standard Fmoc chemistry, dissolved at 10 mg/ml in DMSO, aliquotted and stored at −80°C. The purity was checked by reverse-phase HPLC and was found to be >80%. Two known HLA-A*0201 T-cell epitopes were used: influenza MP 58–66 GILGFVFTL and HCMV pp65 495–503 NLVPMVATV (http://www.syfpeithi.de). Biotinylated recombinant HLA-A*0201 monomers folded with the influenza MP 58–66 or the HMCV pp65 peptides were produced essentially as described, purified by gel filtration and stored as aliquots at −80°C [12]. Fluorescent multimers were obtained by incubation with streptavidin-PE (Molecular Probes, Leiden, The Netherlands), then frozen as aliquots after addition of 0.5% BSA and 16% glycerol. HLA-concentrations of influenza-tetramer and HCMV-tetramers were 700 and 350 μg/ml, respectively. Both tetramers were checked by HPLC and/or validated by staining of a specific CD8+ T-cell line (Influenza) or PBMC from HLA-A2-negative and HLA-A2-positive CMV seronegative donors (CMV). Such tetramers are stable at 4°C for at least 1 month (personal observation) and participants were asked to perform all tests within this time period.
Participating centers
Twelve centers from five European countries participated in the first phase of the monitoring panel. As one of the investigators moved to another institution during the study a 13th center from a 6th European country was added to the group in the second phase of the panel. Participation in the panel was open to all interested laboratories with a focus on T-cell monitoring, independently of membership in the Association for Immunotherapy of Cancer.
Reagent distribution and assay guidelines
Coded PBMC samples, synthetic peptides and HLA-A*0201 tetramers were shipped on dry ice to the participants. Additionally, guidelines for the two T-cell assays were distributed for each phase:
Phase I/2005. A protocol for tetramer staining was included. Briefly, 1 × 106 PBMC per test were transferred directly after thawing into one well of a 96 well u-bottom plate and washed in FACS buffer consisting of PBS, 2% FCS, 2 mM EDTA, 0.02% azide. Incubation with 5 μg/ml HLA-tetramer was then performed in FACS buffer with 50% FCS for 30 min at room temperature in the dark. After one wash in FACS buffer, mAb for T-cell staining were added for 20 min at 4°C. Finally, cells were washed twice before fixing in FACS buffer containing 1% formaldehyde solution. Three mAb combinations were proposed, CD8 alone, CD3 plus CD8, or CD4 plus CD8. Each lab could choose here the antibody clones, fluorescent dye and concentrations used. Stainings were performed in duplicate.
For the functional assays, synthetic peptides were diluted at 1 mg/ml in PBS as a stock solution. Concentrations in further tests were 1–10 μg/ml, left to the choice of the participants. There were no recommendation which functional test should be performed, so that each group could choose the test either routinely used, or to be implemented for its own needs. In this first phase, 11/12 laboratories chose the IFNγ ELISPOT assay, one lab (Z10) a FACS-based intracellular IFNγ staining and one lab performed both assays (Z7). Spot counting was performed locally.
Phase II/2006. Following the results obtained in the first testing phase, requirements were introduced and participants were asked to apply exactly these new criteria (two for the tetramer staining, and four for the ELISPOT, see “Results” section). The assay guidelines were modified accordingly. However, in order to reduce the variability in the FACS analysis of the 13 laboratories, a figure showing exemplary dot-plots, settings of gates and quadrants, and statistical analysis was provided. All laboratories were now required to perform an IFNγ ELISPOT as the functional test, with a fixed peptide concentration of 1 μg/ml. Participants were encouraged to use a distributed model protocol but were allowed to use their local protocol, provided that they applied the four new requirements introduced in the second phase.
Collection and analysis of results
After performing the required tests in each phase, participants returned a completed report form containing all relevant information. Number of cells recovered after thawing was included to assess viability after transport. For the tetramer staining experiments, mAb clone, manufacturer, amount, cytometer type and number of lymphocytes and/or CD8+ cells analyzed were noted. Results were expressed as percentage of tetramer-positive cells among CD8+, CD3+CD8+, or CD4− lymphocytes, depending on which mAb combination was used for the staining. Additionally, FACS dot-plots containing all gates, quadrants and deduced statistical analysis were collected. For the functional test, medium and thawing procedure (e.g. addition of DNAse, of a resting phase, etc.) had to be described, as well as the number of cells per test, the antibodies used (clone, manufacturer, concentration), the final peptide concentration and the incubation times. For the ELISPOT assay, the type of plate, the enzymatic visualization system and the spot reader were also noted. Absolute spot numbers were given by each participant, and filter plates were kept for possible second analysis.
All results from both phases were collected and centrally analyzed. For the tetramer stainings, the number of lymphocytes, number of CD8+ T-cells and frequencies of tetramer-positive cells were calculated on the basis of the stainings and statistics provided by the participants. Apart from these calculated frequencies, a “visual evaluation” was necessary (see “Results”). For the ELISPOT, analysis was performed based on the spot numbers reported by the participants, followed by a student t test. Results were accepted as positive reaction only when the numbers of antigen-specific spots exceeded the number of spots in the background wells by atleast a factor two. The coefficients of variation (CV) were calculated for all results (CV = SD/mean × 100) and are shown in supplementary Tables S1a, b.
The raw data from both panel phases will be provided to interested readers upon request.
Results
Phase I/2005 of the interlaboratory testing project—general aspects
Coded PBMC samples from four HLA-A*0201-positive and one HLA-A*0201-negative healthy donor (D1–D5) were included in this first testing phase. The thawing procedure for PBMC samples in the test centers was not standardized and the recovery of viable cells varied greatly between 45 and 102% (mean 73%) in the 12 labs. However, the number of cells recovered was in all cases sufficient to perform the required analyses. When all the data from the tetramer staining and functional tests were combined it became clear that subjects D1 and D5 had responded to the HLA-A*0201 restricted CMV-derived peptide, consistent with their CMV seropositive-status, and that subjects D1, D2, D3, and D5 had responded to influenza. In total, each laboratory should in theory have been able to measure six positive (2× CMV and 4× influenza) responses.
Detection of antigen-specific T-cells by tetramer staining and IFNγ ELISPOT
Example of tetramer staining results as provided by four selected participating centers Z5, Z12, Z8 and Z1. All stainings were performed on donor D1 from phase I/2005 who showed reactivity with both of the tested tetramers. Cells were gated either on the lymphocyte population (Z1), or the subsets of CD3+CD8+ (Z5) or CD3+ (Z8, Z12), according to the Ab combination used by each lab. The upper panel shows results for tests with the CMV-tetramer, the lower panel shows results for tests with the influenza-tetramer. In all dot-plots, the tetramer staining is displayed on the y-axis and anti-CD8-staining on the x-axis. Number of counted CD8+ T-cells and percentage of tetramer-positive cells among the CD8 subset are indicated
Overview of the tetramer results from phase I/2005 of the CIMT monitoring panel
D1 CMV | D1 Flu | D2 Flu | D3 Flu | D5 CMV | D5 Flu | ||
|---|---|---|---|---|---|---|---|
Min | 448 | 20,000 | 6,667 | 1,818 | 106 | 1,786 | (I) |
Mean | 141 | 8,095 | 1,909 | 1,014 | 80 | 1,106 | |
Max | 35 | 3,774 | 526 | 595 | 26 | 588 | |
Z1 | 129 | 3,774 | 635 | 870 | 106 | 758 | (II) |
Z2 | 112 | – | – | 1,818 | 70 | 1,770 | |
Z3 | n.d | – | – | 701 | n.d | – | |
Z4 | 97 | – | – | 854 | 106 | 1,342 | |
Z5 | 161 | – | – | 909 | 88 | 606 | |
Z6 | 139 | 6,896 | 1,316 | 1,000 | 91 | 651 | |
Z7 | 448 | 20,000 | 6,667 | 1,266 | 82 | 1,724 | |
Z8 | 100 | 5,128 | 545 | 595 | 87 | 877 | |
Z9 | n.d | – | 1,316 | 1,274 | n.d | 1,786 | |
Z10 | 94 | – | 2,597 | 1,360 | 86 | 1,439 | |
Z11 | 99 | 4,675 | 1,667 | 877 | 58 | 588 | |
Z12 | 35 | – | 526 | 645 | 26 | 625 | |
Detected by | 10/10 | 5/12 | 8/12 | 12/12 | 10/10 | 11/12 | (III) |
Detected % | 100 | 42 | 67 | 100 | 100 | 92 |
Overview of the IFNγ ELISPOT results from phase I/2005 of the CIMT monitoring panel
D1 CMV | D1 Flu | D2 Flu | D3 Flu | D5 CMV | D5 Flu | ||
|---|---|---|---|---|---|---|---|
Min | 3,061 | 62,500 | 55,555 | 33,333 | 11,428 | 50,000 | (I) |
Mean | 1,855 | 38,141 | 43,589 | 17,547 | 4,405 | 30,811 | |
Max | 888 | 10,256 | 30,769 | 8,571 | 1,039 | 14,705 | |
Z1 | 1,006 | – | – | 8,571 | – | – | (II) |
Z2 | 2,439 | – | – | – | 2,816 | – | |
Z3 | 1,295 | 62,500 | – | 22,727 | 1,412 | 27,727 | |
Z4 | 1,312 | – | – | 10,909 | 1,980 | – | |
Z5 | – | – | – | – | – | – | |
Z6 | 233* | – | – | 1,960* | 253* | – | |
Z7 | 1,895 | 10,256 | 44,444 | 33,333 | 11,428 | 50,000 | |
Z8 | 3,061 | – | – | – | 6,896 | – | |
Z9 | 888 | 41,666 | 55,555 | 12,195 | 1,039 | 14,705 | |
Z10 | ND | ND | ND | ND | ND | ND | |
Z11 | 1,769 | – | 30,769 | – | – | – | |
Z12 | 3,030 | – | – | – | 5,263 | – | |
Detected by | 10/11 | 3/11 | 3/11 | 6/11 | 8/11 | 3/11 | (III) |
Detected % | 91 | 27 | 27 | 55 | 73 | 27 |
Subgroup analysis reveals that the number of CD8+ T-lymphocytes analyzed affects the sensitivity of the tetramer staining
Although the tetramer stainings were performed with centrally prepared reagents following set guidelines, centers were left free to select several parameters according to their own protocols, and this could have influenced the test results (see “Materials and methods”). Most of the participants used monoclonal antibodies specific for CD3 and CD8 to co-stain the cells. There were no obvious differences in the performance of the centers depending on which antibody clones, antibody combinations or cytometer were used (data not shown).
a Subgroup analysis of tetramer results from phase I/2005. Bars indicate the percentage of positives that could be detected by tetramer staining. The first group of bars shows the results for all of the six detectable positives, the second group shows results from stainings with the CMV-tetramer and the third group of columns shows results from stainings with the influenza-tetramer. The open bars in each group represent all tests performed, grey bars represent results obtained in tests that were performed on more than 3 × 104 CD8+ T-cells and black bars represent results obtained in tests that were performed on less than 3 × 104 CD8+ T-cells. The boxes within each bar indicate the fraction of tests with a positive result. The asterisk indicates a P-value < 0.05 by Chi-square analysis. b Subgroup analysis of ELISPOT results from phase I/2005. The bars indicate the percentage of positive reactivities detected by IFNγ ELISPOT assays. The open bar shows the percentage of all reactivities detected by all 11 centers that performed the ELISPOT assay as the functional test. Criteria for division of centers into two subgroups were based on the following requirements: do not use allo-APC (first subgroup analysis), use a resting time (second subgroup analysis) or use equal or more than 400,000 PBMC per well (third subgroup analysis). Grey bars always represent centers that were in conformity with the indicated minimum requirement, black bars show results from centers that did not fulfil that requirement. The boxes within each column indicate the fraction of centers in each category. The asterisks indicate a P-value < 0.05 in Chi-square analysis
In conclusion, the ability to detect antigen-specific T-cell reactivities by tetramer staining was mainly affected by the number of CD8+ T-cells stained and analyzed, especially when the antigen-specific T-cells were present at low or moderate frequencies. We therefore modified our guidelines for the tetramer assay and recommended staining at least 1 × 106 PBMC and analyzing all cells in the tube. In addition, we provided an example of how optimal cell gates and dot-plot quadrants could be selected.
ELISPOT assays are heterogeneous and require standardization
The ELISPOT analyses were performed according to 11 more or less different protocols. The most discernible differences that were observed in these protocols concerned (1) the different types of multi-screen plates, (2) the serum origin, (3) the use of duplicates, triplicates or quadruplicates, (4) the use of allogeneic APC, (5) the inclusion of a resting phase after thawing the PBMC, (6) the number of PBMC per well, (7) the type of antibodies used, (8) the type of spot-reader, and the (9) enzyme and substrate for staining of the spots. Each center also used a different plate protocol (distribution of the wells, number of replicates, control tests).
The influence of each of these parameters on the number of positive responses was studied by further analysis in which the laboratories were divided into two subgroups. As a result, several criteria were identified which could help to improve the sensitivity and comparability of detection.
All data sets (duplicates, triplicates or quadruplicates) were first analyzed by Student t test for unpaired samples (“Materials and methods”). In our panel, one center used quadruplicates, nine centers used triplicates and one center performed the ELISPOT analysis in duplicates. Due to the variety in the replicates, responses measured by duplicate wells failed to pass the Student t test more often as compared to triplicates.
Overall, the 11 centers were able to detect 50% of all possible reactivities in this panel phase (Table 2; Fig. 2b). In a subgroup of three laboratories (Z5, Z6 and Z8), an allogeneic APC population (T2 or K562-A*0201 cells) was added for binding and presentation of the synthetic peptides. The three centers that used allo-APC detected only 28% of all responses, while the other centers detected 58% of all responses.
In five laboratories (Z3, Z4, Z7, Z8, and Z9) PBMC were thawed, and then incubated in culture medium at 37°C. After this resting phase of 2–20 h, living cells were washed, counted and seeded into ELISPOT plates. Laboratories using a resting phase detected 73% of the positive reactivities (22 out of 30 potentially positive tests). No significant difference in the ability to detect antigen-specific T cells was found using shorter or longer resting-times. In contrast, the laboratories that did not use a resting procedure detected only 30% of all positives (Fig. 2b).
Finally, the number of cells seeded per well differed considerably between all participants and ranged from 1 to 6 × 105 PBMC. We divided the laboratories arbitrarily into two groups, those using either more than 4 × 105 PBMC (Z4, Z7, Z8, and Z9) or less than 4 × 105 PBMC (Z1, Z2, Z3, Z11 and Z12). The first group detected 71% of all positive samples, whereas the second group was able to detect only 43% of all positives (Fig. 2b). Centers Z5 and Z6 used a defined number of separated CD8+ T-cells in the ELISPOT and were therefore not included in this subgroup analysis.
None other of the nine depicted protocol variables had any obvious impact on the detection of specific T-cells. As a conclusion from these results, four minimum requirements were formulated for the ELISPOT protocol: (1) perform triplicates for each test antigen (2) do not use allo-APC (3) add a resting time to increase the proportion of living cells seeded and (4) use a minimum number of 4 × 105 PBMC per well.
Phase II/2006 of the interlaboratory testing project—general aspects
To formally prove that the requirements formulated for tetramer staining and ELISPOT analysis increase the ability of the participants to detect antigen-specific CD8+ T-cells and reduce the inter-center variability, we decided to repeat the analysis in a second phase of the panel, with the same participants (phase II/2006). In this round, all groups were asked to follow our modified guidelines for the tetramer- and the ELISPOT-assays.
Again, all PBMC samples were prepared and pre-tested in one central lab and peptide antigens and PE-conjugated tetramers were also provided from one source. As one investigator had meanwhile moved to another lab, we added a 13th center to the group. PBMC from seven selected healthy HLA-A*0201-positive donors and 1 HLA-A*0201-negative donor (D3) were required to be analyzed for the presence of HLA-A*0201-restricted CMV-specific T cells and for influenza-specific T-cells. The mean number of recovered cells after thawing was sufficient to perform the tests. When all the data were combined, it became clear that subjects D2, D5 and D8 possessed CMV-specific CD8+ T-cell subsets, and D1, D2, D4, D6 and D7 possessed influenza-specific CD8+ T-cells. Therefore, each laboratory could theoretically have measured eight positives (3× CMV and 5× Influenza) in this second phase.
Analysis of CD8+ T-cell tetramer binding using the new guidelines
Overview of tetramer results from phase II/2006 of the CIMT monitoring panel
D1 Flu | D2 CMV | D2 Flu | D4 Flu | D5 CMV | D6 Flu | D7 Flu | D8 CMV | ||
|---|---|---|---|---|---|---|---|---|---|
Min | 7,143 | 77 | 10,000 | 10,000 | 60 | 3,333 | 869 | 42 | (I) |
Medium | 3,739 | 45 | 3,573 | 5,278 | 37 | 1,116 | 347 | 19 | |
Max | 1,538 | 28 | 1,250 | 2,500 | 30 | 571 | 202 | 8 | |
Z1 | 3,333 | 45 | 3,333 | 5,000 | 39 | 769 | 294 | 10 | (II) |
Z2 | 4,000 | 30 | 3,333 | 5,000 | 30 | 1,100 | 250 | 21 | |
Z3 | 4,000 | 47 | – | 5,000 | 45 | 588 | 270 | 20 | |
Z4 | 1,538 | 71 | 2,857 | 6,666 | 33 | 769 | 263 | 42 | |
Z5 | 6,666 | 54 | 2,500 | – | 43 | 1,428 | 377 | 24 | |
Z6 | 5,000 | 77 | – | 10,000 | 60 | 1,666 | 869 | 27 | |
Z7 | 2,857 | 47 | 3,333 | 6,666 | 38 | 833 | 290 | 22 | |
Z8 | 2,000 | 35 | 3,333 | 3,333 | 31 | 666 | 244 | 8 | |
Z9 | 3,333 | 31 | 10,000 | 3,333 | 32 | 3,333 | 625 | 20 | |
Z10 | 3,333 | 28 | 2,222 | – | 30 | 952 | 202 | 13 | |
Z11 | 1,666 | 36 | 1,250 | 2,500 | 32 | 571 | 215 | 15 | |
Z12 | 7,413 | 37 | – | – | 35 | 714 | 260 | 8 | |
Z13 | – | 53 | – | – | 34 | – | – | 20 | |
Detected by | 12/13 | 13/13 | 9/13 | 9/13 | 13/13 | 12/13 | 12/13 | 13/13 | (III) |
Detected % | 92 | 100 | 69 | 69 | 100 | 92 | 92 | 100 |
Analysis of CD8+ T-cell responses by ELISPOT following the introduction of a set of four rules
Overview of IFNγ ELISPOT results from phase II/2006 of the CIMT monitoring panel
D1 Flu | D2 CMV | D2 Flu | D4 Flu | D5 CMV | D6 Flu | D7 Flu | D8 CMV | ||
|---|---|---|---|---|---|---|---|---|---|
Min | 44,118 | 1,791 | 33,803 | 58,824 | 1,745 | 48,387 | 14,720 | 1,698 | (I) |
Medium | 28,823 | 1,088 | 16,395 | 49,960 | 999 | 14,265 | 4,669 | 1,023 | |
Max | 10,345 | 396 | 1,231 | 41,096 | 391 | 4,458 | 1,706 | 269 | |
Z1 | 44,118 | 596 | – | – | 596 | 48,387 | 14,720 | 318 | (II) |
Z2 | – | 1,732 | – | – | 1,745 | 4,739 | 2,222 | 1,698 | |
Z3 | – | 1,333 | – | – | 1,117 | – | 2,982 | 1,292 | |
Z4 | – | – | – | – | 774 | – | 2,273 | 447 | |
Z5 | – | 997 | – | – | 1,157 | 17,647 | 3,827 | 1,209 | |
Z6 | 10,345 | 1,031 | 14,151 | – | 1,006 | – | 1,706 | 1,005 | |
Z7 | – | 1,317 | – | – | 1,061 | 10,000 | 6,000 | 1,661 | |
Z8 | 32,258 | 396 | – | 58,824 | 391 | 4,458 | 3,623 | 269 | |
Z9 | 28,571 | 966 | – | 41,096 | 912 | 14,423 | 8,955 | 1,081 | |
Z10 | – | 847 | 1,231 | – | 806 | 7,426 | 3,052 | 696 | |
Z11 | – | 1,044 | 33,803 | – | 1,087 | 20,339 | 3,670 | 1,273 | |
Z12 | – | 1,791 | – | – | 1,387 | 10,619 | 3,057 | 1,583 | |
Z13 | – | 1,008 | – | – | 949 | 4,615 | 4,615 | 770 | |
Detected by | 4/13 | 12/13 | 3/13 | 2/13 | 13/13 | 10/13 | 13/13 | 13/13 | (III) |
Detected % | 31 | 92 | 23 | 15 | 100 | 77 | 100 | 100 |
Comparison of the results obtained in both phases
Distribution of antigen-specific T-cell frequencies in the two testing phases as obtained by tetramer staining (a) and IFNγ ELISPOT assays (b). The figure shows the six reactivities (filled circle) and the calculated mean of all reactivities from phase I/2005 (filled line) as well as the eight reactivities (open circle) and calculated mean of all reactivities from phase II/2006 (open line). The frequency of antigen-specific T-cells is indicated on the y-axis as 1 per x counted CD8+ T-cells for the tetramer test and as 1 per x seeded PBMC for the ELISPOT assay
Probability of detecting a reactivity by a tetramer staining, or b IFNγ ELISPOT assay. A trendline was inserted on the basis of results from all 14 reactivities from both phases of the panel. The figure shows the six reactivities from phase I/2005 (filled squares) and the eight reactivities from phase II/2006 (open squares). The frequency of antigen-specific T-cells is shown on the x-axis in 1 per x counted CD8+ T-cells for the tetramer assay (a) or 1 per x seeded PBMC for the ELISPOT assay (b). X-values for y = 90% and y = 50% are indicated by the broken lines
For the tetramer assay, T-cell frequencies exceeding 1 per 1,200 CD8+ T-cells were therefore classified as “high”, whereas frequencies of less than 1 per 7,650 CD8+ were classified as “low” (Fig. 4a). Following the same rules for the ELISPOT assay, T-cell responses of at least one IFNγ spot per 2,850 PBMC can be considered as “high” and T-cell responses of less than one spot per 19,000 PBMC as “low” (Fig. 4b).
a Percentage of reactivities actually detected by tetramer staining. The first two groups of bars show the detection rate for the nine high reactivities (>1 per 1,200 CD8+ T-cells) in phase I/2005 and phase II/2006. The next two groups of bars show the detection rate for five moderate to low reactivities (<1 per 1,200 CD8+) in phase I/2005 (third group) or phase II/2006 (fourth group). The open bars represent all tests performed, grey bars represent results obtained in tests that were performed on more than 3 × 104 CD8+ T-cells and filled bars represent results obtained in tests that were performed on less than 3 × 104 CD8+ T-cells. b Percentage of reactivities detected in IFNγ ELISPOT assays. The first two groups of bars show the rate of detection of the four high reactivities (>1 per 2,850 PBMC in phase I/2005 and phase II/2006. The next two groups of columns show the rate of detection for the ten moderate to low reactivities (<1 per 2,850 PBMC) in phase I/2005 and phase II/2006. The open bars represent the performances of all centers in the respective panel phase, grey bars represent results obtained from the five centers that already fulfilled at least three of the four minimum criteria in phase I/2005 and filled bars represent results obtained from centers that fulfilled less than three of the four minimum criteria in phase I/2005
We then analyzed the capacity of the laboratories to measure either high T-cell responses (>1 per 2,850 PBMC) or low to moderate T-cell responses (<1 per 2,850 PBMC) in the ELISPOT assay. This analysis was performed for two defined subgroups of participants. The first subgroup included those five centers (Z3, Z4, Z7, Z8 and Z9) that already fulfilled three or four of the requirements in the first phase of the panel. These five centers did not have to introduce any change or at least no major changes to their protocol for the repetition of the experiments in phase II. The second subgroup included the new center Z13 (led by a colleague that had been in a laboratory that only fulfilled one of four requirements in phase I) and all others that had fulfilled only one or two of the four requirements in the first phase. All laboratories in this second group had to introduce marked changes to their locally established protocols. Similar to the tetramer analysis, the new requirements were not necessary to detect antigen-specific responses among the category of high T-cell frequencies in either the first or second phases (Fig. 5b). However, applying the set of rules defined in phase I markedly improved the capacity of centers to detect the low to moderate T-cell responses. The first subgroup detected a total of 68% of the low to moderate reactivities in phase I, whereas the second subgroup detected only 20% (Fig. 5b). After harmonization of the protocols, both subgroups performed equally well. In addition, the inter-group variability in detecting positive responses was reduced in phase II (percentage of detected responses ranged from 38 to 88% with a mean of 67 ± 16%) as compared to phase I (percentage of detected responses ranged from 0 to 100% with a mean of 55 ± 33%).
Experience does not equal performance
Among the 13 centers that had participated in phase II, tetramer stainings had been performed for 1–8 years. Similarly, the experience in the ELISPOT technology varied between 1 and 10 years. For both techniques, we could not find any correlation between the years of experience and the ability to detect T-cell responses, not even among the subgroups of moderate or low T-cell responses (not shown).
Discussion
Whenever new techniques are introduced to the scientific community, they are first only available to a small group of expert laboratories. If these assays are robust and applicable for specific research or routine applications, they spread to the international community. In general, the “original” protocol then undergoes several adaptations in order to meet specific needs. On the one hand, changes can be beneficial and result in the improvement of protocols. On the other, this evolutionary process leads to employment of many different protocol variants, limiting comparison of the study results obtained by different laboratories. Thus, standardization approaches should be omitted during the initial development but are absolutely required when assays have become firmly established. In recent years, several activities aiming at the harmonization of techniques used to monitor the presence of antigen-specific T-cells have been initiated for ELISPOT [28, 29, 30, 31], tetramer staining [32] and ICS [33, 34, 35, 36]. While these studies showed the feasibility, general applicability and the diversity of performance among participants, they were not designed to either systematically investigate the influence of distinct protocol variables nor to test whether changes to these parameters can lead to a global improvement of the group. The CIMT monitoring panel is the first initiative that has now introduced the two-step approach proposing a strategy where technical variables that influence the performance of a defined assay are first systematically identified (“first step”) followed by a new testing phase where resultant protocol changes are validated under controlled conditions within the same group of investigators (“second step”).
As soon as a number of protocol variables that might have influenced the sensitivity and the quality of the tests were identified in the first phase of our study, it was decided to validate this finding in a second phase. Because this two-step approach was not initially foreseen, the second phase was performed with PBMC samples obtained from different donors than those used in the first round. The distribution and the frequencies of detectable T-cell responses directed against the chosen model antigens were different in the first and second group of donors (Fig. 3) precluding a direct comparison of the results obtained in both phases of the panel. To circumvent this problem, two assay-specific frequency thresholds were introduced that allowed us to distinguish classes of T-cell responses (low, moderate and high) (Fig. 4a, b). Clearly, high-frequency T-cell responses were detected irrespective of the protocol used and as such did not allow the identification of factors that exert a strong influence on the sensitivity and variability of the protocols used. Relevant parameters could only be detected when the comparison was focused on the detection of T-cells that are present at low to moderate frequencies in PBMC. This finding should be taken into account when selecting model antigens for use in monitoring panels [37], in particular by laboratories that are interested in the detection of peripheral tumor-specific T-cells, which are often present at low frequencies, even after vaccination.
Although our experiments do not specifically address the question of detection limits for the ELISPOT and tetramer assays, we could detect a high variability in the sensitivity of protocols used by the different participants. The majority of labs (y = 90%) is able to detect responses with a frequency above 1 per 1,200 CD8+ T cells in the tetramer assay or responses with a frequency above 1 per 2,859 PBMC in the ELISPOT. Note that some of the centers could reliably detect a response with a frequency of about 1 per 8,000 CD8+ T cells in the tetramer assay and about 1 per 40,000 PBMC in the ELISPOT assay. These low frequencies are in the range of that is commonly reported as the detection limit for internally validated protocols for both technologies [39, 40, own unpublished observations]. Another important task of standardization efforts should be to decrease the variation of results obtained in a group of several laboratories down to the stable and low values (15–30%) that can be reproducibly found within single labs. In order to quantify the variation of results among laboratories we calculated the coefficient of variation for all 14 reactivities of the two panel phases. The CVs were determined on the base of centers that were able to detect the respective T cell response and the results are shown in supplementary Tables S1a, b. As expected, the CVs we found in our inter-laboratory testing project were higher than those reported from intra-center analysis [39, 40].
In the ELISPOT assay, the background spot numbers obtained by the different participants varied greatly, but we were unable to correlate this finding to a distinct variable. Since the spontaneous cytokine secretion impacts significantly on the sensitivity of this assay, factors that especially influence the non-specific spot production, possibly the medium type or serum source, will need to be systematically analyzed in a separate study.
The main conclusions from our study have been drawn on the basis of subgroup analyses. Although the CIMT panel in general (13 centers in this initial action), and consequently the subgroups formed during the analysis were rather small, we could already identify statistically significant differences in the ability to detect positive responses. We concluded that the number of counted CD8+ T-cells is the most influential crucial factor for the tetramer assay and that the combination of a resting-time and a high number of PBMC leads to increased sensitivity in the ELISPOT assay. This suggests that the impact of the identified technical variables on the quality of the assays is high. In order to identify those protocol variables that lead to more subtle differences, a larger group of participants would be needed.
In addition to the systematic identification of variables that correlate with sensitivity/insensitivity of various assays, inter-laboratory testing projects also allow the rapid evaluation of individual performance among a group. Interestingly, the finding that experienced laboratories did not perform better than laboratories which recently applied these techniques strongly suggests that non-optimal protocols, once established in a lab, can commonly be maintained for several years. Periodic comparison of local protocols with those of other centers is recommended. Even if a new staff member uses an established protocol, it is recommended to have them participate in inter-laboratory testing/teaching exercise. Regular participation in multi-center comparisons could thereby help to optimize and validate participants’ performance over time and to maintain sensitive protocols or minimal standards. This is of great importance when material from expensive clinical trials has to be analyzed.
All data from the CMV-serology, from the pre-testing experiments and from the results generated by the participating laboratories in ELISPOT and tetramer staining were taken together for each donor in order to qualitatively validate the presence of CMV- and influenza-specific T-cells. To estimate the quantity, i.e. the frequency of specific T-cells in each donor, we calculated the average of all qualitatively positive results, as well as the standard deviations. This procedure constitutes only an approximation of the real number of antigen-specific cells present in a given sample, and cannot be taken as a method for determining absolute T-cell frequencies. Cell samples that contain pre-defined numbers of antigen-specific T-cells (e.g. spiked T-cell clones), especially tumor-reactive T-cells, are not easily available for use in multi-center comparisons, although such standard samples are urgently needed. We see this as one major bottle-neck for the optimization and standardization of immunomonitoring techniques. Methods to generate such standard samples for broader use will therefore be elucidated with high priority in the near future for the next phases of this international collaboration. Another big challenge will be to define accepted rules for the settings of the equipment used in these analyses (flow cytometer or ELISPOT reader) in order to uniformly process and analyze the raw data. Ten from eleven laboratories that performed the ELISPOT assay in the first phase used an ELISPOT reader for spot counting. It is known that spot counts between centers can differ significantly and this may be explained by the use of different reading machines, different settings for the same type of machine or by the experience of the operator. Within this group, four different commercially available reading systems were used (supplementary Table S2). We were not able to identify differences between the types of ELISPOT readers. A new ELISPOT panel phase is currently in preparation, that will specifically focus on the performances of different ELISPOT readers and try to introduce tools to control inter center variation. In addition, none of the participant reported on the use of live/dead cell discrimination on thawed PBMC samples for the FACS-based experiments. Whether the combination of staining with Ab/HLA-tetramers and vital dyes or with a resting phase is beneficial for increasing the sensitivity of the tetramer staining assay could be addressed in future testing actions.
Results from a proficiency panel of 36 laboratories from nine different countries in which the ELISPOT assay was validated are now also being reported [41]. This initiative, conducted under the aegis of the Cancer Vaccine Consortium (CVC), was mainly designed to offer an external validation to the participating laboratories but the in depth analysis of the obtained data sets lead to similar findings and recommendations as the CIMT monitoring panel. It confirmed that a resting phase of cells prior to addition to the ELISPOT plates is advantageous and should therefore be generally recommended. Furthermore, a long year experience in a technology did not guarantee for a sensitive test and failure to detect specific T cell responses concentrated on the weak responses. The fact that two independent initiatives come to similar findings is surely notable and shows the necessity to carry on running proficiency panels.
Last but not least, we would like to stress that even the best guidelines and protocols alone cannot guarantee good performance. Monitoring of antigen-specific T-cell responses requires skills as well as experience. Participation in immunomonitoring panels cannot compensate for the need to constantly educate and train staff and to develop specific expertise for covering individual needs. Nevertheless, we strongly believe that by organizing further two-step inter-laboratory testing projects, the CIMT monitoring panel will be able to improve the sensitivity of the assays used for immunomonitoring as well as to actively participate in the harmonization of these assays, which is required to enable the comparison of immunotherapeutic trials performed in different centers.
Acknowledgments
We thank the executive board members of the Association for Immunotherapy of Cancer (CIMT) for taking over the patronage and partial costs of the monitoring panel. We also thank P. Hrstic for technical assistance (Tuebingen), D. Sigurdardottir (Tuebingen) and C.L.C.M. Franken (Leiden) for HLA tetramers, J. W. Drijfhout for peptides and the Blood Bank at the Johannes-Gutenberg University Mainz for providing HLA-typed buffy coats. C.M.B. is supported by a Dr. Mildred Scheel research grant from the Deutsche Krebshilfe (DKH). We thank Lynne Yakes for editorial assistance.