Equivalence of MammaPrint array types in clinical trials and diagnostics
- 4.2k Downloads
MammaPrint is an FDA-cleared microarray-based test that uses expression levels of the 70 MammaPrint genes to assess distant recurrence risk in early-stage breast cancer. The prospective RASTER study proved that MammaPrint Low Risk patients can safely forgo chemotherapy, which is further subject of the prospective randomized MINDACT trial. While MammaPrint diagnostic results are obtained from mini-arrays, clinical trials may be performed on whole-genome arrays. Here we demonstrate the equivalence and reproducibility of the MammaPrint test. MammaPrint indices were collected for breast cancer samples: (i) on both customized certified array types (n = 1,897 sample pairs), (ii) with matched fresh and FFPE tissues (n = 552 sample pairs), iii) for control samples replicated over a period of 10 years (n = 11,333), and iv) repeated measurements (n = 280). The array type indicated a near perfect Pearson correlation of 0.99 (95 % CI: 0.989–0.991). Paired fresh and FFPE samples showed an excellent Pearson correlation of 0.93 (95 % CI 0.92–0.94), in spite of the variability introduced by intratumoral tissue heterogeneity. Control samples showed high consistency over 10 year's time (overall reproducibility of 97.4 %). Precision and repeatability are overall 98.2 and 98.3 %, respectively. Results confirm that the combination of the near perfect correlation between array types, excellent equivalence between tissue types, and a very high stability, precision, and repeatability demonstrate that results from clinical trials (such as MINDACT and I-SPY 2) are equivalent to current MammaPrint FFPE and fresh diagnostics, and can be used interchangeably.
KeywordsMammaPrint Breast cancer Clinical prognostic value Diagnostic microarray test MINDACT I-SPY
Clinical Laboratory Improvement Amendments
US Food and Drug Administration
Investigational device exempt
In vitro diagnostic
Laboratory developed test
The use of multigene diagnostic testing to guide cancer treatment has increased dramatically, since the clinical introduction of the first US Food and Drug Administration (FDA)-cleared multigene diagnostic signature MammaPrint . These types of genomic tests interrogate gene expression levels in tissues to give information about the disease state and/or prognosis, and have led to new diagnostic approaches that are recognized by international clinical guidelines [2, 3]. MammaPrint determines the expression of 70 signature genes and helps guide physicians to make adjuvant treatment decisions for early-stage breast cancer patients [4, 5]. The test is suitable for both fresh or fresh frozen (fresh) tissues and formalin-fixed paraffin-embedded (FFPE) tissues [4, 6, 7]. MammaPrint is part of Agendia’s In Vitro Diagnostic (IVD) Test Suite that also includes BluePrint, which reveals breast cancer subtype .
MammaPrint was developed and validated based on fresh frozen tissues and cleared by the FDA [1, 4, 9, 10]. However, for diagnostic purposes, shipment of frozen tissue is not always easily implemented in community hospitals. Therefore, RNAretain (for room temperature sample stabilization during shipment) was validated, FDA-cleared and implemented [11, 12] and subsequently, MammaPrint was validated and cleared for use of FFPE tissues [6, 13]. To date, MammaPrint is one of the two (MammaPrint and Prosigna ) FDA-cleared  clinically implemented breast cancer multigene expression tests with 6 clearances, and obtained the EU IVD CE mark, for both fresh and FFPE tissues [1, 11, 13, 16, 17, 18]. Claims on current diagnostic clinical utility include its well-established prognostic value for metastasis risk assessment for early-stage breast cancer patients [5, 9, 10, 19, 20, 21, 22].
Since 2004, MammaPrint has been used for over 90,000 patients to determine the patient’s distant recurrence risk. The activity of the 70 signature genes is compiled in an index that is used to establish the qualitative MammaPrint results: Low Risk or High Risk for distant recurrence. Designated clinically low risk patients with positive hormonal receptor status could well be treated by hormonal therapy in the adjuvant setting , while clinically high risk patients are more suited to receive a combination of chemotherapy and hormonal therapy in an (neo-) adjuvant setting . Clinical studies have shown MammaPrint’s clinical prognostic value in lymph-node negative [10, 22] and lymph-node positive  breast cancer, for women diagnosed at all ages , and irrespective of pathological grade , estrogen receptor status or HER2 expression . The RASTER study  provided the first prospective evidence of clinical utility for MammaPrint. It showed a 98.9 % 5-year metastasis-free survival for patients that were clinically high risk, based on standard clinical parameters, but who chose not to receive adjuvant chemotherapy based on their MammaPrint Low Risk status . These study results indicated that these patients could safely forgo chemotherapy . Confirmation of MammaPrint’s prognostic value is expected in 2016, when results are projected for the large European Organization for Treatment and Cancer (EORTC) prospective randomized clinical MINDACT trial (EORTC 10041; BIG 3-04) [24, 25, 26].
MammaPrint results can be obtained from both customized mini-arrays and from customized whole-genome arrays. Both array types contain multiple replicates of the 70 signature probes complemented with an identical large set of replicated normalization and control probes. Diagnostic MammaPrint results are generally obtained using customized mini-arrays (also referred to as 8-pack arrays), accommodating 8 individual samples per array . In large clinical trials like MINDACT  and I-SPY 2 , MammaPrint expression analysis was performed on whole-genome arrays containing over 32,000 unique probes per array. This array type is currently used in the I-SPY 2 trial with a MammaPrint FDA Investigational Device Exempt (IDE) status. For MammaPrint diagnostic use in Europe, both the customized mini-array and the customized whole-genome array hold an IVD EU directive CE mark. The use of whole-genome arrays allows providing MammaPrint results to clinical trial participants alongside enabling full-genome analysis for the early development of new gene expression signatures. For the MINDACT trial, this resulted in an unprecedented dataset representing almost 6700 early-stage breast cancer patients.
The aim of the current study is to evaluate the equivalence and robustness of the MammaPrint test in different settings. Microarray types [28, 29] as well as tissue preservation techniques  have been known to potentially influence array expression profiles in general. We, therefore, evaluated the agreement of MammaPrint indices between the customized mini-arrays and whole-genome arrays for 1897 array sample pairs and between fresh and FFPE tissues in 552 tissue sample pairs, and assessed reproducibility of >11,000 control samples over the last 10 years as well as the precision and repeatability of the test.
Materials and methods
MammaPrint indices as reported were collected for all patient tumor samples processed for which the sample was hybridized to both a customized mini-array and a customized whole-genome array (n = 1897 sample pairs, totaling 3,794 sample hybridizations, 2005–2015), or for which samples with both fresh (fresh frozen or RNAretain) and FFPE tissues were hybridized to individual arrays (n = 552 totaling 1104 sample hybridizations (2011–2015)). Data for the fresh versus FFPE comparison include the RASTER series  and a data series described previously by Sapino et al. . Clinicopathological data and clinical outcome data were available for those 345 of the RASTER patients with matched samples of fresh and FFPE tissues [5, 12]. Additionally, for assessing the analytical performance of MammaPrint, indices were collected for fresh and FFPE samples representing MammaPrint High Risk and Low Risk samples. A total of 7 samples were measured for 20 days in duplicate, totaling to 280 measurements.
For this study, only data and not samples were collected. All data and analyses used or performed for this study comply with the current ethical laws of the Netherlands. All patient sample data were anonymized in accordance with national ethical guidelines (“Code for Proper Secondary Use of Human Tissues,” Dutch Federation of Medical Scientific Societies), study samples belonging to the used data had Institutional Review Board approvals. For this type of study, formal consent is not required.
High Risk and Low Risk MammaPrint control samples are standardly used as technical and experimental controls within each batch run of samples. Data from MammaPrint control samples, each composed of a pool of clinically representative breast cancer tumors, are continuously monitored in the clinical diagnostic setting, and were available for stability assessment of MammaPrint test indices in this study [4, 6, 31]. All control samples were processed at one of Agendia’s core laboratory facilities in Amsterdam (The Netherlands) or Huntington Beach/Irvine (California, US) from August 2005 until March 2015, totaling n = 11,333 data points for analyses.
Equivalence of MammaPrint indices was determined by the Pearson correlation for assessment of the degree of linear correlation, and a Passing-Bablok regression analysis to obtain the regression equation. Bland–Altman plots were used to visually examine the existence of any constant bias in the difference of measurements between paired samples. The Bland–Altman analysis allows visual identification of any proportional error (if data points show an angled linear association), systemic error (if the horizontal mean difference line of the data points is shifted parallel away from zero), or dependence of a method on the magnitude of measurements (if data points show an association that widens or narrows from left to right).
Clinicopathological data and clinical outcome data of the 345 patient samples were analyzed using the statistical package SPSS 22.0 for Windows (SPSS Inc, Chicago, US). Survival analyses were performed to compare clinical performance of fresh and FFPE tissues. Kaplan–Meier analyses were used to compare the survival distributions of MammaPrint for fresh and FFPE for distant metastasis as first event (DMF) and distant recurrence-free interval (DRFI). DMF was defined as the time from surgery until the diagnosis of a distant metastasis as first event. For DMF, patients who present with a local recurrence, regional recurrence, or second primary tumor before the diagnosis of a distant metastasis were censored at such event, at death or at the end of follow-up. DRFI was defined as the time from surgery until the diagnosis of a first distant metastasis or breast cancer-related death.
Reproducibility of the MammaPrint test was tested by evaluating diagnostic controls over a period of 10 years to assess nearly all sources of variation in n = 11,333 samples. Reproducibility was measured in terms of the relative stability, calculated as 100 minus the relative standard deviation, where the standard deviation was measured as a percentile of the total MammaPrint range . To enable assessment of MammaPrint stability of repeated control sample measurements over time and independent of control sample, MammaPrint indices were centered on their individual sample mean per RNA reference type.
The analytical performance of the current MammaPrint fresh and FFPPE versions was assessed in a precision evaluation experiment according to the EP2-A5 documentation , as described by Delahaye et al. . This analysis included precision and repeatability assessment of MammaPrint High Risk and Low Risk samples. Precision was determined by calculating the relative precision for repeated measurements of High Risk and Low Risk breast cancer samples over a period of 20 days. Repeatability was determined by calculating the relative stability between duplicate runs of the repeated measurements performed each day. Both the relative stability and the relative precision are calculated as 100 minus the relative standard deviation.
Equivalence of MammaPrint indices for mini-arrays versus whole-genome arrays
In conclusion, comparison of 1897 mini-array versus whole-genome array sample pairs showed equivalence and robustness of MammaPrint results between the different arrays types both containing the 70 diagnostic MammaPrint probes.
Equivalence of MammaPrint indices from fresh versus formalin-fixed paraffin-embedded tissues
In summary, comparison of 552 fresh versus FFPE sample pairs showed very high comparability of MammaPrint indices between the tissue preservation techniques.
Impact of tissue preservation technique on MammaPrint patient survival prediction
Highly correlated continuous values may result in a lower concordance when these continuous variables are presented as binary categories. In particular, samples with a result close to a classification threshold that have a very small difference between two measurements may still result in a switch in binary outcome. Therefore, samples in diagnostics with an index close to the MammaPrint classification threshold (area predefined) are tested multiple times to increase measurement precision. We assessed and compared the clinical performance of MammaPrint in a retrospective setting to investigate the overall clinical effect of samples that have MammaPrint results generated from both fresh and FFPE tissues.
Equivalence in Survival percentages for DMF and DRFI in 345 patients with matched fresh and FFPE samples
MammaPrint (Fresh tissues)
MammaPrint (FFPE tissues)
95 % CI
95 % CI
Summarizing, we showed clinical equivalence for MammaPrint results from fresh and FFPE tissues.
MammaPrint stability over time
To assess the stability of the MammaPrint test over time, we investigated the reproducibility of MammaPrint results in a large pool of control samples spanning a 10-year period, and investigated the precision and repeatability of MammaPrint High Risk and Low Risk samples.
While the reproducibility of the MammaPrint test was assessed in control samples consisting of pooled RNA of breast cancer samples, the analytical performance of the MammaPrint test was determined using (non-pooled) individual samples. The analytical performance of MammaPrint fresh and FFPE in this study was assessed by determining the precision and repeatability in samples representing MammaPrint High Risk and Low Risk results that were measured twice per day for 20 days, totaling 280 measurements. The overall precision was 98.2 % (99.0 % for fresh and 97.3 % for FFPE), while repeatability overall was 98.3 % (99.0 % for fresh and 97.6 % for FFPE).
In summary, data on diagnostic control samples of Agendia’s quality control monitoring system, covering 10 years, and repeated measurements of samples show that the MammaPrint test for both fresh and FFPE is constant over time.
The study results show that MammaPrint is a robust and stable test, regardless of whether the used array type is a customized mini-array or customized whole-genome array, or the specimen type is fresh or FFPE. The excellent reproducibility of MammaPrint indices of control samples over the last 10 years and the excellent precision and repeatability of MammaPrint High Risk and Low Risk samples is further testament to the quality and robustness of the MammaPrint test.
A major strength of our current study lies in the large sample size. Almost 2000 sample pairs demonstrated a near perfect agreement in MammaPrint index values between mini and whole-genome array types. Additionally, 552 sample pairs (with clinical data available for 345 sample pairs) contributed to the comparison between fresh and FFPE MammaPrint results, with the results indicating excellent agreement. Assessment of the >11,000 control samples exemplifies the high stability of the MammaPrint results over time. Finally, robustness was shown in the 280 measurements part of the precision and repeatability series.
Other IVD multigene assays are currently available for early breast cancer patients, including the Prosigna Breast Cancer Prognostic Gene Signature Assay (ProSigna, Seattle, WA, USA ), OncotypeDx Breast Cancer Assay (Genomic Health, Redwood City, CA, USA ), Breast Cancer Index (BioTheranostics, San Diego, CA, USA ), and EndoPredict test (Sividon Diagnostics GmbH, Koln, Germany ). These assays are dedicated to FFPE breast cancer tissues and based on either microarray or (RT)-(q)PCR technology. IVD tests, like MammaPrint, need to be accurate, reliable, and clinically meaningful. Regulation under Clinical Laboratory Improvement Amendments (CLIA) is the first crucial step for quality control related to laboratories and personnel. To ensure full patient safety and quality control of IVD tests, including the group of laboratory developed test (LDTs), adhering to FDA regulation is essential . For diagnostic tests performed in the EU the equivalent regulatory oversight is the EU IVD CE mark and laboratory ISO-certification, requirements for which MammaPrint obtained regulatory certification as well. MammaPrint was the first of only two (MammaPrint and Prosigna Breast Cancer Prognostic Gene Signature Assay) of the above described multigene signatures to obtain IVD CE mark and FDA clearance, for which it was required to demonstrate both analytical validity and clinical validity . Moreover, MammaPrint is the first multigene classifier to publicly demonstrate such robust stability over time. Of note is that MammaPrint is independent of clinical factors, whereas the Prosigna gene signature assay is a combination of gene expression assessment (PAM50) and clinical factors (tumor size and proliferation score). MammaPrint diagnostic testing, including array types, is performed per FDA IVD 510 K clearance, or FDA Investigational Device Exempt status (IDE) for tests in the US, and EU IVD CE mark, whichever is required based on the quality regulatory oversight in respective countries.
Large clinical studies featuring IVDs based on microarray technology often use whole-genome arrays to enable whole-genome analyses in relation to clinical endpoints of interest alongside the IVD results. This is a major advantage of the utility of microarrays for IVDs over RT-PCR. When diagnostics is generally performed on dedicated microarrays, it is of critical importance to demonstrate equivalence between the different array types. For MammaPrint, we demonstrated that based on the near perfect agreement (r = 0.99) between different array types, study results based on whole-genome arrays similarly apply to the current diagnostic setting.
Previously, special arrangements were made by physicians to obtain a fresh tissue specimen during surgery for MammaPrint. To broaden utility, we translated MammaPrint to FFPE, thereby enabling the use of FFPE tissues and facilitating the decision of applying a MammaPrint test even after surgery of the primary tumor. A comparison between fresh and FFPE tissues of the same sample is naturally influenced by intrinsic intratumoral heterogeneity and differences in tissue preservation techniques  that are together fully accounting for the observed deviations. A difference of 5 % in MammaPrint binary result can be attributed to these phenomena [6, 31]. Despite this influence, comparison between fresh and FFPE demonstrated very high agreement (r = 0.93) in both MammaPrint Index values  and visualized clinical outcome prognosis, affirming previously described results . Taken together, study results of large clinical studies featuring the 70-gene profile MammaPrint, in particular, the MINDACT trial whose level 1a evidence results will be available in 2016, can be interpreted and transposed to the current diagnostic setting based on FFPE tissues without any reservations.
In summary, study results confirm that MammaPrint indices generated from customized mini-arrays show near perfect agreement to MammaPrint indices obtained from customized whole-genome arrays. Additionally, MammaPrint for fresh and FFPE tissues are robust, equivalent, and stable tests (also at the clinical level) in spite of influences by tissue heterogeneity and tissue preservation techniques. Finally, stability assessment confirmed a very high reproducibility, precision, and repeatability of the MammaPrint test over time. This combination of high equivalence between different array types, between tissue types, and high stability over time demonstrates that results from clinical trials like MINDACT and I-SPY 2 are similar to the current MammaPrint FFPE and fresh diagnostics and can be used interchangeably.
Bas Koster, Tina Treece, and Lisa Blumencranz for critically reading the manuscript and helpful discussions.
For this study, only data and not samples were collected. All data and analyses used or performed for this study comply with the current ethical laws of the Netherlands. All patient sample data were anonymized in accordance with national ethical guidelines (“Code for Proper Secondary Use of Human Tissues,” Dutch Federation of Medical Scientific Societies), study samples belonging to the used data had Institutional Review Board approvals.
Conflict of interest
Authors (except SL) are employed by Agendia (RB and LVV part-time) and CD is a consultant for Agendia, the commercial entity that markets the 70-gene signature as MammaPrint. LVV and RB report being shareholder of Agendia and are named inventors on the patent for the 70-gene signature used in this study. No writing assistance was utilized in the production of this manuscript.
- 1.MammaPrint FDA clearance. See Decision Summary at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?ID=DEN070009
- 2.Coates AS, Winer EP, Goldhirsch A, Gelber RD, Gnant M, Piccart-Gebhart M, Thürlimann B, Senn H-J (2015) Tailoring therapies-improving the management of early breast cancer: St Gallen International Expert Consensus on the Primary Therapy of Early Breast Cancer 2015. Ann Oncol 26:1533–1546CrossRefPubMedPubMedCentralGoogle Scholar
- 3.NCCN Clinical Practice Guidelines in oncology for Breast Cancer version 3.2015. http://www.nccn.org/professionals/physician_gls/f_guidelines.asp
- 5.Drukker CA, Bueno-de-Mesquita JM, Retèl VP, van Harten WH, van’t Tinteren H, Wesseling J, Roumen RMH, Knauer M, van Veer LJ, Sonke GS, Rutgers EJT, van de Vijver MJ, Linn SC (2013) A prospective evaluation of a breast cancer prognosis signature in the observational RASTER study. Int J Cancer 133:929–936CrossRefPubMedPubMedCentralGoogle Scholar
- 6.Sapino A, Roepman P, Linn SC, Snel MHJ, Delahaye LJMJ, van den Akker J, Glas AM, Simon IM, Barth N, de Snoo FA, van’t Veer LJ, Molinaro L, Berns EMJJ, Wesseling J, Riley LB, Anderson D, Nguyen B, Cox CE (2013) MammaPrint molecular diagnostics on formalin-fixed, paraffin-embedded tissue. J Mol Diagn 16:190–197CrossRefPubMedGoogle Scholar
- 9.van de Vijver MJ, He YD, Veer LJV, Dai H, Hart AAM, Voskuil DW, Schreiber GJ, Peterse JL, Roberts C, Marton MJ, Parrish M, Atsma D, Witteveen A, Glas A, Delahaye L, van der Velde T, Bartelink H, Rodenhuis S, Rutgers ET, Friend SH (2002) A gene-expression signature as a predictor of survival in breast cancer. N Engl J Med 347:1999–2009CrossRefPubMedGoogle Scholar
- 10.Buyse M, Loi S, Veer LV, Viale G, Delorenzi M, Glas AM, d’Assignies MS, Bergh J, Lidereau R, Ellis P, Harris A, Bogaerts J, Therasse P, Floore A, Amakrane M, Piette F, Rutgers E, Sotiriou C, Cardoso F, Piccart MJ (2006) Validation and clinical utility of a 70-gene prognostic signature for women with node-negative breast cancer. J Natl Cancer Inst 98:1183–1192CrossRefPubMedGoogle Scholar
- 11.MammaPrint FDA clearance RNAretain. See Decision Summary at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K070675
- 12.Bueno-de-Mesquita JM, van Harten WH, Veer LJV, van Dam FSAM, Karsenberg K, Douma KFL, van Tinteren H, Peterse JL, Wesseling J, Wu TS, Atsma D, Rutgers EJT, Brink G, Floore AN, Glas AM, Roumen RMH, Bellot FE, van Krimpen C, Rodenhuis S, van de Vijver MJ, Linn SC (2007) Use of 70-gene signature to predict prognosis of patients with node-negative breast cancer: a prospective community-based feasibility study (RASTER). Lancet Oncol 8:1079–1087CrossRefPubMedGoogle Scholar
- 13.MammaPrint FDA clearance FFPE. See Decision Summary at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K141142
- 14.Parker JS, Mullins M, Cheang MCU, Leung S, Voduc D, Vickery T, Davies S, Fauron C, He X, Hu Z, Quackenbush JF, Stijleman IJ, Palazzo J, Marron JS, Nobel AB, Mardis E, Nielsen TO, Ellis MJ, Perou CM, Bernard PS (2009) Supervised risk predictor of breast cancer based on intrinsic subtypes. J Clin Oncol 27:1160–1167CrossRefPubMedPubMedCentralGoogle Scholar
- 16.MammaPrint FDA clearance scanner. See Decision Summary at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K080252
- 17.MammaPrint FDA clearance age 61 years. See Decision Summary at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K081092
- 18.MammaPrint FDA clearance US location. See Decision Summary at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K101454
- 19.Mook S, Schmidt MK, Viale G, Pruneri G, Eekhout I, Floore A, Glas AM, Bogaerts J, Cardoso F, Piccart-Gebhart MJ, Rutgers ET, Veer LJV (2008) The 70-gene prognosis-signature predicts disease outcome in breast cancer patients with 1-3 positive lymph nodes in an independent validation study. Breast Cancer Res Treat 116:295–302CrossRefPubMedGoogle Scholar
- 22.Bueno-de-Mesquita JM, Linn SC, Keijzer R, Wesseling J, Nuyten DSA, van Krimpen C, Meijers C, de Graaf PW, Bos MMEM, Hart AAM, Rutgers EJT, Peterse JL, Halfwerk H, de Groot R, Pronk A, Floore AN, Glas AM, Veer LJV, van de Vijver MJ (2008) Validation of 70-gene prognosis signature in node-negative breast cancer. Breast Cancer Res Treat 117:483–495CrossRefPubMedGoogle Scholar
- 32.Tholen DW, Kallner A, Kennedy JW, Krouwer JS, Meier K: Evaluation of precision performance of quantitative measurement methods. Approved Guideline. Second Edition. EP5–A2. National Committee on Clinical Laboratory Standards 2004, 24Google Scholar
- 35.Filipits M, Rudas M, Jakesz R, Dubsky P, Fitzal F, Singer CF, Dietze O, Greil R, Jelen A, Sevelda P, Freibauer C, Müller V, Jänicke F, Schmidt M, Kölbl H, Rody A, Kaufmann M, Schroth W, Brauch H, Schwab M, Fritz P, Weber KE, Feder IS, Hennig G, Kronenwett R, Gehrmann M, Gnant M (2011) A new molecular predictor of distant recurrence in ER-positive, HER2-negative breast cancer adds independent information to conventional clinical risk factors. Clin Cancer Res 17:6012–6020CrossRefPubMedGoogle Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.