The National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) is a library of question items that enables patient reporting of adverse events (AEs) in clinical trials. This study contributes content validity evidence of the PRO-CTCAE by incorporating cancer patient input of the relevance and comprehensiveness of the item library.
Cognitive interviews were conducted among patients undergoing chemotherapy or radiation therapy at multiple sites to evaluate comprehension, memory retrieval, judgment, and response mapping related to AE terms (e.g., nausea), attribute terms (regarding frequency, severity, or interference), response options, and recall period. Three interview rounds were conducted with ≥20 patients completing each item per round. Items were modified and retested if ≥3 patients exhibited cognitive difficulties or if experienced by ≤25 % patients.
One hundred and twenty-seven patients participated (35 % ≤high school, 28 % non-white, and 59 % female). Most AE terms (63/80) generated no cognitive difficulties. The remaining 17 were modified without further difficulties by Round 3. Terms were comprehended regardless of education level. Attribute terms and response options required no modifications. Patient adherence to recall period (7 days) was improved when the reference period was incorporated.
This study provides evidence confirming comprehension of the US English language versions of items in the PRO-CTCAE library for measuring symptomatic AEs from the patient perspective within the context of cancer treatment. Several minor changes were made to the items to improve item clarity, comprehension, and ease of response judgment. This study helps to establish the content validity of PRO-CTCAE items for patient reporting of AEs during cancer treatment.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Trotti, A., Colevas, A. D., Setser, A., Rusch, V., Jaques, D., Budach, V., et al. (2003). CTCAE v3.0: Development of a comprehensive grading system for the adverse events of cancer treatment. Seminars in Radiation Oncology, 13, 176–181.
National Cancer Institute, National Institutes of Health, US Department of Health and Human Services. Common Terminology Criteria for Adverse Events (CTCAE) Version 4.0. Published May 28, 2009; Revised version 4.03 June 14, 2010 (Vol. Available from: http://evs.nci.nih.gov/ftp1/CTCAE/CTCAE_4.03_2010-06-14_QuickReference_5x7.pdf. Accessed 20 May 2013).
Basch, E. (2010). The missing voice of patients in drug-safety reporting. New England Journal of Medicine, 362(10), 865–869. doi:10.1056/NEJMp0911494.
Basch, E., Iasonos, A., McDonough, T., Barz, A., Culkin, A., Kris, M. G., et al. (2006). Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: Results of a questionnaire-based study. Lancet Oncology, 7, 903–909. doi:10.1016/S1470-2045(06)70910-X.
Basch, E., Jia, X., Heller, G., Barz, A., Sit, L., Fruscione, M., et al. (2009). Adverse symptom event reporting by patients versus clinicians: Relationships with clinical outcomes. Journal of the National Cancer Institute, 101(23), 1624–1632. doi:10.1093/jnci/djp386.
Atkinson, T. M., Li, Y., Coffey, C. W., Sit, L., Shaw, M., Lavene, D., et al. (2012). Reliability of adverse symptom event reporting by clinicians. Quality of Life Research, 21(7), 1159–1164. doi:10.1007/s11136-011-0031-4.
Xiao, C., Polomano, R., & Bruner, D. W. (2012). Comparison between patient-reported and clinician-observed symptoms in oncology. Cancer Nursing,. doi:10.1097/NCC.0b013e318269040f.
Greimel, E. R., Bjelic-Radisic, V., Pfisterer, J., Hilpert, F., Daghofer, F., Pujade-Lauraine, E., et al. (2011). Toxicity and quality of life outcomes in ovarian cancer patients participating in randomized controlled trials. Supportive Care in Cancer, 19(9), 1421–1427. doi:10.1007/s00520-010-0969-8.
Flores, L. T., Bennett, A. V., Law, E. B., Hajj, C., Griffith, M. P., & Goodman, K. A. (2012). Patient-reported outcomes versus clinician symptom reporting during chemoradiation for rectal cancer. Gastrointestinal Cancer Research, 5(4), 119–124.
Basch, E., Iasonos, A., Barz, A., Culkin, A., Kris, M. G., Artz, D., et al. (2007). Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. Journal of Clinical Oncology, 25(34), 5374–5380. doi:10.1200/JCO.2007.11.2243.
Tang, B., Giuliani, M., Le, L. W., Higgins, J., Bezjak, A., Brade, A., et al. (2013). Capturing acute toxicity data during lung radiotherapy by using a patient-reported assessment tool. Clinical Lung Cancer, 14(2), 108–112. doi:10.1016/j.cllc.2012.06.003.
US Department of Health and Human Services. Guidance for industry. patient-reported outcome measures: Use in medical development to support labeling claims (December 2009. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. Accessed 20 May 2013).
European Medicines Agency. Committee for medicinal products for human use (CHMP). Pre-authorisation evaluation of medicines for human use: Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products (January 2006, Available from: http://www.ispor.org/workpaper/emea-hrql-guidance.pdf. Accessed 20 May 2013).
Ashley, L., Jones, H., Thomas, J., Forman, D., Newsham, A., Morris, E., et al. (2011). Integrating cancer survivors’ experiences into UK cancer registries: Design and development of the ePOCS system (electronic Patient-reported Outcomes from Cancer Survivors). British Journal of Cancer, 105, S74–S81. doi:10.1038/Bjc.2011.424.
Banerjee, A. K., & Ingate, S. (2012). Web-based patient-reported outcomes in drug safety and risk management: Challenges and opportunities? Drug Safety, 35(6), 437–446. doi:10.2165/11632390-000000000-00000.
Boers-Doets, C. B., Gelderblom, H., Lacouture, M. E., Epstein, J. B., Nortier, J. W., & Kaptein, A. A. (2013). Experiences with the FACT-EGFRI-18 instrument in EGFRI-associated mucocutaneous adverse events. Supportive Care in Cancer,. doi:10.1007/s00520-013-1752-4.
Chan, A., & Tan, E. H. (2011). How well does the MESTT correlate with CTCAE scale for the grading of dermatological toxicities associated with oral tyrosine kinase inhibitors? Supportive Care in Cancer, 19(10), 1667–1674. doi:10.1007/s00520-010-0999-2.
Farnell, D. J., Mandall, P., Anandadas, C., Routledge, J., Burns, M. P., Logue, J. P., et al. (2010). Development of a patient-reported questionnaire for collecting toxicity data following prostate brachytherapy. Radiotherapy and Oncology, 97(1), 136–142. doi:10.1016/j.radonc.2010.05.011.
Quinten, C., Maringwa, J., Gotay, C. C., Martinelli, F., Coens, C., Reeve, B. B., et al. (2011). Patient self-reports of symptoms and clinician ratings as predictors of overall cancer survival. Journal of the National Cancer Institute, 103(24), 1851–1858. doi:10.1093/Jnci/Djr485.
Stacey, D., Bakker, D., Ballantyne, B., Chapman, K., Cumminger, J., Green, E., et al. (2012). Managing symptoms during cancer treatments: evaluating the implementation of evidence-informed remote support protocols. Implementation Science, 7, 110. doi:10.1186/1748-5908-7-110.
Johansen, M. A., Henriksen, E., Horsch, A., Schuster, T., & Berntsen, G. K. R. (2012). Electronic symptom reporting between patient and provider for improved health care service quality: A systematic review of randomized controlled trials. Part 1: State of the Art. Journal of Medical Internet Research, 14(5). doi:10.2196/jmir.2214.
Johansen, M. A., Henriksen, E., Berntsen, G., & Horsch, A. (2011). Electronic symptom reporting by patients: A literature review. Studies in Health Technology and Informatics, 169, 13–17.
Seow, H., King, S., Green, E., Pereira, J., & Sawka, C. (2011). Perspectives of patients on the utility of electronic patient-reported outcomes on cancer care. Journal of Clinical Oncology, 29(31), 4213–4214. doi:10.1200/JCO.2011.37.9750.
King, A., Daniels, J., Lim, J., Cochrane, D. D., Taylor, A., & Ansermino, J. M. (2010). Time to listen: A review of methods to solicit patient reports of adverse events. Quality and Safety in Health Care, 19(2), 148–157. doi:10.1136/qshc.2008.030114.
Zhu, J., Stuver, S. O., Epstein, A. M., Schneider, E. C., Weissman, J. S., & Weingart, S. N. (2011). Can we rely on patients’ reports of adverse events? Medical Care, 49(10), 948–955. doi:10.1097/MLR.0b013e31822047a8.
Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA, 288(23), 3027–3034.
Basch, E. M., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L., Sit, L., et al. (2011). Electronic toxicity monitoring and patient-reported outcomes. Cancer Journal, 17(4), 231–234. doi:10.1097/PPO.0b013e31822c28b3.
Stull, D. E., Leidy, N. K., Parasuraman, B., & Chassany, O. (2009). Optimal recall periods for patient-reported outcomes: Challenges and potential solutions. Current Medical Research and Opinion, 25(4), 929–942. doi:10.1185/03007990902774765.
Kirkova, J., Davis, M. P., Walsh, D., Tiernan, E., O’Leary, N., LeGrand, S. B., et al. (2006). Cancer symptom assessment instruments: A systematic review. Journal of Clinical Oncology, 24(9), 1459–1473. doi:10.1200/JCO.2005.02.8332.
Wild, D., Eremenco, S., Mear, I., Martin, M., Houchin, C., Gawlicki, M., et al. (2009). Multinational trials-recommendations on the translations required, approaches to using the same language in different countries, and the approaches to support pooling the data: The ISPOR patient-reported outcomes translation and linguistic validation good research practices task force report. Value in Health, 12(4), 430–440. doi:10.1111/j.1524-4733.2008.00471.x.
Basch, E., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L. M., Atkinson, T. M., et al. (Under Review). Development of the National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Journal of National Cancer Institute.
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: SAGE Publications, Inc.
Patrick, D. L., Burke, L. B., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., et al. (2011). Content validity-establishing and reporting the evidence in newly Developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part 2-assessing respondent understanding. Value in Health, 14(8), 978–988. doi:10.1016/j.jval.2011.06.013.
Bowen, G. A. (2008). Naturalistic inquiry and the saturation concept: A research note. Qualitative Research, 8(1), 137–152. doi:10.1177/1468794107085301.
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59–82. doi:10.1177/1525822X05279903.
Jobe, J. B., & Mingay, D. J. (1989). Cognitive research improves questionnaires. American Journal of Public Health, 79(8), 1053–1055.
Jobe, J. B. (2003). Cognitive psychology and self-reports: Models and methods. Quality of Life Research, 12(3), 219–227.
Miller, K. (2003). Conducting cognitive interviews to understand question-response limitations. American Journal of Health Behavior, 27(Suppl 3), S264–S272.
Schwarz, N., Oyserman, D., & Petycheva, E. (2010). Cognition, communication, and culture: Implications for the survey response process. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. E. Lyberg, P. P. Mohler, et al. (Eds.), Survey methods in multinational, multicultural, and multiregional contexts (pp. 177–190). Hoboken, NJ: John Wiley & Sons.
Tourangeau, R. (1984). Cognitive science and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.
Namey, E., Guest, G., Thairy, L., & Johnson, L. (2008). Data reduction techniques for large qualitative data sets. In G. Guest & K. M. MacQueen (Eds.), Handbook for team-based qualitative research (pp. 137–161). Lanham, MD: AltaMira Press.
Saldana, J. (2009). The coding manual for qualitative researchers. Los Angeles, CA: Sage Publications.
Basch, E. (2012). Beyond the FDA PRO guidance: Steps toward integrating meaningful patient-reported outcomes into regulatory trials and US drug labels. Value in Health, 15(3), 401–403. doi:10.1016/j.jval.2012.03.1385.
Symptom Management and Quality of Life Steering Committee. Clinical trials planning meeting—building bridges: The identification of core symptoms and health-related quality of life domains for use in cancer research (September 22–23, 2011. Available from: http://transformingtrials.cancer.gov/files/SxQOLSCPROCTPMExecutive%20Summary_FINAL.pdf Accessed 7 June 2012).
Mitchell, S. A., Lang, K., Nichols, C., Clauser, S. B., Federico, V., Lalla, D., et al. (2012). Validation of the NCI Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) in women receiving treatment for metastatic breast cancer. Chicago, IL: Annual Meeting of the American Society of Clinical Oncology.
Dueck, A. C., Mendoza, T., Mitchell, S. A., Reeve, B. B., Castro, K., Denicoff, A., et al. (2012). Validity and reliability of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Chicago, IL: Annual Meeting of the American Society of Clinical Oncology.
Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429. doi:10.1111/j.1524-4733.2008.00470.x.
(2012) Methodological standards and patient-centeredness in comparative effectiveness research—The PCORI perspective. JAMA, 307(15), 1636–1640. doi:10.1001/jama.2012.466.
Work described in this report was supported by contracts from the United States National Cancer Institute, HHSN261201000043C and HHSN261201000063C. Each author met all International Committee of Medical Journal Editors (ICMJE) requirements (i.e., conception/design/acquisition/analysis, drafting/revising article, and final approval) for authorship.
NCI PRO-CTCAE Study Group members are Cindy Geoghegan, Jeff A. Sloan (Mayo Clinic, Rochester, MN, USA), Charles S. Cleeland (University of Texas M.D. Anderson Cancer Center, Houston, TX, USA), Amy P. Abernethy (Duke University Medical Center, Durham, NC, USA), Deborah W. Bruner (University of Pennsylvania School of Nursing, Philadelphia, PA, USA), Allison Barz (The Children’s Hospital of Philadelphia, Philadelphia, PA, USA), Amylou C. Dueck (Mayo Clinic, Scottsdale, AZ, USA), and Yuelin Li (Memorial Sloan-Kettering Cancer Center, New York, NY, USA).
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Hay, J.L., Atkinson, T.M., Reeve, B.B. et al. Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Qual Life Res 23, 257–269 (2014). https://doi.org/10.1007/s11136-013-0470-1
- Drug toxicity
- Reproducibility of results
- Qualitative research