Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)



The National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) is a library of question items that enables patient reporting of adverse events (AEs) in clinical trials. This study contributes content validity evidence of the PRO-CTCAE by incorporating cancer patient input of the relevance and comprehensiveness of the item library.


Cognitive interviews were conducted among patients undergoing chemotherapy or radiation therapy at multiple sites to evaluate comprehension, memory retrieval, judgment, and response mapping related to AE terms (e.g., nausea), attribute terms (regarding frequency, severity, or interference), response options, and recall period. Three interview rounds were conducted with ≥20 patients completing each item per round. Items were modified and retested if ≥3 patients exhibited cognitive difficulties or if experienced by ≤25 % patients.


One hundred and twenty-seven patients participated (35 % ≤high school, 28 % non-white, and 59 % female). Most AE terms (63/80) generated no cognitive difficulties. The remaining 17 were modified without further difficulties by Round 3. Terms were comprehended regardless of education level. Attribute terms and response options required no modifications. Patient adherence to recall period (7 days) was improved when the reference period was incorporated.


This study provides evidence confirming comprehension of the US English language versions of items in the PRO-CTCAE library for measuring symptomatic AEs from the patient perspective within the context of cancer treatment. Several minor changes were made to the items to improve item clarity, comprehension, and ease of response judgment. This study helps to establish the content validity of PRO-CTCAE items for patient reporting of AEs during cancer treatment.

This is a preview of subscription content, access via your institution.

Fig. 1


  1. 1.

    Trotti, A., Colevas, A. D., Setser, A., Rusch, V., Jaques, D., Budach, V., et al. (2003). CTCAE v3.0: Development of a comprehensive grading system for the adverse events of cancer treatment. Seminars in Radiation Oncology, 13, 176–181.

    PubMed  Article  Google Scholar 

  2. 2.

    National Cancer Institute, National Institutes of Health, US Department of Health and Human Services. Common Terminology Criteria for Adverse Events (CTCAE) Version 4.0. Published May 28, 2009; Revised version 4.03 June 14, 2010 (Vol. Available from: Accessed 20 May 2013).

  3. 3.

    Basch, E. (2010). The missing voice of patients in drug-safety reporting. New England Journal of Medicine, 362(10), 865–869. doi:10.1056/NEJMp0911494.

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  4. 4.

    Basch, E., Iasonos, A., McDonough, T., Barz, A., Culkin, A., Kris, M. G., et al. (2006). Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: Results of a questionnaire-based study. Lancet Oncology, 7, 903–909. doi:10.1016/S1470-2045(06)70910-X.

    PubMed  Article  Google Scholar 

  5. 5.

    Basch, E., Jia, X., Heller, G., Barz, A., Sit, L., Fruscione, M., et al. (2009). Adverse symptom event reporting by patients versus clinicians: Relationships with clinical outcomes. Journal of the National Cancer Institute, 101(23), 1624–1632. doi:10.1093/jnci/djp386.

    PubMed  Article  Google Scholar 

  6. 6.

    Atkinson, T. M., Li, Y., Coffey, C. W., Sit, L., Shaw, M., Lavene, D., et al. (2012). Reliability of adverse symptom event reporting by clinicians. Quality of Life Research, 21(7), 1159–1164. doi:10.1007/s11136-011-0031-4.

    PubMed Central  PubMed  Article  Google Scholar 

  7. 7.

    Xiao, C., Polomano, R., & Bruner, D. W. (2012). Comparison between patient-reported and clinician-observed symptoms in oncology. Cancer Nursing,. doi:10.1097/NCC.0b013e318269040f.

    Google Scholar 

  8. 8.

    Greimel, E. R., Bjelic-Radisic, V., Pfisterer, J., Hilpert, F., Daghofer, F., Pujade-Lauraine, E., et al. (2011). Toxicity and quality of life outcomes in ovarian cancer patients participating in randomized controlled trials. Supportive Care in Cancer, 19(9), 1421–1427. doi:10.1007/s00520-010-0969-8.

    PubMed  Article  Google Scholar 

  9. 9.

    Flores, L. T., Bennett, A. V., Law, E. B., Hajj, C., Griffith, M. P., & Goodman, K. A. (2012). Patient-reported outcomes versus clinician symptom reporting during chemoradiation for rectal cancer. Gastrointestinal Cancer Research, 5(4), 119–124.

    PubMed  Google Scholar 

  10. 10.

    Basch, E., Iasonos, A., Barz, A., Culkin, A., Kris, M. G., Artz, D., et al. (2007). Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. Journal of Clinical Oncology, 25(34), 5374–5380. doi:10.1200/JCO.2007.11.2243.

    PubMed  Article  Google Scholar 

  11. 11.

    Tang, B., Giuliani, M., Le, L. W., Higgins, J., Bezjak, A., Brade, A., et al. (2013). Capturing acute toxicity data during lung radiotherapy by using a patient-reported assessment tool. Clinical Lung Cancer, 14(2), 108–112. doi:10.1016/j.cllc.2012.06.003.

    PubMed  Article  Google Scholar 

  12. 12.

    US Department of Health and Human Services. Guidance for industry. patient-reported outcome measures: Use in medical development to support labeling claims (December 2009. Available from: Accessed 20 May 2013).

  13. 13.

    European Medicines Agency. Committee for medicinal products for human use (CHMP). Pre-authorisation evaluation of medicines for human use: Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products (January 2006, Available from: Accessed 20 May 2013).

  14. 14.

    Ashley, L., Jones, H., Thomas, J., Forman, D., Newsham, A., Morris, E., et al. (2011). Integrating cancer survivors’ experiences into UK cancer registries: Design and development of the ePOCS system (electronic Patient-reported Outcomes from Cancer Survivors). British Journal of Cancer, 105, S74–S81. doi:10.1038/Bjc.2011.424.

    PubMed Central  PubMed  Article  Google Scholar 

  15. 15.

    Banerjee, A. K., & Ingate, S. (2012). Web-based patient-reported outcomes in drug safety and risk management: Challenges and opportunities? Drug Safety, 35(6), 437–446. doi:10.2165/11632390-000000000-00000.

    PubMed  Article  Google Scholar 

  16. 16.

    Boers-Doets, C. B., Gelderblom, H., Lacouture, M. E., Epstein, J. B., Nortier, J. W., & Kaptein, A. A. (2013). Experiences with the FACT-EGFRI-18 instrument in EGFRI-associated mucocutaneous adverse events. Supportive Care in Cancer,. doi:10.1007/s00520-013-1752-4.

    Google Scholar 

  17. 17.

    Chan, A., & Tan, E. H. (2011). How well does the MESTT correlate with CTCAE scale for the grading of dermatological toxicities associated with oral tyrosine kinase inhibitors? Supportive Care in Cancer, 19(10), 1667–1674. doi:10.1007/s00520-010-0999-2.

    PubMed  Article  Google Scholar 

  18. 18.

    Farnell, D. J., Mandall, P., Anandadas, C., Routledge, J., Burns, M. P., Logue, J. P., et al. (2010). Development of a patient-reported questionnaire for collecting toxicity data following prostate brachytherapy. Radiotherapy and Oncology, 97(1), 136–142. doi:10.1016/j.radonc.2010.05.011.

    PubMed  Article  Google Scholar 

  19. 19.

    Quinten, C., Maringwa, J., Gotay, C. C., Martinelli, F., Coens, C., Reeve, B. B., et al. (2011). Patient self-reports of symptoms and clinician ratings as predictors of overall cancer survival. Journal of the National Cancer Institute, 103(24), 1851–1858. doi:10.1093/Jnci/Djr485.

    PubMed  Article  Google Scholar 

  20. 20.

    Stacey, D., Bakker, D., Ballantyne, B., Chapman, K., Cumminger, J., Green, E., et al. (2012). Managing symptoms during cancer treatments: evaluating the implementation of evidence-informed remote support protocols. Implementation Science, 7, 110. doi:10.1186/1748-5908-7-110.

    PubMed  Article  Google Scholar 

  21. 21.

    Johansen, M. A., Henriksen, E., Horsch, A., Schuster, T., & Berntsen, G. K. R. (2012). Electronic symptom reporting between patient and provider for improved health care service quality: A systematic review of randomized controlled trials. Part 1: State of the Art. Journal of Medical Internet Research, 14(5). doi:10.2196/jmir.2214.

  22. 22.

    Johansen, M. A., Henriksen, E., Berntsen, G., & Horsch, A. (2011). Electronic symptom reporting by patients: A literature review. Studies in Health Technology and Informatics, 169, 13–17.

    PubMed  Google Scholar 

  23. 23.

    Seow, H., King, S., Green, E., Pereira, J., & Sawka, C. (2011). Perspectives of patients on the utility of electronic patient-reported outcomes on cancer care. Journal of Clinical Oncology, 29(31), 4213–4214. doi:10.1200/JCO.2011.37.9750.

    PubMed  Article  Google Scholar 

  24. 24.

    King, A., Daniels, J., Lim, J., Cochrane, D. D., Taylor, A., & Ansermino, J. M. (2010). Time to listen: A review of methods to solicit patient reports of adverse events. Quality and Safety in Health Care, 19(2), 148–157. doi:10.1136/qshc.2008.030114.

    CAS  PubMed  Article  Google Scholar 

  25. 25.

    Zhu, J., Stuver, S. O., Epstein, A. M., Schneider, E. C., Weissman, J. S., & Weingart, S. N. (2011). Can we rely on patients’ reports of adverse events? Medical Care, 49(10), 948–955. doi:10.1097/MLR.0b013e31822047a8.

    PubMed  Article  Google Scholar 

  26. 26.

    Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA, 288(23), 3027–3034.

    PubMed  Article  Google Scholar 

  27. 27.

    Basch, E. M., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L., Sit, L., et al. (2011). Electronic toxicity monitoring and patient-reported outcomes. Cancer Journal, 17(4), 231–234. doi:10.1097/PPO.0b013e31822c28b3.

    Article  Google Scholar 

  28. 28.

    Stull, D. E., Leidy, N. K., Parasuraman, B., & Chassany, O. (2009). Optimal recall periods for patient-reported outcomes: Challenges and potential solutions. Current Medical Research and Opinion, 25(4), 929–942. doi:10.1185/03007990902774765.

    PubMed  Article  Google Scholar 

  29. 29.

    Kirkova, J., Davis, M. P., Walsh, D., Tiernan, E., O’Leary, N., LeGrand, S. B., et al. (2006). Cancer symptom assessment instruments: A systematic review. Journal of Clinical Oncology, 24(9), 1459–1473. doi:10.1200/JCO.2005.02.8332.

    PubMed  Article  Google Scholar 

  30. 30.

    Wild, D., Eremenco, S., Mear, I., Martin, M., Houchin, C., Gawlicki, M., et al. (2009). Multinational trials-recommendations on the translations required, approaches to using the same language in different countries, and the approaches to support pooling the data: The ISPOR patient-reported outcomes translation and linguistic validation good research practices task force report. Value in Health, 12(4), 430–440. doi:10.1111/j.1524-4733.2008.00471.x.

    PubMed  Article  Google Scholar 

  31. 31.

    Basch, E., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L. M., Atkinson, T. M., et al. (Under Review). Development of the National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Journal of National Cancer Institute.

  32. 32.

    Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: SAGE Publications, Inc.

    Google Scholar 

  33. 33.

    Patrick, D. L., Burke, L. B., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., et al. (2011). Content validity-establishing and reporting the evidence in newly Developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part 2-assessing respondent understanding. Value in Health, 14(8), 978–988. doi:10.1016/j.jval.2011.06.013.

    PubMed  Article  Google Scholar 

  34. 34.

    Bowen, G. A. (2008). Naturalistic inquiry and the saturation concept: A research note. Qualitative Research, 8(1), 137–152. doi:10.1177/1468794107085301.

    Article  Google Scholar 

  35. 35.

    Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59–82. doi:10.1177/1525822X05279903.

    Article  Google Scholar 

  36. 36.

    Jobe, J. B., & Mingay, D. J. (1989). Cognitive research improves questionnaires. American Journal of Public Health, 79(8), 1053–1055.

    CAS  PubMed  Article  Google Scholar 

  37. 37.

    Jobe, J. B. (2003). Cognitive psychology and self-reports: Models and methods. Quality of Life Research, 12(3), 219–227.

    PubMed  Article  Google Scholar 

  38. 38.

    Miller, K. (2003). Conducting cognitive interviews to understand question-response limitations. American Journal of Health Behavior, 27(Suppl 3), S264–S272.

    PubMed  Article  Google Scholar 

  39. 39.

    Schwarz, N., Oyserman, D., & Petycheva, E. (2010). Cognition, communication, and culture: Implications for the survey response process. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. E. Lyberg, P. P. Mohler, et al. (Eds.), Survey methods in multinational, multicultural, and multiregional contexts (pp. 177–190). Hoboken, NJ: John Wiley & Sons.

    Google Scholar 

  40. 40.

    Tourangeau, R. (1984). Cognitive science and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.

    Google Scholar 

  41. 41.

    Namey, E., Guest, G., Thairy, L., & Johnson, L. (2008). Data reduction techniques for large qualitative data sets. In G. Guest & K. M. MacQueen (Eds.), Handbook for team-based qualitative research (pp. 137–161). Lanham, MD: AltaMira Press.

    Google Scholar 

  42. 42.

    Saldana, J. (2009). The coding manual for qualitative researchers. Los Angeles, CA: Sage Publications.

    Google Scholar 

  43. 43.

    Basch, E. (2012). Beyond the FDA PRO guidance: Steps toward integrating meaningful patient-reported outcomes into regulatory trials and US drug labels. Value in Health, 15(3), 401–403. doi:10.1016/j.jval.2012.03.1385.

    PubMed  Article  Google Scholar 

  44. 44.

    Symptom Management and Quality of Life Steering Committee. Clinical trials planning meeting—building bridges: The identification of core symptoms and health-related quality of life domains for use in cancer research (September 22–23, 2011. Available from: Accessed 7 June 2012).

  45. 45.

    Mitchell, S. A., Lang, K., Nichols, C., Clauser, S. B., Federico, V., Lalla, D., et al. (2012). Validation of the NCI Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) in women receiving treatment for metastatic breast cancer. Chicago, IL: Annual Meeting of the American Society of Clinical Oncology.

    Google Scholar 

  46. 46.

    Dueck, A. C., Mendoza, T., Mitchell, S. A., Reeve, B. B., Castro, K., Denicoff, A., et al. (2012). Validity and reliability of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Chicago, IL: Annual Meeting of the American Society of Clinical Oncology.

    Google Scholar 

  47. 47.

    Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429. doi:10.1111/j.1524-4733.2008.00470.x.

    PubMed  Article  Google Scholar 

  48. 48.

    (2012) Methodological standards and patient-centeredness in comparative effectiveness research—The PCORI perspective. JAMA, 307(15), 1636–1640. doi:10.1001/jama.2012.466.

Download references


Work described in this report was supported by contracts from the United States National Cancer Institute, HHSN261201000043C and HHSN261201000063C. Each author met all International Committee of Medical Journal Editors (ICMJE) requirements (i.e., conception/design/acquisition/analysis, drafting/revising article, and final approval) for authorship.

Author information




Corresponding author

Correspondence to Jennifer L. Hay.

Additional information

NCI PRO-CTCAE Study Group members are Cindy Geoghegan, Jeff A. Sloan (Mayo Clinic, Rochester, MN, USA), Charles S. Cleeland (University of Texas M.D. Anderson Cancer Center, Houston, TX, USA), Amy P. Abernethy (Duke University Medical Center, Durham, NC, USA), Deborah W. Bruner (University of Pennsylvania School of Nursing, Philadelphia, PA, USA), Allison Barz (The Children’s Hospital of Philadelphia, Philadelphia, PA, USA), Amylou C. Dueck (Mayo Clinic, Scottsdale, AZ, USA), and Yuelin Li (Memorial Sloan-Kettering Cancer Center, New York, NY, USA).

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 23 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hay, J.L., Atkinson, T.M., Reeve, B.B. et al. Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Qual Life Res 23, 257–269 (2014).

Download citation


  • Drug toxicity
  • Neoplasms
  • Psychometrics
  • Reproducibility of results
  • Qualitative research