Quality of Life Research

, Volume 23, Issue 10, pp 2651–2661 | Cite as

Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment

  • David CellaEmail author
  • Seung Choi
  • Sofia Garcia
  • Karon F. Cook
  • Sarah Rosenbloom
  • Jin-Shei Lai
  • Donna Surges Tatum
  • Richard Gershon



Although the use of patient-reported outcome measures (PROs) has increased markedly, clinical interpretation of scores remains lacking. We developed a method to identify clinical severity thresholds for pain, fatigue, depression, and anxiety in people with cancer.


Using available Patient-Reported Outcomes Measurement Information System (PROMIS) item bank response data collected on 840 cancer patients, symptom vignettes across a range of symptom severity were developed and placed on index cards. Cards represented symptom severity at five-point intervals differences on the T score metric [mean = 50; standard deviation (SD) = 10]. Symptom vignettes for each symptom were anchored on these standardized scores at 0.5 SD increments across the full range of severity. Clinical experts, blind to the PROMIS score associated with each vignette, rank-ordered the vignettes by severity, then arrived at consensus regarding which two vignettes were at the upper and lower boundaries of normal and mildly symptomatic for each symptom. The procedure was repeated to identify cut scores separating mildly from moderately symptomatic, and moderately from severely symptomatic scores. Clinician severity rankings were then compared to the T scores upon which the vignettes were based.


For each of the targeted PROs, the severity rankings reached by clinician consensus perfectly matched the numerical rankings of their associated T scores. Across all symptoms, the thresholds (cut scores) identified to differentiate normal from mildly symptomatic were near a T score of 50. Cut scores differentiating mildly from moderately symptomatic were at or near 60, and those separating moderately from severely symptomatic were at or near 70.


The study results provide empirically generated PROMIS T score thresholds that differentiate levels of symptom severity for pain interference, fatigue, anxiety, and depression. The convergence of clinical judgment with self-reported patient severity scores supports the validity of this methodology to derive clinically relevant symptom severity levels for PROMIS symptom measures in other settings.


PROMIS Patient-reported outcomes Symptom severity levels Standard setting Cancer 



Several colleagues contributed time and expertise to this effort, and we wish to acknowledge their effort. Clinical expert break-out group facilitators included some of the coauthors and also Nan Rothrock, PhD; and Zeeshan Butt, PhD. Expert raters included some of the coauthors and also Amy Peterman, PhD; Janine Gauthier, PhD; Lynne Wagner, PhD; Kimberly Davis, PhD; Margaret Whalen, RN; Gershon Locker, MD; Carmen Griza, MD; Jin-Shei Lai, PhD, OTR/L; Allen Heinemann, PhD; Lauren Mermel Welles, PT; Robin Mieli, MA, OTR/L; Kimberly Brennan, PT; Nan Rothrock, PhD; Zeeshan Butt, PhD; David Victorson, PhD; Judith Paice, PhD, RN, FAAN; Rose Catchatourian, MD; George Carro, PharmD; Mousami Shah, MD. Study coordination was done by Jacquelyn George and Rachel Hanrahan. Supported by a grant from the National Cancer Institute (R01 CA60068; Cella, PI).


  1. 1.
    Carlson, L. E., Groff, S. L., Maciejewski, O., & Bultz, B. D. (2010). Screening for distress in lung and breast cancer outpatients: A randomized controlled trial. Journal of Clinical Oncology, 28(33), 4884–4891.PubMedCrossRefGoogle Scholar
  2. 2.
    Snyder, C., Jensen, R., Geller, G., Carducci, M., & Wu, A. (2010). Relevant content for a patient-reported outcomes questionnaire for use in oncology clinical practice: Putting doctors and patients on the same page. Quality of Life Research, 19(7), 1045–1055.PubMedCrossRefGoogle Scholar
  3. 3.
    Bennett, A. V., Jensen, R. E., & Basch, E. (2012). Electronic patient-reported outcome systems in oncology clinical practice. CA: A Cancer Journal for Clinicians, 62(5), 336–347.Google Scholar
  4. 4.
    Abernethy, A. P., Herndon Ii, J. E., Wheeler, J. L., Day, J. M., Hood, L., Patwardhan, M., et al. (2009). Feasibility and acceptability to patients of a longitudinal system for evaluating cancer-related symptoms and quality of life: Pilot study of an e/tablet data-collection system in academic oncology. Journal of Pain and Symptom Management, 37(6), 1027–1038.PubMedCrossRefGoogle Scholar
  5. 5.
    Berry, D. L., Blumenstein, B. A., Halpenny, B., Wolpin, S., Fann, J. R., Austin-Seymour, M., et al. (2011). Enhancing patient-provider communication with the electronic self-report assessment for cancer: A randomized trial. Journal of Clinical Oncology, 29(8), 1029–1035.PubMedCentralPubMedCrossRefGoogle Scholar
  6. 6.
    Basch, E., & Abernethy, A. P. (2011). Supporting clinical practice decisions with real-time patient-reported outcomes. Journal of Clinical Oncology, 29(8), 954–956.PubMedCrossRefGoogle Scholar
  7. 7.
    Cella, D., Gershon, R., Lai, J.-S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141.PubMedCrossRefGoogle Scholar
  8. 8.
    Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap Cooperative Group during its first two years. Medical Care, 45(5 Suppl 1), S3–S11.PubMedCentralPubMedCrossRefGoogle Scholar
  9. 9.
    Riley, W. T., Rothrock, N., Bruce, B., Christodolou, C., Cook, K., Hahn, E. A., et al. (2010). Patient-Reported Outcomes Measurement Information System (PROMIS) domain names and definitions revisions: further evaluation of content validity in IRT-derived item banks. Quality of Life Research, 19(9), 1311–1321.PubMedCentralPubMedCrossRefGoogle Scholar
  10. 10.
    Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194.PubMedCentralPubMedCrossRefGoogle Scholar
  11. 11.
    Liu, H., Cella, D., Gershon, R., Shen, J., Morales, L. S., Riley, W., et al. (2010). Representativeness of the PROMIS internet panel. Journal of Clinical Epidemiology, 63(11), 1169–1178.PubMedCentralPubMedCrossRefGoogle Scholar
  12. 12.
    Jensen, M. P., Smith, D. G., Ehde, D. M., & Robinsin, L. R. (2001). Pain site and the effects of amputation pain: Further clarification of the meaning of mild, moderate, and severe pain. Pain, 91(3), 317–322.PubMedCrossRefGoogle Scholar
  13. 13.
    Paul, S. M., Zelman, D. C., Smith, M., & Miaskowski, C. (2005). Categorizing the severity of cancer pain: Further exploration of the establishment of cutpoints. Pain, 113(1), 37–44.PubMedCrossRefGoogle Scholar
  14. 14.
    Serlin, R. C., Mendoza, T. R., Nakamura, Y., Edwards, K. R., & Cleeland, C. S. (1995). When is cancer pain mild, moderate or severe? Grading pain severity by its interference with function. Pain, 61(2), 277–284.PubMedCrossRefGoogle Scholar
  15. 15.
    Zelman, D., Hoffman, D., Reed, P., Edwards, L., & Dukes, E. (2003). Quantifying a “day of manageable pain control:” The relationship between pain severity and patient daily functional expectations. Journal of Pain, 4(Suppl 1), 1.Google Scholar
  16. 16.
    Zelman, D. C., Dukes, E., Brandenburg, N., Bostrom, A., & Gore, M. (2005). Identification of cut-points for mild, moderate and severe pain due to diabetic peripheral neuropathy. Pain, 115(1), 29–36.PubMedCrossRefGoogle Scholar
  17. 17.
    Wang, X. S., Zhao, F., Fisch, M. J., O’Mara, A. M., Cella, D., Mendoza, T. R., et al. (2014). Prevalence and characteristics of moderate to severe fatigue. Cancer, 120(3), 425–432.PubMedCrossRefGoogle Scholar
  18. 18.
    Given, B., Sikorskii, A., Given, C. W., Jeon, S., McCorkle, R., Champion, V., et al. (2008). Establishing mild, moderate, and severe scores for cancer-related symptoms: How consistent and clinically meaningful are interference-based severity cut-points? Journal of Pain and Symptom Management, 35(2), 126–135.PubMedCentralPubMedCrossRefGoogle Scholar
  19. 19.
    Hays, R. D., Reise, S., & Calderón, J. L. (2012). How much is lost in using single items? Journal of General Internal Medicine, 27(11), 1402–1403.PubMedCentralPubMedCrossRefGoogle Scholar
  20. 20.
    Angoff, W. H., & Thorndike, R. L. (1971). Scales, norms and equivalent scores. In Educational measurement. Washington, D.C.: American Council on Education.Google Scholar
  21. 21.
    Lewis, D., Mitzel, H. C., & Green, D. R. (1996). Standard setting: A bookmark approach. Paper presented at the Council of Chief State School Officers National Conference on Large Scale Assessment Phoenix, AZ.Google Scholar
  22. 22.
    Impara, J. C., & Plake, B. S. (1977). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 355–368.Google Scholar
  23. 23.
    Baghaei, P. (2007). Applying The Rasch Rating-Scale model to set multiple cut-offs. Rasch Measurement Transactions, 20(4), 1075–1076.Google Scholar
  24. 24.
    Lai, J. S., Cella, D., Choi, S. W., Junghaenel, D. U., Christodolou, C., Gershon, R., et al. (2011). How item banks and their application can influence measurement practice in rehabilitation medicine: A PROMIS fatigue item bank example. Archives of Physical Medicine and Rehabilitation, 92(10 Supplement), S20–S27.PubMedCentralPubMedCrossRefGoogle Scholar
  25. 25.
    Amtmann, D., Cook, K. F., Jensen, M. P., Chen, W.-H., Choi, S., Revicki, D., et al. (2010). Development of a PROMIS item bank to measure pain interference. Pain, 150(1), 173–182.PubMedCentralPubMedCrossRefGoogle Scholar
  26. 26.
    Pilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., & Cella, D. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS):depression, anxiety, and anger. Assessment, 18(3), 263–283.PubMedCentralPubMedCrossRefGoogle Scholar
  27. 27.
    Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(5 Suppl 1), S22–S31.PubMedCrossRefGoogle Scholar
  28. 28.
    King, M. T., Cella, D., Osoba, D., Stockler, M., Eton, D., Thompson, J., et al. (2010). Meta-analysis provides evidence-based interpretation guidelines for the clinical significance of mean differences for the FACT-G, a cancer-specific quality of life questionnaire. Patient Related Outcome Measures, 1(1), 119–126.PubMedCentralPubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • David Cella
    • 1
    Email author
  • Seung Choi
    • 1
  • Sofia Garcia
    • 1
  • Karon F. Cook
    • 1
  • Sarah Rosenbloom
    • 1
  • Jin-Shei Lai
    • 1
  • Donna Surges Tatum
    • 2
  • Richard Gershon
    • 1
  1. 1.Department of Medical Social SciencesNorthwestern UniversityChicagoUSA
  2. 2.Meaningful Measurement Inc.ChicagoUSA

Personalised recommendations