Establishing clinically-relevant terms and severity thresholds for Patient-Reported Outcomes Measurement Information System® (PROMIS®) measures of physical function, cognitive function, and sleep disturbance in people with cancer using standard setting

  • Nan E. RothrockEmail author
  • Karon F. Cook
  • Mary O’Connor
  • David Cella
  • Ashley Wilder Smith
  • Susan E. Yount



Patient-Reported Outcomes Measurement Information System® (PROMIS®) physical function, cognitive function, and sleep disturbance measures are increasingly used in cancer care. However, there is limited guidance for interpreting the clinical meaning of scores. This study aimed to apply bookmarking, a standard setting methodology, to identify PROMIS score thresholds in the context of cancer care.


Using item parameters, we constructed vignettes of five items covering the range of possible scores. Focus groups were held with cancer care providers and people with cancer. Terminology for categorizing levels of severity was explored. Participants rank ordered vignettes by severity and then placed bookmarks between vignettes representing different levels of severity. Group discussion was held until consensus on bookmark placement was reached.


Clinicians selected “within normal limits,” “mild,” “moderate,” and “severe” to describe levels of severity. Both patients and clinicians were able to apply these labels, but there was not unanimous support for any set of descriptors. Clinicians and patients agreed on all severity thresholds for sleep disturbance. For cognitive and physical function, clinicians and patients agreed on the threshold between “within normal limits” and “mild.” However, patients required greater dysfunction than clinicians before applying “moderate” and “severe” labels.


Bookmarking can be applied to develop provisional score interpretation for PROMIS measures. Patients and clinicians were frequently consistent in their bookmark placement. When there was variance, patients required more dysfunction before assigning more severity. Additional research with other cancer samples is needed to evaluate the replicability and generalizability of our findings.


Patient-reported outcomes Physical function Sleep Cognitive function Reference values PROMIS 



This study was funded by the National Institutes of Health Grant U2C CA186878.

Supplementary material

11136_2019_2261_MOESM1_ESM.docx (29 kb)
Supplementary material 1 (DOCX 28 kb)


  1. 1.
    Chen, J., Ou, L., & Hollis, S. J. (2013). A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Services Research, 13, 211–211.CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Howell, D., Molloy, S., Wilkinson, K., Green, E., Orchard, K., Wang, K., et al. (2015). Patient-reported outcomes in routine cancer clinical practice: A scoping review of use, impact on health outcomes, and implementation factors. Annals of Oncology, 26(9), 1846–1858.CrossRefPubMedGoogle Scholar
  3. 3.
    Wagner, L. I., Schink, J., Bass, M., Patel, S., Diaz, M. V., Rothrock, N., et al. (2015). Bringing PROMIS to practice: Brief and precise symptom screening in ambulatory cancer care. Cancer, 121(6), 927–934.CrossRefPubMedGoogle Scholar
  4. 4.
    Seneviratne, M. G., Bozkurt, S., Patel, M. I., Seto, T., Brooks, J. D., Blayney, D. W., et al. (2019). Distribution of global health measures from routinely collected PROMIS surveys in patients with breast cancer or prostate cancer. Cancer, 125(6), 943–951.CrossRefPubMedGoogle Scholar
  5. 5.
    Lohr, K. N., Aaronson, N. K., Alonso, J., Audrey Burnam, M., Patrick, D. L., Perrin, E. B., et al. (1996). Evaluating quality-of-life and health status instruments: Development of scientific review criteria. Clinical Therapeutics, 18(5), 979–992.CrossRefPubMedGoogle Scholar
  6. 6.
    Snyder, C. F., Smith, K. C., Bantug, E. T., Tolbert, E. E., Blackford, A. L., Brundage, M. D., et al. (2017). What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer, 123(10), 1848–1859.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Cappelleri, J. C., & Bushmakin, A. G. (2014). Interpretation of patient-reported outcomes. Statistical Methods in Medical Research, 23(5), 460–483.CrossRefPubMedGoogle Scholar
  8. 8.
    Given, B., Given, C. W., Sikorskii, A., Jeon, S., McCorkle, R., Champion, V., et al. (2008). Establishing mild, moderate, and severe scores for cancer-related symptoms: How consistent and clinically meaningful are interference-based severity cut-points? Journal of Pain and Symptom Management, 35(2), 126–135.CrossRefPubMedGoogle Scholar
  9. 9.
    Palos, G. R., Mendoza, T. R., Mobley, G. M., Cantor, S. B., & Cleeland, C. S. (2006). Asking the community about cutpoints used to describe mild, moderate, and severe pain. The Journal of Pain, 7(1), 49–56.CrossRefPubMedGoogle Scholar
  10. 10.
    Coon, C. D., & Cook, K. F. (2018). Moving from significance to real-world meaning: Methods for interpreting change in clinical outcome assessment scores. Quality of Life Research, 27(1), 33–40.CrossRefPubMedGoogle Scholar
  11. 11.
    Basch, E., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L. M., Dueck, A. C., et al. (2014). Development of the National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JNCI: Journal of the National Cancer Institute, 106(9), 244.CrossRefGoogle Scholar
  12. 12.
    Cizek, G. J. (1993). Reconsidering standards and criteria. Journal of Educational Measurement, 30(2), 93–106.CrossRefGoogle Scholar
  13. 13.
    Perie, M. (2005). Angoff and Bookmark methods. Workshop presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.Google Scholar
  14. 14.
    Cella, D., Choi, S., Garcia, S., Cook, K. F., Rosenbloom, S., Lai, J.-S., et al. (2014). Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment. Quality of Life Research, 23(10), 2651–2661.CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Cook, K. F., Victorson, D. E., Cella, D., Schalet, B. D., & Miller, D. (2015). Creating meaningful cut-scores for Neuro-QOL measures of fatigue, physical functioning, and sleep disturbance using standard setting with patients and providers. Quality of Life Research, 24(3), 575–589.CrossRefPubMedGoogle Scholar
  16. 16.
    Nagaraja, V., Mara, C., Khanna, P. P., Namas, R., Young, A., Fox, D. A., et al. (2018). Establishing clinical severity for PROMIS® measures in adult patients with rheumatic diseases. Quality of Life Research, 27(3), 755–764.CrossRefPubMedGoogle Scholar
  17. 17.
    Morgan, E. M., Mara, C. A., Huang, B., Barnett, K., Carle, A. C., Farrell, J. E., et al. (2017). Establishing clinical meaning and defining important differences for Patient-Reported Outcomes Measurement Information System (PROMIS®) measures in juvenile idiopathic arthritis using standard setting with patients, parents, and providers. Quality of Life Research, 26(3), 565–586.CrossRefPubMedGoogle Scholar
  18. 18.
    Cook, K., Cella, D., & Reeve, B. (2019). PRO-bookmarking to estimate clinical thresholds for patient-reported symptoms and function. Medical Care, 57, S13–S17.CrossRefPubMedGoogle Scholar
  19. 19.
    Rose, M., Bjorner, J. B., Becker, J., Fries, J., & Ware, J. (2008). Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of Clinical Epidemiology, 61(1), 17–33.CrossRefPubMedGoogle Scholar
  20. 20.
    Rose, M., Bjorner, J. B., Gandek, B., Bruce, B., Fries, J. F., & Ware, J. E. (2014). The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency. Journal of Clinical Epidemiology, 67(5), 516–526.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Lai, J. S., Wagner, L. I., Jacobsen, P. B., & Cella, D. (2014). Self-reported cognitive concerns and abilities: two sides of one coin? Psycho-Oncology, 23(10), 1133–1141.CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Buysse, D. J., Yu, L., Moul, D. E., Germain, A., Stover, A., Dodds, N. E., et al. (2010). Development and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments. Sleep, 33(6), 781–792.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194.CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(5), S22–S31.CrossRefPubMedGoogle Scholar
  25. 25.
    R Development Core Team. (2008). R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Medical Social SciencesFeinberg School of Medicine of Northwestern UniversityChicagoUSA
  2. 2.Outcomes Research BranchNational Cancer InstituteBethesdaUSA

Personalised recommendations