Quality of Life Research

, Volume 24, Issue 3, pp 575–589 | Cite as

Creating meaningful cut-scores for Neuro-QOL measures of fatigue, physical functioning, and sleep disturbance using standard setting with patients and providers

  • Karon F. CookEmail author
  • David E. Victorson
  • David Cella
  • Benjamin D. Schalet
  • Deborah Miller



To establish clinically relevant classifications of health outcome scores for four Neuro-QOL measures (lower extremity function, upper extremity function, fatigue, and sleep disturbance).


We employed a modified educational standard-setting methodology to identify cut-scores for symptom severity. Clinical vignettes were developed to represent graduated levels of symptom severity. A clinician panel and a panel of persons with multiple sclerosis (PwMS) were recruited, and, in separate, 1-day meetings, the panelists identified adjacent vignettes they judged to represent the threshold between two levels of severity for a given domain (e.g., threshold between a vignette that indicated “no problems” with sleep and the adjacent one that represented “mild problems” with sleep). Working independently, each panel (PwMS and clinicians) reached consensus on its recommended thresholds for each of the four targeted measures. Cut-scores were defined as the mean location for each pair of threshold vignettes.


PwMS and clinician panels derived identical thresholds for severity levels of lower extremity function and sleep disturbance, but slightly different ones for upper extremity function and fatigue. In every case of divergence, PwMS set higher thresholds for more severe classifications of symptoms (by 0.5 SDs) than did clinicians.


The modified bookmarking method is effective for defining thresholds for symptom severity based on self-reported outcome scores and consensus judgments. Derived cut-scores and severity levels provide an interpretative context for Neuro-QOL scores. Future studies should explore whether these findings can be replicated and evaluate the validity of the classifications compared to external criteria.


Interpretation of HRQOL data Item response theory (IRT) Psychometric methods/scaling Neurology Multiple sclerosis 



This research was supported by The National Multiple Sclerosis Society. The contents represent original work and have not been published elsewhere. No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the author or upon any organization with which the author(s) is/are associated.


  1. 1.
    Hirtz, D., Thurman, D. J., Gwinn-Hardy, K., Mohamed, M., Chaudhuri, A. R., & Zalutsky, R. (2007). How common are the “common” neurologic disorders? Neurology, 68(5), 326–337.CrossRefPubMedGoogle Scholar
  2. 2.
    Hopman, W. M., Coo, H., Edgar, C. M., McBride, E. V., Day, A. G., & Brunet, D. G. (2007). Factors associated with health-related quality of life in multiple sclerosis. Canadian Journal of Neurological Sciences, 34(2), 160–166.CrossRefPubMedGoogle Scholar
  3. 3.
    Miller, D. M., & Allen, R. (2010). Quality of life in multiple sclerosis: Determinants, measurement, and use in clinical practice. Current Neurology and Neuroscience Reports, 10(5), 397–406.CrossRefPubMedGoogle Scholar
  4. 4.
    Mitchell, A. J., Benito-Leon, J., Rivera-Navarro, J., & Morales-Gonzalez, J. M. (2004). Quality of life is not favorable for most patients with multiple sclerosis. Archives of Neurology, 61(11), 1807–1808. Author reply 1808.Google Scholar
  5. 5.
    Kargiotis, O., Paschali, A., Messinis, L., & Papathanasopoulos, P. (2010). Quality of life in multiple sclerosis: Effects of current treatment options. International Review of Psychiatry, 22(1), 67–82.CrossRefPubMedGoogle Scholar
  6. 6.
    Miller, D., Rudick, R. A., & Hutchinson, M. (2010). Patient-centered outcomes: Translating clinical efficacy into benefits on health-related quality of life. Neurology, 74(Suppl 3), S24–S35.PubMedGoogle Scholar
  7. 7.
    Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141.CrossRefPubMedGoogle Scholar
  8. 8.
    Gershon, R. C., Cella, D., Fox, N. A., Havlik, R. J., Hendrie, H. C., & Wagster, M. V. (2010). Assessment of neurological and behavioural function: The NIH Toolbox. Lancet Neurology, 9(2), 138–139.CrossRefGoogle Scholar
  9. 9.
    U S Department of Health and Human Services, F. D. A. (2010). The patient reported outcomes (PRO) Consortium. From Retrieved December 3, 2011, from
  10. 10.
    Cella, D., Lai, J. S., Nowinski, C. J., Victorson, D., Peterman, A., Miller, D., et al. (2012). Neuro-QOL: Brief measures of health-related quality of life for clinical research in neurology. Neurology, 78(23), 1860–1867.CrossRefPubMedCentralPubMedGoogle Scholar
  11. 11.
    Gershon, R. C., Lai, J. S., Bode, R., Choi, S., Moy, C., Bleck, T., et al. (2012). Neuro-QOL: Quality of life item banks for adults with neurological disorders: Item development and calibrations based upon clinical and general population testing. Quality of Life Research, 21(3), 475–486.CrossRefPubMedCentralPubMedGoogle Scholar
  12. 12.
    Wilson, I. B., & Cleary, P. D. (1995). Linking clinical variables with health-related quality of life. A conceptual model of patient outcomes. JAMA, 273(1), 59–65.CrossRefPubMedGoogle Scholar
  13. 13.
    Anastasi, A. (1988). Psychological testing (6th ed.). New York, NY: Macmillian Publishing Company.Google Scholar
  14. 14.
    Jensen, M. P., Smith, D. G., Ehde, D. M., & Robinsin, L. R. (2001). Pain site and the effects of amputation pain: Further clarification of the meaning of mild, moderate, and severe pain. Pain, 91(3), 317–322.CrossRefPubMedGoogle Scholar
  15. 15.
    Paul, S. M., Zelman, D. C., Smith, M., & Miaskowski, C. (2005). Categorizing the severity of cancer pain: Further exploration of the establishment of cutpoints. Pain, 113(1–2), 37–44.CrossRefPubMedGoogle Scholar
  16. 16.
    Serlin, R. C., Mendoza, T. R., Nakamura, Y., Edwards, K. R., & Cleeland, C. S. (1995). When is cancer pain mild, moderate or severe? Grading pain severity by its interference with function. Pain, 61(2), 277–284.CrossRefPubMedGoogle Scholar
  17. 17.
    Zelman, D. C., Dukes, E., Brandenburg, N., Bostrom, A., & Gore, M. (2005). Identification of cut-points for mild, moderate and severe pain due to diabetic peripheral neuropathy. Pain, 115(1–2), 29–36.CrossRefPubMedGoogle Scholar
  18. 18.
    Zelman, D. C., Smith, M. Y., Hoffman, D., Edwards, L., Reed, P., Levine, E., et al. (2004). Acceptable, manageable, and tolerable days: Patient daily goals for medication management of persistent pain. Journal of Pain and Symptom Management, 28(5), 474–487.CrossRefPubMedGoogle Scholar
  19. 19.
    Zelman, D. C., Hoffman, D. L., Seifeldin, R., & Dukes, E. M. (2003). Development of a metric for a day of manageable pain control: Derivation of pain severity cut-points for low back pain and osteoarthritis. Pain, 106(1–2), 35–42.CrossRefPubMedGoogle Scholar
  20. 20.
    Given, B., Given, C. W., Sikorskii, A., Jeon, S., McCorkle, R., Champion, V., et al. (2008). Establishing mild, moderate, and severe scores for cancer-related symptoms: How consistent and clinically meaningful are interference-based severity cut-points? Journal of Pain and Symptom Management, 35(2), 126–135.CrossRefPubMedCentralPubMedGoogle Scholar
  21. 21.
    Hays, R. D., Reise, S., & Calderon, J. L. (2012). How much is lost in using single items? Journal of General Internal Medicine, 27(11), 1402–1403.CrossRefPubMedCentralPubMedGoogle Scholar
  22. 22.
    Buckendahl, C., Smith, R., Impara, J., & Plake, B. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(4), 253–263.CrossRefGoogle Scholar
  23. 23.
    Huynh, H. (2010). Psychometric aspects of item mapping for criterion-referenced interpretation and bookmark standard setting. Journal of Applied Measurement, 11(1), 91–98.PubMedGoogle Scholar
  24. 24.
    Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996). Standard setting: A bookmark approach. In D. R. Green (Chair), IRT-based standard-setting procedures utilizing behavioral anchoring (symposium), Council of Chief State School Officers National Conference on Large-scale Assessment, Phoenix, AZ.Google Scholar
  25. 25.
    Cizek, G. J. (1993). Reconsidering standards and criteria. Journal of Educational Measurement, 30(2), 93–106.CrossRefGoogle Scholar
  26. 26.
    Perie, M. (2005). The Angoff and bookmark methods: Presentation at the annual meeting of the National Council on Measurement in Education. Montreal: National Council on Measurement in Education.Google Scholar
  27. 27.
    Zieky, M. J., Perie, M., & Livingston, S. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.Google Scholar
  28. 28.
    Messick, S. (1998). Test validity: A matter of consequence. Social Indicators Research, 45(1–3), 35–44.CrossRefGoogle Scholar
  29. 29.
    Amato, M. P., Ponziani, G., Rossi, F., Liedl, C. L., Stefanile, C., & Rossi, L. (2001). Quality of life in multiple sclerosis: The impact of depression, fatigue and disability. Multiple Sclerosis, 7(5), 340–344.CrossRefPubMedGoogle Scholar
  30. 30.
    Coleman, C. I., Sidovar, M. F., Roberts, M. S., & Kohn, C. (2013). Impact of mobility impairment on indirect costs and health-related quality of life in multiple sclerosis. PLoS ONE, 8(1), e54756.CrossRefPubMedCentralPubMedGoogle Scholar
  31. 31.
    Mills, R. J., & Young, C. A. (2011). The relationship between fatigue and other clinical features of multiple sclerosis. Multiple Sclerosis, 17(5), 604–612.CrossRefPubMedGoogle Scholar
  32. 32.
    Yozbatiran, N., Baskurt, F., Baskurt, Z., Ozakbas, S., & Idiman, E. (2006). Motor assessment of upper extremity function and its relation with fatigue, cognitive function and quality of life in multiple sclerosis patients. Journal of the Neurological Sciences, 246(1–2), 117–122.CrossRefPubMedGoogle Scholar
  33. 33.
    Cella, D., Choi, S., Garci, A. S., Cook, K. F., Rosenbloom, S., Lai, J.-S., et al. (2014). Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment. Quality of Life Research. doi: 10.1007/s11136-014-0732-6.
  34. 34.
    Cella, D., Choi, S., Rosenbloom, S., Surges, Tatum, D., Garcia, S., et al. (2008). A Novel IRT-based case-ranking approach to derive expert standards for symptom severity (paper presentation), International Society for Quality of Life Research Annual Scientific Meeting. Montevideo, Uruguay.Google Scholar
  35. 35.
    (NINDS), N. I. o. N. D. a. S. (2010). User manual for the quality of life in neurological disorders (Neuro-QOL) Measures. Version 1.0.Google Scholar
  36. 36.
    Team, R. D. C. (2008). A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. ISBN 3-900051-07-0.
  37. 37.
    Choi, S. W., Schalet, B. D., Cook, K. F., & Cella, D. (2014). Establishing a common metric for depressive symptoms: Linking BDI-II, CES-D, and PHQ-9 to PROMIS depression. Psychological Assessment, 26(2), 513–527.CrossRefPubMedGoogle Scholar
  38. 38.
    Schalet, B. D., Cook, K. F., Choi, S. W., & Cella, D. (2014). Establishing a common metric for anxiety symptoms: Linking GAD-7, MASQ, and PANAS to PROMIS anxiety. Journal of Anxiety Disorders, 28(1), 88–96.CrossRefPubMedCentralPubMedGoogle Scholar
  39. 39.
    Wikipedia. List of most common surnames in North America. Retrieved 02/04/2014, from
  40. 40.
    Shepard, L. A. (1979). Setting standards. In M. A. Bunda & J. R. Sanders (Eds.), Practices and problems in competency-based measurement (pp. 72–88). Washington, DC: National Council on Measurement in Education.Google Scholar
  41. 41.
    Lampic, C., & Sjoden, P. O. (2000). Patient and staff perceptions of cancer patients’ psychological concerns and needs. Acta Oncologica, 39(1), 9–22.CrossRefPubMedGoogle Scholar
  42. 42.
    Radowsky, J. S., Baines, L., Howard, R. S., Shriver, C. D., Buckenmaier, C. C., I. I. I., & Stojadinovic, A. (2012). Pain ratings by patients and their providers of radionucleotide injection for breast cancer lymphatic mapping. Pain Medicine, 13(5), 670–676.CrossRefPubMedGoogle Scholar
  43. 43.
    Basch, E., Iasonos, A., McDonough, T., Barz, A., Culkin, A., Kris, M. G., et al. (2006). Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: Results of a questionnaire-based study. The lancet Oncology, 7(11), 903–909.CrossRefPubMedGoogle Scholar
  44. 44.
    McColl, E., Junghard, O., Wiklund, I., & Revicki, D. A. (2005). Assessing symptoms in gastroesophageal reflux disease: How well do clinicians’ assessments agree with those of their patients? American Journal of Gastroenterology, 100(1), 11–18.CrossRefPubMedGoogle Scholar
  45. 45.
    Lazarus, R. S., & Folkman, S. (1987). Transactional theory and research on emotions and coping. European Journal of Personality, 1, 141–169.CrossRefGoogle Scholar
  46. 46.
    Salander, P., Bergenheim, T., & Henriksson, R. (1996). The creation of protection and hope in patients with malignant brain tumours. Social Science and Medicine, 42(7), 985–996.CrossRefPubMedGoogle Scholar
  47. 47.
    Burgess, C., Morris, T., & Pettingale, K. W. (1988). Psychological response to cancer diagnosis–II. Evidence for coping styles (coping styles and cancer diagnosis). Journal of Psychosomatic Research, 32(3), 263–272.CrossRefPubMedGoogle Scholar
  48. 48.
    Carver, C. S., Pozo, C., Harris, S. D., Noriega, V., Scheier, M. F., Robinson, D. S., et al. (1993). How coping mediates the effect of optimism on distress: A study of women with early stage breast cancer. Journal of Personality and Social Psychology, 65(2), 375–390.CrossRefPubMedGoogle Scholar
  49. 49.
    Dembo, T., Leviton, G. L., & Wright, B. A. (1956). Adjustment to misfortune; a problem of social-psychological rehabilitation. Artificial Limbs, 3(2), 4–62.PubMedGoogle Scholar
  50. 50.
    Cizek, G. J. (2012). An introduction to contemporary standard setting. In G. J. Cizek (Ed.), Setting performance standards (2nd ed., pp. 3–14). New York, NY: Routledge.Google Scholar
  51. 51.
    Calvert, M., Draper, H., & Kyte, D. (2014). Patient-reported outcome alert monitoring—reply. JAMA, 311(4), 420–421.CrossRefPubMedGoogle Scholar
  52. 52.
    Thombs, B. D., Ells, C., & Black, L. (2014). Patient-reported outcome alert monitoring. JAMA, 311(4), 420.CrossRefPubMedGoogle Scholar
  53. 53.
    Deshpande, P. R., Rajan, S., Sudeepthi, B. L., & Abdul Nazir, C. P. (2011). Patient-reported outcomes: A new era in clinical research. Perspectives in Clinical Research, 2(4), 137–144.CrossRefPubMedCentralPubMedGoogle Scholar
  54. 54.
    Brettschneider, C., Luhmann, D., & Raspe, H. (2011). Informative value of patient reported outcomes (PRO) in health technology assessment (HTA). GMS Health Technology Assessment, 7, Doc01.Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Karon F. Cook
    • 1
    Email author
  • David E. Victorson
    • 1
  • David Cella
    • 1
  • Benjamin D. Schalet
    • 1
  • Deborah Miller
    • 2
  1. 1.Department of Medical Social SciencesNorthwestern University Feinberg School of MedicineChicagoUSA
  2. 2.Mellen CenterCleveland Clinic FoundationClevelandUSA

Personalised recommendations