Quality of Life Research

, Volume 22, Issue 8, pp 1889–1905 | Cite as

ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research

  • Bryce B. ReeveEmail author
  • Kathleen W. Wyrwich
  • Albert W. Wu
  • Galina Velikova
  • Caroline B. Terwee
  • Claire F. Snyder
  • Carolyn Schwartz
  • Dennis A. Revicki
  • Carol M. Moinpour
  • Lori D. McLeod
  • Jessica C. Lyons
  • William R. Lenderking
  • Pamela S. Hinds
  • Ron D. Hays
  • Joanne Greenhalgh
  • Richard Gershon
  • David Feeny
  • Peter M. Fayers
  • David Cella
  • Michael Brundage
  • Sara Ahmed
  • Neil K. Aaronson
  • Zeeshan Butt



An essential aspect of patient-centered outcomes research (PCOR) and comparative effectiveness research (CER) is the integration of patient perspectives and experiences with clinical data to evaluate interventions. Thus, PCOR and CER require capturing patient-reported outcome (PRO) data appropriately to inform research, healthcare delivery, and policy. This initiative’s goal was to identify minimum standards for the design and selection of a PRO measure for use in PCOR and CER.


We performed a literature review to find existing guidelines for the selection of PRO measures. We also conducted an online survey of the International Society for Quality of Life Research (ISOQOL) membership to solicit input on PRO standards. A standard was designated as “recommended” when >50 % respondents endorsed it as “required as a minimum standard.”


The literature review identified 387 articles. Survey response rate was 120 of 506 ISOQOL members. The respondents had an average of 15 years experience in PRO research, and 89 % felt competent or very competent providing feedback. Final recommendations for PRO measure standards included: documentation of the conceptual and measurement model; evidence for reliability, validity (content validity, construct validity, responsiveness); interpretability of scores; quality translation, and acceptable patient and investigator burden.


The development of these minimum measurement standards is intended to promote the appropriate use of PRO measures to inform PCOR and CER, which in turn can improve the effectiveness and efficiency of healthcare delivery. A next step is to expand these minimum standards to identify best practices for selecting decision-relevant PRO measures.


Patient-reported outcomes Comparative effectiveness Patient-centered outcomes research Psychometrics Questionnaire 



This study was funded by the Patient-Centered Outcomes Research Institute (PCORI-SOL-RMWG-001: PIs: Zeeshan Butt, PhD, Northwestern University; Bryce Reeve, PhD, University of North Carolina at Chapel Hill). The views expressed in this article are those of the authors and do not necessarily reflect those of PCORI.

Supplementary material

11136_2012_344_MOESM1_ESM.pdf (13 kb)
Supplementary material 1 (PDF 12 kb)


  1. 1.
    US Food and Drug Administration. (2009). Patient-reported outcome measures: Use in medical product development to support labeling claims. Guidance for industry. Accessed November 26, 2011.
  2. 2.
    Scientific Advisory Committee of the Medical Outcomes Trust. (2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11(3), 193–205.CrossRefGoogle Scholar
  3. 3.
    Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2006). Protocol of the COSMIN study: COnsensus-based standards for the selection of health measurement INstruments. BMC Medical Research Methodology, 6, 2. doi: 10.1186/1471-2288-6-2.PubMedCrossRefGoogle Scholar
  4. 4.
    Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology, 63(7), 737–745. doi: 10.1016/j.jclinepi.2010.02.006.PubMedCrossRefGoogle Scholar
  5. 5.
    Snyder, C. F., Aaronson, N. K., Choucair, A. K., Elliott, T. E., Greenhalgh, J., Halyard, M. Y., et al. (2011). Implementing patient-reported outcomes assessment in clinical practice: A review of the options and considerations. Quality of Life Research,. doi: 10.1007/s11136-011-0054-x.Google Scholar
  6. 6.
    Basch, E. M., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L., Sit, L., et al. (2011). Electronic toxicity monitoring and patient-reported outcomes. Cancer Journal, 17(4), 231–234. doi: 10.1097/PPO.0b013e31822c28b3.CrossRefGoogle Scholar
  7. 7.
    Revicki, D. A., Gnanasakthy, A., & Weinfurt, K. (2007). Documenting the rationale and psychometric characteristics of patient reported outcomes for labeling and promotional claims: The PRO Evidence Dossier. Quality of Life Research, 16(4), 717–723. doi: 10.1007/s11136-006-9153-5.PubMedCrossRefGoogle Scholar
  8. 8.
    Schunemann, H. J., Akl, E. A., & Guyatt, G. H. (2006). Interpreting the results of patient reported outcome measures in clinical trials: The clinician’s perspective. Health and Quality of Life Outcomes, 4, 62. doi: 10.1186/1477-7525-4-62.PubMedCrossRefGoogle Scholar
  9. 9.
    Deyo, R. A., & Patrick, D. L. (1989). Barriers to the use of health status measures in clinical investigation, patient care, and policy research. Medical Care, 27(3 Suppl), S254–S268.PubMedCrossRefGoogle Scholar
  10. 10.
    International Society for Quality of Life Research. Accessed July 30, 2012.
  11. 11.
    Guyatt, G., & Schunemann, H. (2007). How can quality of life researchers make their work more useful to health workers and their patients? Quality of Life Research, 16(7), 1097–1105. doi: 10.1007/s11136-007-9223-3.PubMedCrossRefGoogle Scholar
  12. 12.
    Revicki, D. A., Osoba, D., Fairclough, D., Barofsky, I., Berzon, R., Leidy, N. K., et al. (2000). Recommendations on health-related quality of life research to support labeling and promotional claims in the United States. Quality of Life Research, 9(8), 887–900.PubMedCrossRefGoogle Scholar
  13. 13.
    Lipscomb, J., Donaldson, M. S., Arora, N. K., Brown, M. L., Clauser, S. B., Potosky, A. L., et al. (2004). Cancer outcomes research. Journal of the National Cancer Institute Monographs (33), 178–197. doi: 10.1093/jncimonographs/lgh039.
  14. 14.
    US Patient-Centered Outcomes Research Institute Accessed 26 November, 2011.
  15. 15.
    Methodology Committee of the Patient-Centered Outcomes Research, I. (2012). Methodological standards and patient-centeredness in comparative effectiveness research: The PCORI perspective. Journal of the American Medical Association, 307(15), 1636–1640. doi: 10.1001/jama.2012.466.CrossRefGoogle Scholar
  16. 16.
    Terwee, C. B., Jansma, E. P., Riphagen, I. I., & de Vet, H. C. (2009). Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Quality of Life Research, 18(8), 1115–1123. doi: 10.1007/s11136-009-9528-5.PubMedCrossRefGoogle Scholar
  17. 17.
    Qualtrics Labs Inc. Why choose qualtrics survey software? Accessed November 26, 2011.
  18. 18.
    US Food and Drug Administration. (2010). Qualification process for drug development tools. Draft Guidance for Industry. Accessed November 26, 2011.
  19. 19.
    Erickson, P., Willke, R., & Burke, L. (2009). A concept taxonomy and an instrument hierarchy: Tools for establishing and evaluating the conceptual framework of a patient-reported outcome (PRO) instrument as applied to product labeling claims. Value in Health, 12(8), 1158–1167. doi: 10.1111/j.1524-4733.2009.00609.x.PubMedCrossRefGoogle Scholar
  20. 20.
    Patrick, D. L., Burke, L. B., Powers, J. H., Scott, J. A., Rock, E. P., Dawisha, S., et al. (2007). Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value in Health, 10(Suppl 2), S125–S137. doi: 10.1111/j.1524-4733.2007.00275.x.PubMedCrossRefGoogle Scholar
  21. 21.
    Angst, F. (2011). The new COSMIN guidelines confront traditional concepts of responsiveness. BMC Medical Research Methodology, 11, 152; author reply 152. doi: 10.1186/1471-2288-11-152.
  22. 22.
    Mokkink, L. B., Terwee, C. B., Gibbons, E., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2010). Inter-rater agreement and reliability of the COSMIN (COnsensus-based standards for the selection of health status measurement instruments) checklist. BMC Medical Research Methodology, 10, 82. doi: 10.1186/1471-2288-10-82.PubMedCrossRefGoogle Scholar
  23. 23.
    Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2010). The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Medical Research Methodology, 10, 22. doi: 10.1186/1471-2288-10-22.PubMedCrossRefGoogle Scholar
  24. 24.
    Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al. (2010). The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: An international Delphi study. Quality of Life Research, 19(4), 539–549. doi: 10.1007/s11136-010-9606-8.PubMedCrossRefGoogle Scholar
  25. 25.
    Terwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W., Bouter, L. M., & de Vet, H. C. (2012). Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Quality of Life Research, 21(4), 651–657. doi: 10.1007/s11136-011-9960-1.PubMedCrossRefGoogle Scholar
  26. 26.
    Johnson, C., Aaronson, N., Blazeby, J. M., Bottomley, A., Fayers, P., Koller, M., et al. (2011). EORTC Quality of life group: Guidelines for developing questionnaire modules. Accessed November 26, 2011.
  27. 27.
    Cella, D. (1997). Manual of the functional assessment of chronic illness therapy (FACIT) measurement system. Evanston, IL: Northwestern University.Google Scholar
  28. 28.
    Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429. doi: 10.1111/j.1524-4733.2008.00470.x.PubMedCrossRefGoogle Scholar
  29. 29.
    Wild, D., Grove, A., Martin, M., Eremenco, S., McElroy, S., Verjee-Lorenz, A., et al. (2005). Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR task force for translation and cultural adaptation. Value in Health, 8(2), 94–104. doi: 10.1111/j.1524-4733.2005.04054.x.PubMedCrossRefGoogle Scholar
  30. 30.
    Rothman, M., Burke, L., Erickson, P., Leidy, N. K., Patrick, D. L., & Petrie, C. D. (2009). Use of existing patient-reported outcome (PRO) instruments and their modification: The ISPOR good research practices for evaluating and documenting content validity for the use of existing instruments and their modification PRO task force report. Value in Health, 12(8), 1075–1083. doi: 10.1111/j.1524-4733.2009.00603.x.PubMedCrossRefGoogle Scholar
  31. 31.
    Wild, D., Eremenco, S., Mear, I., Martin, M., Houchin, C., Gawlicki, M., et al. (2009). Multinational trials-recommendations on the translations required, approaches to using the same language in different countries, and the approaches to support pooling the data: the ISPOR patient-reported outcomes translation and linguistic validation good research practices task force report. Value in Health, 12(4), 430–440. doi: 10.1111/j.1524-4733.2008.00471.x.PubMedCrossRefGoogle Scholar
  32. 32.
    Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1998). Standards for educational and psychological testing Washington DC: American Psychological Association.Google Scholar
  33. 33.
    Magasi, S., Ryan, G., Revicki, D., Lenderking, W., Hays, R. D., Brod, M., et al. (2011). Content validity of patient-reported outcome measures: Perspectives from a PROMIS meeting. Quality of Life Research,. doi: 10.1007/s11136-011-9990-8.PubMedGoogle Scholar
  34. 34.
    Revicki, D., Hays, R. D., Cella, D., & Sloan, J. (2008). Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61(2), 102–109. doi: 10.1016/j.jclinepi.2007.03.012.PubMedCrossRefGoogle Scholar
  35. 35.
    Valderas, J. M., Ferrer, M., Mendivil, J., Garin, O., Rajmil, L., Herdman, M., et al. (2008). Development of EMPRO: A tool for the standardized assessment of patient-reported outcome measures. Value in Health, 11(4), 700–708. doi: 10.1111/j.1524-4733.2007.00309.x.PubMedCrossRefGoogle Scholar
  36. 36.
    Dewolf, L., Koller, M., Velikova, G., Johnson, C., Scott, N., & Bottomley, A. (2009). EORTC quality of life group: Translation procedure. Accessed November 26, 2011.
  37. 37.
    Hays, R. D., & Hadorn, D. (1992). Responsiveness to change: An aspect of validity, not a separate dimension. Quality of Life Research, 1(1), 73–75.PubMedCrossRefGoogle Scholar
  38. 38.
    Wyrwich, K. W., Norquist, J. M., Lenderking, W. R., Acaster, S., & the Industry Advisory Committee of International Society for Quality of Life, R. (2012). Methods for interpreting change over time in patient-reported outcome measures. Quality of Life Research, 2012 Apr 17. [Epub ahead of print.]. doi: 10.1007/s11136-012-0175-x.
  39. 39.
    Ahmed, S., Berzon, R. A., Revicki, D., Lenderking, W., Moinpour, C. M., Basch, E., et al. (2012). The use of patient-reported outcomes (PRO) within comparative effectivenss research: Implications for clinical practice and healthcare policy. Medical Care, 50(12), 1060–1070.PubMedCrossRefGoogle Scholar
  40. 40.
    Terwee, C. B., Bot, S. D., de Boer, M. R., van der Windt, D. A., Knol, D. L., Dekker, J., et al. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60(1), 34–42. doi: 10.1016/j.jclinepi.2006.03.012.PubMedCrossRefGoogle Scholar
  41. 41.
    Nunnally, J. C., & Berstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Google Scholar
  42. 42.
    Norquist, J. M., Girman, C., Fehnel, S., Demuro-Mercon, C., & Santanello, N. (2011). Choice of recall period for patient-reported outcome (PRO) measures: Criteria for consideration. Quality of Life Research,. doi: 10.1007/s11136-011-0003-8.PubMedGoogle Scholar
  43. 43.
    Revicki, D. A., Cella, D., Hays, R. D., Sloan, J. A., Lenderking, W. R., & Aaronson, N. K. (2006). Responsiveness and minimal important differences for patient reported outcomes. Health and Quality of Life Outcomes, 4, 70. doi: 10.1186/1477-7525-4-70.PubMedCrossRefGoogle Scholar
  44. 44.
    Brozek, J. L., Guyatt, G. H., & Schunemann, H. J. (2006). How a well-grounded minimal important difference can enhance transparency of labelling claims and improve interpretation of a patient reported outcome measure. Health and Quality of Life Outcomes, 4, 69. doi: 10.1186/1477-7525-4-69.PubMedCrossRefGoogle Scholar
  45. 45.
    Norman, G. R., Sridhar, F. G., Guyatt, G. H., & Walter, S. D. (2001). Relation of distribution- and anchor-based approaches in interpretation of changes in health-related quality of life. Medical Care, 39(10), 1039–1047.PubMedCrossRefGoogle Scholar
  46. 46.
    Koller, M., Kantzer, V., Mear, I., Zarzar, K., Martin, M., Greimel, E., et al. (2012). The process of reconciliation: Evaluation of guidelines for translating quality-of-life questionnaires. Expert Review of Pharmacoeconomics & Outcomes Research, 12(2), 189–197. doi: 10.1586/erp.11.102.CrossRefGoogle Scholar
  47. 47.
    Jordan, J. E., Osborne, R. H., & Buchbinder, R. (2011). Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses. Journal of Clinical Epidemiology, 64(4), 366–379. doi: 10.1016/j.jclinepi.2010.04.005.PubMedCrossRefGoogle Scholar
  48. 48.
    Acquadro, C., Conway, K., Hareendran, A., & Aaronson, N. (2008). Literature review of methods to translate health-related quality of life questionnaires for use in multinational clinical trials. Value in Health, 11(3), 509–521. doi: 10.1111/j.1524-4733.2007.00292.x.PubMedCrossRefGoogle Scholar
  49. 49.
    Crosby, R. D., Kolotkin, R. L., & Williams, G. R. (2003). Defining clinically meaningful change in health-related quality of life. Journal of Clinical Epidemiology, 56(5), 395–407.PubMedCrossRefGoogle Scholar
  50. 50.
    Frost, M. H., Reeve, B. B., Liepa, A. M., Stauffer, J. W., Hays, R. D., & Mayo, F. D. A. P.-R. O. C. M. G. (2007). What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value in Health, 10(Suppl 2), S94–S105. doi: 10.1111/j.1524-4733.2007.00272.x.PubMedCrossRefGoogle Scholar
  51. 51.
    Hays, R. D., & Woolley, J. M. (2000). The concept of clinically meaningful difference in health-related quality-of-life research. How meaningful is it? PharmacoEconomics, 18(5), 419–423.PubMedCrossRefGoogle Scholar
  52. 52.
    Kemmler, G., Zabernigg, A., Gattringer, K., Rumpold, G., Giesinger, J., Sperner-Unterweger, B., et al. (2010). A new approach to combining clinical relevance and statistical significance for evaluation of quality of life changes in the individual patient. Journal of Clinical Epidemiology, 63(2), 171–179. doi: 10.1016/j.jclinepi.2009.03.016.PubMedCrossRefGoogle Scholar
  53. 53.
    Kottner, J., Audige, L., Brorson, S., Donner, A., Gajewski, B. J., Hrobjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64(1), 96–106. doi: 10.1016/j.jclinepi.2010.03.002.PubMedCrossRefGoogle Scholar
  54. 54.
    Schmidt, S., & Bullinger, M. (2003). Current issues in cross-cultural quality of life instrument development. Archives of Physical Medicine and Rehabilitation, 84(4 Suppl 2), S29–S34. doi: 10.1053/apmr.2003.50244.PubMedCrossRefGoogle Scholar
  55. 55.
    Sprangers, M. A., Moinpour, C. M., Moynihan, T. J., Patrick, D. L., Revicki, D. A., & Clinical Significance Consensus Meeting Group. (2002). Assessing meaningful change in quality of life over time: A users’ guide for clinicians. Mayo Clinic Proceedings, 77(6), 561–571. doi: 10.4065/77.6.561.PubMedCrossRefGoogle Scholar
  56. 56.
    Turner, R. R., Quittner, A. L., Parasuraman, B. M., Kallich, J. D., Cleeland, C. S., & Mayo, F. D. A. P.-R. O. C. M. G. (2007). Patient-reported outcomes: Instrument development and selection issues. Value in Health, 10(Suppl 2), S86–S93. doi: 10.1111/j.1524-4733.2007.00271.x.PubMedCrossRefGoogle Scholar
  57. 57.
    Schunemann, H. J., & Guyatt, G. H. (2005). Commentary–goodbye M(C)ID! Hello MID, where do you come from? Health Services Research, 40(2), 593–597. doi: 10.1111/j.1475-6773.2005.00374.x.PubMedCrossRefGoogle Scholar
  58. 58.
    Schunemann, H. J., Puhan, M., Goldstein, R., Jaeschke, R., & Guyatt, G. H. (2005). Measurement properties and interpretability of the Chronic respiratory disease questionnaire (CRQ). Copd, 2(1), 81–89.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Bryce B. Reeve
    • 1
    • 20
    Email author
  • Kathleen W. Wyrwich
    • 2
  • Albert W. Wu
    • 3
  • Galina Velikova
    • 4
  • Caroline B. Terwee
    • 5
  • Claire F. Snyder
    • 3
  • Carolyn Schwartz
    • 6
  • Dennis A. Revicki
    • 2
  • Carol M. Moinpour
    • 7
  • Lori D. McLeod
    • 8
  • Jessica C. Lyons
    • 20
  • William R. Lenderking
    • 2
  • Pamela S. Hinds
    • 9
    • 10
  • Ron D. Hays
    • 11
  • Joanne Greenhalgh
    • 4
  • Richard Gershon
    • 12
  • David Feeny
    • 13
  • Peter M. Fayers
    • 14
    • 15
  • David Cella
    • 12
  • Michael Brundage
    • 16
  • Sara Ahmed
    • 17
  • Neil K. Aaronson
    • 18
    • 19
  • Zeeshan Butt
    • 12
  1. 1.Department of Health Policy and Management, Gillings School of Global Public HealthUniversity of North Carolina at Chapel HillChapel HillUSA
  2. 2.United BioSource CorporationBethesdaUSA
  3. 3.Johns Hopkins School of MedicineBaltimoreUSA
  4. 4.University of LeedsLeedsUK
  5. 5.VU University Medical CenterAmsterdamThe Netherlands
  6. 6.DeltaQuest Foundation, Inc.ConcordUSA
  7. 7.Fred Hutchinson Cancer Research CenterSeattleUSA
  8. 8.Research Triangle Institute Health SolutionsDurhamUSA
  9. 9.Children’s National Medical CenterWashingtonUSA
  10. 10.The George Washington University School of MedicineWashingtonUSA
  11. 11.David Geffen School of Medicine at UCLALos AngelesUSA
  12. 12.Northwestern University Feinberg School of MedicineChicagoUSA
  13. 13.University of AlbertaAlbertaCanada
  14. 14.University of AberdeenAberdeenUK
  15. 15.Norwegian University of Science and Technology (NTNU)TrondheimNorway
  16. 16.Queen’s UniversityKingstonCanada
  17. 17.McGill UniversityMontrealCanada
  18. 18.The Netherlands Cancer InstituteAmsterdamThe Netherlands
  19. 19.University of AmsterdamAmsterdamThe Netherlands
  20. 20.Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel HillChapel HillUSA

Personalised recommendations