ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research

Abstract

Purpose

An essential aspect of patient-centered outcomes research (PCOR) and comparative effectiveness research (CER) is the integration of patient perspectives and experiences with clinical data to evaluate interventions. Thus, PCOR and CER require capturing patient-reported outcome (PRO) data appropriately to inform research, healthcare delivery, and policy. This initiative’s goal was to identify minimum standards for the design and selection of a PRO measure for use in PCOR and CER.

Methods

We performed a literature review to find existing guidelines for the selection of PRO measures. We also conducted an online survey of the International Society for Quality of Life Research (ISOQOL) membership to solicit input on PRO standards. A standard was designated as “recommended” when >50 % respondents endorsed it as “required as a minimum standard.”

Results

The literature review identified 387 articles. Survey response rate was 120 of 506 ISOQOL members. The respondents had an average of 15 years experience in PRO research, and 89 % felt competent or very competent providing feedback. Final recommendations for PRO measure standards included: documentation of the conceptual and measurement model; evidence for reliability, validity (content validity, construct validity, responsiveness); interpretability of scores; quality translation, and acceptable patient and investigator burden.

Conclusion

The development of these minimum measurement standards is intended to promote the appropriate use of PRO measures to inform PCOR and CER, which in turn can improve the effectiveness and efficiency of healthcare delivery. A next step is to expand these minimum standards to identify best practices for selecting decision-relevant PRO measures.

This is a preview of subscription content, log in to check access.

References

  1. 1.

    US Food and Drug Administration. (2009). Patient-reported outcome measures: Use in medical product development to support labeling claims. Guidance for industry. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM071975.pdf. Accessed November 26, 2011.

  2. 2.

    Scientific Advisory Committee of the Medical Outcomes Trust. (2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11(3), 193–205.

    Article  Google Scholar 

  3. 3.

    Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2006). Protocol of the COSMIN study: COnsensus-based standards for the selection of health measurement INstruments. BMC Medical Research Methodology, 6, 2. doi:10.1186/1471-2288-6-2.

    PubMed  Article  CAS  Google Scholar 

  4. 4.

    Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology, 63(7), 737–745. doi:10.1016/j.jclinepi.2010.02.006.

    PubMed  Article  Google Scholar 

  5. 5.

    Snyder, C. F., Aaronson, N. K., Choucair, A. K., Elliott, T. E., Greenhalgh, J., Halyard, M. Y., et al. (2011). Implementing patient-reported outcomes assessment in clinical practice: A review of the options and considerations. Quality of Life Research,. doi:10.1007/s11136-011-0054-x.

    Google Scholar 

  6. 6.

    Basch, E. M., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L., Sit, L., et al. (2011). Electronic toxicity monitoring and patient-reported outcomes. Cancer Journal, 17(4), 231–234. doi:10.1097/PPO.0b013e31822c28b3.

    Article  Google Scholar 

  7. 7.

    Revicki, D. A., Gnanasakthy, A., & Weinfurt, K. (2007). Documenting the rationale and psychometric characteristics of patient reported outcomes for labeling and promotional claims: The PRO Evidence Dossier. Quality of Life Research, 16(4), 717–723. doi:10.1007/s11136-006-9153-5.

    PubMed  Article  Google Scholar 

  8. 8.

    Schunemann, H. J., Akl, E. A., & Guyatt, G. H. (2006). Interpreting the results of patient reported outcome measures in clinical trials: The clinician’s perspective. Health and Quality of Life Outcomes, 4, 62. doi:10.1186/1477-7525-4-62.

    PubMed  Article  Google Scholar 

  9. 9.

    Deyo, R. A., & Patrick, D. L. (1989). Barriers to the use of health status measures in clinical investigation, patient care, and policy research. Medical Care, 27(3 Suppl), S254–S268.

    PubMed  Article  CAS  Google Scholar 

  10. 10.

    International Society for Quality of Life Research. http://www.isoqol.org/. Accessed July 30, 2012.

  11. 11.

    Guyatt, G., & Schunemann, H. (2007). How can quality of life researchers make their work more useful to health workers and their patients? Quality of Life Research, 16(7), 1097–1105. doi:10.1007/s11136-007-9223-3.

    PubMed  Article  Google Scholar 

  12. 12.

    Revicki, D. A., Osoba, D., Fairclough, D., Barofsky, I., Berzon, R., Leidy, N. K., et al. (2000). Recommendations on health-related quality of life research to support labeling and promotional claims in the United States. Quality of Life Research, 9(8), 887–900.

    PubMed  Article  CAS  Google Scholar 

  13. 13.

    Lipscomb, J., Donaldson, M. S., Arora, N. K., Brown, M. L., Clauser, S. B., Potosky, A. L., et al. (2004). Cancer outcomes research. Journal of the National Cancer Institute Monographs (33), 178–197. doi:10.1093/jncimonographs/lgh039.

  14. 14.

    US Patient-Centered Outcomes Research Institute http://www.pcori.org. Accessed 26 November, 2011.

  15. 15.

    Methodology Committee of the Patient-Centered Outcomes Research, I. (2012). Methodological standards and patient-centeredness in comparative effectiveness research: The PCORI perspective. Journal of the American Medical Association, 307(15), 1636–1640. doi:10.1001/jama.2012.466.

    Article  Google Scholar 

  16. 16.

    Terwee, C. B., Jansma, E. P., Riphagen, I. I., & de Vet, H. C. (2009). Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Quality of Life Research, 18(8), 1115–1123. doi:10.1007/s11136-009-9528-5.

    PubMed  Article  Google Scholar 

  17. 17.

    Qualtrics Labs Inc. Why choose qualtrics survey software? https://www.qualtrics.com/why-survey-software. Accessed November 26, 2011.

  18. 18.

    US Food and Drug Administration. (2010). Qualification process for drug development tools. Draft Guidance for Industry. http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM230597.pdf. Accessed November 26, 2011.

  19. 19.

    Erickson, P., Willke, R., & Burke, L. (2009). A concept taxonomy and an instrument hierarchy: Tools for establishing and evaluating the conceptual framework of a patient-reported outcome (PRO) instrument as applied to product labeling claims. Value in Health, 12(8), 1158–1167. doi:10.1111/j.1524-4733.2009.00609.x.

    PubMed  Article  Google Scholar 

  20. 20.

    Patrick, D. L., Burke, L. B., Powers, J. H., Scott, J. A., Rock, E. P., Dawisha, S., et al. (2007). Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value in Health, 10(Suppl 2), S125–S137. doi:10.1111/j.1524-4733.2007.00275.x.

    PubMed  Article  Google Scholar 

  21. 21.

    Angst, F. (2011). The new COSMIN guidelines confront traditional concepts of responsiveness. BMC Medical Research Methodology, 11, 152; author reply 152. doi:10.1186/1471-2288-11-152.

  22. 22.

    Mokkink, L. B., Terwee, C. B., Gibbons, E., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2010). Inter-rater agreement and reliability of the COSMIN (COnsensus-based standards for the selection of health status measurement instruments) checklist. BMC Medical Research Methodology, 10, 82. doi:10.1186/1471-2288-10-82.

    PubMed  Article  Google Scholar 

  23. 23.

    Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2010). The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Medical Research Methodology, 10, 22. doi:10.1186/1471-2288-10-22.

    PubMed  Article  Google Scholar 

  24. 24.

    Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al. (2010). The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: An international Delphi study. Quality of Life Research, 19(4), 539–549. doi:10.1007/s11136-010-9606-8.

    PubMed  Article  Google Scholar 

  25. 25.

    Terwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W., Bouter, L. M., & de Vet, H. C. (2012). Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Quality of Life Research, 21(4), 651–657. doi:10.1007/s11136-011-9960-1.

    PubMed  Article  Google Scholar 

  26. 26.

    Johnson, C., Aaronson, N., Blazeby, J. M., Bottomley, A., Fayers, P., Koller, M., et al. (2011). EORTC Quality of life group: Guidelines for developing questionnaire modules. http://groups.eortc.be/qol/sites/default/files/archives/guidelines_for_developing_questionnaire-_final.pdf. Accessed November 26, 2011.

  27. 27.

    Cella, D. (1997). Manual of the functional assessment of chronic illness therapy (FACIT) measurement system. Evanston, IL: Northwestern University.

    Google Scholar 

  28. 28.

    Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429. doi:10.1111/j.1524-4733.2008.00470.x.

    PubMed  Article  Google Scholar 

  29. 29.

    Wild, D., Grove, A., Martin, M., Eremenco, S., McElroy, S., Verjee-Lorenz, A., et al. (2005). Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR task force for translation and cultural adaptation. Value in Health, 8(2), 94–104. doi:10.1111/j.1524-4733.2005.04054.x.

    PubMed  Article  Google Scholar 

  30. 30.

    Rothman, M., Burke, L., Erickson, P., Leidy, N. K., Patrick, D. L., & Petrie, C. D. (2009). Use of existing patient-reported outcome (PRO) instruments and their modification: The ISPOR good research practices for evaluating and documenting content validity for the use of existing instruments and their modification PRO task force report. Value in Health, 12(8), 1075–1083. doi:10.1111/j.1524-4733.2009.00603.x.

    PubMed  Article  Google Scholar 

  31. 31.

    Wild, D., Eremenco, S., Mear, I., Martin, M., Houchin, C., Gawlicki, M., et al. (2009). Multinational trials-recommendations on the translations required, approaches to using the same language in different countries, and the approaches to support pooling the data: the ISPOR patient-reported outcomes translation and linguistic validation good research practices task force report. Value in Health, 12(4), 430–440. doi:10.1111/j.1524-4733.2008.00471.x.

    PubMed  Article  Google Scholar 

  32. 32.

    Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1998). Standards for educational and psychological testing Washington DC: American Psychological Association.

  33. 33.

    Magasi, S., Ryan, G., Revicki, D., Lenderking, W., Hays, R. D., Brod, M., et al. (2011). Content validity of patient-reported outcome measures: Perspectives from a PROMIS meeting. Quality of Life Research,. doi:10.1007/s11136-011-9990-8.

    PubMed  Google Scholar 

  34. 34.

    Revicki, D., Hays, R. D., Cella, D., & Sloan, J. (2008). Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61(2), 102–109. doi:10.1016/j.jclinepi.2007.03.012.

    PubMed  Article  Google Scholar 

  35. 35.

    Valderas, J. M., Ferrer, M., Mendivil, J., Garin, O., Rajmil, L., Herdman, M., et al. (2008). Development of EMPRO: A tool for the standardized assessment of patient-reported outcome measures. Value in Health, 11(4), 700–708. doi:10.1111/j.1524-4733.2007.00309.x.

    PubMed  Article  Google Scholar 

  36. 36.

    Dewolf, L., Koller, M., Velikova, G., Johnson, C., Scott, N., & Bottomley, A. (2009). EORTC quality of life group: Translation procedure. http://groups.eortc.be/qol/sites/default/files/archives/translation_manual_2009.pdf. Accessed November 26, 2011.

  37. 37.

    Hays, R. D., & Hadorn, D. (1992). Responsiveness to change: An aspect of validity, not a separate dimension. Quality of Life Research, 1(1), 73–75.

    PubMed  Article  CAS  Google Scholar 

  38. 38.

    Wyrwich, K. W., Norquist, J. M., Lenderking, W. R., Acaster, S., & the Industry Advisory Committee of International Society for Quality of Life, R. (2012). Methods for interpreting change over time in patient-reported outcome measures. Quality of Life Research, 2012 Apr 17. [Epub ahead of print.]. doi:10.1007/s11136-012-0175-x.

  39. 39.

    Ahmed, S., Berzon, R. A., Revicki, D., Lenderking, W., Moinpour, C. M., Basch, E., et al. (2012). The use of patient-reported outcomes (PRO) within comparative effectivenss research: Implications for clinical practice and healthcare policy. Medical Care, 50(12), 1060–1070.

    PubMed  Article  Google Scholar 

  40. 40.

    Terwee, C. B., Bot, S. D., de Boer, M. R., van der Windt, D. A., Knol, D. L., Dekker, J., et al. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60(1), 34–42. doi:10.1016/j.jclinepi.2006.03.012.

    PubMed  Article  Google Scholar 

  41. 41.

    Nunnally, J. C., & Berstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

    Google Scholar 

  42. 42.

    Norquist, J. M., Girman, C., Fehnel, S., Demuro-Mercon, C., & Santanello, N. (2011). Choice of recall period for patient-reported outcome (PRO) measures: Criteria for consideration. Quality of Life Research,. doi:10.1007/s11136-011-0003-8.

    PubMed  Google Scholar 

  43. 43.

    Revicki, D. A., Cella, D., Hays, R. D., Sloan, J. A., Lenderking, W. R., & Aaronson, N. K. (2006). Responsiveness and minimal important differences for patient reported outcomes. Health and Quality of Life Outcomes, 4, 70. doi:10.1186/1477-7525-4-70.

    PubMed  Article  Google Scholar 

  44. 44.

    Brozek, J. L., Guyatt, G. H., & Schunemann, H. J. (2006). How a well-grounded minimal important difference can enhance transparency of labelling claims and improve interpretation of a patient reported outcome measure. Health and Quality of Life Outcomes, 4, 69. doi:10.1186/1477-7525-4-69.

    PubMed  Article  Google Scholar 

  45. 45.

    Norman, G. R., Sridhar, F. G., Guyatt, G. H., & Walter, S. D. (2001). Relation of distribution- and anchor-based approaches in interpretation of changes in health-related quality of life. Medical Care, 39(10), 1039–1047.

    PubMed  Article  CAS  Google Scholar 

  46. 46.

    Koller, M., Kantzer, V., Mear, I., Zarzar, K., Martin, M., Greimel, E., et al. (2012). The process of reconciliation: Evaluation of guidelines for translating quality-of-life questionnaires. Expert Review of Pharmacoeconomics & Outcomes Research, 12(2), 189–197. doi:10.1586/erp.11.102.

    Article  Google Scholar 

  47. 47.

    Jordan, J. E., Osborne, R. H., & Buchbinder, R. (2011). Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses. Journal of Clinical Epidemiology, 64(4), 366–379. doi:10.1016/j.jclinepi.2010.04.005.

    PubMed  Article  Google Scholar 

  48. 48.

    Acquadro, C., Conway, K., Hareendran, A., & Aaronson, N. (2008). Literature review of methods to translate health-related quality of life questionnaires for use in multinational clinical trials. Value in Health, 11(3), 509–521. doi:10.1111/j.1524-4733.2007.00292.x.

    PubMed  Article  Google Scholar 

  49. 49.

    Crosby, R. D., Kolotkin, R. L., & Williams, G. R. (2003). Defining clinically meaningful change in health-related quality of life. Journal of Clinical Epidemiology, 56(5), 395–407.

    PubMed  Article  Google Scholar 

  50. 50.

    Frost, M. H., Reeve, B. B., Liepa, A. M., Stauffer, J. W., Hays, R. D., & Mayo, F. D. A. P.-R. O. C. M. G. (2007). What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value in Health, 10(Suppl 2), S94–S105. doi:10.1111/j.1524-4733.2007.00272.x.

    PubMed  Article  Google Scholar 

  51. 51.

    Hays, R. D., & Woolley, J. M. (2000). The concept of clinically meaningful difference in health-related quality-of-life research. How meaningful is it? PharmacoEconomics, 18(5), 419–423.

    PubMed  Article  CAS  Google Scholar 

  52. 52.

    Kemmler, G., Zabernigg, A., Gattringer, K., Rumpold, G., Giesinger, J., Sperner-Unterweger, B., et al. (2010). A new approach to combining clinical relevance and statistical significance for evaluation of quality of life changes in the individual patient. Journal of Clinical Epidemiology, 63(2), 171–179. doi:10.1016/j.jclinepi.2009.03.016.

    PubMed  Article  CAS  Google Scholar 

  53. 53.

    Kottner, J., Audige, L., Brorson, S., Donner, A., Gajewski, B. J., Hrobjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64(1), 96–106. doi:10.1016/j.jclinepi.2010.03.002.

    PubMed  Article  Google Scholar 

  54. 54.

    Schmidt, S., & Bullinger, M. (2003). Current issues in cross-cultural quality of life instrument development. Archives of Physical Medicine and Rehabilitation, 84(4 Suppl 2), S29–S34. doi:10.1053/apmr.2003.50244.

    PubMed  Article  Google Scholar 

  55. 55.

    Sprangers, M. A., Moinpour, C. M., Moynihan, T. J., Patrick, D. L., Revicki, D. A., & Clinical Significance Consensus Meeting Group. (2002). Assessing meaningful change in quality of life over time: A users’ guide for clinicians. Mayo Clinic Proceedings, 77(6), 561–571. doi:10.4065/77.6.561.

    PubMed  Article  Google Scholar 

  56. 56.

    Turner, R. R., Quittner, A. L., Parasuraman, B. M., Kallich, J. D., Cleeland, C. S., & Mayo, F. D. A. P.-R. O. C. M. G. (2007). Patient-reported outcomes: Instrument development and selection issues. Value in Health, 10(Suppl 2), S86–S93. doi:10.1111/j.1524-4733.2007.00271.x.

    PubMed  Article  Google Scholar 

  57. 57.

    Schunemann, H. J., & Guyatt, G. H. (2005). Commentary–goodbye M(C)ID! Hello MID, where do you come from? Health Services Research, 40(2), 593–597. doi:10.1111/j.1475-6773.2005.00374.x.

    PubMed  Article  Google Scholar 

  58. 58.

    Schunemann, H. J., Puhan, M., Goldstein, R., Jaeschke, R., & Guyatt, G. H. (2005). Measurement properties and interpretability of the Chronic respiratory disease questionnaire (CRQ). Copd, 2(1), 81–89.

    PubMed  Article  Google Scholar 

Download references

Acknowledgments

This study was funded by the Patient-Centered Outcomes Research Institute (PCORI-SOL-RMWG-001: PIs: Zeeshan Butt, PhD, Northwestern University; Bryce Reeve, PhD, University of North Carolina at Chapel Hill). The views expressed in this article are those of the authors and do not necessarily reflect those of PCORI.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Bryce B. Reeve.

Additional information

This study was conducted on behalf of the International Society for Quality of Life Research (ISOQOL).

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 12 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Reeve, B.B., Wyrwich, K.W., Wu, A.W. et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res 22, 1889–1905 (2013). https://doi.org/10.1007/s11136-012-0344-y

Download citation

Keywords

  • Patient-reported outcomes
  • Comparative effectiveness
  • Patient-centered outcomes research
  • Psychometrics
  • Questionnaire