Skip to main content
SpringerLink
Log in

Assessment Methods for Patient-Reported Outcomes

  • Practical Disease Management
  • Published:
Disease Management & Health Outcomes

Abstract

The methods used to collect information on patient-reported outcomes (PROs) can affect the validity, reliability and sensitivity of the data. This review examines the influence of a variety of methodological issues that occur. In particular, when patients are asked to rely on their memory to aggregate and summarize their experience, a variety of inaccuracies and biases can affect the data. As the extent and magnitude of recall bias have become more well-known, researchers have increasingly turned toward collecting data in the field, closer in time to events and experiences of interest. Diary methods require patients to self-monitor their experiences, disease episodes and healthcare utilization over time. The collection of diary data from patients places a premium on adherence to the data collection protocol. Recent evidence suggests that patients routinely fail to adhere to diary protocols, thus introducing retrospective error and biases into the data. Reasons for patients’ non-adherence to data collection protocols include their desire to appear to be ‘good’ patients which can result in back-filling PRO entries before a site visit, simply forgetting, and data collection methods that fail to help patients be adherent to the data collection protocol. Principles that can be used to enhance patient adherence to PRO data collection methods include good patient training, creating simple and intuitive user interfaces for PRO assessments, and actively driving the protocol by helping patients remember to complete PRO measures as required by the protocol.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Burke L. Acceptable evidence for pharmaceutical advertising and labeling. Poster presented at the Drug Information Association Workshop on Pharmacoeconomic and Quality Of Life Labeling and Marketing Claims. New Orleans (LA): 2000 Oct.

  2. Hufford MR, Shiffman S, Paty J, et al. Ecological momentary assessment: real world, real-time measurement of patient experience. In: Fahrenberg J, Myrtek M, editors. Progress in ambulatory assessment. Seattle (WA): Hogrefe & Huber Publishers, 2001:69–92.

    Google Scholar 

  3. Gill TM, Feinstein AR. A critical appraisal of the quality-of-life measurements. JAMA 1994; 272: 619–26.

    Article  PubMed  CAS  Google Scholar 

  4. Muldoon MF, Barger SD, Flory JD, et al. What are the quality of life measurements measuring? BMJ 1998; 316: 542–5.

    Article  PubMed  CAS  Google Scholar 

  5. Schipper H, Clinch J, Powell V. Definitions and conceptual issues. In: Spilker B, editor. Quality of life assessments in clinical trials. New York (NY): Raven Press, 1990: 11–24.

    Google Scholar 

  6. Willke RJ, Burke LB, Erickson P. PROs as clinical endpoints for registration and labeling. Poster presented at the Drug Information Association Quality of Life Symposium. Hilton Head Island (SC): 2002 Mar 24–6.

    Google Scholar 

  7. Kind P. Measuring up to the task. Value Health 2001; 4: 344–5.

    Google Scholar 

  8. Bersoff DM, Bersoff DN. Ethical issues in the collection of self-report data. In: Stone AA, Turkkan JA, Bachrach CA, et al., editors. The science of self-report: implications for research and practice. Mahwah (NJ): Lawrence Erlbaum Associates, 2000: 9–24.

    Google Scholar 

  9. Singer E, Frankel MR. Informed consent procedures in telephone interviews. Am Sociol Rev 1982; 47: 416–27.

    Article  PubMed  Google Scholar 

  10. Stanton AL, Burker EJ, Kershaw D. Effects of researcher follow-up on distressed subjects: trade-off between validity and ethical responsibility? Ethics Behav 1991; 1: 105–12.

    Article  PubMed  Google Scholar 

  11. Turner CF, Ku L, Rogers SM, et al. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 1998; 280: 847–8.

    Article  Google Scholar 

  12. Göbel H, Heinze A, Kuhn K, et al. Effects of operationalized computer diagnosis on the therapeutic results of sumatriptan in general practice. Cephalalgia 1998; 18: 481–6.

    Article  PubMed  Google Scholar 

  13. Shiffman S, Paty JA, Gnys M, et al. First lapses to smoking: within-subjects analysis of real-time reports. J Consult Clin Psychol 1996; 64: 366–79.

    Article  PubMed  CAS  Google Scholar 

  14. Drummond HE, Ghosh S, Ferguson A, et al. Electronic quality of life questionnaires: a comparison of pen-based electronic questionnaires with convention paper in a gastrointestinal study. Qual Life Res 1995; 4: 21–6.

    Article  PubMed  CAS  Google Scholar 

  15. Rabin JM, McNett J, Badlani GH. ‘Compu-void II’: the computerized voiding diary. J Med Syst 1996; 20: 19–34.

    Article  PubMed  CAS  Google Scholar 

  16. Tiplady B, Crompton GK, Dewar MH, et al. The use of electronic diaries in respiratory studies. Drug Inf J 1997; 31: 759–64.

    Article  Google Scholar 

  17. Pickering TG, Coats A, Mallion JM, et al. Blood pressure monitoring: task force V: white-coat hypertension. Blood Press Monit 1999; 4: 333–41.

    PubMed  CAS  Google Scholar 

  18. Nived O, Sturfelt G, Eckernas SA, et al. A comparison of 6 months’ compliance of patients with rheumatoid arthritis treated with tenoxicam and naproxen: use of patient computer data to assess response to treatment. J Rheumatol 1994; 21: 1537–41.

    PubMed  CAS  Google Scholar 

  19. Nelson RO. Assessment and therapeutic functions of self-monitoring. In: Hersen M, Eisler RM, Miller P, editors. Progress in behavioral modification. New York (NY): Academic Press, 1977: 3–41.

    Google Scholar 

  20. Beck AT, Ward CH, Mendelson M, et al. An inventory for measuring depression. Arch Gen Psychiatry 1964; 4: 561–71.

    Article  Google Scholar 

  21. Mark MM, Sinclair RC, Wellens TR. The effect of completing the Beck Depression Inventory on self-reported mood state: contrast and assimilation. Pers Soc Psychol Bull 1991; 17:457–65.

    Article  Google Scholar 

  22. Bailey AS, Martin ML. In quality of life questionnaires, what do patients understand “past month” to mean? Poster presented at the International Society for Quality of Life Research conference. Vancouver (BC), 2000 Oct 29–31.

  23. Bradburn NM, Rips LJ, Shevell SK. Answering autobiographical questions: the impact of memory and inference on surveys. Science 1987; 236: 157–61.

    Article  PubMed  CAS  Google Scholar 

  24. Tourangeau R. Remembering what happened: memory errors and survey reports. In: Stone AA, Turkkan JA, Bachrach CA, et al., editors. The science of self-report: implications for research and practice. Mahwah (NJ): Lawrence Erlbaum Associates, 2000: 29–48.

    Google Scholar 

  25. Erickson JR, Jemison CR. Relations among measures of autobiographical memory. Bull Psychom Soc 1991; 29: 233–6.

    Google Scholar 

  26. Linton SJ. Memory for chronic pain intensity: correlates of accuracy. Percept Mot Skills 1991; 72: 1091–5.

    Article  PubMed  CAS  Google Scholar 

  27. Wagenaar WA. My memory: a study of autobiographical memory over six years. Cogn Psychol 1986; 18: 225–52.

    Article  Google Scholar 

  28. Joffe RT, MacDonald C, Kutcher SP. Life events and mania: a case-controlled study. Psychiatry Res 1989; 30: 213–6.

    Article  PubMed  CAS  Google Scholar 

  29. Menon G, Yorkton EA. The use of memory and contextual cues in the formation of behavioral frequency judgments. In: Stone AA, Turkkan JA, Bachrach CA, et al., editors. The science of self-report: implications for research and practice. Mahwah (NJ): Lawrence Erlbaum Associates, 2000: 63–80.

    Google Scholar 

  30. Shiffman S, Hufford M, Hickox M, et al. Remember that? A comparison of realtime versus retrospective recall of smoking lapses. J Consult Clin Psychol 1997; 65: 292–300.

    Article  PubMed  CAS  Google Scholar 

  31. Clark DM, Teasdale JD. Constraints on effects of mood on memory. J Pers Soc Psychol 1985; 48: 1595–608.

    Article  Google Scholar 

  32. Clark DM, Teasdale JD. Diurnal variation in clinical depression and accessibility of memories of positive and negative experiences. J Abnorm Psychol 1982; 91: 87–95.

    Article  PubMed  CAS  Google Scholar 

  33. Teasdale JD, Fogarty SJ. Differential effects of induced mood on retrieval of pleasant and unpleasant events from episodic memory. J Abnorm Psychol 1979; 88: 248–57.

    Article  PubMed  CAS  Google Scholar 

  34. Brown GW, Harris T. Social origins of depression. London: Tavistock, 1978.

    Google Scholar 

  35. Eich E, Reeves JL, Jaeger B, et al. Memory for pain: relation between past and present pain intensity. Pain 1985; 23: 375–9.

    Article  PubMed  CAS  Google Scholar 

  36. Means B, Swan GE, Jobe JB, Esposito JL. The effects of estimation strategies on the accuracy of respondents’ reports of cigarette smoking. In: Schwartz N, Sudman S, editors. Autobiographical memory and the validity of retrospective reports. New York (NY): Springer-Verlag, 1994: 107–19.

    Chapter  Google Scholar 

  37. Sprangers MAG, Van Dam FS, Broersen J, et al. Revealing response shift in longitudinal research on fatigue: the use of the thentest approach. Acta Oncol 1999; 38: 709–18.

    Article  PubMed  CAS  Google Scholar 

  38. Redelmeier D, Kahneman D. Patients’ memories of painful medical treatments: real-time and retrospective evaluations of two minimally invasive procedures. Pain 1996; 66: 3–8.

    Article  PubMed  CAS  Google Scholar 

  39. Stone AA, Broderick JB, Kaell AT, et al. Does the peak-end phenomenon observed in laboratory pain studies apply to real-world pain in rheumatoid arthritics? J Pain 2000; 1: 203–18.

    Google Scholar 

  40. Baldwin W. Information no one else knows: the value of self-report. In: Stone AA, Turkkan JA, Bachrach CA, et al., editors. The science of self-report: implications for research and practice. Mahwah (NJ): Lawrence Erlbaum Associates, 2000: 3–7.

    Google Scholar 

  41. Hufford MR, Shiffman SS. Methodological issues affecting the value of patientreported outcomes data. Expert Rev Pharmacoeconomics Health Outcomes 2002; 2(2): 119–28.

    Article  Google Scholar 

  42. Shiffman S. Real-time self-report of momentary states in the natural environment: computerized ecological momentary assessment. In: Stone AA, Turkkan JA, Bachrach CA, et al., editors. The science of self-report: implications for research and practice. Mahwah (NJ): Lawrence Erlbaum Associates, 2000: 277–96.

    Google Scholar 

  43. Houpt JB, McMillan R, Wein C, et al. Effect of glucosamine hydrochloride in the treatment of pain of osteoarthritis of the knee. J Rheumatol 1999; 26: 2423–30.

    PubMed  CAS  Google Scholar 

  44. Norman GR, McFarlane AH, Streiner DL, et al. Health diaries: strategies for compliance and relation to other measures. Med Care 1982; 20: 623–9.

    Article  PubMed  CAS  Google Scholar 

  45. Verbrugge LM. Health diaries. Med Care 1980; 18: 73–95.

    Article  PubMed  CAS  Google Scholar 

  46. Lewis B, Lewis D, Cumming G. The comparative analgesic efficacy of transcutaneous electrical nerve stimulation and a non-steroidal anti-inflammatory drug for painful osteoarthritis. Br J Rheumatol 1994; 33: 455–60.

    Article  PubMed  CAS  Google Scholar 

  47. Neugebauer R. Reliability of seizure diaries in adult epileptic patients. Neuroepidemiology 1989; 8: 228–33.

    Article  PubMed  CAS  Google Scholar 

  48. Stone AA, Schwartz JE, Neale JM, et al. A comparison of coping assessed by ecological momentary assessment and retrospective recall. J Pers Soc Psychol 1998; 74: 1670–80.

    Article  PubMed  CAS  Google Scholar 

  49. Stone AA, Shiftman S. Ecological momentary assessment: measuring real world processes in behavioral medicine. Ann Behav Med 1994; 16: 199–202.

    Google Scholar 

  50. Patterson R. The new focus: integrating behavioral science into disease management. In: Patterson R, editor. Changing patient behavior: improving outcomes in health and disease management. San Francisco (CA): Jossey-Bass, 2001: 1–21.

    Google Scholar 

  51. Urquhart J. Role of patient compliance in clinical pharmacokinetics: a review of recent research. Clin Pharmacokinet 1994; 27: 202–15.

    Article  PubMed  CAS  Google Scholar 

  52. Urquhart J, De Klerk E. Contending paradigms for the interpretation of data on patient compliance with therapeutic drug regimens. Stat Med 1998; 17: 251–67.

    Article  PubMed  CAS  Google Scholar 

  53. Stone AA, Shiftman S, Schwartz JE, et al. Patient noncompliance with paper diaries. BMJ 2002; 324: 1193–4.

    Article  PubMed  Google Scholar 

  54. Shiftman S, Elash CE, Paton SM, et al. Comparative efficacy of 24-hour and 16-hour transdermal nicotine patches for relief of morning craving. Addiction 2000; 95: 1185–95.

    Article  Google Scholar 

  55. Chmelik F, Doughty A. Objective measurements of compliance in asthma treatment. Ann Allergy 1994; 73: 527–32.

    PubMed  CAS  Google Scholar 

  56. Jonasson G, Carlsen K, Sodal A, et al. Patient compliance in a clinical trial with inhaled budesonide in children with mild asthma. Eur Respir J 1999; 14: 150–4.

    Article  PubMed  CAS  Google Scholar 

  57. Mazze RS, Shamoon H, Pasmantier R, et al. Reliability of blood glucose monitoring by patients with diabetes mellitus. Am J Med 1984; 77: 211–7.

    Article  PubMed  CAS  Google Scholar 

  58. Milgrom H, Bender B, Ackerson L, et al. Noncompliance and treatment failure in children with asthma. J Allergy Clin Immunol 1996; 98: 1051–7.

    Article  PubMed  CAS  Google Scholar 

  59. Simmons MS, Nides MA, Rand CS, et al. Unpredictability of deception in compliance with physician prescribed bronchodilator inhaler use in a clinical trial. Chest 2000; 118:290–5.

    Article  PubMed  CAS  Google Scholar 

  60. Spector S, Kinsman R, Mawhinney H, et al. Compliance of patients with asthma with an experimental aerosolized medication: implications for controlled clinical trials. J Allergy Clin Immunol 1986; 77: 65–70.

    Article  PubMed  CAS  Google Scholar 

  61. Straka R, Fish J, Benson S, et al. Patient self-reporting of compliance does not correspond with electronic monitoring: an evaluation using isosorbide dinitrate as a model drug. Pharmacotherapy 1997; 17: 126–32.

    PubMed  CAS  Google Scholar 

  62. Verscheiden P, Cartier A, L’Archeveque A, et al. Compliance with and accuracy of daily assessment of peak expiratory flows (PEF) in asthmatic subjects over a three month period. Eur Respir J 1996; 9: 880–5.

    Article  Google Scholar 

  63. Boudes P. Drug compliance in therapeutic trials: a review. Control Clin Trials 1998; 19: 257–268.

    Article  PubMed  CAS  Google Scholar 

  64. Raskin J. The humane interface: new directions for designing interactive systems. Reading (MA): Addison Wesley Longman, 2000.

    Google Scholar 

  65. Trick LR. Patient compliance: don’t count on it! J Am Optom Assoc 1993; 64: 264–70.

    PubMed  CAS  Google Scholar 

  66. Hufford MR, Shields AL. Electronic diaries: an examination of applications and what works in the field. Appl Clin Trials 2002; 11(4): 46–56.

    Google Scholar 

  67. Kamarck TW, Shiftman SM, Smithline L, et al. Effects of task strain, social conflict, and emotional activation on ambulatory cardiovascular activity: daily life consequences of recurring stress in a multiethnic adult sample. Health Psychol 1998; 17: 17–29.

    Article  PubMed  CAS  Google Scholar 

  68. Shiffman S, Stone AA. Introduction to the special section: ecological momentary assessment in health psychology. Health Psychol 1998; 17: 3–5.

    Article  Google Scholar 

Download references

Acknowledgements

Drs Hufford and Shiffman have a financial investment in invivodata, inc., which provides software for electronic diary data collection in clinical trials. No funding sources were used to support this publication.

Author information

Authors and Affiliations

  1. Invivodata inc., 2100 Wharton St., Ste. 505, Pittsburgh, Pennsylvania, 15203, USA

    Michael R. Hufford Ph.D. (Vice President) & Saul Shiffman

  2. Departments of Psychology, Psychiatry and Pharmaceutical Sciences, University of Pittsburgh, Pittsburgh, Pennsylvania, USA

    Saul Shiffman

Authors
  1. Michael R. Hufford Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saul Shiffman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael R. Hufford Ph.D..

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hufford, M.R., Shiffman, S. Assessment Methods for Patient-Reported Outcomes. Dis-Manage-Health-Outcomes 11, 77–86 (2003). https://doi.org/10.2165/00115677-200311020-00002

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.2165/00115677-200311020-00002

Keywords

Search

Navigation