Skip to main content
SpringerLink
Log in

Issues in the design of Internet-based systems for collecting patient-reported outcomes

  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Background and objectives

Although there is a growing interest in using patient-reported outcomes (PRO) to monitor disease progression and/or therapeutic response, to improve care, and to screen for physical or psychosocial problems in routine clinical practice, PRO instruments can be difficult to administer, score, and interpret in this setting. Internet-based approaches to PRO collection may help overcome these obstacles. This paper discusses the rationale for using the Internet for routine PRO collection, summarizes relevant literature and ongoing projects, and raises several key design and development issues that should guide further efforts in this area.

Major findings

A small number of Internet-based PRO collection applications have been or are currently being developed. The major characteristics of several of these projects are reviewed and summarized. Successful Internet-based PRO collection applications must address patient and clinician-specific needs related to workflow and to the way in which results are presented. A growing number of instruments have been adapted for and evaluated in a web-based format.

Conclusions

Collecting PROs via the Internet has the potential to overcome many of the challenges associated with efforts to routinely use PROs in the clinical encounter.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Abbreviations

BMT:

Bone Marrow Therapy

CAT:

Computerized Adaptive Testing

CHADIS:

Child Health and Development Interactive System

CTCAE:

Common Terminology Criteria for Adverse Events

DSM-PC:

Diagnostic and Statistical Manual for Primary Care

EDC:

Electronic Data Collection

EHR:

Electronic Health Record

EQ-5D:

EuroQOL 5-D

EORTC-QLQ C30:

European Organization for Research and Treatment of Cancer Quality of Life Questionnaire

FHCRC:

Fred Hutchinson Cancer Research Center

HADS:

Hospital Anxiety and Depression Scale

HIPAA:

Health Insurance Portability and Accountability Act

HRQOL:

Health-Related Quality of Life

HSCT:

Hematopoietic Stem Cell Transplantation

IRT:

Item Response Theory

IVR:

Interactive Voice Response System

MSKCC:

Memorial Sloan Kettering Cancer Center

NCI:

National Cancer Institute

PRO:

Patient Reported Outcome(s)

PROMIS:

Patient Reported Outcome Measurement Information System

SCNS:

Supportive Care Needs Survey

References

  1. Ellwood, P. M. (1988). Shattuck lecture–outcomes management. A technology of patient experience. The New England Journal of Medicine, 318, 1549–1556.

    Article  PubMed  CAS  Google Scholar 

  2. List of 1000 instruments in alphabetical order. Retrieved 8/29, 2006, from http://www.proqolid.org/public/list/a.html.

  3. Bielli, E., Carminati, F., La Capra, S., Lina, M., Brunelli, C., & Tamburini, M. (2004). A wireless health outcomes monitoring system (WHOMS): Development and field testing with cancer patients using mobile phones. BMC Medical Informatics and Decision Making, 4, 7.

    Article  PubMed  Google Scholar 

  4. Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA, 288, 3027–3034.

    Article  PubMed  Google Scholar 

  5. Greenhalgh, J., Long, A. F., & Flynn, R. (2005). The use of patient reported outcome measures in routine clinical practice: Lack of impact or lack of theory? Social Science & Medicine, 60, 833–843.

    Article  Google Scholar 

  6. Donaldson, M. (2006). Using patient-reported outcomes in clinical oncology practice: Benefits, challenges and next steps. Expert Review of Pharmacoeconomics & Outcomes Research, 6, 87–95.

    Article  Google Scholar 

  7. Detmar, S. B., & Aaronson, N. K. (1998). Quality of life assessment in daily clinical oncology practice: A feasibility study. European Journal of Cancer, 34, 1181–1186.

    Article  PubMed  CAS  Google Scholar 

  8. McLachlan, S. A., Allenby, A., Matthews, J., Wirth, A., Kissane, D., Bishop, M., Beresford, J., & Zalcberg, J. (2001). Randomized trial of coordinated psychosocial interventions based on patient self-assessments versus standard care to improve the psychosocial functioning of patients with cancer. Journal of Clinical Oncology, 19, 4117–4125.

    PubMed  CAS  Google Scholar 

  9. Wu, A. W., Cagney, K. A., & St John, P. D. (1997). Health status assessment. Completing the clinical database. Journal of General Internal Medicine, 12, 254–255.

    Article  PubMed  CAS  Google Scholar 

  10. Velikova, G., Brown, J. M., Smith, A. B., & Selby, P. J. (2002). Computer-based quality of life questionnaires may contribute to doctor-patient interactions in oncology. British Journal of Cancer, 86, 51–59.

    Article  PubMed  CAS  Google Scholar 

  11. Taenzer, P., Bultz, B. D., Carlson, L. E., Speca, M., DeGagne, T., Olson, K., Doll, R., & Rosberger, Z. (2000). Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psychooncology, 9, 203–213.

    Article  PubMed  CAS  Google Scholar 

  12. Athale, N., Sturley, A., Skoczen, S., Kavanaugh, A., & Lenert, L. (2004). A web-compatible instrument for measuring self-reported disease activity in arthritis. The Journal of Rheumatology, 31, 223–228.

    PubMed  Google Scholar 

  13. McDaniel, A. M., Benson, P. L., Roesener, G. H., & Martindale, J. (2005). An integrated computer-based system to support nicotine dependence treatment in primary care. Nicotine & Tobacco Research, 7(Suppl 1), S57–S66.

    Article  Google Scholar 

  14. Saleh, K. J., Radosevich, D. M., Kassim, R. A., Moussa, M., Dykes, D., Bottolfson, H., Gioe, T. J., & Robinson, H. (2002). Comparison of commonly used orthopaedic outcome measures using palm-top computers and paper surveys. Journal of Orthopaedic Research, 20, 1146–1151.

    Article  PubMed  Google Scholar 

  15. Anhoj, J., & Moldrup, C. (2004). Feasibility of collecting diary data from asthma patients through mobile phones and SMS (short message service): Response rate analysis and focus group evaluation from a pilot study. Journal of Medical Internet Research, 6, e42.

    Article  PubMed  Google Scholar 

  16. Crawley, J. A., Kleinman, L., & Dominitz, J. (2000). User preferences for computer administration of quality of life instruments. Drug Information Journal, 34, 137.

    Google Scholar 

  17. Jacobsen, P. B., Davis, K., & Cella, D. (2002). Assessing quality of life in research and clinical practice. Oncology (Williston.Park), 16, 133–139.

    Google Scholar 

  18. Bliven, B. D., Kaufman, S. E., & Spertus, J. A. (2001). Electronic collection of health-related quality of life data: Validity, time benefits, and patient preference. Quality of Life Research, 10, 15–22.

    Article  PubMed  CAS  Google Scholar 

  19. Wright, E. P., Selby, P. J., Crawford, M., Gillibrand, A., Johnston, C., Perren, T. J., Rush, R., Smith, A., Velikova, G., Watson, K., Gould, A., & Cull, A. (2003). Feasibility and compliance of automated measurement of quality of life in oncology practice. Journal of Clinical Oncology, 21, 374–382.

    Article  PubMed  CAS  Google Scholar 

  20. Bushnell, D. M., Martin, M. L., & Parasuraman, B. (2003). Electronic versus paper questionnaires: A further comparison in persons with asthma. The Journal of Asthma, 40, 751–762.

    Article  PubMed  CAS  Google Scholar 

  21. Bayliss, M. S., Dewey, J. E., Dunlap, I., Batenhorst, A. S., Cady, R., Diamond, M. L., & Sheftell, F. (2003). A study of the feasibility of internet administration of a computerized health survey: The headache impact test (HIT). Quality of Life Research, 12, 953–961.

    Article  PubMed  CAS  Google Scholar 

  22. Pealer, L. N., Weiler, R. M., Pigg, R. M. Jr, Miller, D., & Dorman, S. M. (2001). The feasibility of a web-based surveillance system to collect health risk behavior data from college students. Health Education & Behavior, 28, 547–559.

    Article  CAS  Google Scholar 

  23. Hanscom, B., Lurie, J. D., Homa, K., & Weinstein, J. N. (2002). Computerized questionnaires and the quality of survey data. Spine, 27, 1797–1801.

    Article  PubMed  Google Scholar 

  24. Wilson, A. S., Kitas, G. D., Carruthers, D. M., Reay, C., Skan, J., Harris, S., Treharne, G. J., Young, S. P., & Bacon, P. A. (2002). Computerized information-gathering in specialist rheumatology clinics: An initial evaluation of an electronic version of the short form 36. Rheumatology (Oxford), 41, 268–273.

    Article  CAS  Google Scholar 

  25. Bush, N., Donaldson, G., Moinpour, C., Haberman, M., Milliken, D., Markle, V., & Lauson, J. (2005). Development, feasibility and compliance of a web-based system for very frequent QOL and symptom home self-assessment after hematopoietic stem cell transplantation. Quality of Life Research, 14, 77–93.

    Article  PubMed  CAS  Google Scholar 

  26. Boyes, A., Newell, S., & Girgis, A. (2002). Rapid assessment of psychosocial well-being: Are computers the way forward in a clinical setting? Quality of Life Research, 11, 27–35.

    Article  PubMed  Google Scholar 

  27. Hambleton, R. K. (2005). Applications of item response theory to improve health outcomes assessment: developing item banks, linking instruments, and computer-adaptive testing. In J. Lipscomb, C. C. Gotay, & C. Snyder (Eds.), Outcomes assessment in cancer: Measures, methods and applications (pp. 445–464). Cambridge: Cambridge University Press.

    Google Scholar 

  28. Litaker, D. (2003). New technology in quality of life research: Are all computer-assisted approaches created equal? Quality of Life Research, 12, 387–393.

    Article  PubMed  Google Scholar 

  29. Velikova, G., Wright, E. P., Smith, A. B., Cull, A., Gould, A., Forman, D., Perren, T., Stead, M., Brown, J., & Selby, P. J. (1999). Automated collection of quality-of-life data: A comparison of paper and computer touch-screen questionnaires. The Journal of Clinical Oncology, 17, 998–1007.

    CAS  Google Scholar 

  30. More Americans turning to web for health information, poll shows. The Wall Street Journal Online (2006).

  31. Liederman, E. M., & Morefield, C. S. (2003). Web messaging: A new tool for patient-physician communication. Journal of the American Medical Informatics Association, 10, 260–270.

    Article  PubMed  Google Scholar 

  32. Casebeer, L., Bennett, N., Kristofco, R., Carillo, A., & Centor, R. (2002). Physician internet medical information seeking and on-line continuing education use patterns. The Journal of Continuing Education in the Health Professions, 22, 33–42.

    Article  PubMed  Google Scholar 

  33. Bennett, N. L., Casebeer, L. L., Kristofco, R. E., & Strasser, S. M. (2004). Physicians’ internet information-seeking behaviors. Journal of Continuing Education in the Health Professions, 24, 31–38.

    Article  PubMed  Google Scholar 

  34. Howsyourhealth.org. from www.howsyourhealth.org.

  35. Child health and development interactive system (CHADIS) & CHADIS EMR. Retrieved 11/2, 2006, from http://www.childhealthcare.org/marketing/datasheet.shtml.

  36. Sturner, R., Morrel, T., & Howard, B. J. (2004). Mental health diagnoses among children being seen for child health supervision visits: Typical practice and DSM-PC diagnoses. Presented at Pediatric Academic Society Annual Meeting, 441.

  37. Sturner, R., Morrel, T., & Howard, B. (2005). Preliminary validation of the DSM-PC. Journal of Developmental and Behavioral Pediatrics, 24, 393–394.

    Google Scholar 

  38. Sturner, R., Howard, B. J., Morrel, T., & Rogers-Senuta, K. (2003). Validation of a computerized parent questionnaire for identifying child mental health disorders and implementing DSM-PC. Presented at Pediatric Academic Society Annual Meeting, 3801.

  39. Basch, E., Artz, D., Dulko, D., Scher, K., Sabbatini, P., Hensley, M., Mitra, N., Speakman, J., McCabe, M., & Schrag, D. (2005). Patient online self-reporting of toxicity symptoms during chemotherapy. Journal of Clinical Oncology, 23, 3552–3561.

    Article  PubMed  Google Scholar 

  40. Cohen, S. R., Boston, P., Mount, B. M., & Porterfield, P. (2001). Changes in quality of life following admission to palliative care units. Palliative Medicine, 15, 363–371.

    Article  PubMed  CAS  Google Scholar 

  41. Cull, A., Gould, A., House, A., Smith, A., Strong, V., Velikova, G., Wright, P., & Selby, P. (2001). Validating automated screening for psychological distress by means of computer touchscreens for use in routine oncology practice. British Journal of Cancer, 85, 1842–1849.

    Article  PubMed  CAS  Google Scholar 

  42. Taenzer, P. A., Speca, M., Atkinson, M. J., Bultz, B. D., Page, S., Harasym, P., & Davis, J. L. (1997). Computerized quality-of-life screening in an oncology clinic. Cancer Practice, 5, 168–175.

    PubMed  CAS  Google Scholar 

  43. Hodge, J. G. Jr. (2003). Health information privacy and public health. The Journal of Law, Medicine & Ethics, 31, 663–671.

    Article  Google Scholar 

  44. Lumpkin, J. R. (2000). e-Health, HIPAA, and beyond. Health Affairs (Millwood) 19, 149–151.

    Article  CAS  Google Scholar 

  45. Garner, J. C. (2003). Final HIPAA security regulations: A review. Managed Care Quarterly, 11, 15–27.

    PubMed  Google Scholar 

  46. Fairfield, K. M., Chen, W. Y., Colditz, G. A., Emmons, K. M., & Fletcher, S. W. (2004). Colon cancer risk counseling by health-care providers: Perceived barriers and response to an internet-based cancer risk appraisal instrument. Journal of Cancer Education, 19, 95–97.

    Article  PubMed  Google Scholar 

  47. Sittig, D. F. (2003). Results of a content analysis of electronic messages (email) sent between patients and their physicians. BMC Medical Informatics and Decision Making, 3, 11–15.

    Article  PubMed  Google Scholar 

  48. Katz, S. J., Nissan, N., & Moyer, C. A. (2004). Crossing the digital divide: Evaluating online communication between patients and their providers. The American Journal of Managed Care, 10, 593–598.

    PubMed  Google Scholar 

  49. Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16, 133–141.

    Article  PubMed  Google Scholar 

  50. Thissen, D., Reeve, B. B., Bjorner, J. B., & Chang, C. H. (2007). Methodological issues for building item banks and computerized adaptive scales. Quality of Life Research, 16, 109–119.

    Article  Google Scholar 

  51. Berry, D. L., Trigg, L. J., Lober, W. B., Karras, B. T., Galligan, M. L., Austin-Seymour, M., & Martin, S. (2004). Computerized symptom and quality-of-life assessment for patients with cancer part I: Development and pilot testing. Oncology Nursing Forum, 31, E75–E83.

    Article  PubMed  Google Scholar 

  52. Bensley, R. J., & Lewis, J. B. (2002). Analysis of internet-based health assessments. Health Promotion Practice, 3, 463–476.

    Article  Google Scholar 

  53. Mullen, K. H., Berry, D. L., & Zierler, B. K. (2004). Computerized symptom and quality-of-life assessment for patients with cancer part II: Acceptability and usability. Oncology Nursing Forum, 31, E84–E89.

    Article  PubMed  Google Scholar 

  54. NCI announces new research-based web design guidelines for improving health information web sites. 2005. Retrieved 11/2, 2006, from http://www.cancer.gov/newscenter/usabilityguidelines.

  55. Williams, P., Nicholas, D., Huntington, P., & McLean, F. (2002). Surfing for health: User evaluation of a health information website. part one: Background and literature review. Health Information and Libraries Journal, 19, 98–108.

    Article  PubMed  Google Scholar 

  56. Brundage, M., Feldman-Stewart, D., Leis, A., Bezjak, A., Degner, L., Velji, K., Zetes-Zanatta, L., Tu, D., Ritvo, P., & Pater, J. (2005). Communicating quality of life information to cancer patients: A study of six presentation formats. Journal of Clinical Oncology, 23, 6949–6956.

    Article  PubMed  CAS  Google Scholar 

  57. Bjorner, J. B., Chang, C. H., Thissen, D., & Reeve, B. B. (2007). Developing tailored instruments: Item banking and computerized adaptive assessment. Quality of Life Research, 16, 95–108.

    Article  Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge the input and assistance of the members of the “Website for Outpatient QOL Assessment” Research Network: Neil Aaronson, PhD; Michael Brundage, MD, MSc, FRCP(C); Carolyn Gotay, PhD; James Hodge, JD; Denise Hynes, RN, MPH, PhD; John Wasson, MD; Susan Yount, PhD, Brad Zebrack, PhD, MSW, MPH. Funding for this project was provided by R21 CA113223-01 from the National Cancer Institute.

Author information

Authors and Affiliations

  1. Department of Health Policy and Management, Bloomberg School of Public Health, The Johns Hopkins University, 624 North Broadway, 6th Floor, Baltimore, MD, 21205-1901, USA

    James B. Jones, Claire F. Snyder & Albert W. Wu

  2. Center for Health Research, Geisinger Medical Center, Danville, PA, USA

    James B. Jones

  3. Department of Medicine, School of Medicine, The Johns Hopkins University, Baltimore, MD, USA

    Claire F. Snyder & Albert W. Wu

Authors
  1. James B. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Claire F. Snyder
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Albert W. Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Albert W. Wu.

Additional information

James B. Jones, Claire F. Snyder, and Albert W. Wu for the “Website for Outpatient QOL Assessment” Research Network.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jones, J.B., Snyder, C.F. & Wu, A.W. Issues in the design of Internet-based systems for collecting patient-reported outcomes. Qual Life Res 16, 1407–1417 (2007). https://doi.org/10.1007/s11136-007-9235-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-007-9235-z

Keywords

Search

Navigation