Quality of Life Research

, Volume 16, Issue 8, pp 1407–1417 | Cite as

Issues in the design of Internet-based systems for collecting patient-reported outcomes

  • James B. Jones
  • Claire F. Snyder
  • Albert W. Wu


Background and objectives

Although there is a growing interest in using patient-reported outcomes (PRO) to monitor disease progression and/or therapeutic response, to improve care, and to screen for physical or psychosocial problems in routine clinical practice, PRO instruments can be difficult to administer, score, and interpret in this setting. Internet-based approaches to PRO collection may help overcome these obstacles. This paper discusses the rationale for using the Internet for routine PRO collection, summarizes relevant literature and ongoing projects, and raises several key design and development issues that should guide further efforts in this area.

Major findings

A small number of Internet-based PRO collection applications have been or are currently being developed. The major characteristics of several of these projects are reviewed and summarized. Successful Internet-based PRO collection applications must address patient and clinician-specific needs related to workflow and to the way in which results are presented. A growing number of instruments have been adapted for and evaluated in a web-based format.


Collecting PROs via the Internet has the potential to overcome many of the challenges associated with efforts to routinely use PROs in the clinical encounter.


Electronic health record Internet Patient-reported outcomes Web-based 



Bone Marrow Therapy


Computerized Adaptive Testing


Child Health and Development Interactive System


Common Terminology Criteria for Adverse Events


Diagnostic and Statistical Manual for Primary Care


Electronic Data Collection


Electronic Health Record


EuroQOL 5-D


European Organization for Research and Treatment of Cancer Quality of Life Questionnaire


Fred Hutchinson Cancer Research Center


Hospital Anxiety and Depression Scale


Health Insurance Portability and Accountability Act


Health-Related Quality of Life


Hematopoietic Stem Cell Transplantation


Item Response Theory


Interactive Voice Response System


Memorial Sloan Kettering Cancer Center


National Cancer Institute


Patient Reported Outcome(s)


Patient Reported Outcome Measurement Information System


Supportive Care Needs Survey


  1. 1.
    Ellwood, P. M. (1988). Shattuck lecture–outcomes management. A technology of patient experience. The New England Journal of Medicine, 318, 1549–1556.PubMedCrossRefGoogle Scholar
  2. 2.
    List of 1000 instruments in alphabetical order. Retrieved 8/29, 2006, from http://www.proqolid.org/public/list/a.html.
  3. 3.
    Bielli, E., Carminati, F., La Capra, S., Lina, M., Brunelli, C., & Tamburini, M. (2004). A wireless health outcomes monitoring system (WHOMS): Development and field testing with cancer patients using mobile phones. BMC Medical Informatics and Decision Making, 4, 7.PubMedCrossRefGoogle Scholar
  4. 4.
    Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA, 288, 3027–3034.PubMedCrossRefGoogle Scholar
  5. 5.
    Greenhalgh, J., Long, A. F., & Flynn, R. (2005). The use of patient reported outcome measures in routine clinical practice: Lack of impact or lack of theory? Social Science & Medicine, 60, 833–843.CrossRefGoogle Scholar
  6. 6.
    Donaldson, M. (2006). Using patient-reported outcomes in clinical oncology practice: Benefits, challenges and next steps. Expert Review of Pharmacoeconomics & Outcomes Research, 6, 87–95.CrossRefGoogle Scholar
  7. 7.
    Detmar, S. B., & Aaronson, N. K. (1998). Quality of life assessment in daily clinical oncology practice: A feasibility study. European Journal of Cancer, 34, 1181–1186.PubMedCrossRefGoogle Scholar
  8. 8.
    McLachlan, S. A., Allenby, A., Matthews, J., Wirth, A., Kissane, D., Bishop, M., Beresford, J., & Zalcberg, J. (2001). Randomized trial of coordinated psychosocial interventions based on patient self-assessments versus standard care to improve the psychosocial functioning of patients with cancer. Journal of Clinical Oncology, 19, 4117–4125.PubMedGoogle Scholar
  9. 9.
    Wu, A. W., Cagney, K. A., & St John, P. D. (1997). Health status assessment. Completing the clinical database. Journal of General Internal Medicine, 12, 254–255.PubMedCrossRefGoogle Scholar
  10. 10.
    Velikova, G., Brown, J. M., Smith, A. B., & Selby, P. J. (2002). Computer-based quality of life questionnaires may contribute to doctor-patient interactions in oncology. British Journal of Cancer, 86, 51–59.PubMedCrossRefGoogle Scholar
  11. 11.
    Taenzer, P., Bultz, B. D., Carlson, L. E., Speca, M., DeGagne, T., Olson, K., Doll, R., & Rosberger, Z. (2000). Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psychooncology, 9, 203–213.PubMedCrossRefGoogle Scholar
  12. 12.
    Athale, N., Sturley, A., Skoczen, S., Kavanaugh, A., & Lenert, L. (2004). A web-compatible instrument for measuring self-reported disease activity in arthritis. The Journal of Rheumatology, 31, 223–228.PubMedGoogle Scholar
  13. 13.
    McDaniel, A. M., Benson, P. L., Roesener, G. H., & Martindale, J. (2005). An integrated computer-based system to support nicotine dependence treatment in primary care. Nicotine & Tobacco Research, 7(Suppl 1), S57–S66.CrossRefGoogle Scholar
  14. 14.
    Saleh, K. J., Radosevich, D. M., Kassim, R. A., Moussa, M., Dykes, D., Bottolfson, H., Gioe, T. J., & Robinson, H. (2002). Comparison of commonly used orthopaedic outcome measures using palm-top computers and paper surveys. Journal of Orthopaedic Research, 20, 1146–1151.PubMedCrossRefGoogle Scholar
  15. 15.
    Anhoj, J., & Moldrup, C. (2004). Feasibility of collecting diary data from asthma patients through mobile phones and SMS (short message service): Response rate analysis and focus group evaluation from a pilot study. Journal of Medical Internet Research, 6, e42.PubMedCrossRefGoogle Scholar
  16. 16.
    Crawley, J. A., Kleinman, L., & Dominitz, J. (2000). User preferences for computer administration of quality of life instruments. Drug Information Journal, 34, 137.Google Scholar
  17. 17.
    Jacobsen, P. B., Davis, K., & Cella, D. (2002). Assessing quality of life in research and clinical practice. Oncology (Williston.Park), 16, 133–139.Google Scholar
  18. 18.
    Bliven, B. D., Kaufman, S. E., & Spertus, J. A. (2001). Electronic collection of health-related quality of life data: Validity, time benefits, and patient preference. Quality of Life Research, 10, 15–22.PubMedCrossRefGoogle Scholar
  19. 19.
    Wright, E. P., Selby, P. J., Crawford, M., Gillibrand, A., Johnston, C., Perren, T. J., Rush, R., Smith, A., Velikova, G., Watson, K., Gould, A., & Cull, A. (2003). Feasibility and compliance of automated measurement of quality of life in oncology practice. Journal of Clinical Oncology, 21, 374–382.PubMedCrossRefGoogle Scholar
  20. 20.
    Bushnell, D. M., Martin, M. L., & Parasuraman, B. (2003). Electronic versus paper questionnaires: A further comparison in persons with asthma. The Journal of Asthma, 40, 751–762.PubMedCrossRefGoogle Scholar
  21. 21.
    Bayliss, M. S., Dewey, J. E., Dunlap, I., Batenhorst, A. S., Cady, R., Diamond, M. L., & Sheftell, F. (2003). A study of the feasibility of internet administration of a computerized health survey: The headache impact test (HIT). Quality of Life Research, 12, 953–961.PubMedCrossRefGoogle Scholar
  22. 22.
    Pealer, L. N., Weiler, R. M., Pigg, R. M. Jr, Miller, D., & Dorman, S. M. (2001). The feasibility of a web-based surveillance system to collect health risk behavior data from college students. Health Education & Behavior, 28, 547–559.CrossRefGoogle Scholar
  23. 23.
    Hanscom, B., Lurie, J. D., Homa, K., & Weinstein, J. N. (2002). Computerized questionnaires and the quality of survey data. Spine, 27, 1797–1801.PubMedCrossRefGoogle Scholar
  24. 24.
    Wilson, A. S., Kitas, G. D., Carruthers, D. M., Reay, C., Skan, J., Harris, S., Treharne, G. J., Young, S. P., & Bacon, P. A. (2002). Computerized information-gathering in specialist rheumatology clinics: An initial evaluation of an electronic version of the short form 36. Rheumatology (Oxford), 41, 268–273.CrossRefGoogle Scholar
  25. 25.
    Bush, N., Donaldson, G., Moinpour, C., Haberman, M., Milliken, D., Markle, V., & Lauson, J. (2005). Development, feasibility and compliance of a web-based system for very frequent QOL and symptom home self-assessment after hematopoietic stem cell transplantation. Quality of Life Research, 14, 77–93.PubMedCrossRefGoogle Scholar
  26. 26.
    Boyes, A., Newell, S., & Girgis, A. (2002). Rapid assessment of psychosocial well-being: Are computers the way forward in a clinical setting? Quality of Life Research, 11, 27–35.PubMedCrossRefGoogle Scholar
  27. 27.
    Hambleton, R. K. (2005). Applications of item response theory to improve health outcomes assessment: developing item banks, linking instruments, and computer-adaptive testing. In J. Lipscomb, C. C. Gotay, & C. Snyder (Eds.), Outcomes assessment in cancer: Measures, methods and applications (pp. 445–464). Cambridge: Cambridge University Press.Google Scholar
  28. 28.
    Litaker, D. (2003). New technology in quality of life research: Are all computer-assisted approaches created equal? Quality of Life Research, 12, 387–393.PubMedCrossRefGoogle Scholar
  29. 29.
    Velikova, G., Wright, E. P., Smith, A. B., Cull, A., Gould, A., Forman, D., Perren, T., Stead, M., Brown, J., & Selby, P. J. (1999). Automated collection of quality-of-life data: A comparison of paper and computer touch-screen questionnaires. The Journal of Clinical Oncology, 17, 998–1007.Google Scholar
  30. 30.
    More Americans turning to web for health information, poll shows. The Wall Street Journal Online (2006).Google Scholar
  31. 31.
    Liederman, E. M., & Morefield, C. S. (2003). Web messaging: A new tool for patient-physician communication. Journal of the American Medical Informatics Association, 10, 260–270.PubMedCrossRefGoogle Scholar
  32. 32.
    Casebeer, L., Bennett, N., Kristofco, R., Carillo, A., & Centor, R. (2002). Physician internet medical information seeking and on-line continuing education use patterns. The Journal of Continuing Education in the Health Professions, 22, 33–42.PubMedCrossRefGoogle Scholar
  33. 33.
    Bennett, N. L., Casebeer, L. L., Kristofco, R. E., & Strasser, S. M. (2004). Physicians’ internet information-seeking behaviors. Journal of Continuing Education in the Health Professions, 24, 31–38.PubMedCrossRefGoogle Scholar
  34. 34.
    Howsyourhealth.org. from www.howsyourhealth.org.Google Scholar
  35. 35.
    Child health and development interactive system (CHADIS) & CHADIS EMR. Retrieved 11/2, 2006, from http://www.childhealthcare.org/marketing/datasheet.shtml.
  36. 36.
    Sturner, R., Morrel, T., & Howard, B. J. (2004). Mental health diagnoses among children being seen for child health supervision visits: Typical practice and DSM-PC diagnoses. Presented at Pediatric Academic Society Annual Meeting, 441.Google Scholar
  37. 37.
    Sturner, R., Morrel, T., & Howard, B. (2005). Preliminary validation of the DSM-PC. Journal of Developmental and Behavioral Pediatrics, 24, 393–394.Google Scholar
  38. 38.
    Sturner, R., Howard, B. J., Morrel, T., & Rogers-Senuta, K. (2003). Validation of a computerized parent questionnaire for identifying child mental health disorders and implementing DSM-PC. Presented at Pediatric Academic Society Annual Meeting, 3801.Google Scholar
  39. 39.
    Basch, E., Artz, D., Dulko, D., Scher, K., Sabbatini, P., Hensley, M., Mitra, N., Speakman, J., McCabe, M., & Schrag, D. (2005). Patient online self-reporting of toxicity symptoms during chemotherapy. Journal of Clinical Oncology, 23, 3552–3561.PubMedCrossRefGoogle Scholar
  40. 40.
    Cohen, S. R., Boston, P., Mount, B. M., & Porterfield, P. (2001). Changes in quality of life following admission to palliative care units. Palliative Medicine, 15, 363–371.PubMedCrossRefGoogle Scholar
  41. 41.
    Cull, A., Gould, A., House, A., Smith, A., Strong, V., Velikova, G., Wright, P., & Selby, P. (2001). Validating automated screening for psychological distress by means of computer touchscreens for use in routine oncology practice. British Journal of Cancer, 85, 1842–1849.PubMedCrossRefGoogle Scholar
  42. 42.
    Taenzer, P. A., Speca, M., Atkinson, M. J., Bultz, B. D., Page, S., Harasym, P., & Davis, J. L. (1997). Computerized quality-of-life screening in an oncology clinic. Cancer Practice, 5, 168–175.PubMedGoogle Scholar
  43. 43.
    Hodge, J. G. Jr. (2003). Health information privacy and public health. The Journal of Law, Medicine & Ethics, 31, 663–671.CrossRefGoogle Scholar
  44. 44.
    Lumpkin, J. R. (2000). e-Health, HIPAA, and beyond. Health Affairs (Millwood) 19, 149–151.CrossRefGoogle Scholar
  45. 45.
    Garner, J. C. (2003). Final HIPAA security regulations: A review. Managed Care Quarterly, 11, 15–27.PubMedGoogle Scholar
  46. 46.
    Fairfield, K. M., Chen, W. Y., Colditz, G. A., Emmons, K. M., & Fletcher, S. W. (2004). Colon cancer risk counseling by health-care providers: Perceived barriers and response to an internet-based cancer risk appraisal instrument. Journal of Cancer Education, 19, 95–97.PubMedCrossRefGoogle Scholar
  47. 47.
    Sittig, D. F. (2003). Results of a content analysis of electronic messages (email) sent between patients and their physicians. BMC Medical Informatics and Decision Making, 3, 11–15.PubMedCrossRefGoogle Scholar
  48. 48.
    Katz, S. J., Nissan, N., & Moyer, C. A. (2004). Crossing the digital divide: Evaluating online communication between patients and their providers. The American Journal of Managed Care, 10, 593–598.PubMedGoogle Scholar
  49. 49.
    Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16, 133–141.PubMedCrossRefGoogle Scholar
  50. 50.
    Thissen, D., Reeve, B. B., Bjorner, J. B., & Chang, C. H. (2007). Methodological issues for building item banks and computerized adaptive scales. Quality of Life Research, 16, 109–119.CrossRefGoogle Scholar
  51. 51.
    Berry, D. L., Trigg, L. J., Lober, W. B., Karras, B. T., Galligan, M. L., Austin-Seymour, M., & Martin, S. (2004). Computerized symptom and quality-of-life assessment for patients with cancer part I: Development and pilot testing. Oncology Nursing Forum, 31, E75–E83.PubMedCrossRefGoogle Scholar
  52. 52.
    Bensley, R. J., & Lewis, J. B. (2002). Analysis of internet-based health assessments. Health Promotion Practice, 3, 463–476.CrossRefGoogle Scholar
  53. 53.
    Mullen, K. H., Berry, D. L., & Zierler, B. K. (2004). Computerized symptom and quality-of-life assessment for patients with cancer part II: Acceptability and usability. Oncology Nursing Forum, 31, E84–E89.PubMedCrossRefGoogle Scholar
  54. 54.
    NCI announces new research-based web design guidelines for improving health information web sites. 2005. Retrieved 11/2, 2006, from http://www.cancer.gov/newscenter/usabilityguidelines.
  55. 55.
    Williams, P., Nicholas, D., Huntington, P., & McLean, F. (2002). Surfing for health: User evaluation of a health information website. part one: Background and literature review. Health Information and Libraries Journal, 19, 98–108.PubMedCrossRefGoogle Scholar
  56. 56.
    Brundage, M., Feldman-Stewart, D., Leis, A., Bezjak, A., Degner, L., Velji, K., Zetes-Zanatta, L., Tu, D., Ritvo, P., & Pater, J. (2005). Communicating quality of life information to cancer patients: A study of six presentation formats. Journal of Clinical Oncology, 23, 6949–6956.PubMedCrossRefGoogle Scholar
  57. 57.
    Bjorner, J. B., Chang, C. H., Thissen, D., & Reeve, B. B. (2007). Developing tailored instruments: Item banking and computerized adaptive assessment. Quality of Life Research, 16, 95–108.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2007

Authors and Affiliations

  • James B. Jones
    • 1
    • 2
  • Claire F. Snyder
    • 1
    • 3
  • Albert W. Wu
    • 1
    • 3
  1. 1.Department of Health Policy and Management, Bloomberg School of Public HealthThe Johns Hopkins UniversityBaltimoreUSA
  2. 2.Center for Health ResearchGeisinger Medical CenterDanvilleUSA
  3. 3.Department of Medicine, School of MedicineThe Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations