Quality of Life Research

, Volume 25, Issue 3, pp 575–583 | Cite as

Evaluation of point-of-care PRO assessment in clinic settings: integration, parallel-forms reliability, and patient acceptability of electronic QOL measures during clinic visits

  • Pranav Sharma
  • Rodney L. Dunn
  • John T. Wei
  • James E. Montie
  • Scott M. Gilbert
Special Section: PROs in Non-Standard Settings (by invitation only)



Assessment of patient-reported outcomes (PROs), such as health-related quality of life, has become an important component of healthcare that measures the impact of disease and medical treatment on patient health. Collecting PROs during point-of-care assessments and integrating them into the clinical setting, however, remains challenging. The objective of this pilot study was to evaluate the reliability, usability, and acceptability of point-of-care electronic PRO assessments implemented in a prostate cancer clinic.


Fifty subjects completed paper–pencil and computerized formats of the Expanded Prostate Cancer Index Composite (EPIC), a validated, condition-specific QOL instrument, at separate times before treatment. Parallel-forms reliability was evaluated by comparing mean scores, variations in response distribution, and correlations between administration formats. Correlation coefficients of at least 0.70 were used for reliability testing. Differences between administration forms, indicating potential bias, were compared using the signed-rank test. A 6-item acceptability scale was also used to evaluate patient acceptability and satisfaction with the electronic format.


Mean scores and standard deviations were similar between the paper–pencil and electronic forms across all EPIC instrument domains, and no assessment bias was found. Each EPIC domain demonstrated a high reliability between administration formats (correlation coefficients: 0.70–0.98). The majority (>90 %) of respondents found that the computerized QOL format was user friendly and simple to use.


Point-of-care computerized QOL assessments were reliable and acceptable to patients in this study, supporting the feasibility of PRO integration at the point-of-care in clinical settings.


Electronic questionnaire Quality of life Patient-reported outcomes Clinical tools 



Supported, in part, by the American Urological Association Foundation.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.


  1. 1.
    Santana, M. J., Haverman, L., Absolom, K., Takeuchi, E., Feeny, D., Grootenhuis, M., & Velikova, G. (2015). Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Quality of Life Research, 24(7), 1707–1718. CrossRefPubMedGoogle Scholar
  2. 2.
    Detmar, S. B., & Aaronson, N. K. (1998). Quality of life assessment in daily clinical oncology practice: A feasibility study. European Journal of Cancer, 34(8), 1181–1186.CrossRefPubMedGoogle Scholar
  3. 3.
    Velikova, G., Booth, L., Smith, A. B., Brown, P. M., Lynch, P., Brown, J. M., & Selby, P. J. (2004). Measuring quality of life in routine oncology practice improves communication and patient well-being: A randomized controlled trial. Journal of Clinical Oncology, 22(4), 714–724.CrossRefPubMedGoogle Scholar
  4. 4.
    Jensen, R. E., Snyder, C. F., Abernethy, A. P., Basch, E., Potosky, A. L., Roberts, A. C., et al. (2014). Review of electronic patient-reported outcomes systems used in cancer clinical care. Journal of Oncology Practice, 10(4), e215–e222.PubMedCentralCrossRefPubMedGoogle Scholar
  5. 5.
    Wu, A. W., Kharrazi, H., Boulware, L. E., & Snyder, C. F. (2013). Measure once, cut twice—Adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. Journal of Clinical Epidemiology, 66(8 Suppl), S12–S20.PubMedCentralCrossRefPubMedGoogle Scholar
  6. 6.
    Silver, S. M. (2008). Cancer care for the whole patient-a new institute of medicine report. Journal of Oncology Practice, 4(3), 131.PubMedCentralCrossRefPubMedGoogle Scholar
  7. 7.
    Deyo, R. A., & Patrick, D. L. (1989). Barriers to the use of health status measures in clinical investigation, patient care, and policy research. Medical Care, 27(3 Suppl), S254–S268.CrossRefPubMedGoogle Scholar
  8. 8.
    Hjollund, N. H., Larsen, L. P., Biering, K., Johnsen, S. P., Riiskjaer, E., & Schougaard, L. M. (2014). Use of patient-reported outcome (PRO) measures at group and patient levels: Experiences from the generic integrated PRO system, WestChronic. Interactive Journal of Medical Research, 3(1), e5.PubMedCentralCrossRefPubMedGoogle Scholar
  9. 9.
    Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health, 11(2), 322–333.CrossRefPubMedGoogle Scholar
  10. 10.
    Velikova, G. (2004). Use of electronic quality of life applications in cancer research and clinical practice. Expert Review of Pharmacoeconomics & Outcomes Research, 4(4), 403–411.CrossRefPubMedGoogle Scholar
  11. 11.
    Velikova, G., Brown, J. M., Smith, A. B., & Selby, P. J. (2002). Computer-based quality of life questionnaires may contribute to doctor-patient interactions in oncology. British Journal of Cancer, 86(1), 51–59.PubMedCentralCrossRefPubMedGoogle Scholar
  12. 12.
    Davis, K. M., Kelly, S. P., Luta, G., Tomko, C., Miller, A. B., & Taylor, K. L. (2014). The association of long-term treatment-related side effects with cancer-specific and general quality of life among prostate cancer survivors. Urology, 84(2), 300–306.CrossRefPubMedGoogle Scholar
  13. 13.
    Hamoen, E. H., De Rooij, M., Witjes, J. A., Barentsz, J. O., & Rovers, M. M. (2015). Measuring health-related quality of life in men with prostate cancer: A systematic review of the most used questionnaires and their validity. Urologic Oncology, 33(2), 69-e19–69-e28.CrossRefGoogle Scholar
  14. 14.
    Montague, D. K., Barada, J. H., Belker, A. M., Levine, L. A., Nadig, P. W., Roehrborn, C. G., et al. (1996). Clinical guidelines panel on erectile dysfunction: Summary report on the treatment of organic erectile dysfunction. The American Urological Association. The Journal of Urology, 156(6), 2007–2011.CrossRefPubMedGoogle Scholar
  15. 15.
    Rosen, R. C., Riley, A., Wagner, G., Osterloh, I. H., Kirkpatrick, J., & Mishra, A. (1997). The international index of erectile function (IIEF): A multidimensional scale for assessment of erectile dysfunction. Urology, 49(6), 822–830.CrossRefPubMedGoogle Scholar
  16. 16.
    Litwin, M. S., Hays, R. D., Fink, A., Ganz, P. A., Leake, B., & Brook, R. H. (1998). The UCLA Prostate Cancer Index: Development, reliability, and validity of a health-related quality of life measure. Medical Care, 36(7), 1002–1012.CrossRefPubMedGoogle Scholar
  17. 17.
    Wei, J. T., Dunn, R. L., Litwin, M. S., Sandler, H. M., & Sanda, M. G. (2000). Development and validation of the expanded prostate cancer index composite (EPIC) for comprehensive assessment of health-related quality of life in men with prostate cancer. Urology, 56(6), 899–905.CrossRefPubMedGoogle Scholar
  18. 18.
    Gilbert, S. M., Dunn, R. L., Wittmann, D., Montgomery, J. S., Hollingsworth, J. M., Miller, D. C., et al. (2015). Quality of life and satisfaction among prostate cancer patients followed in a dedicated survivorship clinic. Cancer, 121(9), 1484–1491.CrossRefPubMedGoogle Scholar
  19. 19.
    Chang, P., Szymanski, K. M., Dunn, R. L., Chipman, J. J., Litwin, M. S., Nguyen, P. L., et al. (2011). Expanded prostate cancer index composite for clinical practice: Development and validation of a practical health related quality of life instrument for use in the routine clinical care of patients with prostate cancer. Journal of Urology, 186(3), 865–872.CrossRefPubMedGoogle Scholar
  20. 20.
    Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human–Computer Interaction, 7(1), 57–78.CrossRefGoogle Scholar
  21. 21.
    Blum, D., Raj, S. X., Oberholzer, R., Riphagen, II, Strasser, F., Kaasa, S., & Euro Impact, E. I. M. P. C. R. T. (2014). Computer-based clinical decision support systems and patient-reported outcomes: A systematic review. The Patient-Patient-Centered Outcomes Research. doi: 10.1007/s40271-014-0100-1.
  22. 22.
    Tariman, J. D., Berry, D. L., Halpenny, B., Wolpin, S., & Schepp, K. (2011). Validation and testing of the acceptability E-scale for web-based patient-reported outcomes in cancer care. Applied Nursing Research, 24(1), 53–58.PubMedCentralCrossRefPubMedGoogle Scholar
  23. 23.
    Mohler, J., Bahnson, R. R., Boston, B., Busby, J. E., D’Amico, A., Eastham, J. A., et al. (2010). NCCN clinical practice guidelines in oncology: Prostate cancer. Journal of the National Comprehensive Cancer Network, 8(2), 162–200.PubMedGoogle Scholar
  24. 24.
    Basch, E., & Abernethy, A. P. (2011). Supporting clinical practice decisions with real-time patient-reported outcomes. Journal of Clinical Oncology, 29(8), 954–956.CrossRefPubMedGoogle Scholar
  25. 25.
    Efficace, F., Jacobs, M., Pusic, A., Greimel, E., Piciocchi, A., Kieffer, J. M., et al. (2014). Patient-reported outcomes in randomised controlled trials of gynaecological cancers: Investigating methodological quality and impact on clinical decision-making. European Journal of Cancer, 50(11), 1925–1941.CrossRefPubMedGoogle Scholar
  26. 26.
    Basch, E., Artz, D., Dulko, D., Scher, K., Sabbatini, P., Hensley, M., et al. (2005). Patient online self-reporting of toxicity symptoms during chemotherapy. Journal of Clinical Oncology, 23(15), 3552–3561.CrossRefPubMedGoogle Scholar
  27. 27.
    Velikova, G., Wright, E. P., Smith, A. B., Cull, A., Gould, A., Forman, D., et al. (1999). Automated collection of quality-of-life data: A comparison of paper and computer touch-screen questionnaires. Journal of Clinical Oncology, 17(3), 998–1007.PubMedGoogle Scholar
  28. 28.
    Rutherford, C., Mercieca-Bebber, R., Rice, H., Costa, D., & King, M. (2014). Mode of administration of patient-reported outcomes (pro) measures: A systematic review. In Paper presented at the AsiaPacific Journal of Clinical Oncology.Google Scholar
  29. 29.
    Vickers, A. J., Savage, C. J., Shouery, M., Eastham, J. A., Scardino, P. T., & Basch, E. M. (2010). Validation study of a web-based assessment of functional recovery after radical prostatectomy. Health Quality Life Outcomes, 8, 82.Google Scholar
  30. 30.
    Abernethy, A. P., Herndon, J. E, 2nd, Wheeler, J. L., Day, J. M., Hood, L., Patwardhan, M., et al. (2009). Feasibility and acceptability to patients of a longitudinal system for evaluating cancer-related symptoms and quality of life: Pilot study of an e/Tablet data-collection system in academic oncology. Journal of Pain and Symptom Management, 37(6), 1027–1038.CrossRefPubMedGoogle Scholar
  31. 31.
    McCann, L., Maguire, R., Miller, M., & Kearney, N. (2009). Patients’ perceptions and experiences of using a mobile phone-based advanced symptom management system (ASyMS) to monitor and manage chemotherapy related toxicity. European Journal of Cancer Care, 18(2), 156–164.CrossRefPubMedGoogle Scholar
  32. 32.
    Litwin, M. S., & McGuigan, K. A. (1999). Accuracy of recall in health-related quality-of-life assessment among men treated for prostate cancer. Journal of Clinical Oncology, 17(9), 2882–2888.PubMedGoogle Scholar
  33. 33.
    Judson, T. J., Bennett, A. V., Rogak, L. J., Sit, L., Barz, A., Kris, M. G., et al. (2013). Feasibility of long-term patient self-reporting of toxicities from home via the Internet during routine chemotherapy. Journal of Clinical Oncology, 31(20), 2580–2585.PubMedCentralCrossRefPubMedGoogle Scholar
  34. 34.
    Nama, V., Nordin, A., & Bryant, A. (2013). Patient-reported outcome measures for follow-up after gynaecological cancer treatment. Cochrane Database System Review, 11, CD010299.Google Scholar
  35. 35.
    Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44(11 Suppl 3), S78–S94.PubMedCentralCrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Pranav Sharma
    • 1
  • Rodney L. Dunn
    • 2
  • John T. Wei
    • 2
  • James E. Montie
    • 2
  • Scott M. Gilbert
    • 1
    • 3
  1. 1.Genitourinary Oncology ProgramH. Lee Moffitt Cancer Center & Research InstituteTampaUSA
  2. 2.Department of UrologyUniversity of MichiganAnn ArborUSA
  3. 3.Health Outcomes and Behavior Program, Department of Genitourinary OncologyH. Lee Moffitt Cancer Center & Research InstituteTampaUSA

Personalised recommendations