Quality of Life Research

, Volume 28, Issue 2, pp 345–356 | Cite as

Making a picture worth a thousand numbers: recommendations for graphically displaying patient-reported outcomes data

  • Claire SnyderEmail author
  • Katherine Smith
  • Bernhard Holzner
  • Yonaira M. Rivera
  • Elissa Bantug
  • Michael Brundage
  • PRO Data Presentation Delphi Panel



Patient-reported outcomes (PROs) can promote patient-centered care in multiple ways: (1) using an individual patient’s PRO data to inform his/her management, (2) providing PRO results from comparative research studies in patient educational materials/decision aids, and (3) reporting PRO results from comparative research studies in peer-reviewed publications. Patients and clinicians endorse the value of PRO data; however, variations in how PRO measures are scored and scaled, and in how the data are reported, make interpretation challenging and limit their use in clinical practice. We conducted a modified Delphi process to develop stakeholder-engaged, evidence-based recommendations for PRO data display for the three above applications to promote understanding and use.


The Consensus Panel included cancer survivors/caregivers, oncologists, PRO researchers, and application-specific end-users (e.g., electronic health record vendors, decision aid developers, journal editors). We reviewed the data display issues and their evidence base during pre-meeting webinars. We then surveyed participants’ initial perspectives, which informed discussions during an in-person meeting to develop consensus statements. These statements were ratified via a post-meeting survey.


Issues addressed by consensus statements relevant to both individual and research data applications were directionality (whether higher scores are better/worse) and conveying score meaning (e.g., none/mild/moderate/severe). Issues specific to individual patient data presentation included representation (bar charts vs. line graphs) and highlighting possibly concerning scores (absolute and change). Issues specific to research study results presentation included handling normed data, conveying statistically significant differences, illustrating clinically important differences, and displaying proportions improved/stable/worsened.


The recommendations aim to optimize accurate and meaningful interpretation of PRO data.


Patient-reported outcomes Consensus statements Cancer Data display Clinical practice 



The study was funded by the Patient-Centered Outcomes Research Institute (PCORI). Drs. Snyder and Smith are members of the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins (P30CA006973).

PRO Data Presentation Delphi Panel members

The Consensus Panel Participants included the project Stakeholder Advisory Board (SAB) and other invitees, in addition to the project team named authors. The SAB members were Daniel Weber (National Coalition for Cancer Survivorship); Ethan Basch, MD (Lineberger Cancer Center at the University of North Carolina); Neil Aaronson, PhD (Netherlands Cancer Institute); Bryce Reeve, PhD (Duke University); Galina Velikova, BMBS(MD), PhD (University of Leeds, Leeds Institute of Cancer and Pathology); Andrea Heckert, PhD, MPH (Patient-Centered Outcomes Research Institute); Eden Stotsky-Himelfarb (patient representative); Cynthia Chauhan (patient representative); Vanessa Hoffman, MPH (caregiver representative); Patricia Ganz, MD* (UCLA Jonsson Comprehensive Cancer Center); Lisa Barbera, MD, MPA (University of Toronto and Cancer Care Ontario). Additional Consensus Panel participants included Elizabeth Frank* (patient advocate); Mary Lou Smith, JD (patient advocate); Arturo Durazo (patient advocate); Judy Needham (patient advocate); Shelley Fuld Nasso (National Coalition for Cancer Survivorship); Robert Miller, MD (American Society of Clinical Oncology); Tenbroeck Smith, MA (American Cancer Society); Deborah Struth, MSN, RN, PhD(c) (Oncology Nursing Society); Alison Rein, MS (AcademyHealth); Andre Dias, PhD* (International Consortium for Health Outcomes Measurement); Charlotte Roberts, MBBS, BSc (International Consortium for Health Outcomes Measurement); Nancy Smider, PhD* (Epic); Gena Cook* (Navigating Cancer); Jakob Bjorner, MD, PhD* (Optum); Holly Witteman, PhD* (Laval University); James G. Dolan, MD* (University of Rochester); Jane Blazeby, MD, MSc (University of Bristol); Robert M. Golub, MD (JAMA); Christine Laine, MD, MPH (Annals of Internal Medicine); Scott Ramsey, MD, PhD (Fred Hutchinson Cancer Research Center). Additional details on all meeting participants are listed in Appendix Table 3. (*Meeting 1 only; Meeting 2 only.)


This study was funded by the Patient-Centered Outcomes Research Institute (PCORI). Drs. Snyder and Smith are members of the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins (P30CA006973).

Compliance with ethical standards

Conflict of interest

No authors declare a conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. 1.
    U. S. Food and Drug Administration. (2009). Guidance for industry. Patient reported outcome measures: Use in medical product development to support labeling claims. Federal Register, 74(35), 65132–65133.Google Scholar
  2. 2.
    Acquadro, C., Berzon, R., Dubois, D., Leidy, N. K., Marquis, P., Revicki, D., et al. (2003). Incorporating the patient’s perspective into drug development and communication: An ad hoc task force report of the patient-reported outcomes (PRO) harmonization group meeting at the food and drug administration, February 16, 2001. Value Health, 6, 522–531.CrossRefGoogle Scholar
  3. 3.
    Greenhalgh, J. (2009). The applications of PROs in clinical practice: What are they, do they work, and why? Quality Life Research, 18, 115–123.CrossRefGoogle Scholar
  4. 4.
    Snyder, C. F., & Aaronson, N. K. (2009). Use of patient-reported outcomes in clinical practice. Lancet, 374, 369–370.CrossRefGoogle Scholar
  5. 5.
    Jensen, R. E., Snyder, C. F., Abernethy, A. P., Basch, E., Reeve, B. B., Roberts, A., et al. (2014). A review of electronic patient reported outcomes systems used in cancer clinical care. Journal of Oncology Practice, 10, e215–e222.CrossRefGoogle Scholar
  6. 6.
    Till, J. E., Osoba, D., Pater, J. L., & Young, J. R. (1994). Research on health-related quality of life: Dissemination into practical applications. Quality Life Research, 3(4), 279–283.CrossRefGoogle Scholar
  7. 7.
    Au, H.-J., Ringash, J., Brundage, M., Palmer, M., Richardson, H., Meyer, R. M., et al. (2010). Added value of health-related quality of life measurement in cancer clinical trials: The experience of the NCIC CTG. Expert Review of Pharmacoeconomics & Outcomes Research, 10(2), 119–128.CrossRefGoogle Scholar
  8. 8.
    Lipscomb, J., Gotay, C. C., & Snyder, C. (Eds.). (2005). Outcomes assessment in cancer: Measures, methods, and applications. Cambridge: Cambridge Univ Press.Google Scholar
  9. 9.
    Brundage, M., Bass, B., Ringash, J., & Foley, K. (2011). A knowledge translation challenge: Clinical use of quality of life data from cancer clinical trials. Quality Life Research, 20, 979–985.CrossRefGoogle Scholar
  10. 10.
    Bezjak, A., Ng, P., Skeel, R., Depetrillo, A. D., Comis, R., & Taylor, K. M. (2001). Oncologists’ use of quality of life information: Results of a survey of Eastern Cooperative Oncology Group physicians. Quality Life Research, 10, 1–13.CrossRefGoogle Scholar
  11. 11.
    Snyder, C. F., Blackford, A. L., Wolff, A. C., Carducci, M. A., Herman, J. M., Wu, A. W., et al. (2013). Feasibility and value of PatientViewpoint: A web system for patient-reported outcomes assessment in clinical practice. Psychooncology, 22, 895–901.CrossRefGoogle Scholar
  12. 12.
    Brundage, M. D., Smith, K. C., Little, E. A., Bantug, E. T., & Snyder, C. F., PRO Data Presentation Stakeholder Advisory Board (2015). Communicating patient-reported outcome scores using graphic formats: Results from a mixed methods evaluation. Quality Life Research, 24, 2457–2472.CrossRefGoogle Scholar
  13. 13.
    Bantug, E. T., Coles, T., Smith, K. C., Snyder, C. F., Rouette, J., Brundage, M. D., et al. (2016). Graphical displays of patient-reported outcomes (PRO) for use in clinical practice: What makes a PRO picture worth a thousand words? Patient Education and Counselling, 99, 483–490.CrossRefGoogle Scholar
  14. 14.
    Smith, K. C., Brundage, M. D., Tolbert, E., Little, E. A., Bantug, E. T., Snyder, C., et al. (2016). Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice. Support Care Cancer, 24, 4149–4157.CrossRefGoogle Scholar
  15. 15.
    Snyder, C. F., Smith, K. C., Bantug, E. T., Tolbert, E. E., Blackford, A. L., Brundage, M. D., et al. (2017). What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer, 123, 1848–1859.CrossRefGoogle Scholar
  16. 16.
    Tolbert, E., Snyder, C., Bantug, E., Blackford, A., Smith, K., Brundage, M., et al. (2016). Graphing group-level data from research studies for presentation to patients in educational materials and decision aids. Quality Life Research, 15, 17.Google Scholar
  17. 17.
    Brundage, M. D., Blackford, A., Tolbert, E., Smith, K., Bantug, E., Snyder, C., et al. (2018). Presenting comparative study PRO results to clinicians and researchers: Beyond the eye of the beholder. Quality Life Research, 27, 75–90.CrossRefGoogle Scholar
  18. 18.

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Claire Snyder
    • 1
    • 2
    • 3
    Email author
  • Katherine Smith
    • 2
    • 3
  • Bernhard Holzner
    • 4
  • Yonaira M. Rivera
    • 2
  • Elissa Bantug
    • 3
  • Michael Brundage
    • 5
  • PRO Data Presentation Delphi Panel
  1. 1.Johns Hopkins School of MedicineBaltimoreUSA
  2. 2.Johns Hopkins Bloomberg School of Public HealthBaltimoreUSA
  3. 3.Sidney Kimmel Comprehensive Cancer Center at Johns HopkinsBaltimoreUSA
  4. 4.Medical University of InnsbruckInnsbruckAustria
  5. 5.Queens Cancer Research InstituteKingstonCanada

Personalised recommendations