INTRODUCTION

Quality improvement is an important goal for learning health systems, which have increasingly acknowledged the role of patient contributions in closing critical quality and performance gaps.1 Some health systems, including those administered by the U.S. Department of Veterans Affairs (VA),2 have established patient advisory groups that provide input on research and quality improvement priorities, best practices, and development of interventions.3–5 Recognizing the importance of appropriately balancing the input of families and caregivers with other health system stakeholders, we undertook an effort to incorporate patient and family input in quality measurement and improvement.

Historically, the development and prioritization of quality measures has relied on the input of experts in clinical medicine, while incorporating patient voices is more novel.6, 7 One effort incorporating input from both clinicians and patients underscored the importance of including both kinds of stakeholders, with patients placing higher priority on measures of care access and communication compared to clinicians.8, 9 Clearly, patients and their caregivers bring a distinct voice from professionals to the exercise of standard setting.10, 11 Patients and caregivers can potentially contribute to implementation as well, but potential for health systems to more fully realize this framework is unclear.5

Palliative cancer care is an optimal area to explore the role of patients and caregivers in health system quality improvement because palliative cancer care decisions should be driven by patient preferences and goals. Using an approach modified from the UCLA-RAND Appropriateness Method, we endeavored to examine priorities for palliative cancer care quality measurement from the perspectives of people with experience as cancer patients or cancer caregivers.

METHODS

The ImPACS Project

We conducted this work as part of the Improving Palliative Measurement Application with Computer-Assisted Abstraction (ImPACS) Project (VA HSR&D Investigator-Initiated Research [IIR] 17-277). ImPACS explores how VA can integrate the experience and priorities of cancer patient and cancer caregivers into system-wide quality measurement and improvement initiatives, including among dual users of VA and non-VA care. This research was approved by the VA Palo Alto and Stanford University, VA Greater Los Angeles, and Dana-Farber Cancer Institute institutional review boards.

Overview of the Patient-Caregiver Panel Process

We convened a panel of cancer patients and caregivers to identify priorities for palliative cancer care quality measurement and improvement by rating and prioritizing process quality measures. We included caregivers in the panel because in national surveys, seriously ill patients indicate that impact of their choices on caregivers is a major quality-of-life-relevant concern.12 Patient and caregiver panelists participated in a two-stage modified RAND-UCLA Appropriateness Method.13 This method has been previously used to generate quantitative ratings by expert stakeholders of available palliative cancer care quality measures.14

In contrast to traditional appropriateness panels where similar ratings of items would not be a problem, similar ratings of all items in this prioritization exercise would not be a desirable outcome because a clear set of priorities would not emerge. Anticipating a scenario in which panelists might rate all items similarly, we asked panelists to nominate a set of five most important (“top 5”) measure concepts from the set15 in addition to the numeric ratings.

We solicited input from the VA Palo Alto Healthcare System Patient and Family Advisory Council (PFAC) for Research to tailor our communication with patients and caregivers, and we worked with a patient panelist chair with experience serving on national quality of care panels. We created detailed non-technical explanations of the factual background (e.g., care gaps) in the form of a monograph that summarized each measure concept with a description of supporting evidence. We also decided to meet in person rather than remotely due to complexity of content and panel discussions.

Panel Participants

We identified individuals with personal experience as a cancer patient or cancer caregiver via referral and a snowball sampling process following a maximal variation approach. We solicited referrals to patients and caregivers from VA clinicians, including oncologists and multidisciplinary palliative care clinicians, VA program leaders, and members of the study team across the USA. We aimed to recruit patients and caregivers representing geographic and demographic diversity, cancer diagnoses and stages, and cancer experiences (e.g., curative and palliative care). Our desired panel size was nine participants to promote diversity while permitting all members to participate in conversations.16 Panelists had to be able to travel for a one-day, in-person meeting in Santa Monica, CA. Each panelist participated in a brief phone interview with a study team member (KG or KL) to confirm biographic details (e.g., cancer experience), expectations for participation, and study goals. Participants’ travel expenses were covered, but they were not otherwise compensated. They are listed as an author group on the manuscript and by name in the acknowledgements.

Overview of Measure Concepts Rated by Patient-Caregiver Panelists

To identify measure concepts, we scanned the literature for process measures relevant to palliative and end-of-life cancer care.17 Briefly, we followed the search strategy from a previous systematic review by Kamal and colleagues18 to identify 226 individual process measures from 23 measure sources, which we grouped into 64 measure concepts. A panel of palliative care expert clinician stakeholders reviewed evidence, discussed, and rated the measure concepts in an expert stakeholder RAND-UCLA Appropriateness Panel process in September 2019.19

We used these data to create a shortlist of 20 measure concepts (Table 1) that we anticipated would be feasible for knowledgeable lay patients and caregivers to consider in a 1-day panel. Included measure concepts had a median rating of 7 or greater and were nominated by at least one expert clinician panelist as a “top 5” measure concept with respect to either of two criteria: importance to providing patient- and family-centered care and potential for improving quality of palliative and end-of-life cancer care.19 We followed this approach to identify measure concepts that shared a high priority among both VA leaders and cancer patients and caregivers because we anticipated operationalizing 10 or fewer measures and we were focused on eventual measure implementation. Although we considered discussing all measure concepts, we were concerned about this option’s feasibility based on our experience with the expert stakeholder panel and discussions with the patient and caregiver panel chair.

Table 1 Twenty-One Measure Concepts Considered by the Patient and Caregiver Panel
Full size table

Preparation of Patient-Friendly Written Materials

In advance of the panel discussion, we provided patients and caregivers with written materials modified from the expert stakeholder panel. Materials included a table explaining the measure concepts and a monograph summarizing measure concepts and their relevant evidence. We iteratively rewrote these materials into lay language, incorporating feedback from the panel chair, a VA patient who had previously served as a lay representative for quality measurement (JB), and the panel moderator, a physician member of the study team with extensive panel experience (SA). We asked patients to read the measure concept table (Supplementary Appendix Table) and evidence monograph (available from the authors upon request). We made the original expert stakeholder versions of these documents available to participants as an optional resource.

Orientation to Written Materials

Panelists participated in a 1-hour, one-on-one phone conversation with a study team member (KG or RG) to orient them to the packet materials, discuss each measure concept and how to complete the rating sheet, answer questions, and encourage panelists to reach out for additional support if needed. Finally, the study team member described the panel process. Study team members proactively primed participants for their role as a representative of other patients with advanced cancer or at the end of life, encouraging them to speak on behalf of other patients who may be too ill to speak for themselves and explicitly inviting them to express disagreement with other panelists.

Panel Ratings, Top 5 Nominations, Additional Measure Concepts

Before and during the in-person meeting, panelists rated each measure concept on a discrete scale from 1 to 9 on “importance to cancer patients and their caregivers” (1 being “not important” and 9 being “very important”). To further distinguish between measures with similarly high ratings, panelists also nominated a set of five most important (“top 5”) measure concepts.15 Finally, patient and caregivers were asked to review the original list of 64 measure concepts presented to expert stakeholders, and note any measure concepts that were even more important than the 20 they had been asked to rate. They were also asked if there were additional important concepts that we had not considered.

In-Person Patient and Caregiver Panel Meeting

The 1-day, in-person panel meeting occurred on March 5, 2020, at the RAND Corporation in Santa Monica, CA. To facilitate discussions focused on understanding different perspectives, panelists were given a summary of their own initial ratings, anonymized counts of other panelists’ ratings, median ratings, and measures of dispersion for each rating (mean absolute deviation [MAD] from the median rating). The panel chair and moderator emphasized the importance of using the ratings for prioritization, and that using the full range of numbers would be helpful for achieving that objective. Each measure concept was discussed separately, focusing on areas of disagreement, and then rated again individually by each panelist. No attempt was made to force the panel to achieve consensus around prioritization. Participants also discussed additional measure concepts and ideas for which there were no measures. At the end of the meeting, panelists submitted their choices of their “top 5” most important measure concepts. We digitally recorded the meeting.

Analyses

We determined each concept’s median rating, mean average deviation (MAD) from the median, and pre- and post-panel rating differences in medians. The number of times panelists nominated measure concepts into the most important “top 5” was counted.

RESULTS

Patient/Caregiver Panel Composition

Our study team approached 14 potential panel members. Two never responded and 12 agreed to participate. Three dropped out due to unanticipated health and personal issues, yielding nine panel participants. The panel included seven men and two women, aged 43 to 72 years, one of whom identified as a racial/ethnic minority. Panelists lived in eight states representing different regions: Michigan, Massachusetts, California, Florida, New Jersey, Washington, Wisconsin, and Virginia. The panel was chaired by a patient participant, and a physician member study team member with experience moderating appropriateness panels (SA). Although the requirement to travel limited our ability to recruit patients with advanced cancer, three participants had current advanced cancer and one was a caregiver whose spouse died from cancer. Prostate cancer and lung cancer were the most common cancers among participants, aligning with VA rates generally.20 Other panelists had experiences with ongoing or past surgery, chemotherapy, and radiation treatment. Even though we recruited panel members based on our knowledge of their experience as patients and/or caregivers, participants drew on their experiences with health care systems as both patients and caregivers in the context of cancer and non-cancer illness and injury during discussions.

Pre-panel Ratings

In the pre-panel ratings, panelists rated all measure concepts as highly important (median rating ≥ 7); none were rated as medium importance (median rating greater than ≥ 4 and < 7) or low importance (median rating < 4) (Table 2). Five of 20 had high dispersion (MAD ≥ 1); mean MAD was 0.7. Patients and caregivers nominated a total of 14 of 20 measure concepts in their top five. Twelve were nominated by more than one patient or caregiver.

Table 2 Patient and Caregiver Panel Ratings and Top Five Nominations of 21 Measure Concepts.
Full size table

Post-panel Ratings

After discussion during the panel meeting, panelists completed the second round of ratings. As before, panelists rated all measure concepts (20 of 20) as highly important (median rating 7 or above), with none receiving medium or low ratings. Again, five of 20 had high dispersion (MAD ≥ 1), though only two of those also had high dispersion in the pre-panel ratings. The mean MAD was similar to the pre-panel ratings (0.8). Patients and caregivers nominated a total of 15 of 21 measure concepts (the original 20, plus the additional measure concept pain screening) in their sets of the top five most important measure concepts. Thirteen measure concepts were nominated by more than one patient or caregiver.

Additional Measure Concept

Following rating of measure concepts in the monograph, open discussion and broad agreement among patients and caregivers resulted in the study team adding one measure concept (pain screening) from the original list of 64 measure concepts back into consideration, bringing the total number of considered measure concepts at the in-person meeting to 21. This measure concept received a median (MAD) rating of 9 (0.7) and was nominated as a most important measure by two panelists.

Comparison of Pre-panel Ratings to Post-panel Ratings

Discussion generally reduced ratings of importance (11 of 20 median ratings decreased, 7 of 20 were the same, and 2 of 20 increased). The largest increases and decreases in medians were + 1 (family meetings and MRI for spinal cord compression) and − 2 (advance directive and following treatment preferences).

DISCUSSION

Although the participants in our patient and caregiver panel indicated that all the palliative care quality measures presented were important to the quality of life of patients and caregivers with advanced cancer, this process helped distill the most important quality measure concepts and potential priorities for quality improvement efforts. Given that all the measure concepts had been previously rated as important to providing patient- and family-centered care by expert clinician stakeholders, it is unsurprising that numerical ratings were overall similar across measure concepts. Because numerical ratings were similar, a forced choice nomination process, which in our case was asking patients and caregivers to nominate the five most important measure concepts, was crucial to emphasizing the highest priority measure concepts. The forced choice nomination process emphasized the special importance of physical and psychosocial symptoms to participants in the patient and caregiver panel compared to the expert stakeholder panel, especially with respect to mental health care (comprehensive assessments, which include emotional needs, and depression management) and pain.

The patient and caregiver panel placed special emphasis on the importance of routine pain screening, management, and follow-up. The weight patients and caregivers placed on pain screening was notable, as it was not among the most important measures prioritized by the expert stakeholder panel.19 Despite longstanding evidence of the burden of pain on patients and caregivers,21 pain screening has become controversial in recent years for its contribution to opioid over-prescribing in the USA.22, 23 In VA, pain quality gaps persist; even when pain is identified through screening, it is often ignored.24 Other recent VA studies have found that patients believe pain screening is critical to identifying pain and symptoms that might not be identified unless solicited directly.25

Among limitations, a small panel recruited through referrals from VA providers cannot comprehensively or proportionally represent all potentially important demographic groups. Our panel was relatively homogeneous, including few women and people of color. Our requirement for panelists to attend the 1-day panel meeting in person may have caused people with full-time employment, disabilities, or caregiving responsibilities to self-exclude from participation, especially in the absence of compensation. However, panel approaches in quality improvement like this aim to promote consensus through nuanced discussion of both evidence and personal experiences and values, not just an aggregation of population preferences. Further, participants were diverse with respect to geography, caregiver status, type and stage of cancer, and diagnosis. While several of the panel participants had educational backgrounds and work or volunteer experiences relevant to health care quality improvement, most did not. Since we aimed to elicit participants’ expertise as patients and/or caregivers and not their technical expertise, which was solicited from expert stakeholders beforehand, this should not impact our findings. Generalizability of these panel findings may also be limited due to the focus on VA care; to ensure broader salience, we included two non-Veteran panelists.

This work will inform the next stages of the ImPACS project, which will operationalize one or more high-priority measures. These will be inaugural measures for creating a VA-wide palliative care quality dashboard to monitor palliative care quality at scale. In addition to using structured data like visits and prescriptions to monitor care, we anticipate being able to monitor care processes described in unstructured data like clinical notes by developing natural language processing tools.

The ImPACS patient and caregiver panel demonstrates that it is feasible and informative to elicit the perspectives of cancer patients and caregivers to prioritize palliative cancer care quality measures. Our effort was time-intensive, as it required tailoring materials describing cancer-related quality measures and quality measurement in general to a patient and caregiver audience. A forced choice (“top 5”) exercise was important for discriminating between similar numeric ratings to distinguish the most important measure concepts. To provide patient- and family-centered cancer care, involving patients and caregivers in setting health system quality improvement priorities should be the rule, rather than the exception.