Background

More than 30,000 people in the United States have cystic fibrosis (CF), which is a rare, life-shortening, multi-organ disease that can lead to severe respiratory and digestive problems as well as other complications such as infections and diabetes [1]. Until fairly recently, most persons affected by CF were children, but today, with increased medical interventions, more than 50% of people with CF (PwCF) are adults with a median survival of almost 45 years [1]. The CF community is widely recognized for its long-standing tradition of including PwCF and families to help shape research affecting their community [2, 3]. However, this patient participation has been limited to only discrete parts of the research process, such as participating on data safety monitoring boards, prioritizing research topic areas, providing feedback on study questionnaires or reviewing grant proposals. While this level of involvement, according to the spectrum of patient/stakeholder engagement, allows patients to provide input, it falls short of genuine engagement and partnership with researchers [4]. Part of what makes bringing PwCF together onto research teams so difficult are strict infection control guidelines that restrict in-person contact between patients to avoid the spread of deadly pathogens [5]. This is especially problematic for traditional methods of patient engagement, which are mostly geared for “in-person” group interactions.

Patient-centered outcomes research (PCOR), or patient and public involvement (PPI), entails meaningfully engaging patients, caregivers and other stakeholders (such as clinicians, payers and policy makers) throughout the research process and is increasingly gaining traction among research teams in the United States [6]. Patients, in particular, are valuable to include on research teams because they provide expertise in living daily with their disease. Ideally, research teams using PCOR methodology invite patients as partners to bring ideas and questions based on their lived experience, with researchers then sharing a variety of possible approaches to study them. With this exchange, patient-partners begin to understand the research process more fully and can move towards authentically participating in all phases of research. PCOR has shown to improve research quality, increase patient trust in both the research and researchers, and positively affect health outcomes [6,7,8].

The process of inviting patients as full partners onto research teams requires a cultural shift among researchers who prize efficiencies, and rarely have to contend with diverse perspectives, new unconventional possibilities, or members of the team who are unfamiliar with research terms and processes [9]. For researchers and patients who want to learn about PCOR, training is available. Current PCOR training curricula, however, do not address certain cultural aspects unique to the CF community. For example, power dynamics exist in all areas of medicine, but the hierarchical patient-doctor relationship in CF is considered to be especially apparent because of the life-long and complex nature of the disease [10, 11]. Clinicians are unsure how to ask patients to serve as partners without patients feeling a sense of obligation [12]. Additionally, as a rare disease, PwCF, caregivers, clinicians and researchers tend to already know one-another, and thus the change from role as patient to patient-partner is more difficult. For example, when we first started our PCOR team, patient partners were reluctant to speak freely in front of researchers and clinicians due to the fear that something they would say would get back to their personal clinician [13]. This made patients want to keep their discussions confidential. Thus, based on our prior needs assessment, we found the most important training areas to address for both the patient/caregiver and provider/researcher groups included: (1) knowing the time commitment required to learn PCOR methodology, and (2) learning how to develop and maintain trust when patients/caregivers are active members of the research team [14]. PCOR training is necessary to achieve a critical mass of researchers employing this methodology. Additionally, more research funders require stakeholder engagement on grant submissions. This study aims to evaluate a virtual training adapted for the CF community on perceived PCOR knowledge acquisition, confidence in engaging stakeholders and satisfaction of the training program.

Methods

We report this study according to the GRIPP2 guidelines in the reporting of patient and public involvement (PPI) in research [15]. We report as many elements on the checklist as relevant to this study.

Design

We employed a qualitative descriptive design to co-develop four training sessions with patient and advocacy organization stakeholders using a framework adapted from the Model for Improvement [16]. The Model for Improvement framework uses four iterative phases (1) Plan, (2) Do, (3) Study, (4) Act (see Fig. 1) [16]. Our focus for this present study were phases (2) and (3). Phase 1 (Plan) included a needs assessment co-produced and conducted by our team, which has been published previously [14].

Fig. 1
figure 1

Adapted Model For Improvement framework to build patient-centered outcomes research capacity in the cystic fibrosis community

Full size image

Setting

This study was conducted virtually using Zoom for synchronous meetings and training sessions, Slack and email for asynchronous communications between team members, and Google Docs for document sharing. The origins of this co-production patient-engaged team came from the Cystic Fibrosis Reproductive and Sexual Health Collaborative (CFReSHC, cfreshc.org). CFReSHC is a U.S-based, nationwide, English-speaking, online patient-driven collaborative committed to responding to research gaps articulated by adult PwCF. In 2018, having built a successful patient-engagement structure, members of CFReSHC wanted to build PCOR capacity within the greater CF community. CFReSHC researcher and patient members co-wrote a successfully funded grant application to the Patient-Centered Outcomes Research Institute (PCORI) to conduct a needs assessment and develop and evaluate an educational PCOR training program for PwCF, researchers and health care providers. CFReSHC is hosted by the University of Washington Department of Family Medicine.

Training co-developers: participant characteristics

The training development and evaluation team consisted of several different types of stakeholders, including clinician-researchers (n = 3), PwCF (n = 3), public health researcher (n = 1) and a CF advocacy organization staff member (n = 1). The project team was led by a CFReSHC co-founder and a practicing family physician and clinician-researcher with patient engagement methodological expertise. A second clinician-researcher specialized in CF, and the third clinician-researcher was an expert in educational design and evaluation. Project co-leads included three current CFReSHC patient-partners: (1) one with a career as a sexual health educator and trainer, (2) another with CF community advocacy connections, and (3) another with experience in media and marketing. Our community advocate was a staff member for the largest non-for-profit organization in the CF community in the United States, the Cystic Fibrosis Foundation (CFF). A Master-level student in the University of Washington School of Public Health managed the team and performed the data analysis.

Training program development

We developed, led and evaluated four separate pilot trainings between February 2019 and August 2020. Members of the development/evaluation team met weekly throughout the study period. The team’s activities are depicted in the timeline in Fig. 2, which were guided by best practices for training and development provided by the University of Washington Institute of Translational Health Sciences [17]. Our initial steps included developing four core competencies and learning objectives for each competency, informed by findings from a prior CF community educational needs assessment [14]. Over the course of several weeks, the team co-created four separate core competencies for two distinct learner groups: (1) researchers/providers and (2) patients/caregivers. We iteratively developed 3–6 learning objectives within each core competency using Bloom's taxonomy until consensus was reached by all team members [18]. To create the training format, we applied the key adult learning principles, which included pre-training materials, learning aids, and multi-modal learning strategies (e.g., didactics, small group activities, case scenario discussions, and question/answer sessions) [19, 20]. The public health researcher and a patient-partner with sex-education experience then collaboratively scouted the internet for existing PCOR training programs with materials or items that met our learning objectives (see Appendix 1). The patient-partner with CF advocacy organizations connections sought pre-existing training programs already available for the CF community. At each weekly meeting, the public health researcher and patient-partners would present power-point slides or material they found on the internet, and the remaining team members adapted these to meet items specifically mentioned during our needs assessment [14]. The public health researcher and two patient-partners additionally identified information from the peer-reviewed literature on PCOR, which supplemented aspects of the CF-specific training program that had not been identified in pre-existing PCOR training [21,22,23,24,25,26,27,28,29]. The literature was reviewed by the clinician-researchers and the educational specialist. Because in-person contact between patients with CF is restricted, we included information about the use of web-based platforms for virtual PCOR collaborations based on an in-depth interview study performed by our team [30].

Fig. 2
figure 2

PCOR training development activities Feb 2019-Jul 2020

Full size image

Data collection

Once the materials to conduct a training session were finalized, we co-led the online PCOR training program with participants from the CF community throughout the United States. We invited adults with CF and their caregivers, CF providers, researchers and research staff to participate in and provide feedback about our training sessions. We advertised the training through CFReSHC, CFF, Cystic Fibrosis Research Institute (CFRI), and the University of Washington. The patient-partner with marketing skills created recruitment flyers for each training and widely advertised the training throughout the CF community. Participants who attended a training session and completed the surveys were provided a $15 gift card.

All team members helped conduct four separate pilot PCOR training sessions between April 2019 and October 2019. We evaluated each training session by administering a survey to participants before and immediately after the training session, except for Training 1, where we surveyed participants only after the training. The survey questions asked participants to rate their agreement about their own perceived PCOR knowledge (a.k.a PCOR knowledge), confidence with engaging stakeholders and training session satisfaction (post-training only) using a 5-point Likert scale. At the end of each survey, we asked open-ended questions regarding what the participants liked about the training session and how we could improve. After each training session, the development team met for 45 min to discuss what went well and what could be improved, which were captured as notes to implement into the next iteration of the training program. A single, summative PCOR session for the CF community was provided by the researcher–clinician co-lead and a patient-partner as a podcast through CFRI in July, 2020 [31]. This podcast was not evaluated by our team.

Data analysis

The public health researcher performed the statistical analysis of the survey responses and created the data tables and figures. We conducted descriptive statistics for participant characteristics and median scores with inter quartile range (IQR) for post Training 1 (Research 101). For Trainings 2–4, we calculated the median pre- and post-training score for each PCOR knowledge and confidence in engagement questions. We also assessed the difference between training attendees’ self-rated pre- and post-training responses using the sign test. Because the sample size was small and not normally distributed, we used non-parametric summary statistics and tests. We used the sign test specifically to assess whether there was a significant directional change in the pre- and post-training responses for individual responses (alpha = 0.05) [32]. We performed statistical analysis using R version 3.6.3 with RStudio version 1.3.1093 [33]. We summarized responses to open-ended questions after each training session regarding suggestions for improvement. The patient-partners provided input on how best to display the results with tables and figures.

Results

Training program content

Our pilot PCOR training program consisted of four separate core competencies: (1) Understand the principles of research; (2) Understand the science of PCOR; (3) Participate in and maintain a PCOR team; (4) Design and implement a PCOR study. Each core competency served as a separate session title and within each competency we listed learning objectives, training format, presenters/facilitators and the learner group/audience for which the session was intended (see Appendix 2). Our first training was intended for patients/caregivers only (Research 101), and included a 25-min asynchronous, self-directed learning seminar intended to be viewed before the subsequent interactive PCOR sessions. The remaining three training programs were synchronous, interactive training sessions, lasting approximately 1.5 h each. Two of these sessions included both learner groups (patients/caregivers and researchers/providers) together (PCOR 101 and PCOR Team Dynamics) and one session (PCOR Study Design) was for researchers/healthcare providers only. The pilot training sessions can be downloaded here: familymedicine.uw.edu/pcor-guide/.

Training program evaluation

Training program participants included 28 patients and caregivers, and 31 researchers and providers. Several participants attended more than a single PCOR session. Detailed training program participant characteristics, including type of participant, job title, and attendees per session are reported in Table 1.

Table 1 Training program participant characteristics
Full size table

Overall, participants significantly improved self-assessed PCOR knowledge. The median Likert scale responses post-training for training session 1 and a test of the difference between knowledge perception questions administered before and after training sessions 2, 3, and 4 are presented in Table 2.

Table 2 Participant self-assessment of PCOR knowledge Likert scale responses pre- and post-training
Full size table

After Training 1 (Research 101), we found the majority of patients/caregivers reported being able to describe the different types of research methods (15/17, 88%), terminology (14/17, 82%), and modes of dissemination used in PCOR (17/17, 100%).

After Training 2 (PCOR 101), we found a significant difference in change of PCOR knowledge related to engagement levels compared to before among patients/caregivers and CF researchers/providers, (p values = 0.008 and 0.031, respectively). Following the session, patients/caregivers reported being significantly better able to identify barriers and enablers to adopting PCOR compared to pre-training (p = 0.016), whereas CF researchers/providers reported being significantly better able to describe and provide examples of the core principles of PCOR (p = 0.017).

During Training 3 (PCOR Team Dynamics), patients/caregivers significantly improved their PCOR knowledge in every aspect of the training except knowing how confidentiality of patient partners are maintained on PCOR teams. In contrast, we found no reported significant PCOR knowledge changes among CF researchers/providers. After this session, patients/caregivers reported being significantly better able to describe elements of an effective PCOR team (p = 0.039), how to be an active participant throughout the research process (p = 0.012), how to identify barriers to successfully functioning PCOR teams (p = 0.021).

In Training 4 (PCOR Study Design for CF Researchers/Providers only), participants reported significant improvement of their PCOR knowledge in every aspect of the training, including how to identify patients and caregivers to participate as partners in research, formulate research questions from patient-driven priorities, articulate successful components of an engagement plan in grant applications, and describe the patient partner role at every stage of the research project (p-values = 0.039, 0.039, 0.006, 0.001).

Confidence with engaging partners in PCOR

Confidence was only solicited in surveys related to Training 2–4. The median confidence score of patient/caregiver participants attending training sessions 2 (PCOR 101) or 3 (PCOR Team Dynamics) to engage as a partner in research was a “4” (fairly confident), which did not change significantly after either training. Similarly, the median confidence score of CF researchers/providers attending training sessions 2, 3 or 4 (PCOR Study Design) did not change significantly: self-rated confidence to engage PwCF in research before each training was either “3” (neutral) or “4” (fairly confident), and after each training was “4” (fairly confident).

Training session satisfaction

Overall, both learner groups were satisfied with the format of each training session (Fig. 3). The highest proportion of participants from either the patient or researcher/provider group who reported being very satisfied with the training occurred with Training 3 (PCOR team dynamics) compared with the other training sessions.

Fig. 3
figure 3

Satisfaction with the interactive training session format, by participant type

Full size image

Training satisfaction: open-ended questions

Training 1 (Research 101) The most common beneficial aspects participants noted was learning about research terminology, the grant submission process, how to design research questions, and the difference between types of research studies (e.g., quantitative vs qualitative and retrospective vs prospective). Most participants appreciated the inclusion of multiple speakers, including people with CF and the incorporation of visuals.

Training 2 (PCOR 101) Most participants liked the interaction between patients, caregivers, researchers and healthcare providers and the fact that the learners were taught together. Some participants reported not liking required participation in the interactive portions of the training, while others thought the breakout sessions were too small and too short. One person suggested having facilitators participate in each group to help move the conversation along.

Training 3 (PCOR Team Dynamics) This training included two case scenarios related to: (1) creating a respectful space for collective sharing, and (2) building and maintaining trust. Within each scenario, participants were asked to identify barriers to a successfully functioning PCOR team and articulate potential solutions to address those barriers. All participants commented that they enjoyed the discussion of both case scenarios, and thought they clearly illustrated potential challenges PCOR teams face. Participants also liked how facilitators engaged to advance the discussions. For example, one researcher/provider participant noted that all participants were “encouraged to speak”. Participants suggested including more role playing for this training.

Training 4 (PCOR Study Design) Participants favorably rated the video conferencing format and thought the session was easy to join. Several suggestions for improvement included: Offering closed captioning versions of the presentation for viewing in different languages (including English for Deaf viewers); shorten the panel discussion and bring back the interactive breakout session format; provide more specific examples of how including patients or caregivers on the research team improves the quality and relevance of research; include example documents, such as a patient partner biographies and a list of potential patient partner roles; include more discussion of ways to include a diverse representation of patients on the research team.

Discussion

In this study, our team, consisting of researchers, clinicians, PwCF and CF advocates, developed and evaluated four unique virtual training sessions related to patient/caregiver engagement on CF research teams. Based on the results of our prior educational needs assessment [14], we worked with an educational specialist to identify competencies and used an iterative process to specify our learning objectives based on the Model of Improvement [16]. We customized existing PCOR materials to meet the needs of the CF community, which called for incorporation of key adult learning principles, including pre-training materials, learning aids, and multi-modal learning strategies (e.g., didactics, small group activities, discussions, and question/answer sessions). Our findings suggest participants improved their knowledge about PCOR after each training session. Participants had fairly high confidence about their own PCOR skills at baseline, a measurement which did not significantly change with the PCOR training. Regardless of change in PCOR knowledge or confidence, participants in both learner groups (researchers/providers and patients/caregivers) were very satisfied with the teaching sessions. With high confidence and satisfaction, it is likely that learners who attended the sessions felt favorably about PCOR going into the sessions, but the change in baseline knowledge suggests that this filled a learning gap for the CF community.

This study is unique because of the level of engagement of our patient- and stakeholder-partners who maintained input into the project starting at the grant generation stage through dissemination of the findings. Our process of highlighting patient-partner skills (in addition to their insights as persons with the disease) allowed our patient-partners to more fully contribute to team activities. Additionally, open-ended comments from training participants indicated positive feedback in having PwCF and researchers teaching together. The input of the patient-partners helped make the didactic material more relatable to the audience. The impact of PPI in this work enabled our team’s patient-partners to intimately learn about PCOR. As a result, they wrote and published an article about PCOR in a widely read quarterly periodical by CF patients [34] and produced an asynchronous podcast for the CF community that is publicly available [31].

As major funding agencies increasingly encourage and expect the inclusion of patient stakeholders on grant applications (i.e., by making such engagement a requirement for funding) [35], patient and caregiver partners will increasingly begin to join CF clinical research teams and accordingly, CF researchers will need adequate skills to successfully integrate these members. Engaging patients and caregivers for the long-term requires a sustained approach to support CF researchers and team collaborations, and to ensure PCOR principles of belonging and collaborative learning are actualized [20]. Other PCOR training programs such as one developed by the National Organization of Rare Disorders (NORD) in conjunction with the University of Maryland also contains the notion of sustaining support for PCOR by developing a pipeline of qualified and skilled mentors in PCOR methodology for new PCOR teams [23]. A separate study found that training priorities should include helping team members identify appropriate patient partners, devising an engagement strategy that clarifies roles and expectations, and building skills for positive team dynamics [36]. Prior training suggests that learning is not a singular one-time event, but comes from the act of “doing.” Thus, ideally research teams should have an opportunity to participate in a PCOR mentoring program in which they can integrate patient/caregiver partners and have experts with whom to consult.

Our study had some limitations. Per the GRIPP2 guidelines, we did not quantitatively measure the impact of PPI in this study. We did, however, collect qualitative feedback about the impact of PPI from training participants, many of whom indicated positive comments about seeing PwCF as leaders of this work. Another limitation of this study was the low participation rates, which we believe was primarily due to our recruitment strategy. We had intended for the program to be presented sequentially with the same participants attending each training session. Thus, we initially limited our invitations to only those participants who had attended the prior session. We eventually opened our invitations to CF community members who had participated in our prior needs assessment, but finding a mutual time for synchronous training was difficult to achieve with busy work schedules. Our program evaluation was limited to feedback occurring immediately post-training. We did not include an evaluation process that assessed whether attendees later incorporated PCOR onto their research teams, or whether patients/caregivers joined research teams after receiving the training.

Conclusions

A core team of researchers, patient-partners and advocacy stakeholders successfully co-developed four pilot PCOR training programs and a publicly available podcast about how to increase patient-engagement capacity on research teams. This CF-specific PCOR training was well received by patients, caregivers, health care providers and CF researchers. The program significantly improved PCOR knowledge with core PCOR learning items. The training development team is now creating a new, comprehensive PCOR training manual with input from stakeholders from the CF community, with the goal of increasing interest in PCOR skills and methods among CF clinical researchers.