BACKGROUND

Stakeholder engagement in health research has become increasingly common as investigators, journal editors, and funders recognize its potential influence on the evidence we produce.1, 2 With the expansion in recent years of patient-oriented and translational research, engagement of stakeholders—patients, clinicians, policy makers, and others, each including multiple members—is increasingly expected.3,4,5 However, despite a growing number of completed and ongoing stakeholder-engaged research projects,6,7,8,9,10,11,12 little is known about the quality of stakeholder engagement work, or its impact on evidence.13

Experience shows that researchers have questions about engaging with stakeholders: what is it, why is it important, who should be involved, how should they be involved, when in the research process should they be involved, and what difference will it make?14 To address these questions, published frameworks describe how to identify stakeholders and how to synthesize the views of multiple groups.15,16,17,18,19,20,21,22,23,24 Previous work also characterizes stages of research in which stakeholders can be engaged, such as involvement in prioritizing research questions before a study is underway, or developing robust dissemination and implementation plans once it is completed.16, 20, 21, 24

In this paper, we synthesize expert opinion and research resources from four countries into practical guidance on carrying out the activities that are needed between stakeholder identification and evidence implementation: what are the roles (i.e., what research activities can stakeholders be involved in) and modes (i.e., how do stakeholders and researchers interact?) by which researchers and stakeholders can work together, and how can a study team select from among the many options? To find answers, we formed a consortium—dubbed Multi Stakeholder Engagement or MuSE—of individuals from Canada, Australia, the UK, and the USA who represent a variety of viewpoints about research in health. Our consortium identified several challenges for stakeholder-engaged research and formed a writing team and review panel (Online Appendix A) composed of individuals to address this particular need (Online Appendix B for a summary of MuSE projects).

Our primary audience is health researchers who are embarking on stakeholder-engaged work. The paper will be helpful especially to students and early career investigators who want to build a portfolio of stakeholder-engaged research. It will also be helpful to experienced scientists who recognize the need to involve stakeholders in their work. Finally, it should be useful to researchers who are already working with stakeholders but need a concise resource for designing future studies. Non-researchers are an important secondary audience; this work might be adapted specifically for individual stakeholder communities.

We define “stakeholder” as an individual or group who is responsible for or affected by health- and healthcare-related decisions. We use the term stakeholder engagement when describing activities the researcher can take in this relationship and stakeholder involvement when describing the activities that either researchers or stakeholders can take. In this paper, the term roles is used to describe the research activities in which stakeholders may be involved. The term modes of engagement is used to describe the processes by which researchers and stakeholders interact with each other. Finally, engaging with stakeholders is not the same as studying their views. This paper addresses involvement of stakeholders as partners in, not as, subjects of research.

CONSIDERATIONS BEFORE EMBARKING ON STAKEHOLDER-ENGAGED RESEARCH

When it comes to developing an engagement plan, the study team faces a bewildering array of options: How will we build relationships and trust with stakeholders before the work begins? When should the activities start and finish and how does uncertainty about funding for the work play into the plan? Will contact be sustained in person or by phone, email, web, or other tele-communications? How frequently should the research team and stakeholders be in contact with each other? Will contacts be made individually or in groups? Will individuals representing different stakeholder communities work separately or in mixed groups? Will opinions be gathered in structured discussions, formal interviews, or a survey? Are consensus techniques needed? Taken together, these and other considerations make developing an engagement plan a daunting task, especially for those new to multi-stakeholder engagement.

These decisions may be easier to tackle in steps. Before diving into the details, it will help to reflect on why to involve stakeholders, who should be involved, and how extensively they may be involved. Once a team has articulated these aspects of the plan, the specific roles and modes of involvement may become clearer. We discuss these four constructs—the rationale, extent, roles, and modes of involving stakeholders—in more detail below. Text box 1 presents this material as a series of questions.

Text Box 1. Considerations before embarking on stakeholder-engaged research

What is the rationale for engaging stakeholders?

 ○ What are the intrinsic reasons for working with stakeholders?

 ○ How do you expect working with stakeholders can improve your research?

 ○ How to you expect working with stakeholders will improve relevant health care or outcomes?

Which stakeholder communities will be engaged?

 ○ What model will you use to identify relevant stakeholders (Online Appendix B)?

 ○ Which of the stakeholder communities in your framework make decisions the research is meant to inform?

 ○ Which stakeholder communities are affected by decisions the research is meant to inform?

 ○ What are the preferences of stakeholder communities for how they wish to be engaged?

How extensively will the stakeholders be engaged?

 ○ How will stakeholders be involved in preparing for research?

 ○ How will stakeholders be involved in conducting the research?

 ○ How will stakeholders be involved in using the research?

 ○ How intensively can stakeholders be involved in each activity?

 ○ What resources and time that can be devoted to engagement activities?

What are the appropriate roles and modes by which stakeholders may be engaged?

 ○ Will stakeholders have control over the course of the project?

 ○ Will stakeholders help the research team carry out the research?

 ○ Will stakeholder provide input but neither direct nor help with the research directly?

 ○ Will activities be conducted in person or remotely?

 ○ Will activities be conducted with individuals?

 ○ Will activities be conducted with groups?

 ○ Will stakeholder communities be mixed in multi-stakeholder activities?

What conflict of interest procedures and conflict management resources are needed?

What Is the Rationale for Engaging Stakeholders?

Pre-specifying the rationale—or desired outcomes—of engagement is a critical factor in developing an engagement plan. By establishing the desired outcomes in advance, furthermore, the study team can later evaluate the extent to which expectations have been met. This section points at ways to characterize the expected value of stakeholder engagement.

To articulate the rationale for engagement, the team might consider both its intrinsic and instrumental imperatives. Intrinsic imperatives suggest that engagement is an end in itself. In other words, involving stakeholders may simply be the right thing to do, especially if public dollars are used. The principles of engagement call researchers and stakeholders to pursue intrinsic goods like autonomy, dignity, equity, inclusiveness, partnership, and participation25, 26; in contrast, superficial involvement can be insulting to stakeholders.26 Instrumental imperatives suggest that engagement produces some other good worth having. For instance, involving stakeholders may make study questions more relevant, methods and approaches more transparent, findings more useful, and evidence more likely to be used in practice.16 If researchers wish our work to have detectable impact on health-related decisions, we should involve decision makers as we carry the work out.

Whether involving stakeholders meets intrinsic and instrumental aims can be assessed through quantitative and qualitative evaluation. Do stakeholders report a sense of autonomy, dignity, self-determination, equity, inclusiveness, partnership, and participation? To what extent do stakeholders use the research results in decision-making? The ultimate goal of health research is to improve health at affordable costs. For this reason, it may be productive to map explicitly how stakeholder involvement is expected to lead to improved research outcomes, without exceeding a desired budget.

Which Stakeholder Groups Will Be Involved?

Previous frameworks have sought to help researchers identify stakeholders in their work. Concannon et al. identified seven types of stakeholders in the Tufts-RAND 7Ps taxonomy for engagement16; Deverka et al. identified eight types in the Center for Medical Technology Policy framework for engagement17; PCORI identified nine types in its engagement “rubric.”19 Cochrane identified four audiences in its recent knowledge translation strategy.20 Tugwell et al. named six types in a WHO Bulletin on knowledge translation for systematic reviews.21 By putting these models in a single table (Online Appendix C), it is possible to see that differences in the number of stakeholder types are largely a matter of classification, not a disagreement about who should be included. At least two of the approaches16, 17 were developed and published simultaneously, suggesting independent agreement about which stakeholders are key to improving health research.

Each of these frameworks recognizes that identifying the right individuals to represent stakeholder perspectives is a challenge, as stakeholders within a single group may hold different views. Choosing a multi-stakeholder approach is even more complex given the need to consider how to assure that underrepresented voices are heard, to ensure that the financial or academic interests of one group do not dominate the discussion, to manage group interactions and potential power imbalances, and to synthesize the views of different groups. It is important to be transparent about who was involved and why.

How Extensively Will Stakeholders Be Involved?

The frameworks described above agree that stakeholders can be involved throughout the research process and that contact should be sustained over time. The PCORI model identifies nine distinct research activities in three groups: (1) planning the study, (2) conducting the study, (3) disseminating the study results. Similarly, the Tufts-RAND 7Ps model lumps research activities into three stages: preparing for, conducting, and using research. The Cochrane model describes stakeholder involvement in the topic and question selection, design, execution, interpretation, and dissemination of their research.

When preparing for a study, researchers may engage stakeholders in a variety of activities, such as capacity building, team building, training, and topic selection. During the conduct of a study, activities like question development, selection of outcomes, participant recruitment, data collection, analysis, and identification and interpretation of findings may involve stakeholders. Once findings have been established and a study is concluded, researchers are often engaged in helping decision-makers use their findings. Stakeholders can be valuable partners during this stage by supporting or leading implementation activities.

Researchers may also consider how intensive the engagement of stakeholders should be. Some partner with stakeholders as co-investigators, sharing full control over the direction, management, and budget of a study. Others may arrange for stakeholder involvement at the level of technical advice. Thus, stakeholders may be empowered with differing levels of control over a study, from providing direction to collaborating, consulting, and providing information. It is possible, however, to mix approaches over the course of one or more studies and across different stakeholder groups.

What Are the Appropriate Roles and Modes by Which Stakeholders May Be Involved?

The core challenges we address include how to identify the roles that stakeholders will have and the modes by which their involvement will be facilitated. In designing the roles and modes of engagement, research teams should be responsive to stakeholder views about how they would like to be involved.27,28,29,30,31 Researchers might try to learn from stakeholders about their interest in the research topic, including what the stakeholders want to get out of being involved. Furthermore, researchers should consider communicating about the interests and potential roles of stakeholders before a plan is settled. Once the plan is drafted, study teams may find that summarizing the engagement plan in a written document will facilitate communication. One recent work suggested summarizing roles and responsibilities in a “terms of reference” document for use by the whole research team throughout the project.32

Roles

All study teams—even those in basic and clinical sciences—have experience working with independent peers who review study protocols and manuscripts. This is a form of stakeholder engagement, in which external researchers with an interest in safeguarding the ethical conduct and rigor of research use commonly held standards to review the proposed or completed work.

Engagement with non-research stakeholders is similar. This might involve assembling a panel of individuals who have an interest in the outcomes of the research and can potentially use it to support decisions. Expert panels can review documents, run practice tests of survey instruments, vote on the relevance or importance of evidence for decisions they make, or work together to identify the implications of study results for their communities. They can develop study inclusion criteria, discuss and revise study protocols and materials, and identify outcomes that are most important to them.33, 34

Engagement activities may not be oriented toward finding a group’s consensus about research, but rather on identifying the way that different stakeholder groups view the work. Understanding whether and how stakeholders disagree about the research can be just as important as understanding whether and how they agree. Knowing about these views may be critical to public release of the evidence with appropriate messaging and context.

Investigators working in community-based participatory research (CBPR), partnered action research, and co-produced research have experience partnering with communities in the direction, management, and oversight of studies. Involvement in this work may include deciding which work to prioritize, setting or sharing budgets, and managing personnel. It is critical to remember that principal investigators—whether researchers or stakeholders—retain full responsibility for the ethical conduct, quality, and rigor of research. The process of pre-publication review by scientific peers safeguards these dimensions of quality, and these reviews may be conducted transparently in the presence of stakeholders.

Modes

Distinct from the roles stakeholders play are the modes of interaction. While roles refer to the activities of stakeholders as they become involved in the work, modes have to do with the format and structure of interactions between researchers and stakeholders. When stakeholders are invited to serve as co-investigators or collaborators, the modes of engagement typically include all of the routine communication channels and interaction opportunities of the research workplace, such as in-person meetings, chance meetings, telephone calls, e-mails, and web-enabled communications. In instances where stakeholders serve in a consulting or advisory role, the modes of interaction may involve specialized communications in person, by telephone, by e-mail, or over the web. Alternatively, modes of interaction may include group communications such as town meetings, or group discussions, and these can also be held in-person or virtually. Finally, passive modes of communication are also used, such as public comment periods for research prioritizations or plans, and comments may be collected in writing, electronically, or by telephone.

WRITING AN ENGAGEMENT PLAN

Once these considerations have been reviewed, researchers and stakeholders can work together to write an engagement plan. To assist, we introduce a matrix that may be used to summarize a stakeholder engagement approach for a program of research or a single study (Table 1).

Table 1 Summarizing a stakeholder engagement plan
Full size table

The rows describe stages and illustrative activities in a research project. While the rows may imply a sequential process from top to bottom, in practice, studies involve several iterative steps that loop backward and jump forward. The columns describe stakeholder groups, derived from a combination of the frameworks16, 17, 19,20,21 (Online Appendix C). The collection of activities and stakeholders might be described at length in a legend. Researchers can use the matrix to develop a plan, share and revise the plan after consultation with stakeholders about how they would like to be involved, or summarize completed work involving stakeholders of different types in activities of different types. The table facilitates critical appraisal of the engagement plan by providing a quick view over the whole project. Researchers may decide to publish a completed matrix in their proposals or manuscripts.

Each cell in the matrix may be filled in with information summarizing the roles and modes of engagement for a specific stakeholder group in the research activity. Filling in many or most cells in this matrix will make the table quite large. As the table grows in size, it will become clear that the time and resource costs of engaging with stakeholders are significant. It also serves to point out where there are gaps in the plan.

This matrix is meant to help in planning well-targeted engagement activities. It is not meant to imply that all research activities require engagement, nor that all stakeholder groups must be included in all activities. There may be good reasons why certain cells are left blank; in these cases, researchers might note the rationale for blank cells or gaps in the matrix. In many cases, the roles and modes of involvement will be identical for stakeholders representing different communities. For instance, a technical advisory board composed of patients, clinicians, payers, and policy makers might be charged with several activities: (1) reviewing the study design; (2) reviewing data analyses; and (3) participating in identifying and interpreting findings.

Finally, any stakeholder who represents one community may belong to one or more additional communities. For example, some purchasers are also payers and some payers provide care. Patients and their advocates may also be providers or employers with policy-making responsibilities. Overlap may be inevitable, but care should be taken that multiple roles do not create unacceptable conflicts of interest. Building relationships with stakeholders, addressing conflict of interest, and dealing with conflict between those with competing interests are topics that have been explored elsewhere. Research teams need formal processes to build relationships and address conflicts, and resources are available to help.35,36,37,38

DISCUSSION

We developed this guidance to assist researchers to choose appropriate roles and modes of engagement with multiple stakeholder communities. Stakeholder engagement is in need of further experimentation, and the exact path toward meaningful engagement will not be uniform across every research institution and project. Our recommendations for developing stakeholder engagement plans follow a plan-do-study-act (PDSA) approach, and they can be applied to any program:

  1. 1.

    Set forth engagement roles and modes (plan). Researchers should plan engagement roles and activities throughout the course of a study, and funders might incorporate expectations about doing so in funding opportunity announcements.

  2. 2.

    Experiment with alternative strategies (do). Research institutions, investigators, and stakeholder groups should begin to experiment with organizational- and project-level strategies to support stakeholder engagement. Institutions might initiate stakeholder activities to support multiple studies, such as a priority-setting process to inform the organization’s research agenda. Investigators can begin to experiment with alternative approaches to engagement in their own research.

  3. 3.

    Evaluate alternative approaches (study). Funders and investigators can begin right away to identify appropriate intermediate and long-term benchmarks for evaluating the effectiveness of roles and modes, keeping in mind that the optimal approach will vary by institution and project. Future research on what works and what does not might consider whether and what kind of stakeholder engagement leads to informed decision-making and improved uptake of evidence into practice. The matrix offered in Table 1 can help with this challenge by pointing at specific activities we hope to change (in rows) and how we hope to change them (in cells). The resources listed in Online Appendix D can help researchers adapt their approach to different contexts.

  4. 4.

    Report on outcomes, implement changes as needed, and iterate (act). Investigators should consider reporting stakeholder activities in manuscripts and contract reports. Journals might consider publishing both quantitative and qualitative research on this topic, to continue establishing an evidence base about what works in various settings. As the evidence base grows, funders, research institutions, and investigators need to be prepared to implement changes in their engagement programs. As changes are adopted, an iterative assessment process should follow.

In this paper, we have offered practical guidance on designing and carrying out an engagement plan. Research teams and funders may use pre-specified aims and roles and modes of engagement to support formal evaluations of stakeholder engagement work, an important next step.