Knowledge Management Research & Practice

, Volume 9, Issue 3, pp 203–214

Indicators at the interface: managing policymaker-researcher collaboration

  • Anita Kothari
  • Lynne MacLean
  • Nancy Edwards
  • Allison Hobbs
Article

DOI: 10.1057/kmrp.2011.16

Cite this article as:
Kothari, A., MacLean, L., Edwards, N. et al. Knowl Manage Res Pract (2011) 9: 203. doi:10.1057/kmrp.2011.16

Abstract

The knowledge transfer literature encourages partnerships between researchers and policymakers for the purposes of policy-relevant knowledge creation. Consequently, research findings are more likely to be used by policymakers during policy development. This paper presents a set of practice-based indicators that can be used to manage the collaborative knowledge creation process or assess the performance of a partnership between researchers and policymakers. Indicators for partnership success were developed from 16 qualitative interviews with health policymakers and researchers involved with eight research transfer partnerships with government. These process and outcomes indicators were refined through a focus group. Resulting qualitative and quantitative indicators were judged to be clear, relevant, credible, and feasible. New findings included the need to have different indicators to evaluate new vs mature partnerships, as well as specific indicators common to researcher-policymaker partnerships in general.

Keywords

knowledge transferpolicymakerknowledge utilizationknowledge management toolsknowledge creationhealth research

Introduction

Gibbons and colleagues (Gibbons et al., 1994; Nowotny et al., 2001) have put forth a new model of knowledge production that calls for research that is interdisciplinary, based on needs of end users and context-driven. Mode 2 research, as it is called, is positioned in contrast to Mode 1 research, characterized by traditional norms of scholarship. Mode 2 research represents a revised social contract between society and science. Among other implications, the new social contract demands collaborations with other stakeholders for problem-focused scientific inquiry. This approach is predominant in the health sector, where health researchers are being encouraged to conduct research in partnership with policymakers (Lomas, 2000; Hanney et al., 2003; Lavis et al., 2003; Tetroe et al., 2008; Graham & Tetroe, 2009; Bullock et al., 2010), with good reason. The two groups of individuals come from quite different organizational settings – researchers and policymakers work within different time lines, use different jargon, practice different norms, and expect different things from research studies in terms of content and rigour (Lomas, 2000; Goering et al., 2003). Partnerships are a means by which to bridge the gap between the two groups (Caplan, 1979) for the common goal of solving complex health problems. Frequent interactions between the two groups are thought to assist researchers in developing research questions that are relevant to political priorities and contextual opportunities. In turn, researchers can help policymakers interpret research findings in light of local realities (Huberman, 1994; Lavis, 2006).

Underpinning these interactions is the expectation that research findings are then more likely to be used (or at least considered) by policymakers when developing health policies (Innvaer et al., 2002), bridging the science to policy gap. Researchers have found that the use of health research and other types of information in the policymaking process was most influenced by interactions between policymakers and researchers (Landry et al., 2001; Paluck et al., 2001; Griffin & Edwards, 2002; Lavis et al., 2002; Landry et al., 2003; Lavis et al., 2003). To clarify, in this paper the term ‘policymaker’ is also inclusive of the term ‘decision-maker’ and refers to those individuals who make clinical, administrative, or legislative policy decisions in organizations or governments. Thus, we are considering a variety of civil servants, in different roles and at different levels, but we are not dealing with elected parliamentarians. Elected officials such as Ministers were not involved in the research partnerships we studied. Rather, the policymakers in their ministries were the direct research users.

The challenge is how researchers and policymakers can interact or partner in an effective and transparent manner. The community collaboration literature – partnerships between researchers and community groups or community members – is vast, and offers a number of strategies to support these collaborations (Israel et al., 1998). The concept of partnerships between researchers and policymakers has not yet received the same attention, an area to which this paper contributes. The purpose of this paper is to present a set of indicators, reflecting both processes and outcomes, which can be used to manage collaborative knowledge generation or assess the performance of a partnership between health researchers and policymakers. Compared to other sectors, the policy area of health is unique in that ‘need’ (illness) is unpredictable yet experienced universally. Nevertheless, it shares features with other social policy sectors – a complex issue with multiple (professional) stakeholders, differential funding mechanisms and incentives, cross-jurisdictional and cross-legislative considerations – that make this study relevant to those outside of the health domain.

Research partnerships: drawing from the broader collaboration literature

What makes collaborations work?

Not all research is intended for or amenable to application in the policy context. Those researchers, however, who are interested in increasing the likelihood of research uptake are encouraged to work in collaboration with policymakers or community groups. Partners involved in a variety of collaborations have reported key features of successful collaborations based on their experiences. A common suggestion is to ensure that partners have been involved in the research process at an early stage to reflect respectful engagement of partners (Reback et al., 2002; Ross et al., 2003; Jansson et al., 2009). As one moves into the research process, nurturing constant communication is seen as vital to understanding each other's needs and priorities (Reback et al., 2002; Denis et al., 2003; Golden-Biddle et al., 2003). As well, maintaining equality among collaborators, both in terms of the research process but also in terms of the research products, has also been seen as important (LeGris et al., 2000; Reback et al., 2002). This overall way of positioning oneself with partners has been called ‘seeing below the waterline’, or being mindful of the relational stance in a collaboration (Golden-Biddle et al., 2003).

Researchers in particular are encouraged to share their knowledge through the provision of research articles, on-going presentation of research findings, and providing educational resources to assist the subsequent change process (Golden-Biddle et al., 2003). Further, creative initiatives might be required to engage partners fully to allow for shared interpretation of findings (Denis et al., 2003; Golden-Biddle et al., 2003; Jansson et al., 2009). Recent work has shown that decision-makers are more likely to be involved in the research process if the process aligns with their regular professional activities or expertise (Ross et al., 2003). As well, a realistic allowance for the amount of time required to develop relationships is vital (Ross et al., 2003).

Collaboration challenges and shortcomings

The challenges to collaborative research are well documented. Professional differences between researchers and decision-makers or community-based partners challenge the partnership process throughout all stages of its existence (Elliott & Popay, 2000; Lomas, 2000; Innvaer et al., 2002; Gaskill et al., 2003; Ross et al., 2003; Walter et al., 2003). For example, researchers focus on clearly defined questions that are appropriately narrow for rigorous study; in comparison, decision-makers must address an entire problem with all of its complexities (Frenk, 1992). Even researchers used to addressing broader questions, such as those in the areas of social science, the humanities, and population health must operationalize their questions, measures, and methods to some subset of the wealth of interrelated variables, dimensions, and themes existing in real life problems. Those researchers who are academics will, most often, take on a focus relevant to their own programmes of research. Decision-makers, on the other hand, must weigh and synthesize information from a variety of researchers, and combine that with the perspectives and needs of related policies in other ministries, task force recommendations, current directives of their own Minister, and competing pressures from advocacy groups, to name a few of the considerations that go into defining a policy-related problem and determining its solution. Adding to this are the competing goals and agendas that both groups bring to a partnership (Gaskill et al., 2003; Walter et al., 2003). Also, partners often have different perceptions about the nature of the final product of a partnership initiative; for researchers, the product is the published article while for others, research continues until it has influenced a decision (Frenk, 1992).

The time commitment is a common stumbling block for many partnerships (Brown, 1994; Elliott & Popay, 2000; Goering et al., 2003; Ross et al., 2003; Walter et al., 2003). The rate at which a partnership effort progresses may also prove to be a challenge (Frenk, 1992; Kouri, 1999; Lomas, 2000; Denis et al., 2003; Eagar et al., 2003; Ross et al., 2003; Walter et al., 2003). Communication barriers, often related to professional culture, can pose a challenge to partnerships (Frenk, 1992; Kouri, 1999; Lomas, 2000; Eagar et al., 2003). Researchers focus on precise, scientific means of communication such as specialized publications (e.g., systematic reviews) found through detailed searches whereas decision-makers or community partners may value accessible expert opinion or information written in plain language (Frenk, 1992).

Subtler shortcomings of partnerships, such as issues around power, distrust, and sustainability are also emerging in the collaborative research literature. When a partnership is composed of members with varying levels of experience and expertise, issues of power and control may arise (Beattie et al., 1996; Innvaer et al., 2002; Gaskill et al., 2003). Right from the start, collaborators may find themselves on unequal on cultural or financial ground (Srinivasan & Colman, 2005). In terms of the process, researchers may be concerned that strong collaborator influence will unduly affect the researchers’ approaches, research questions, or reporting methods (Lomas, 2000; Ross et al., 2003). From the perspective of the collaborator, research domination is a risk: researchers’ training and expertise naturally gives them control of almost all stages of the research process, while decision-maker involvement is often limited to the stages of data interpretation and dissemination (Cousins & Simon, 1996). There appears to be an optimal closeness for working partnerships beyond which partners pose the risk of having too much influence on one another (Goering et al., 2003). Roles and structures must be put into place that limit the amount of formal decision-making input into the research by the decision-making partner, the grant competition process must be transparent, there must be clear distinctions between formal and informal communications, and conflicts of interest, real and perceived, must be surfaced and removed (Goering et al., 2003). Otherwise, it is difficult for both parties to navigate the tension between effective collaboration and independence.

Another issue that has not yet received a great deal of attention is the potential volatility of research findings (Kouri, 1999; Lomas, 2000). Since partnerships have no guaranteed outcomes, the research with which decision-makers associate themselves could have unforeseen implications for the policies and practices which they govern. Future sustainability, beyond the grant funding period, seems to be an unresolved issue especially given the prior investment of time and resources by all partners (Srinivasan & Colman, 2005).

Despite these shortcomings, the benefits offered to members in a partnership exceed what either party could achieve or access on their own, and consequently we assume a positive stance towards collaborative research. Clearly a great deal of preparation and continual planning is necessary to keep partnerships moving beyond the many obstacles that will inevitably arise. Consequently, partnerships need to be monitored so that improvements can be implemented. Indicators provide a transparent, diagnostic checklist guide by which to guide the development of a partnership.

Previous work and the need for indicators

The need for a set of such indicators was suggested as a result of a series of research activities. In 2000–2001, a study was conducted to examine research receptor capacity and research utilization needs within the Ontario Ministry of Health and Long Term Care (MOHLTC) (Edwards et al., 2001). The research receptor capacity study focused on the abilities of Ministry staff to find, understand, and use evidence-based research in policy development processes. The study was conducted using the blended methods of a cross-sectional survey followed by qualitative interviews. It resulted in detailed recommendations to improve access to research information, enhance use of the information once accessed, and promote an organizational culture supportive of research utilization.

The study was accompanied by a literature review on research receptor capacity and research utilization (Griffin & Edwards, 2002). The purpose of the literature review was to identify and describe (i.e. ‘best strategies’) to build research capacity among health policy decision-makers and to facilitate the utilization of research in policy decision-making. As a result of the literature review it became evident that well-functioning relationships between researchers and Ministry decision-makers was a key component in enhancing transfer of research findings into government decision-making. (MacLean et al., 2003). Previous work in identifying partnership indicators had focused primarily on somewhat shorter relationships and on community member-public health system partnerships (MacLean et al., 1997). Determination of appropriate and feasible indicators of linkage and communication with academic-government research partnerships is an underdeveloped area.

A more immediate impetus was the desire of the MOHLTC to have viable, evidence-based indicators to evaluate its involvement with research partnerships through the evaluations that it was required to perform. Thus, development of the partnership indicators presented here was guided by a number of principles identified by the authors at the outset of the study. It was envisioned that members of a partnership would use the indicators in a self-evaluative fashion. For this to occur, the indicators needed to be understandable. It was also assumed that those partnerships that took the time to self-evaluate would also take the time to improve the functioning of the partnership. Therefore the indicators had to clearly indicate areas requiring improvement. Related to this was the idea that the indicators needed to go beyond measuring frequencies (e.g., number of meetings) to include some indication of quality. They had to be relevant for the purpose of this type of partnership; for example, indicators developed for the purposes of community collaboration were not suitable for research/government collaboration. Finally, indicators had to be perceived as credible to members of a partnership, and the information had to be feasible to collect. It was decided to use a qualitative approach in order to allow a more in-depth exploration of areas which previous work on research receptor capacity had highlighted, but which did not reveal sufficient information on new measures about the quality of relationship, allowing exploration of linkage processes.

Methods

This research involved developing indicators that reflected possible linkage mechanisms between Health System Linked Research Units and MOHLTC partners. Combined with the information on research receptor capacity and research utilization needs from the literature (Griffin & Edwards, 2002), the findings of previous work (Edwards et al., 2001), Health System-Linked Research Unit (HSLRU) 10-year outcome reports, and other documents, a tentative set of indicators was developed. The HSLRU programme was selected for these initial steps in indicator development. This programme is one in which participants often had much experience working with Ministry partners, that is, developing research directly intended for transfer into government decision-making.

All eight of Ontario's HSLRUs, and their designated partners at the Ministry of Health and Long Term Care, were used as the study participants with which to develop and validate indicators. Their directors (or designates) participated, along with specific Ministry partners purposively selected by the Ontario Ministry of Health and Long-term Care's Research Unit. As well as providing information of use to this specific partnership programme, it was felt that the broad range of experiences, government partners, and different foci among the HSLRUs, would provide information transferable to other such partnerships.

The HSLRUs were initially established to foster collaboration between academic researchers and the managers of health-care services organizations. These units received core infrastructure funding from the Ontario Ministry of Health and Long-term Care for research staff, trainees, and research projects. The early years of the programme were awarded long-term funding (5 years, renewable). However, the last 15 years were only yearly contracts as the programme's future within the government was uncertain. In the late 1990s, the Ministry required that as a condition of ongoing funding, the HSLRUs were to negotiate research projects to be undertaken with Ministry partners. These projects were to address priority research questions identified by different branches within the Ministry. Thus, as well as their historical partnerships with health services organizations, the academics now had partnerships with the MOHLTC. Further, the HSLRU active researchers now in partnership with the Ministry were not just the academics, but also included researchers from the partner health services organizations. Research initiatives that were deemed relevant by the Ministry and identified as a good fit with the research expertise and interest of the participating HSLRU were negotiated on an annual or bi-annual basis. The project described in this paper was one of these projects.

The first step of the study involved the generation of indicators. To understand formal partnership experiences, telephone interviews with Research Unit directors (or their designates) and their Ministry partners from all eight HSLRUs were conducted in 2002 by the first two authors. Most Research Unit directors were the senior researchers in most of the Ministry work conducted by their HSLRU. Designates became involved if the Research Unit directors felt another senior researcher was more involved with the research and with the specific Ministry partnership.

Interviews were approximately 60 min long. Using a semi-structured interview schedule, participants were asked to describe characteristics of their partnership in its past and current states, key moments in the partnership (positive or negative), suggestions for indicators of successful partnership linkage, and descriptions of how they identified the successful translation of research findings into the policy domain. Sixteen individuals (eight from the Research Units, and their eight partners) participated in the interviews, which were tape recorded and transcribed. Participants were offered the chance to review a copy of their transcript to confirm its accuracy. Interviews were analysed using a modified content analysis approach by two of the authors. Themes were developed from a mix of expected and emerging concepts. Initial codes were developed based on indicators from the literature. As well, open coding occurred such that new ideas were captured. Further, codes were developed based on processes operating within broader constructs. Concepts were organized by themes, and both themes and concepts were systematically compared, triangulated, and findings confirmed or disconfirmed, using matrix methods. For example, the idea of some indicators being the result of maturation time came from observing that some indicators were present for all partnerships, and some only in younger or older ones. The research team came together repeatedly to discuss findings and deliberate possible refutations. Qualitative analysis was supported by QSR N6 software.

The findings from the interviews were used to revise and expand an initial set of indicators, based on a literature review (Griffin & Edwards, 2002), the findings of a previous study (Edwards et al., 2001) and the HSLRUs’ 10-year outcome reports and other documents.

The second step of the study was concerned with the face and content validity of the revised set of indicators. Study participants from the interviews were invited to attend a 2-h focus group, facilitated by two of the authors, to discuss the indicators (and a broader evaluation workbook within which they were incorporated). Questions related to the indicators asked about clarity, feasibility, credibility, relevance, level of specificity, and their ability to support each evaluation question. Six participants from the HSLRUs and one Ministry partner attended the discussion. The majority of the focus group members had been interview participants, though a few were designates (including the Ministry partner participant). Ultimately, this provided an opportunity to both see whether the indicators reflected accurately views of participants in general, as well as providing fresh input. The focus group was tape recorded and transcribed, and the transcript was analysed manually for information about the indicators, as well as for information about partnership experiences as a whole. The indicators were refined in light of this focus group discussion.

The Health Sciences and Science Research Ethics Board at the University of Ottawa approved this study.

Results and discussion

The partners had undertaken a wide range of research studies of policy interest over several different major health content areas (e.g., community health, cancer, dental health, rehabilitation, child health, arthritis, mental health, health information). Many partnerships involved multiple projects, and some projects involved community partners as well as research and government. Projects included literature reviews, surveys, programme and service evaluation, costing estimates for policy initiatives, policy analysis, health system human resource analysis, intervention studies, knowledge dissemination to government and community, and knowledge transfer studies. Each HSLRU worked with different government partners as appropriate to their content area.

The partnerships were developed to support research for health policymaking among different Ministry departments. Within this broad mandate, participants’ descriptions demonstrated that there was diversity across the research projects. The partnerships that supported these projects revealed a variety of general dimensions (column 1 on all Tables) related to policy-relevant research. Study participants described indicators of successful attainment of each dimension (column 2 on all Tables). When probed further, participants were able to identify specific, measurable examples or sub-indicators related to each indicator (column 3 on all Tables). In some cases, the research team was able to draw on their own experiences to identify measurable sub-indicators. The tables also show (by *) which indicators (and sub-indicators) were developed inductively as a result of data gathered from participants solely as a result of this project. The other indicators were developed deductively from the literature, and were supported by the findings of this study.

An important finding was related to the maturity of the partnership. The interviews demonstrated that partnerships within their first 2 years of coming together exhibited different characteristics than those partnerships of an earlier vintage. Below we present ‘common’ indicators, ‘early partnership’ indicators and ‘mature partnership’ indicators.

Common partnership indicators

Table 1 outlines indicators related to communication, collaboration, and dissemination, all of which emerged as key elements throughout all stages of a partnership.
Table 1

Common partnership indicators

Dimensions of partnerships

Indicators of successful dimensions

Potential sub-indicators

Communication

1.0 Communication is clear

1.1 Communication is on-going

1.2 Communication involves face-to-face meetings as well as telephone, mail, email, and fax methods

1.3 The same contact people continue over the life of the project

1.4 A common language/lexicon is used by both parties

 

2.0 Communication is relevant

2.1 **Roles, expectations, and criteria for deliverables are explicit

 

3.0Communication is timely

3.1 Communication is frequent

 

4.0 **Communication is respectful

4.1 **Partners value each other's contributions

4.2 **Partners are acknowledged in project documents

Collaborative research

1.0. Joint meetings occur at most stages of research

1.1 Joint identification of research questions

1.2 Each partner's needs and constraints expressed

1.3 Joint designing of research protocol

1.4 If relevant, joint data collection

1.5 If relevant, joint data analysis

1.6 Joint ongoing evaluation of relevance of research (e.g. current project, new findings, new partner needs etc.)

1.7 Joint discussion of findings and implications

 

2.0 Joint meetings occur to discuss research dissemination and utilization plans

2.1 **Feedback about research report is provided before final draft

2.2 **Response to feedback is prompt

2.3 **Only a few rounds of revisions before deliverable is acceptable to all

2.4 Feedback is given after the final deliverable is received

Dissemination of research

1.0 Multiple formats of written and/or other forms of presentation (e.g., newsletter, website summary, interim report, oral presentation)

1.1 Stakeholders and Ministry partners received relevant documents

 

2.0 Presentation formats in layman's terms

 
 

3.0 Presentation formats include recommendations for action

3.1 Recommendations for action reflect current program and policy challenges

 

4.0 Where appropriate, presentation formats are concise (e.g., less than two pages)

4.1 Presentation formats are similar to those used for other communications within the Ministry (e.g. briefing notes)

 

5.0 **Community stakeholders contacted researcher or government partner to discuss the research findings

 

Note: * shows which indicators were developed from participant data.

The Communication dimension emerged unanimously as an important factor related to the success of a partnership, as also noted by other researchers (Browne, 1999; Kothari et al., 2005a, 2005b). Participants noted that communication needed to be clear, relevant, timely, and respectful in order for an effective partnership to function.

I guess it would be fairly safe to say that at the beginning we didn’t know each other's worlds and so we would talk with each other but we would not be able to identify where the research question was. And we’re so far beyond that now but that's an example of an early signpost of how things were going. That we would spend an hour and a half together and still there was no discernable research question. [researcher participant]

The motivation behind these researcher and policymaker partnerships is to produce policy-relevant research. Not surprisingly, then, the dimension of Collaborative Research arose from our interviews, and it was further characterized as joint meetings during the research process and joint meetings to simply discuss dissemination and utilization plans related to the research results. Joint meetings during most stages of research (question formulation, designing the research, data collection, data analysis, and interpretation) are perhaps the gold standard that partnerships are striving towards, and were reflected in the more successful partnerships. On the other hand, we suggest that joint meetings to discuss the dissemination of the results probably reflect a more realistic image of, or a starting point for, Collaborative Research.

In the context of knowledge translation partnerships, appropriate Dissemination of Research was noted as an important dimension for success. The findings reflect the recommendation of Lavis et al. (2002) that research results be presented in a variety of formats, including short reports; that communication about the study is provided in plain language (not academic jargon) and that policy recommendations accompany the research results. In addition, participants noted the importance of stakeholder involvement: that stakeholders (e.g., patient advocacy groups) were included in the dissemination plans for research results.

Early partnership indicators

Table 2 outlines indicators of specific importance to early stages of partnerships. These include thorough discussion of Research Findings, Frequent Negotiations of a range of partnership factors and Enhancement of the partnership itself.
Table 2

Early partnership indicators

Dimensions of partnerships

Indicators of successful partnerships

Potential sub-indicators

Research findings

1.0 Research findings are discussed in policy deliberations

1.1 Research findings are presented in policy-related format and language

1.2 Implications of findings are understood by all

1.3 Documentation of feedback to researchers

1.4 Ministry senior staff are aware of research findings

1.5 Research findings are discussed or are reflected in government meeting material and research documents

Negotiation

1.0 Negotiation occurs at various stages of the research process

1.1 Roles and responsibilities are documented

1.2 Written terms of reference for research project (or similar document)

 

2.0 Negotiated items are clearly understood by all

2.1 Requirements for deliverables and timelines are documented

2.2 Partners make their needs explicit (i.e., in terms of accountabilities, priorities, and long-term interest)

2.3 Partners document the above needs

Partnership enhancement

1.0. Clear leadership with respect to partnership management

1.1 Key players and senior management, where relevant, are visibly involved and supportive

 

2.0. Development of team mentality

2.1 Discussion of potential long-term plans or structure to ensure continuity of relationship

 

3.0. **Early engagement of people

3.1 **Staff with previous linkages with each other are incorporated into partnership

 

4.0  Exposure to team/organization structures of research partners

4.1 Discussion of respective organizational realities of research partners

Note: * shows which indicators were developed from participant data.

Participants in new partnerships spent a lot of time discussing the dimension of Research Findings. As others have noted, our participants suggested that one indication of a successful partnership is that research results are discussed in policy deliberations. According to a research partner,

So the most important indicator to us, or how we would judge things, success would be whether the research information is used in the development of policy, or considered in the development of policy.

This view was echoed by a policymaker participant:

We will do an assessment evaluation of that which the research unit is doing for us in [specific health area] and then we’ll build it up into policy options and try and have that translated into new or standard program for the Province … . Now if they provide us with a good research piece and if we’ve done this right, and I’ve told them this, we should be able to take what they’ve done and build it up into a real policy piece which would mean implementation.

A noteworthy contribution to understanding these partnerships is the way in which this indicator was elaborated upon. The breadth of issues addressed here – the policy-relevant presentation of research results, the level of awareness about results at senior policy levels, and attention to the utilization of the results in policy discussions – speak to the many determinants of policy development, and suggest multiple strategies to try and ensure that the findings are ultimately discussed in policy deliberations.

Through the interviews it became apparent that Negotiation was an explicit dimension in the partnership. One indicator of successful negotiation was the need for repeated ‘give and take’ throughout the research process, complemented by a second indicator that ensured a common understanding of what was actually being negotiated. This two-pronged aspect to Negotiation has received minimal attention in the researcher-policymaker literature.

Similarly, the idea that members of partnerships must invest in themselves as an entity, captured in the dimension entitled Partnership Enhancement, is distinctly spotlighted in this study. Indicators that capture successful actions around this Partnership Enhancement include the identification of clear leadership, development of a team mentality, early engagement, and exposure to each other's organizational structures.

I think there were a lot of initial issues around somebody thinking they were a higher level than who they were dealing with in the organization. Like if they were dealing with my counterpart, [but] thinking that they should be dealing with a Director. We always make Directors accessible but we have a very strong feeling about the program, and we’re using input from the field to decide on what the deliverables are. So, often the people who know a lot are lower in the chain. [policymaker participant]

Mature partnership indicators

The interviews demonstrated that partnerships that had had the time to develop, demonstrated unique partnership characteristics. Meeting information needs, the level of rapport and the commitment to the partnership were more pronounced in mature partnerships (Table 3).
Table 3

Mature partnership indicators

Dimensions of partnerships

Indicators of successful partnerships

Potential sub-indicators

Meeting information needs

1.0 **Partners are flexible about meeting partner's changing needs and revising research plans and timelines

1.1 Project timelines and changes have been tracked through documentation

1.2 Roles and responsibilities have been defined up front

1.3 Research purpose and objectives have been defined, documented, and referred to in an on-going fashion as the research progresses

 

2.0 **Partners understand the limits of each other's flexibility

 
 

3.0 Partners understand research findings, their limits, and their implications for Ministry work

 

Level of rapport

1.0 Conflict is dealt with openly, informally, and promptly

1.1 More informal communication occurs, though formal meetings and communication continues

 

2.0 **Trust has increased between partners

2.1 **Appreciation is shown of each other's efforts

 

3.0 **Comfort has increased between partners

3.1 **Partners support each other publicly

 

4.0 **Openness has increased between partners

4.1 **Partners provide advance notice of surprising or potentially contentious research findings or government decisions

 

5.0 **Partners begin speaking a common language regarding research

 
 

6.0 **Partners facilitate removal of barriers for each other's work

6.1 Partners understand:

• **how things are communicated within the partner organization;

• **how senior level people work and what their concerns are;

• agendas, priorities, expectations, and limits;

• dissemination opportunities within the partner organization;

• opportunities for research use and impact within the partner organization;

• costs of monitoring, influencing, and incorporating research into decision-making

 

7.0 **Linkage with partner enhances partner linkage with community/other stakeholders

7.1 **Linkage with partner does not detract from previously established linkages with other partners

Commitment

1.0 **There is joint commitment to the research project

1.1 The partners contribute more resources, material and otherwise to the research project

1.2 **Partners willingly provide ‘extras’, such as extra time or staff, to the project

 

2.0 **There is an increase in joint activity around the project

2.1 **Partners take on new roles with each other

2.2 **On-going dialogue moves a research programme forward over a series of projects

 

3.0 **Partners are perceived as experts in the research/ policy area and are referred to as such to others

3.1 **Partners introduce each other to new networks

3.2 Partners think of each other in relation to projects, committees, etc., outside of the research project relationship

 

4.0 **An informal or formal infrastructure exists for linking and transferring research between partners

4.1 The partnership's work becomes integrated with work associated with other stakeholders

Note: * shows which indicators were developed from participant data.

The idea that partnerships ought to Meet Information Needs seems straightforward, and our participants’ elaboration on this dimension supports the ‘two-community’ metaphor that has been used to position researcher and policymaker worlds (Dunn, 1980). In a successful partnership, our participants reported an acknowledgement of each other's needs, time lines, and limits of each other's flexibility. Furthermore, there is a mutual understanding of the implications of the research results for each other's worlds.

I would say in the middle of that project [we] started to see a lot of flexibility, dedication, understanding or merging understanding and appreciation of the other parties efforts and that's both ways … [policymaker participant]

In this study, participants identified that the Level of Rapport was an important dimension to consider when discussing partnerships. Rapport was associated with a number of possible indicators revolving around conflict, trust, comfort, openness, and common language between partners. Rapport was also linked to the removal of barriers for each other's work (e.g., easing the way for appropriate communication of research results). As well, partners with strong rapport described how the relationship led to a strengthening of linkages with other community stakeholders.

To work with people who research this work all the time, have done all kinds of work already, so you feel a huge imbalance in terms of your comfort level in having that preliminary conversation. But as I say, over time that changed. I mean now the lead researcher, you know, he and I talk periodically and you know he comes to me for advice and every once in a while I call him up and say what do you think of this and have involved him in other things. [policymaker participant]

In particular, there were some unintended outcomes related to a Commitment dimension that are worth highlighting. Participants revealed that more mature partnerships tended to exchange more information or services outside the immediate scope of the project. In other words, partners were more apt to exchange and expect ‘freebies’ within a comfortable partnership.

By the time we came to a second year, to figure out what we wanted to do together for a second year, that was a much more effortless discussion and probably ah, you know I remember it as a very productive discussion where we were able to build on each other's ideas and it became, you know, started to consult with them on other matters and involve them in other things. [policymaker participant]

And, according to another participant:

And [the HSLRU] also donated time and effort like well beyond what would have been expected or expected to be contributed at the beginning. [policymaker participant]

These indicators, relating to common, early, and mature partnerships, improve our current understanding of partnerships and linkages for knowledge translation. They provide detailed guidance about the facets of common labels like ‘communication’. Most importantly, the indicators imply that the way linkages are measured (for success) needs to reflect the stage of their development. A new linkage might be unfairly judged if measured against, for example, the ideal standards of effective, informal communication channels that develop with more mature partnerships.

Thus, our suggestion for using these indicators is as follows. A new researcher-policymaker collaboration could begin with a discussion about the common and early partnerships indicators. In this way, expectations about future partnership assessments are clear, and the documentation required to complete future assessments is collected from the outset. This discussion might also establish the points at which partnership assessments will occur. As well, partners may need to determine how or who will be reviewing the documentation and/or collecting new data if needed; it is assumed that the indicators can be used in a self-assessment fashion. Researcher-policymaker collaborations that have existed for more than 2 years may initiate a similar discussion using the common and mature partnership indicators.

The indicators presented here are one way to manage knowledge creation partnerships, or evaluate partnerships, between researchers and policymakers. They were developed with consideration to previous studies as well as to the experiences of those currently in such a partnership; in particular, data were collected from both the researchers and the decision-makers. This ‘bottom-up’ input represents a strength of this study. The indicators are based on a variety of projects that were part of a formalized agreement, required as a condition of HSLRU funding. That is, the indicators were derived from activities of well-established research units that had a committed group of investigators and strong infrastructure support. Furthermore, some of the HSLRUs had worked with the same Ministry partners on a variety of research projects over a few years, while in other cases these were new partnerships. These features contribute to an ideal study setting from which to derive indicators.

As mentioned earlier, the indicators were developed further into an Evaluation Workbook to assist the Ontario Ministry of Health and Long-term Care in their evaluation of research transfer activities between Ministry-sponsored external research projects and associated Ministry partners. As part of the validation process, a retrospective pilot process was carried out. The information in the project grant files of six selected projects was applied to the workbook, and based on the findings, the Evaluation Workbook was refined further. Most importantly, this experience highlights the need for a future prospective pilot study to generate evidence on the applicability of the tool in practice. Other future studies using these indicators might focus on prioritizing them, determining optimal frequency of measurement, usefulness in modifying the partnership midway through the partnership, or determining the extent to which they predict the use of research by policymakers. Alternatively, one might study which indicators are better suited for partnerships with bureaucrats, and which are better for collaborations with elected officials. Validation and reliability work would be required to optimize issues of reliability, validity, and generalizability. Such a study would also want to consider whether there are instances in which the indicators may obstruct the partnership. For example, if followed too dogmatically, could partners lose sight of the relationship forest for the indicator trees? When would other approaches to evaluation be more productive?

Another area for further study is the maturation of such partnerships. For us, in our case study, the terms ‘early’ and ‘mature’ reflect the dimension of time alone, as, consistently, a 2-year period showed the relationship change. However, it is quite likely that other factors are at play in the maturation of a partnership. For example, a partnership could exist that is producing minimally useful research but not moving forward in terms of relationship development (e.g. filled either with conflict or lack of sufficient engagement). Such a partnership could last 3 years, but not progress to a partnership with the more ‘mature’ characteristics. This would be consistent with other theories of how people work together over time, such as in the community partnership literature (MacLean et al., 1997). More research, however, needs to be done further delineating and evaluating these aspects of partnerships under different conditions.

The broader science-policy interface is wrought with political determinants. What is interesting to note for our purposes is the presence of socio-political conditions that could provide incentives or disincentives for the possible formation of partnerships. For example, policymakers can be faced with science that is often contested, or only partially available, for complex policy issues. One strategic manoeuvre in this case might be to conceal the uncertainties in the science (perhaps by not involving the scientists). Another manoeuvre might be to characterize the uncertainties through risk assessments and the like, thereby redefining the political problem to one that involves researchers and science. And it might be argued that policymakers are becoming more involved in partnerships to craft data that align with current ideologies and policy practices. An example of another condition influencing potential collaborations is the notion that knowledge generation has moved out of the scientific domain and now involves stakeholder involvement through a new social contract. Our own experiences have shown that a common stimulus for partnerships can be traced to health funding agencies that require collaborations with decision-makers as part of their funding requirements. Our study participants pointed to organizational arrangements as an important disincentive; when yearly funding contracts became the norm, research projects became more short term in nature.

The question arises as to the importance of shared values and ideologies in achieving successful partnerships. In our data, it appeared that having a good partnership allowed the partners to overcome actual differences of values and ideologies that might have impeded the work. The one set of shared values and ideology that was critical, however, was the commitment to open collaboration and to the work required to develop and maintain the key elements of success: communication, rapport, and negotiation. When these were not valued or when the partners did not have the skills to manifest them effectively, difficulties ensued. The role of a facilitator in helping the two communities to work through partnership processes was not explored. However, the role of culture brokers, people with experience in both worlds, serving as bridges would be one avenue. It is not uncommon for such people to be available, as the flow of academics into government management, and more rarely, the reverse flow, would attest (Goering et al., 2003). Other potential facilitators include those trained in group processes, with special understanding of the content areas involved. MacLean et al. (2000) make such a suggestion in the area of research facilitation for multi-disciplinary teams, where some of the processes are similar to what we recount here. Who currently exists in these roles and what would make the optimal facilitation approach under which circumstances would be of interest for future studies.

At the start of this article we noted that this study contributes to the literature related to researcher-policymaker partnerships, while acknowledging the tremendous amount of work done already in the area of community-based partnerships. The Mode 2 research model, with its emphasis on context and user-driven knowledge, raises further questions about the function and roles of partnership models that include all three stakeholders: policymakers, researchers, and community agencies/members. Community involvement can, theoretically at least, lead to research and policy priorities and/or solutions that are more relevant, credible, and feasible than those that are designed without on-the-ground input. Interested readers are directed to Lencucha et al. (2010) for a fuller discussion of these issues. Future studies could begin to unravel the complexities inherent in such relationships.

Concluding implications for partnerships and knowledge generation

The indicators identified through this project will be of relevance to other researchers and research units working with government partners. Similarly, findings yield useful indicators for consideration by government partners who are working with researchers based in academic institutions. Clearly these relationships focus on research with practical application, and we recognize that not all research is meant for this purpose. Thus, the indicators are aligned with research with an intended usability function, that is, Mode 2 research.

While the study here was conducted with research units who had long-term relationships with government partners, sometimes over multiple projects and programmes, we maintain that results are still transferable beyond this set of partners: as described previously, the units represented a broad range of research type and content. Further, these findings are supported by previous collaborative research about somewhat shorter-term partnerships between community health organizations and community members (MacLean et al., 1997). As with the community partnerships, it is important to heed the different sets of indicators that are relevant for early and mature partnerships. Discussions of the pros and cons of collaborative partnerships between researchers and decision-makers will continue. For those instances where the use of such partnerships has been decided upon, and where situations of accountability apply, indicators are important. Besides supporting well-meaning partners in developing positive, affirmative, productive relationships, use of these indicators would provide a means to monitor processes for policymaker-researcher partnerships and may provide guidance for an alternative set of ‘deliverables’ to be considered in negotiated agreements.

Copyright information

© Operational Research Society 2011

Authors and Affiliations

  • Anita Kothari
    • 1
  • Lynne MacLean
    • 2
  • Nancy Edwards
    • 3
  • Allison Hobbs
    • 4
  1. 1.School of Health Studies, University of Western OntarioCanada
  2. 2.Population Health Improvement Research Network, University of OttawaCanada
  3. 3.School of Nursing, and Department of Epidemiology and Community Medicine University of OttawaCanada
  4. 4.Schulich School of Medicine, University of Western OntarioCanada