Background

Research co-production shows great promise for connecting science to societal problems. Research co-production can be rigorous and ethical [1,2,3,4,5,6,7] and serve as a vehicle for generating and translating scientific findings into action [8]. Research on implementation science and scaling science [9] demonstrates that the use of rigorous research designs is only one consideration when implementing and scaling innovations—context, user/beneficiary perspectives, and systems matter just as much. The active involvement of users (those who may move research findings in action) and beneficiaries (those who may be affected) can be a crucial predictor of success [8,9,10,11].

Research co-production comes in many forms and under many different names. Among others, community academic partnership [11]; community-based participatory research [12]; co-creation [13]; and integrated knowledge translation [14,15,16]. A research study involving experts from a range of five research co-production traditions [17] found that the definitions and motivations of each type of co-production research were very similar. While there are many different names, engaging the users of research in the research process is a common goal. Therefore, we anticipate the results of this study to hold potential beyond the immediate sample. See Table 1 for general definitions of selected co-production traditions.

Table 1 Research co-production traditions
Full size table

The need for better evaluation approaches for co-production

There is growing dissatisfaction with the approaches available for assessing the quality of research co-production. Traditional approaches to research quality assessment do not take into account engagement with knowledge users and, as such, do not address a key factor in the hypothesis behind research co-production: that meaningful researcher - knowledge user partnerships make a difference to the quality of the evidence that research produces [8, 10].

Table 2 outlines the predominant forms of research evaluation—as classified and further discussed in Chapter 4.3 of Research Coproduction in Healthcare [25]—and describes how these approaches can undervalue research co-production.

Table 2 Mainstream evaluation stacked against co-production [25]
Full size table

Objectives of the RQ+ 4 Co-Pro field test

The purpose of this study is to field test the relevance and utility of an adapted research quality evaluation approach that was first developed and validated by the International Development Research Centre. This approach, called Research Quality Plus (RQ+) [29, 30], has previously been used to assess applied and use-oriented research. For a full explication, see McLean et al. [29]. With this study, we will test whether RQ+ can be adapted for assessing the quality of co-production researchFootnote 1. This prototype adaptation is called the Research Quality Plus for Co-Production, or RQ+ 4 Co-Pro, framework (25). See Fig. 1 below for key definitions of RQ+ and RQ+ 4 Co-Pro.

Fig. 1
figure 1

Key definitions

Full size image

Two research questions guide the field test:

  1. 1.

    Is the RQ+ 4 Co-Pro framework relevant for the evaluation of research co-production?

  2. 2.

    Is the RQ+ 4 Co-Pro framework useful for the evaluation of research co-production?

The Research Quality Plus for Co-Production (RQ+ 4 Co-Pro) framework

The adaptation of RQ+ into the RQ+ 4 Co-Pro framework is illustrated in Fig. 2. RQ+ 4 Co-Pro was first proposed by authors of this paper following their experience designing and using the initial RQ+ framework at IDRC, doing research evaluations internationally, and doing research co-production (25). This is a prototype rendition. The study described in this manuscript aims to field test the protype. In Additional file 1: Appendix I, the fully detailed RQ+ 4 Co-Pro framework template is provided for the interested reader; it includes the definitions of each framework component and the associated evaluative rubrics. Additional file 2: Appendix II provides a crosswalk of the components of the RQ+ framework with the components of the RQ+ 4 Co-Pro framework.

Fig. 2
figure 2

The RQ+ 4 Co-Pro framework [adapted from infographic originally published by authors (RKDM, IDG, FC), and secondly in 25]

Full size image

The RQ+ 4 Co-Pro framework embraces the three tenets of the RQ+ Approach. These are as follows: (1) context matters, (2) quality is multi-dimensional, and (3) assessments should be empirical and systematic. These are modified from the RQ+ framework to reflect the particularities of co-production research. Here we provide a description of how each tenet was tailored, and then introduce an all-of-framework infographic (Fig. 2) to show how the three tenets fit together.

Contextual factors

Research always occurs in a context. Research is affected by and affects the socio-economic, historical, cultural, and political contexts as well as the geographic and institutional setting.Footnote 2 Attention to context is particularly important in the evaluation of the quality of research co-production [31]. We identify three contextual factors that can be monitored and categorized in a co-production evaluation. The goal in examining contextual factors is to gather information that can help to understand and navigate the enabling environment for co-production research. Understanding context is important to research design, management, and funding decisions as it helps clarify potential risks and opportunities and might also help with the development of strategies to capitalize on these and monitor progress. The contextual factors are not intended to affect the ratings of research quality dimensions or sub-dimensions, nor is any rating of a contextual factor necessarily “better” than another. Rather, they help to provide a deeper understanding of the enabling environment.

The three RQ+ 4 Co-Pro contextual factors are as follows: (1) Knowledge Use Environment, (2) Research Environment, and (3) Capacities for Co-Production. In the International Development Research Centre’s current RQ+ framework, there are five contextual factors. Three are closely aligned to those here, given some tailoring to match co-production specifically. The additional two contextual factors, Data Environment and Maturity of the Research Field, are not included in RQ+ 4 Co-Pro as they present less immediate alignment with the aims of co-production. The decision to reduce the number and tailor the contextual factors for RQ+ 4 Co-Pro was the result of consultations between authors of this paper, and their shared experiences with co-production and co-production evaluations [25]. With this field test, we will further examine the relevance of these three contextual factors and determine the need to modify, exclude, or include new elements on grounds of relevance and/or utility (see research questions above). Additional file 2: Appendix II provides a crosswalk of the RQ+ contextual factors vis-à-vis the RQ+ 4 Co-Pro contextual factors.

Quality dimensions and sub-dimensions

To assess co-production quality, we identify three dimensions and eight sub-dimensions. These are summarized in Fig. 2 and presented in detail in Additional file 1: Appendix I. Additional file 2: Appendix II crosswalks these dimensions and sub-dimensions with those of the RQ+ framework.

As with all research, Scientific Rigour is central to co-production research and therefore comprises the first dimension. Two sub-dimensions are identified under Scientific Rigour: 1.1. Protocol which addresses issues of study design, and 1.2. Methodological Integrity which assesses the rigor and integrity of the application of the study design. Research Legitimacy is the second dimension of RQ + 4 Co-Pro. There are four sub-dimensions to Research Legitimacy that assess the fidelity of the research to the operating environment. These are as follows: 2.1 Inclusion of Local Knowledge and Ways of Knowing, 2.2. Trust, Power and Mutually Beneficial Partnerships, 2.3. Intersectionality, and 2.4. Attention to Potentially Negative Consequences. The third and final dimension is Positioning for Use. It assesses the utility of the co-production research through examining 3.1. Relevance or how well the work is aligned to a current problem, and 3.2. Openness and Actionability which addresses accessibility and usefulness of the research findings.

In the RQ+ 4 Co-Pro framework, all dimensions are interrelated and should not be considered as variables that are independent of each other. They are disaggregated to promote a deeper understanding of the multiple dimensions of research quality—ultimately, they must be considered as a set. We assign equal weight to the dimensions and sub-dimensions; others may choose to prioritize or highlight some sub-dimensions over others in any assessment they design.

Empirical and systematic appraisal

Column 3 in Fig. 2 outlines the scale to be used for measurement. RQ+ 4 Co-Pro users apply a rubric for measurement which ensures transparency in the results and promotes a systematic approach across all the research that is being assessed. A combination of qualitative explanations and quantitative measures of sub-dimensions should be used to reach conclusions about the quality of the co-production research. In the following sections, we outline how empirical evidence will be gathered in our field test.

Table 3 below outlines how RQ+ 4 Co-Pro addresses some key recommendations from studies on research co-production approaches.

Table 3 What RQ+ 4 Co-Pro can learn, benefit from, and build on from existing frameworks, experiences, and systematic reviews [25]
Full size table

Study design

This field test will use a multiple method qualitative design. It will include training of participants, standardized data collection using desk-based templates, and follow-up qualitative interviews with both the assessors and those whose projects have been assessed. As well, it will include a consultative process with the project team for revising RQ+ 4 Co-Pro based on the outcomes of the field test [35, 36].

The study will take a research co-production approach. To do so, the study is being undertaken as a partnership between researchers and knowledge users. All activities and responsibilities will be shared, yet, five team members (authors: RKDM, FC, IDG, AK, CM) are primarily responsible for study design and execution. Thirteen team members (authors: ABA, RA, JB, CEC, OD, EDR, LAF, MG, AMH, RK, SK, JR, GS) hold primary responsibility for identifying knowledge uptake and use opportunities. These “knowledge user” team members represent critical stewardship roles for research co-production broadly, including funders, university administrators/leaders, research evaluation specialists, journal editors, co-production trainees, research managers, and co-production scientists. By working together to field test RQ+ 4 Co-Pro, we hope to spark reasoned and appropriate uptake of the framework into settings where current co-production evaluation techniques demand revision and innovation.

The field test will be implemented in four phases, which comprise eight steps. Figure 3 presents an illustration of the complete research life cycle.

Fig. 3
figure 3

Outline of the research life cycle

Full size image

Phase 1—Study preparation

Sample recruitment and participant consent

Researchers in the Integrated Knowledge Translation Research Network (IKTRN) will be invited to submit projects for assessment and volunteer to assess another project. IKTRN is a network of researchers with an interest in both using and carrying out research on integrated knowledge translation. IKTRN is funded by a multi-year grant from the Canadian Institutes of Health Research [19]. The sample will be a convenience sample. This sample will be drawn from researchers with IKT research experience (members of the IKTRN) and who have recently completed an IKT project (have an IKT case published in the IKTRN casebook series). This invitation will be delivered by email from the study PI to eligible members of the IKTRN, until the desired sample size of 16 projects is reached, with a maximum of 20 projects. Enrolled participants will be arranged in dyads based on research topic familiarity for the assessment.

The sample range (16–20 projects) is based on two factors. The first is viability given resource requirements of past experiences using the RQ+ approach, and our own study timelines and resources for this project. The second is the anticipated saturation point of qualitative data collected in the field test [37].

Eligibility criteria

We will consider eligibility at two levels: (1) the IKT research project and (2) the individuals participating in our study, as described below in Tables 4 and 5 respectively. The research team will gather informed written consent from all participants.

Table 4 Project eligibility criteria
Full size table
Table 5 Participant eligibility criteria
Full size table

RQ+ 4 Co-Pro framework training

Participants will receive training in RQ+ 4 Co-Pro. Training will be provided by the core research team, with the aim to introduce the RQ+ 4 Co-Pro framework, the definitions and meaning of its components (contextual factors, quality dimensions and sub-dimensions, evaluative rubrics), and systematize the approach to its use by participants. The 2-h training will be completed prior to the initiation of all data collection.

Phase 2—Data collection

Data collection will involve four steps: (1) completion of a participant socio-demographic form, (2) completion of a project information form, (3) dyadic RQ+ 4 Co-Pro assessments, and (4) participating in an interview with the research team on the strengths and limitations of the RQ+ 4 Co-Pro framework.

Step 1—Participant socio-demographic form

All participants will be sent a link to an online socio-demographic form. This form will collect information on participant demographics and their experience/background developing and/or delivering IKT research projects. We will ask that participants complete this form prior to taking part in the training session (5 min).

Step 2—Project information form

All participants will be sent a link to a project information form. This form will be pre-populated by the study team as much as possible to profile the project included in the field test. The study participant will verify and complete any missing information on the project profile prepared by the study team (10 min).

Step 3—Dyadic project assessment exercise

Each participant in the paired dyad will provide the other, who will serve as assessor, with publicly available documentation on the project they will be assessing (inter alia publications, manuscripts, reports, briefs, blogs, etc). These assessors (study participants in dyads [38]) will review this material to gain an understanding of the project context, as well as its strengths and weaknesses (1-2 hours). Next, the assessors within each dyad will engage in an assessment interview about their projects using the RQ+ 4 Co-Pro training and the field test template (see Annex 1) provided by the research team. Assessment interviews may be done in one virtual call or split in two as the two determine. It is estimated they will last 60 min per project. The field test template will be used by the assessors for recording results of the assessment during the interview.

Step 4—Research interview with RQ+ 4 Co-Pro study team

On completion of the dyadic assessments, members of the research team (RKDM, FC) will interview the assessors (study participants) individually using a semi-structured interview guide to elicit their views as both assessor and assessed on the utility and relevance of using RQ+ 4 Co-Pro to assess the quality of IKT research [36]. The interviews will be completed by phone or video conference, depending on participant preference. Interviews will take approximately 60 min.

Phase 3—Analysis and revision

Data analysis

Data analysis will be conducted for each data source independently (Participant demographic forms, Project information forms, Interviews with study participants), and triangulation will be conducted across the independent lines of evidence for congruence as well as instances of discordance.

Step 1—Participant demographic form analysis

Frequencies will be generated for all closed-ended questions. Responses to open-ended questions will be analyzed for common and disparate themes using content analysis. Analysis will provide an overview of participants’ backgrounds and experiences brought to the field test.

Step 2—Project information form analysis

The project profile forms will be analyzed using content analysis to provide an overview of the nature of the projects included in the field test.

Step 3—Interview analysis

Qualitative interview data collection and analysis will occur simultaneously so that identified themes can be incorporated into future interviews. Interviews will be audio recorded with permission of the interviewee. Where permission for transcription or recording are not granted, the interview notes will be sent to the interviewee for review.

We will use thematic analysis [39] to identify patterns in the interview data. We will use an inductive or data-driven approach, without using a pre-existing coding frame. The coding will be modified based on new findings and in collaboration among interviewers. The first two interviews will be coded by two researchers independently and the results compared. Differences will be discussed to ensure agreement on a common approach for the remaining interviews. If agreement is not achieved between the two researchers, a third researcher will arbitrate opposing views and provide a third opinion to reach majority decision if consensus is not achieved.

Step 4—Triangulation and analysis

As a final step in data analysis, we will look for similarities and differences of note in the study data by comparing findings across the lines of evidence. We will conduct triangulation by data source and by data collection method. Data will be considered in triangulation by using identified codes and themes to compare data. For example, we may cross tabulate all projects with a timeline of more than 4 years (as identified in the project information form), by perspectives around the importance, or lack thereof, of using contextual factors in the RQ+ 4 Co-Pro framework. This is a hypothetical example. Triangulation will be driven by identified themes in the data.

Revision of the RQ+ 4 Co-Pro framework

Based on the findings of the research, we will revise the prototype version of the RQ+ 4 Co-Pro framework. To facilitate this revision, the research team will host a meeting of all team members (including our knowledge users) to review and discuss preliminary research results and how these may induce the desire for change to the framework or its components. The reasons for changing a framework component will relate to the two research questions driving the study: relevance of the framework components and utility of the framework components and its application. Following team iteration, we will prepare any required revisions and represent the revised framework to the study participants for review/member checking.

Phase 4—Results sharing

The final version of the RQ+ 4 Co-Pro framework will be published in a study findings report. This report will be submitted to an open access peer-reviewed journal for external assessment by co-production specialists. Uptake and use strategies will be developed by/among knowledge user perspectives represented on our co-production team.

Discussion

This study will yield knowledge about a new way to assess research co-production. Specifically, it will address the relevance and utility of using RQ+ 4 Co-Pro, a framework that includes context as an inseparable component of research, identifies dimensions of quality matched to the aims of co-production, and applies a systematic and transferable evaluative method for reaching conclusions. As we have argued in this paper (see Table 2), evaluation is a needed area of innovation for research co-production to reach its full potential. As we have presented (see Table 3), we are not alone in raising this call.

Limitations

There are limitations with our study design and study methods. The first is while we propose RQ+ 4 Co-Pro should apply to all research co-production approaches, we have limited our sample to IKT projects as a specific sub-domain of co-production which may limit generalization to other partnered research approaches. Second, our sample of IKTRN projects will prioritize experiences of the global North, as IKTRN membership is largely comprised of members from Canada, Australia, and the UK. Third, we further limited our sample to completed projects and so the study will not test the potential use of RQ+ 4 Co-Pro at the design and implementation stages of co-production projects. Fourth, given the evaluands in our field test are research projects, generalizability to other evaluands such as organizations, project portfolios, or grant applications should be tempered. Fifth, the approach to applying the framework will focus on discussion with principal investigators and documentary analysis. In future uses of the framework, users may wish to be more holistic and include more data sources, for example interviews with end users (however, if the framework is not considered useful to researchers, it will likely be problematic for knowledge users.) At the same time, other applications may go into less depth than we do in this field test, for example using a checklist for project design or application review. Using our design, we cannot be certain our field test experience will generalize to these other potential uses of the framework. Sixth, this field test is limited to health research projects although we recognize that co-production is an approach used in multiple domains of science. Finally, all study participants are part of the same network; this may risk a more positive assessment of each other’s projects due to social bias. However, the goal of the research is to assess the relevance and utility of the framework, not to draw a final conclusion about the quality of co-production research endeavors sampled.

Looking ahead

We expect that our findings will add to the existing options for assessing co-production research that may benefit researchers but also other stewards of research co-production. Accordingly, we undertake this study as a co-production team with varied experiences and constituents we currently represent. Some potential uses may include funders interested in new ways to select, encourage, and/or evaluate co-production, including at different phases of the research life cycle. It may also give journal editors a higher level of comfort with the quality of research co-production they publish. Research institutions, such as universities or think tanks, may benefit from assessing the quality of co-production they do using a framework tailored to their values, objectives, and context. In the final research report of this RQ+ 4 Co-Pro field test, a section discussing users and uses of RQ+ 4 Co-Pro will be elaborated. We will also tailor outputs and use strategies to the identified needs of knowledge users within our team. These efforts may not appear in peer-reviewed journals or other scholarly publication formats, but instead as use-oriented outputs and activities.