Background

The concept of active involvement in research of stakeholders was founded on the principle that the public have a moral right to contribute to decisions about what research is undertaken and in what way [1–3]. We define stakeholders as any potential knowledge user whose primary job is not directly in research [1], including people with a healthcare condition, their families, friends and caregivers, health professionals, decision makers and others. It is now widely accepted that active stakeholder involvement is beneficial to the quality, relevance and impact of health research [2, 3], and this has driven national strategies in many countries to ensure involvement in all research activities [4]. There is now an expectation from funding bodies, including government and charities, that researchers will actively involve patients and the public in their research, including systematic reviews [2, 5–8].

Systematic reviews aim to inform and support the delivery of evidence-based practice, by finding and bringing together, in an explicit and transparent way, all the research evidence that addresses a particular topic or healthcare question. Active stakeholder involvement within systematic reviews has been proposed as a way to enhance the actual and perceived usefulness of synthesised research evidence, addressing barriers to the uptake of evidence into practice [9]. While there are a number of examples of active stakeholder involvement in systematic reviews, the approaches to, and extent of, involvement have varied considerably [5, 10, 11]. Cochrane, an international organisation which produces systematic reviews of healthcare evidence, has had patient and public (described by Cochrane as “consumer”) involvement as an explicit principle of the organisation since it began in 1993 [12, 13]. However a recent review of Cochrane consumer activity concluded that while Cochrane consumer contributors comment on protocols, reviews and plain language summaries, there are few examples of active consumer involvement in the conduct of Cochrane reviews [5, 14]. This review focussed on consumer involvement at the organisational level (with an emphasis on Cochrane), rather than activities and roles of individual researchers and how they may involve stakeholders in their reviews. Thus there remains a lack of evidence about the best ways to actively involve stakeholders in systematic reviews [15], and the impact of involvement on research activity and uptake of evidence [2].

The aims of this systematic review are to:

  1. 1.

    find and bring together evidence relating to stakeholder involvement in systematic reviews in order to provide a broad map of the current evidence-base (a scoping review), and,

  2. 2.

    use this evidence to:

    1. A.

      Describe the methods or approaches which have been used in relation to stakeholder involvement in systematic reviews

    2. B.

      Summarise the evidence relating to the effect of stakeholder involvement on the quality, relevance or impact of systematic reviews

Methods

Design

We plan a mixed method evidence synthesis, using a contingent design to systematically map and assimilate evidence in relation to our research objectives. Systematic maps of research evidence are useful for informing the planning, conduct and interpretation of an evidence synthesis [16]. Contingent designs comprise a cycle of research syntheses, conducted to address clearly defined objectives, and assimilate evidence according to its relevance to a clear objective or question, rather than grouping studies according to whether they have a qualitative or quantitative research design [17]. An outline of the planned contingent design is illustrated in Fig. 1. The design will therefore incorporate a broad map of evidence relating to stakeholder involvement in systematic reviews (a scoping review, [18, 19]), followed by two contingent syntheses each specifically addressing a different research objective.

Fig. 1
figure 1

Outline of contingent review design

Full size image

For the scoping review, and following the Arksey and O’Malley (2005) framework, we will use an iterative team approach to ensure clarity of purpose and balance between breadth and comprehensiveness of the review [18, 20]. In addition to traditional scoping review methods, in order to facilitate efficient identification of relevant up-to-date literature, we will implement a stepwise approach [21] to the identification of literature for inclusion. This approach involves a series of pre-planned searches, with progression from one stage (or ‘step’) to the next dependent on consideration of the results of the previous step, aimed at enabling efficient identification of the most relevant evidence. Pre-agreed criteria and contingencies will be used to inform discussions and reach consensus on whether to progress to the next step of searching, and to define the parameters of that search, or whether to cease searching. Year of publication has been selected as central to the proposed searching steps as there is evidence of rapid changes in stakeholder involvement in research over time [6], and we anticipate an increase over time in reported stakeholder involvement in systematic reviews. The proposed stepwise approach to searching for the scoping review is illustrated in Fig. 2. We acknowledge that this is a novel approach to searching, and we hope to learn methodological lessons from use of this approach.

Fig. 2
figure 2

Stepwise approach to searching

Full size image

Scoping review methods

Search methods: step 1

Step 1 of the stepwise search strategy will involve comprehensive searching of electronic databases [22], from 01/01/2014 - current date 2016, and will be supplemented with searching other sources. Databases will include CENTRAL (CDSR, DARE, HTA, Cochrane Methodology Register), AMED (OVID), DoPHER (EPPI-centre database), Embase (OVID), Medline (OVID), CINAHL (EBSCO) and Joanna Briggs database. A comprehensive search strategy will be used, adapted for each database, using established search filters. An example search strategy is provided in Additional file 1.

Other searching to be included in Step 1 will include hand searching of ProQuest Dissertations & Theses (UK & Ireland), Epistemonikos, PDQ-Evidence and Research Involvement and Engagement journal. We will consider the papers identified in Cochrane Consumer Function Review [14], the reference lists of known reviews relating to involvement e.g., [6, 23], and the reference lists of all included studies. In order to identify unpublished data, expert experience and ‘grey’ literature relating to involvement in systematic reviews, we will contact authors of published papers, contact relevant organisations and promote this review via social media.

Search methods: subsequent steps

To inform the team decision as to whether to progress to the next step of searching, and if so the details of the search strategy for the next step, data will be collected on the number of unique references identified for inclusion from each source searched. Redundancy of individual databases will be considered in relation to the number of unique citations judged to meet inclusion criteria. Efficiency will be considered by exploring the proportion of papers included in relation to the number of titles within the search results from each individual database. These data will inform the iterative team decisions for subsequent steps, which will consider expanding the searches of each individual source by a further 2 years. This process will continue in a cyclical fashion, until it is perceived that the optimal breadth and comprehensiveness of evidence has been identified, with consideration of data saturation and available project resources (Fig. 2). To ensure transparency of the iterative team decisions, we will audio record each of the meetings and will report key themes related to all decision making.

Criteria for inclusion in the scoping review

Selection criteria for inclusion in the broad map of research evidence will be purposefully wide. We will include any paper, published or unpublished, regardless of study design, including commentaries, letters and expert opinion, which investigates, reports or discusses any aspect of involvement in a systematic review.

Types of evidence

We anticipate that we will include (but will not be limited to) the following types of evidence:

  • Published systematic reviews which report involvement

  • Reports of methods of involvement in an individual systematic review

  • Reports of methods of involvement in an organisation which commissions, undertakes or supports systematic reviews

  • Studies quantitatively or qualitatively evaluating involvement in individual systematic reviews, or in organisations which commission, undertake or support systematic reviews

  • Opinions, commentary and discussion relating to involvement in systematic reviews or organisations that commission, undertake or support systematic reviews

Definition of stakeholder

As stated above, we define a stakeholder as any potential knowledge user whose primary job is not directly in research. Potential knowledge users include a broad range of people, including those who are actual or potential recipients of health or social care, where this may include patients, carers and family members, or people interested in remaining healthy who are seeking information about a health condition or treatment for personal use [24]; members of organisations that represent people who use services; people with a professional role in health and social care; policy makers and managers. Given the variety of terms used to describe stakeholders (e.g., “consumers”, “patients and the public”), the different types of stakeholders who could be involved, and the importance of distinguishing between the perspectives of the public and the perspectives of people who have a professional role [25], we will document, categorise and report the types of people involved within any evidence included in this review.

Definition of systematic review

We will define a systematic review as a research process in which literature relevant to a stated question is identified and brought together (synthesised) using explicit methods [26], including reporting of inclusion/exclusion criteria, search methods and details of included studies. We will include studies which report involvement in systematic reviews regardless of the type of evidence synthesised in the systematic review (i.e., quantitative, qualitative, mixed-methods) and the type of question addressed (e.g., intervention effectiveness, diagnostic test accuracy, patient experiences).

Definition of involvement in a systematic review

We will apply a wide definition to stakeholder involvement in a systematic review (or organisation which commissions, undertakes or supports systematic reviews). We will include reports relating to any role or contribution toward the development of a review protocol, completion of any of the stages of a systematic review, or dissemination of the findings of a review.

We will exclude:

  • Reports of involvement focused specifically on the generation of research priorities and questions, unless these are explicitly questions for systematic reviews.

  • Systematic reviews focused on synthesising evidence relating to stakeholder involvement in primary research.

  • Discussion and commentary relating to stakeholder involvement in research more broadly, or in guideline development, unless there is explicit mention of involvement in systematic reviews.

Methods of selection of studies

One member of the review team (PC) will run the search strategy and exclude any obviously irrelevant titles. Two review authors (PC, AP) will independently review the abstracts of all remaining records, applying selection criteria to identify eligible studies. Full papers will be obtained for all studies considered potentially relevant by at least one reviewer, and will be independently assessed by two reviewers. Any disagreements between reviewers will be resolved through discussion, involving a third reviewer (CS) where necessary.

Data extraction and mapping the evidence

Data will be extracted into structured tables, categorising each included paper as either:

  1. A.

    Report specifically focused on the methods of, or effect of, stakeholder involvement in a systematic review(s)

  2. B.

    A systematic review that reports stakeholder involvement in the review process

  3. C.

    Other report that describes, discusses or comments on stakeholder involvement in a systematic review(s)

  4. D.

    Unpublished data or expert experience relating to stakeholder involvement in systematic reviews

One reviewer will extract the following data, and a second reviewer will independently check the data entry:

  • Bibliographic information

  • Year of publication

  • Publication status (published/unpublished)

  • Methodological focus/study methodology

  • Description of reported method(s) or approach(es) to involvement of members of public

  • Details/experience of people involved (patients, carers, professionals, policy makers etc.)

Two reviewers will independently judge the focus and comprehensiveness of each of the included papers, using the criteria below (Table 1) (adapted from [27]). Where there are disagreements between the two reviewers, this will be resolved through discussion with the involvement of a third reviewer where necessary.

Table 1 Criteria for judgement of focus and comprehensiveness of reports of involvement in systematic reviews
Full size table

Synthesis A methods

Included studies

Studies from the scoping review that are judged as ‘green’ or ‘amber’ for comprehensiveness of description of methods or approaches to involvement (table 1) will be included.

Data extraction and synthesis

A narrative description of the methods or approaches to involvement will be extracted. In addition, we will tabulate the following information:

  • Stated aim of involvement

  • Number of people involved

  • Characteristics of people involved

  • How people were recruited

  • Format of involvement (face-to-face meeting, telephone meeting, written consultation, online survey, other)

  • Stage(s) of review at which there was involvement (question, protocol, search & identification of studies, data extraction, quality appraisal & synthesis, data interpretation, dissemination)

  • Amount of involvement (number of meetings, number of days involved)

  • Formal research methods used (e.g., participatory action research, nominal group technique, Delphi method)

  • Any evaluation of the methods which was done

  • Ethical approval obtained for involvement?

  • Financial compensation (or alternative) for people involved?

  • Tools or method of reporting involvement?

The quality of the studies contributing to this synthesis will not be systematically appraised, as this is not contingent with the aim of this synthesis, which is focused on providing a description of methods or approaches of involvement (rather than synthesising or interpreting outcome data).

Synthesis B methods

Included studies

All papers previously identified, as ‘studies of effectiveness’ will be included, regardless of whether they were judged as green, amber or red for comprehensiveness. This approach will be taken for this synthesis as the quality of the study or effectiveness will be assessed, and subsequent decisions will be based on quality of evidence (rather than comprehensiveness of reporting).

We will include any study designs, including experimental and observational studies. We will include studies with or without a control intervention.

Appraisal of study quality

Two independent reviewers (AP, PC) will assess and report the quality of all studies of effectiveness using CASP appraisal tools [28], with any disagreements resolved through discussion, involving a third reviewer (CS) if necessary.

Data extraction and synthesis

We will extract data into structured tables, including the following:

  • Study aim

  • Study design

  • Focus of systematic review

  • Summary of involvement – including numbers, who was involved, at what stages in the review process, and methods used.

  • Outcomes assessed, relating to the effect of involvement

  • Results (measured effect or impact of involvement)

The different types of outcomes assessed will be synthesised, creating one list of unique methods of assessing the effect of involvement in systematic reviews. These methods will be categorised as measures of review quality, relevance of review, impact of review, or other effect. Any data relating to the psychometric properties, benefits or limitations of these outcomes will be systematically tabulated from the included studies.

For each of the included studies, the measured effect on the identified outcome measures will be documented. This may comprise quantitative or qualitative outcome data. A summary of the judgement of the quality of the study contributing the data will be incorporated into the table. A summary of findings table will be produced, summarising any evidence for effect on the measured outcomes, for different methods or approaches to stakeholder involvement. We will write a description of the reported beneficial effects of involvement, using a similar format to that used by Brett 2014 when describing the impact of patient and public involvement on health and social care research [29].

Integration of data from syntheses A and B

We will identify studies that were (i) judged to have a ‘green’ or ‘amber’ comprehensive description of the methods or approach (in synthesis A) and (ii) demonstrated evidence of a beneficial effect (in synthesis B). For these studies we will tabulate the key components of the methods or approaches, identifying areas of agreement or dissonance between studies. We will contact the authors of all of these studies and seek any additional material or resources associated with the method or approach.

Discussion

To ensure that stakeholder involvement is beneficial to the quality, relevance and impact of systematic reviews, it is essential to have clearly described approaches to involvement. This must be supported by evidence relating to the effect or impact associated with involvement, as well as effective strategies for measuring the impact of involvement on the uptake and use of systematic review evidence. This planned systematic review will synthesise evidence relating to stakeholder involvement in systematic reviews. Using a contingent design, an initial scoping review will provide a broad map of evidence relating to involvement, with two more focused syntheses to describe the methods and approaches to involvement that have been used, and to explore the evidence of effectiveness of involvement. These syntheses will identify the current methods that have been used to assess the effect of involvement on systematic reviews, and summarise the evidence of effectiveness for these outcomes.

Use of an innovative stepwise search strategy will ensure the balance of breadth and comprehensiveness of data within a review for which searching and identification of relevant studies is anticipated to be challenging. Audio-recording and documentation of iterative team decisions associated with the stepwise and scoping methods will ensure transparency in the review process. Contacting researchers and stakeholders involved in systematic reviews in which there is evidence of a beneficial effect of involvement will enable a richer description of methods and approaches, providing descriptions which can support future replication and improvement.

There is widespread consensus that high-quality training material, reporting guidelines and examples of best practice are urgently required to support active patient and public involvement and enhance the relevance, usefulness and accessibility of systematic reviews [3, 11, 15, 30, 31]. When we have completed our review, the findings will be used to produce training material and resources, as part of the ACTIVE project (http://training.cochrane.org/ACTIVE), which is being carried out in collaboration with Cochrane Training. The focus of these resources will be the production of clearly described methods of how stakeholders may be usefully involved in systematic reviews. We encourage anyone with knowledge, information or experience relating to stakeholder involvement in systematic reviews to contact the ACTIVE project, via the project website or email (ACTIVE@gcu.ac.uk).