Review

Introduction

The primary aim of research priority setting is to gain consensus about areas where increased research effort including collaboration, coordination and investment will have wide benefits to society. Priority-driven research has a clearly defined purpose, with an emphasis on answering questions of key importance that are likely to have a significant impact on knowledge or practice in the short to medium term [1]. The use of a systematic, explicit and transparent process of setting health research priorities ensures that research is funded that has the greatest potential public health benefit, that research funding and outputs are aligned with the needs of decision makers [2], and that there is efficient and equitable use of limited resources, with less duplication of research effort [3]. Priority setting should be as evidence-based as possible, while also incorporating the views of a wide range of stakeholders [4].

A wide range of approaches for setting research priorities exist. While there is no consensus about a gold standard or best practice model, general principles for setting health research priorities have been suggested [5–8]. In 2010, Viergever and colleagues published a nine item checklist for priority setting providing recommendations about processes that should be considered before, during and after undertaking priority setting [6]. However to date, there has been no comprehensive review of health research priority setting in high income countries, the barriers and facilitators of different approaches, nor the effectiveness of different models of priority setting in terms of outcomes. This information is needed to provide guidance to research and policy makers of high income countries who wish to undertake priority setting exercises.

The aim of this review is to:

  1. 1.

    Examine methods, models and frameworks used to set health research priorities;

  2. 2.

    Identify barriers and facilitators to priority setting processes; and

  3. 3.

    Determine the outcomes of priority setting processes in relation to their objectives, and impact on policy or practice.

Methods

Literature search

Medline, Cochrane, and PsycINFO databases were searched for relevant peer-reviewed studies published from 1990 to March 2012. A combination of MeSH and keywords were used (see Table  1). Each identified article or report was examined by two reviewers (JB and JW) to assess relevance. The reference lists of relevant articles were also reviewed to identify further potentially relevant sources of information. Google and Google Scholar searches were also conducted to search for additional grey literature, including government and organisational reports of priority setting processes, with the first one hundred results from each search reviewed. The tables of contents of the following five relevant journals were manually searched between 2007 and February 2012: Cost Effectiveness and Resource Allocation; Health Research Policy and Systems; Journal of Public Health Practice and Management; International Journal for Equity in Health; and Health Policy.

Table 1 Search terms
Full size table

Inclusion and exclusion criteria

Any priority setting exercise that aimed to develop research priorities for population health and health services programs in Australia, New Zealand, North America, Europe and the UK were included. The content areas for health research were not restricted. The range of research methodologies included within frameworks was broadly defined and could include: intervention research, implementation and translation research, comparative effectiveness research, quality improvement research, and other forms of rigorous program evaluation. Only priority setting frameworks which were actually implemented were included. Priority setting processes carried out in non-health related fields, or approaches that were implemented to set health care or service delivery priorities were excluded. The final priority setting processes included in the review were selected to ensure that a diverse range of priority setting approaches were considered within the scope of this narrative review.

Data extraction

Authors JB and JW extracted data from included priority setting exercises. Data included: the aim of the priority setting process; the scope of the process in terms of the specificity of developed priorities; the methods and criteria used to generate priorities; the methods and criteria used to rank priorities; the stakeholders included in the priority setting process; and the success of the model in meeting its stated objectives.

Results

Search outcomes

Figure  1 shows the outcomes of the implemented search strategy.

Figure 1
figure 1

Flowchart of search strategy.

Full size image

Identified models

Eleven diverse priority setting exercises carried out in Australia [9–12], the United Kingdom [13–18], the United States [19] and Canada [20–23] were identified (see Table  2). Three priority setting exercises related specifically to Indigenous health [9–11, 23], two related to primary and secondary care research [14, 15, 17], and one each related to cancer [12], anaesthesiology [13], asthma [16], general health [20–22], service delivery research [18] and comparative effectiveness research [19].

Table 2 Summary of identified priority setting processes (N = 11)
Full size table

Level of priority setting and specificity of developed priorities

Nine of the eleven reviewed priority setting exercises aimed to set health research priorities at a national level [9, 10, 12–16, 18, 20–23]. One priority setting exercise aimed to generate local priorities [17], and one aimed to establish priorities to guide decision-making about developing and funding specific projects [11]. There were differences in the specificity of the priorities that each model aimed to generate. Three of the eleven reviewed models identifying only broad research themes [12, 14, 15, 18] while eight models identified specific research questions [9–11, 13, 16, 17, 19–23]. Of those identifying specific questions, several used a staged process, first deciding on broad priority areas, then generating specific research questions [9, 10, 20–22].

Methods for generating priorities

A diverse range of strategies including calls for submission, stakeholder surveys and questionnaires, stakeholder interviews, workshops, focus groups, roundtables, the Nominal Group technique and the Delphi technique were used to generate research priorities. Each of these strategies are outlined below and the advantages and disadvantages of each are outlined in Table  3.

Table 3 Advantages and disadvantages of different methods of generating priorities
Full size table

Workshops, focus groups or roundtables

The majority of approaches used workshops [14, 15], roundtables [23], focus groups [12, 18], or approaches that included focus groups, workshops or round tables in combination with other approaches [9–12, 20–22] to bring key stakeholders together to generate research priorities. Each differed in the format used. Most merged representatives from multiple stakeholder groups together [9, 10, 14, 15, 23], while two used a combination of merged and separate stakeholder groups. For example, in the priority setting exercise carried out by the National Health and Medical Research Council (NHMRC) Breast Centre to identify agreed areas of priority [12], merged stakeholder focus groups were conducted in each state in Australia, followed by three separate focus groups with Aboriginal and Torres Strait Islander people, people from non-English speaking backgrounds, and people from rural and remote areas. The use of special workshops was considered to provide unique insights, raising issues not identified in representative workshops [12]. The priority setting exercise to set research priories for service delivery research by the National Health Service (NHS) in England also used a combination of approaches. Sixteen mixed groups were held across health regions, with an additional six focus groups held comprising experts in specific groups only; one group was held specifically with consumers, one group with educators, one group with research funders, one group with innovators and two groups with researchers [18].

Stakeholder surveys or questionnaires

Stakeholder surveys were used to generate priority areas in four of the included priority setting exercises [13, 16, 19–22]. Responses were typically collated, refined and arranged into categories or themes which become the basis for further discussion or prioritisation. The size and scope of the surveys and questionnaires differed. In the Institute of Medicine (IOM) priority setting exercise [19], a web-based questionnaire was sent to more than 20,000 individuals on the institutes’ database as well as the media, academics, policy makers, researchers, physicians, health care providers, federal government agencies and individuals and organisations interested in health policy. A total of 1,758 responses were received.

Calls for submission or comment

Calls for submission or comment seek to utilise stakeholders’ personal and/or professional perspectives and expertise to generate priority areas. Two priority setting exercises carried out to establish Indigenous research priorities sought submissions from the public. The Canadian Social Science and Humanities Research Council exercise put out a national call for briefs as the first step of the priority setting process [23]. Five criteria developed by a steering committee were provided to shape the direction of responses from participants. The Australian NHMRC process used both a call for submissions and a series of workshops to generate priorities [9, 10]. Calls for submissions were sought within six research themes pre-determined by a Research Advisory Working Group.

Nominal group technique

The nominal group technique is a structured group information gathering process that aims to combine idea generation and consensus building into a single meeting [26]. A question is posed, then responses from participants are sought, collated, and disseminated to the wider group. Participants are then asked to prioritize the ideas put forward by group members. The nominal group technique was used to generate priorities in two priority setting processes [12, 17]. The London Region Research and Development Programme process to set priorities on the effectiveness, cost and quality of prescribing in primary care [17] used a structured nominal group interview in which individual participants presented ideas for research questions to a group. This process provided an opportunity for discussion and helped to establish a common representation of research questions from apparently divergent ideas. In the NHMRC Breast Cancer Centre priority setting exercise [12], representative stakeholders were asked to generate a list of priorities from an individual perspective prior to a group meeting. The structured nominal groups process were reported to result in reasonably high levels of agreement about priorities across the workshops held.

Delphi technique

The Delphi technique is a structured model undertaken predominantly using questionnaires. Participants answer a questionnaire, then the results (usually a statistical representation of the group response including reasons for judgements [25]) are circulated to all participants. Participants are encouraged to revise their original responses in light of the responses of other participants, allowing sharing of information and reasoning among participants. Generally two or more rounds are conducted, with the answers of participants converging towards consensus. The Delphi technique was used in the London Region Research and Development Programme priority setting exercise carried out to set priorities on the effectiveness, cost and quality of prescribing in primary care [17]. A two-round postal Delphi process was undertaken with pharmacists, general practitioners and nurses. Low response rates were obtained in both the round one (53%) and round two (38%) questionnaires.

Public input session

The IOM exercise to set research priorities for comparative effectiveness research also utilised a public meeting to seek input from stakeholders [19]. Fifty-four experts were invited to address the committee, making 3 minute long presentations as well as written statements which were made publicly available on the IOM website.

Ranking generated priorities

Prioritisation is a process whereby individuals or groups place in rank order identified research priorities in terms of their importance or significance. Specific criteria are normally provided to aid this process. Seven of the eleven working examples used a variety of processes to rank priorities (see Table  3). Techniques included subjective ranking based on perception of social and scientific merit [11], simple counting of the number of times a priority area was mentioned with the most frequently mentioned ranked first [20–22], ranking based on sophisticated criteria (including data on prevalence, mortality, morbidity, cost and variability, utility of area for decision making, information gaps, variability in care, and gaps in translation [19]) and ranking using a five point scales designed to capture the need for the research (likely benefit of research to the organisation and patient care; relevance to policy initiatives; burden of disease; costs to the service and to patients; and practice variation) and research potential (feasibility, degree of management commitment to the issue; study design; and participants) [14, 15]. Three use structured processes of the nominal group technique [12, 16] and the Delphi technique [17] to rank priorities.

Advisory group oversight and stakeholder involvement

Nine of the eleven priority setting exercises reported the use of a core steering or advisory group to oversee and supervise the priority setting process [9–11, 13–19, 23]. The composition, level of involvement and function of each steering or advisory group varied. The most common function of the core steering or advisory groups was synthesising, refining and or translating into themes priority areas generated by stakeholders [14, 16–18, 23]. Other roles of the core steering or advisory group included: developing criteria to guide the generation of priority areas by stakeholders [23]; ranking developed priorities [11]; generating criteria to guide ranking of priorities [19]; agreeing on a set of common terms and/or definitions to be used during the priority setting process [16, 19]; and determining which stakeholders should be consulted [18].

Stakeholder input was a feature of all eleven reviewed priority setting processes. However, the breadth of stakeholder involvement and the way stakeholders were involved differed. Some priority setting approaches conducted separate consultation processes with individual groups of stakeholders, while others identified a broad range of stakeholders and brought people from each one of these groups together. Nine of the eleven reviewed models sought consumer input. Six of these models used consultative consumer participation [9, 10, 12, 13, 18, 19, 23] and three used collaborative approaches [11, 14–16] while none used a consumer controlled approach.

Outcome evaluation

None of the models identified conducted a systematic assessment of the outcomes of the priority setting processes, or assessed whether the generated priorities had any impact on policy or practice [3]. There was very little evidence of the reliability and validity of priorities generated through priority setting frameworks, and no evaluation or reflection on whether the generated research priorities resulted in improvements to important outcomes. Three approaches did however assess whether identified priorities reflected participants perceptions of discussions by carrying out validation surveys [12, 18, 20–22] all with positive results. One additional exercise assessed participant satisfaction with the process [12].

Identified barriers to carrying out research priority setting

A number of barriers to conducting priority setting exercises were identified.

Multi-component methods are resource intensive

Each of the priority setting exercises utilised a multi-component approach, which required considerable time and resources. In some instances the level of demand placed on resources was initially underestimated or later criticised. The James Lind Alliance encountered difficulties conducting the asthma exercise due to constraints on clinician and patient time, which prolonged the process to approximately six months [16]. The Cooperative Research Centre for Aboriginal Health also received criticism that the facilitated development approach was too time-consuming and resource-intensive, requiring the involvement and co-ordination of many participants across multiple stages [11].

Difficulties conceptualising the process and intended outcomes and generating initial priorities

In several priority setting exercises, there was difficulty among both experts and non- experts in conceptualising some aspects of the process. In the James Lind Alliance exercise which involved setting priorities for asthma research, patients involved in the priority setting process had difficulties conceptualising research opportunities which were referred to as treatment ‘uncertainties’ [16]. Despite repeated clarification, many of the patient responses had to be removed as they did not address the area of research interest. The NHS priority setting exercise also found problems with the advisory group struggling to clearly define the area of research and distinguish the interface between primary and secondary care [14, 15]. In the National Institute of Academic Anaesthesia exercise [13], individuals found the process of generating research ideas burdensome and challenging, which contributed to a relatively low response rate from participants. Lay members in particular reported difficulties in forming specific research questions as required.

Difficulties making decisions within the advisory group

In the NHS priority setting exercise, there was a perception of a lack of background knowledge of the current state of the literature within the advisory group, which made it difficult for the group to make decisions [14, 15]. The advisory group also struggled with distinguishing between the importance of research and its feasibility.

Ideas not being suggested by researchers out of fear that idea would be appropriated by others

Priority setting exercises may be inhibited by the traditionally competitive nature of research. In the National Institute of Academic Anaesthesia exercise, there was a perception that some anaesthesiologists did not submit research ideas out of concern their idea would be appropriated by others [13].

Identified facilitators to conducting research priority setting

Structured techniques were perceived as useful

Three priority setting processes used the nominal group method to generate and/or decide on priorities [12, 16, 17]. In all three instances, the structured approach was perceived as useful in facilitating agreement about priorities. Giving participants the opportunity to discuss the rationale for presented ideas was thought to help build group cohesion, facilitate the refinement of disparate ideas [17], and assist in reaching consensus [16].

Piloting of questionnaire

Prior to distributing a questionnaire to determine national priorities for anaesthesia and perioperative medicine, the National Institute for Academic Anaesthesia piloted the questionnaire with 32 respondents from two teaching hospitals [13]. This proved useful as the initial questionnaire, which asked respondents to suggest a research question and the primary outcome measure and patient group, was considered ‘too complex and demanding’. As a result, the questionnaire was altered to ask respondents to suggest research topics only in more general terms. The priorities were then developed into specific research questions by the advisory group.

Separate consultation exercises for ‘non-professional’ consumers or special groups

In two instances, the use of separate consultation exercises for consumers or special groups has been found to be valuable. In setting priorities for the NHMRC Breast Centre, special workshops were held to consult with Aboriginal and Torres Strait Islander women, women of non-English speaking background, and women living in rural or remote areas [12]. These workshops used a less structured method of gaining consensus than other workshops, but were considered important to consider the needs of women who had more difficulty accessing information or services. After analysis of priorities identified by each of the workshops held, the special workshops were considered to provide unique insights, and raised issues not identified in other workshops. Similarly, in the NHS exercise, a separate consumer consultation exercise, in the form of a focus group of informal carers, was convened as one attempt to sample the views of ordinary users as opposed to the “professional” consumers who responded to the formal consultation exercise [14, 15]. This input was helpful to the advisory group as a prompt to consider the needs of individual users when setting priorities.

Discussion

This review shows a failure to evaluate priority setting processes and a lack of consensus about appropriate priority setting methodologies [6]. It is therefore not possible to provide strong evidence-based recommendations about optimum methods to set research priorities. Consequently, any attempts to develop a priority setting process must rely on the critical appraisal of the existing literature, expert consensus, and the relevance and necessities of the local socio-political and policy environment. The following recommendations are made:

A multi-disciplinary advisory group should oversee the priority setting process

A well-managed and resourced multi-disciplinary advisory group should oversee the priority setting exercise. Such an advisory group provides credibility to the process of determining research priorities, and ensures the developed priorities are relevant and feasible. The group should elect a member to chair the group. A process should be put in place to manage any potential conflicts of interest.

Broad representation of stakeholders is critical

Involvement of a broad representation of stakeholders was seen in the majority of priority setting models as a strength of the process, consistent with notions that such inclusion provides credibility and transparency to the process, and ensures that developed priorities are relevant, feasible, and meet actual health care needs [27–29]. Involvement of researchers may mean they are more likely to commit to undertaking research within identified priority areas [30], and involvement of policy makers may mean a greater likelihood of knowledge transfer and implementation of research outcomes [6]. The guiding principle in selecting stakeholder groups for consultation should be one of inclusivity [31]. Depending upon the technicality of the selected approach, it may be necessary to provide additional support to consumers members of the workgroup through individual sessions outside working group meetings to clarify objectives and ensure members are comfortable with what is being asked of them [32].

Objective, clearly defined criteria should guide the generation of priorities

Clear and specific criteria for eliciting and ranking potential priorities should be determined by an advisory group before seeking stakeholder input. Whether specific research questions or broad priority areas are generated as an outcome of the priority setting process, these should be determined based on the purpose and context of the priority setting process [6, 7].

The impact of the priority setting processes should be evaluated

An outcome evaluation should be integrated into the priority setting process to provide evidence for future priority setting processes. Although difficult to undertake, as has been suggested elsewhere [33], an outcome evaluation could include tracking and reporting of acceptability and perceived usefulness of individuals involved in the process; the number and type of new initiatives generated that relate to the priority areas in defined time period after implementation; the number and type of research projects undertaken related to each endorsed research question; the key outputs of any initiatives funded as a direct or indirect result of the priority setting process; and the outcomes associated with each one of the research initiatives, including indices such as number of people trained in research methodology, number of publications, number of publications by designated priority determined area, the amount of measurement, descriptive and intervention research, the impact of the research in informing policy, program or health service delivery design.

Strengths and limitations of this review

This narrative review examined priority setting processes carried out in selected high income countries with similar healthcare systems. Only priority setting processes actually implemented were examined to ensure the feasibility, acceptability and barriers to carrying out the priority setting activity could be examined. While systematic methods were used to identify relevant priority setting processes, the scope of this review was limited to a diverse but relatively small sample of priority setting processes, and is not intended to be exhaustive. While we aimed to determine the outcomes of priority setting processes in relation to their objectives and impact on policy or practice, we only assessed the impacts as reported in the examined publication, rather than conducting an additional broad and comprehensive literature review. Therefore it is possible that some broad policy and practice implications have been missed. Additionally, the review focuses on priority setting exercises carried out only in selected high income countries and so should be considered within this context only.

Conclusions

There is a no consensus about a gold standard or best practice model for health research prioritization [6]. Nevertheless, the recommendations arising from this review are consistent with other guidance literature on this topic [5–8, 33]. The method used should be selected based upon the context of the priority setting process, as well as time and resource constraints. Priority setting should be overseen by a multi-disciplinary advisory group, should involve a broad representation of stakeholders, should utilise objective and clearly defined criteria for generating priorities and should be evaluated.