Data was collected by reviewing documents published by WHO technical programmes at the headquarters in Geneva from 2002 to 2017 and contained in the institutional repository, IRIS. Deposition in IRIS ensures the document is free-to-access in a digital archive and is assigned a permanent url, as opposed to the url for a web page where links can break and be lost over time with editing and updates. This ensures users can always locate the original source document. The data collection was undertaken by interns working within TDR, the Special Programme for Research and Training in Tropical Diseases, under the guidance of the first author of this paper in three, 12-week phases (May–July 2013, May–July 2015 and May–July 2018). The types of publication included guidelines, rapid advice guidelines, strategy documents, action plans, fact sheets, roadmaps, blueprints, meeting notes and research prioritisation reports. The search criteria in IRIS searched the full-text of documents containing the phrases ‘research priority’ and/or ‘research priorities’. While many of the WHO publications are available in the six official languages of WHO, this work utilised the English language version of the documents.
The inclusion criteria required that the document was (1) covering global health research priorities identified by technical programmes at the WHO headquarters and published in English; (2) publicly available in the WHO IRIS with a permanent url; (3) an official WHO publication that has been through the internal clearance process and has the WHO logo within the publication; and (4) published between the years 2002 and 2017.
The inclusion criteria were created to produce a manageable dataset of publications that shared the same objectives and that represented official WHO publications concentrating on health research priorities from a global perspective.
The exclusion criteria were (1) documents produced by WHO regional and country offices; (2) publications by other UN agencies or global health research agencies, for example, research funders; and (3) journal articles and systematic reviews. The exclusion criteria were chosen to keep the number of documents under review manageable to answer our question about the value of standardisation with respect to comparing priorities across disease areas.
An initial search of IRIS from 2002 to 2017 retrieved 2840 documents. A review of these 2840 documents identified 280 that merited more in-depth review according to the inclusion criteria. Of these 280 documents, 116 were eventually retained for use in this study.
Short summaries (‘data captures’) were created to transfer the relevant information within each of the 116 documents to a standard template. Identical priorities were reported in two separate documents and therefore there were 115 data captures. The information captured included publication title, publication date, permanent url, disease/health topic, contact (author, editor, etc.), executive summary, key findings/areas needing further research, priority-setting approach/methodology, measurement/ranking criteria, research financing estimates, limitations, other relevant information (pipeline/timeline/feasibility), and key figures and tables.
The data captures were sent to the WHO department representatives and/or authors of the documents for confirmation that their document had been summarised accurately. This sharing of information was accompanied by a request for other relevant documents. In addition, representatives from 34 of the 36 technical programmes at WHO headquarters were interviewed. These interviews often provided additional information regarding the methods and the priorities identified. However, to maintain the integrity of the study, only the information contained in published documents was analysed.
The individual priorities were then extracted and entered into a Microsoft Excel 2010 database to enable data analysis. Phrases and spellings were transcribed as reported in the original document. See PRISMA diagram flow Fig. 1.
WHO Health Research Priorities database (Additional file 1)
The data were entered into seven integrated worksheets. Table 1 explains the relationship between the worksheets.
The 115 data captures summarising publications containing research priorities were analysed using the Excel database to capture three main categories of data, namely (1) general publication information, (2) prioritisation methodology, and (3) research type in relation to the WHO Strategy on Research for Health.
The database was designed to study the distribution of research priorities across the research cycle of individual diseases/health topic categories. The database also allows analysis of the methodologies used to develop individual research priorities. Each priority has a specific entry number in the database.
General publication information added to the database includes source location, health topic(s) covered, publication date, and type of document (research prioritisation as self-described in title or introduction; report, namely an official WHO publication, but not specifically presented as a research prioritisation publication; or meeting notes, namely unpublished or less formal WHO documents). Research prioritisation documents were official prioritisation reports, while reports were formal WHO reports such as guidelines, action plans and strategy outlines.
Disease/health topic research priorities were narrowed to the most specific category referenced in the publication. For example, both ‘Neglected Tropical Diseases’ and ‘Leishmaniasis’ exist in the database as disease/health topic, but research priorities were categorised as ‘Neglected Tropical Diseases’ if they did not address one of the specific neglected tropical diseases.
The ‘expert consultation’ method of priority-setting was further classified by the composition of the expert group, including academic, medical organisations, international organisations, government, non-government, funding bodies, private sector, and advocacy/stakeholders. The priority ranking criteria used in this study were as follows: public need, magnitude of impact, scientific feasibility, cost, equity, sustainability, curative versus preventative, critical need, and pro-poor.
The methods used to identify research priorities were added to the database. Relevant information on methodology included research prioritisation methods (e.g. expert consultation, literature review, quantitative prioritisation, interviews, Delphi, CHNRI), priority ranking criteria, disease/health topic per priority, and broad and specific research gaps and priorities. For a discussion of research priority methods see Viergever et al. .
Priorities were categorised at two levels under a heading of Identified Gaps. Broad research gaps and priorities identified a research area or a research classification (e.g. basic or operational research). The broad gaps and priorities were included only if they grouped together more specific gaps and priorities in the document; therefore, there are a number of blanks in this column. The specific research gaps and priorities were then categorised as a ‘research cycle type’ by mapping against one of the five categories in the research cycle first defined in the WHO Strategy on Research for Health and further elaborated in the World Health Report 2013 [8, 9]. We decided to use this framework as it is simple and applicable to the wide range of public health issues that WHO works with, including infectious and non-communicable diseases, child and maternal health, health systems, environmental health, food safety and many more.
The research cycle categories were adapted and defined in this research as follows:
Problem: research to measure the size of the health problem through epidemiology, estimating the burden of disease and other forms of data collection.
Cause: research to understand the causal agents, risk factors and determinants of the health issue. May include, for example, study of infection cycle, vectors, role of socioeconomic factors, environment, diet and the interaction of multiple factors.
Solution: research to develop new interventions; includes therapeutics, devices and procedures as well as policy interventions, public health campaigns, etc.
Implementation: research to translate new interventions into policy and practice and understanding the barriers to delivering known interventions.
Evaluation: research to monitor and evaluate the effectiveness or health impact of an intervention or programme.
As research priority-setting often has a focus on the research and development of products (diagnostics, pharmaceuticals and vaccines) and other health interventions, this research cycle type was broken down further to include the following categories: drug, vaccine, diagnostic or screening intervention, vector control, device, basic science, or other/unspecified. An additional three other categories allowed us to record when the research priority was focused on improving an existing intervention (e.g. testing a new combination therapy) or a new intervention and whether there was a specific target population for women or children (paediatric formulations). Each data capture was reviewed by a minimum of two members of the study team and the lead authors/contact point for each document was also asked to verify the data captures. While not all lead authors replied to our request to verify the data captures, of those that did provide their input, not one challenged the premise of the study or the categorisation method. This provided a positive support that our approach was understood and accepted by those technical programmes undertaking research priority-setting.
The database was constructed as a binary code. As each individual priority was categorised across the various Excel columns, a number 1 was placed in each applicable box and a 0 in non-applicable boxes. Using this binary method, number patterns could be analysed for comparison between categories and for summation (quantification) of specific responses. For example, positive responses for each category can easily be summed at the end of a column, while simple Excel equations can identify positive responses across a sheet or various columns. For the intent of this research, the results of selected filters were set-up to display on the Statistics worksheet. Source documents are archived in the WHO repository IRIS with a permanent url so a user can always locate the priority within the context of the original document from where it was extracted.