Fostering access to and use of contextualised knowledge to support health policy-making: lessons from the Policy Information Platform in Nigeria
Contextualising evidence to inform policy-making is increasingly recognised as key to developing and implementing effective health policies. Creating a one-stop shop for evidence is an approach that can facilitate timely access to the best evidence to inform policy decisions. We report outcomes after implementation of the Policy Information Platform (PIP), a pilot one-stop evidence repository in Nigeria designed to alleviate barriers to accessing policy-relevant knowledge.
This cross-sectional study involved five phases, namely (1) consultation with Nigerian policy-makers to identify priority policy issues, areas of health policy information needs, and challenges and capacity constraints in accessing evidence for policy-making; (2) a stakeholder engagement workshop to formally launch the PIP; (3) extraction of data and other information from scientific articles, policy briefs, evaluation reports, grey literature and health policy documents relevant to policy-making in Nigeria (identified by Google and PubMed searches and by examination of websites of relevant Nigerian government ministries, agencies and parastatals), for use in developing the PIP website; (4) promotion of the PIP in national and state health policy meetings; and (5) evaluation of the PIP using a stakeholder survey questionnaire distributed via email and critical appraisal of the grey literature included in the PIP using the authority, accuracy, coverage, objectivity, date and significance (AACODS) checklist.
Priority policy areas identified by policy-makers were disease control and prevention, population health issues and health administration. Challenges identified by policy-makers were a lack of adequate capacity to access policy-relevant evidence and transform the evidence into policy. Policy-makers suggested using systematic reviews, policy briefs and rapid response mechanisms and involving policy-makers in research as ways of increasing evidence uptake for policy. A total of 126 policy-relevant, peer-reviewed scientific articles, 85 health policy documents and 201 policy-relevant grey literature documents were selected for inclusion in the PIP. Of the 195 individuals contacted via email to evaluate the PIP, 31 (15.9%) provided a response. Respondents noted that the PIP facilitated access to information based on local evidence and context-sensitive data. Barriers identified included lack of knowledge about the PIP and limited capacity of end-users to use the data compiled in the platform.
An easily accessible one-stop shop of policy-relevant evidence can considerably improve policy-makers’ access to evidence for use in policy-making and practice.
KeywordsPolicy information platform Policy-makers Evidence Capacity One-stop shop
Authority, accuracy, coverage, objectivity, date and significance
low- and middle-income countries
maternal, newborn and child health
National Strategic Health Development Plan
Policy Information Platform
Contextualising evidence to inform policy-making is increasingly recognised as key to developing and implementing effective health policies. Several previous studies have indicated that research evidence can enhance the processes of health policy development and implementation [1, 2, 3, 4, 5]. Despite the importance of science in health policy-making, a considerable gap remains between research evidence and the formulation and implementation of health policies, particularly in low- and middle-income countries (LMICs) . Several studies have attempted to understand the suboptimal use of research evidence in policy-making, with findings suggesting the following key factors impeding evidence uptake: (1) lack of available research; (2) poor dissemination of research findings; (3) limited access to research, for various reasons including cost; (4) lack of clarity, relevance and reliability of research findings; and (5) unsuitability of the format of research output [5, 7, 8].
WHO has underlined that policy-makers need access to the right information at the right time to support evidence-informed decision-making . Timely, suitably packaged and policy-relevant research can foster the use of evidence in policy processes in LMICs [10, 11, 12, 13]. Additionally, the importance of developing concise materials and tools to communicate various types of information to policy-makers and those supporting them is well recognised . Evidence exists for the value of information-packaging efforts designed to support action based on messages arising from research and other policy-relevant information [11, 12, 13]. Additional evidence suggests that effective communication relies on various factors, including readily understandable research, presentation of timely data in visually compelling formats, use of illustrative anecdotes where appropriate, creation of clear key messages about the meaning of the data, suggestions of ways to use research findings for answering important policy questions, involvement of policy-makers in the planning and execution of health research, as well as involvement of researchers in the planning and execution of health programmes, prompts of relevant articles or evidence briefs targeted to appropriate policy-makers, and establishment of relationships of trust and credibility with policy-makers [14, 15, 16, 17].
Furthermore, the complexity of decision-making requires inputs from a broad evidence base beyond scientific research, including knowledge generated from local evidence and good practices, as well as tacit knowledge . For instance, Pang  noted that epidemiological studies, qualitative research, experience, know-how, consensus and local knowledge should all be taken into account in health policy-making. Although the vast majority of research evidence is published in peer-reviewed journals, much contextualised and policy-relevant knowledge is confined to the grey literature and is not widely shared. The grey literature comprises a wide range of material, including policy documents, statistical publications, newsletters, fact sheets, working papers, technical reports, conference proceedings, dissertations and multimedia content . There is increasing interest in the pivotal role played by the grey literature in the evidence-to-policy process . For example, in May 2014, the Pisa Declaration on Policy Development for Grey Literature Resources was signed . Its signatories called for increased recognition of the grey literature’s role and value by governments, academics and other stakeholders, particularly in terms of its importance for open access to research, open science, innovation, evidence-based policy and knowledge transfer . Despite this interest, knowledge from the grey literature is not optimised by decision-makers, researchers and other stakeholders in the development of policies and programmes . In 2012, the WHO Strategy on Health Policy and Systems Research suggested synthesising and consolidating relevant research evidence and other knowledge via country-specific national repositories of evaluations, best practices and grey literature to enable greater access to existing knowledge that could improve decision-making . One of the approaches for achieving this synthesis and consolidation is the development of ‘one-stop shops’ of high-quality research evidence and policy-relevant knowledge products.
A one-stop shop for research evidence can allow health system policy-makers, stakeholders and researchers to efficiently find and use the best available research evidence in the limited time they have available to make, inform or advocate for a decision . Several one-stop shops already in existence were designed to provide information related to clinical programmes and services, prescription drugs, and public health programmes and services [23, 24, 25, 26, 27, 28]. In recent years, global one-stop shops have been developed to address issues related to health policy and health systems, with a focus on decision-makers, including the Health Systems Evidence repository, the Health Technology Assessment Database, EVIPNet (the Evidence to Policy Network), the Virtual Health Library and the PDQ-Evidence repository [23, 28, 29, 30, 31]. However, to date, few one-stop shops have been established in LMICs to address their needs for health policy and health systems evidence. More specifically, there has been no health policy repository in Nigeria that could be regarded as a one-stop shop for various types of high-quality evidence (e.g. peer-reviewed research publications, expert opinion, policy documents and grey literature) specifically relevant to policy-makers’ needs for decision-making in that country.
To address this gap, the WHO Alliance for Health Policy and Systems Research and the African Institute for Health Policy & Health Systems of Ebonyi State University established a pilot repository called the Policy Information Platform (PIP) , to alleviate barriers to accessing the existing policy-relevant knowledge. The platform was designed to include both indexed publications and the grey literature, as well as to ensure that relevant and reliable information is available in a user-friendly format for policy and management decisions in the country. The aim of the current study was to assess the implementation of the PIP in Nigeria, with a view to documenting its functioning and policy influence, as well as understanding decision-makers’ perceptions of and satisfaction with this resource.
This cross-sectional study involved five phases in developing the PIP, as follows: (1) consultation with policy-makers and identification of priority policy issues; (2) a stakeholder engagement workshop, with formal launch of the PIP; (3) extraction of data from policy-relevant publications and development of the PIP website; (4) promotion of the PIP website; and (5) evaluation of the PIP.
Consultation with policy-makers and identification of priority policy issues
Phases of development of the Policy Information Platform (PIP) in Nigeria
Consultation with policy-makers and identification of priority policy issues
Interaction with key policy-makers representing various areas of the policy-making sector in Nigeria to identify priorities in the health policy-making process
Key policy priority areas identified by policy-makers (see Table 3)
Stakeholder engagement workshop and formal launch of PIP
Formal presentation of the PIP Nigeria website and administration of structured questionnaire to elicit information to aid in the operational effectiveness of the platform
Information generated by questionnaire: (1) areas of health policy information needs; (2) challenges and capacity constraints in accessing evidence for policy-making; (3) how evidence is utilised in the policy-making; (4) suggested ways and formats in which policy-relevant information can be made easily available and accessible to policy-makers (see Table 5)
Extraction of policy-relevant publications and development of PIP website
Publication extraction process using PubMed, Google Scholar, etc. Extracted publications classified into five main categories: (1) scientific articles, (2) policy briefs, (3) evaluation reports, (4) grey literature and (5) health policy documents
Policy-relevant publications extracted: scientific articles (n = 126), policy briefs (n = 46), evaluation reports (n = 23), grey literature (n = 201), health policy documents (n = 85) (see Fig. 1)
Promotion of PIP website at national and state health policy meetings
Presentation of the PIP during a stakeholder engagement event organised by West African Health Organization in Abuja in October 2015 and two state meetings in Ebonyi State in November 2015 and April 2016
Participants (n = 195) made aware of the existence of the PIP and given first-hand information on how to use it
Evaluation of the PIP using stakeholder survey questionnaire
Stakeholder evaluation survey undertaken via email 6 months after establishment of the PIP
Of 195 individuals contacted via email with survey questionnaire, 30 (15.4%) provided a response; respondents commended the PIP initiative and made suggestions for its improvement and sustenance
Key Nigerian policy-makers consulted to identify priority health policy issues that should be addressed by the Policy Information Platform
Director of Public Health
Ministry of Health
To coordinate the formulation of public health policies and guidelines and to support their implementation and evaluation in Nigeria through health promotion, surveillance and prevention
Director of Nursing Services
Ministry of Health
To improve nursing services to patients in all public healthcare facilities
Health Systems Information Services Officer
Ministry of Health
To collect, transmit, store and manage health-related data to inform and support health management practices
Coordinator of Reproductive Health Services
Ministry of Health
To ensure improvement of maternal and newborn health, with reduction in maternal mortality ratio
Head, Department of Family Medicine
Federal Teaching Hospital, Abakaliki, Nigeria
To develop and coordinate the implementation of policies and programmes that promote the health of the family through efficient, integrated health services
Director of Primary Healthcare
Local Government Service Commission
To ensure the functioning, planning, implementation, supervision and monitoring of primary healthcare services
Chief Executive Officer
National health-based non-governmental organisation
To advocate for efficient and effective health services that will lead to improvement in health outcomes
Stakeholder engagement workshop and formal launch of the PIP
A 1-day stakeholder engagement event was convened in September 2015 at Abakaliki, Nigeria, during which the PIP was formally launched. The purposes of the meeting were to bring together policy-makers, researchers and other stakeholders in the health sector policy-making process (including health practitioners, civil society organisations and media practitioners), to formally present the PIP Nigeria, including the website, and to elicit insights on the implementation and effectiveness of the platform to support health policy-making. A structured questionnaire (Additional file 1) was administered to participants to assess (1) their health policy-relevant information needs; (2) the challenges and capacity constraints they experienced in accessing evidence; (3) the ways in which they utilised evidence in policy-making; and (4) their suggestions of ways and formats in which policy-relevant information could be made easily available and accessible through the PIP.
Extraction of data and other information from policy-relevant publications and development of the PIP website
The potential content of the PIP was classified into five main categories, namely (1) scientific articles, (2) policy briefs, (3) evaluation reports, (4) grey literature and (5) health policy documents. A description of the process for extracting data and other information from these publication types is provided below.
Scientific publications reporting research done in Nigeria related to policy-making in the field of maternal, newborn and child health (MNCH) were sought. The emphasis on MNCH was a recommendation arising from the stakeholder engagement event. A PubMed search of the MEDLINE database was performed in August 2015, and studies published in English that investigated MNCH in Nigeria in relation to health policy were identified. To be included as scientific articles, these publications had to be original studies and had to contain policy recommendations. Eligible studies were retrieved and indexed in the PIP, with links to the full articles in PubMed.
For the purpose of the PIP, a policy brief was defined as a policy document that clarifies a health policy problem, renders the evidence for addressing the problem concise and understandable, explains why the evidence is important, describes evidence-informed policy options that would be suitable actions for policy-makers to take, and provides key implementation considerations [33, 34, 35]. A Google search was performed in August 2015 with the keywords ‘policy brief’, ‘health’ and ‘Nigeria’, yielding a total of 313 entries, of which 46 policy briefs were selected. Each selected document fulfilled the following inclusion criteria: (1) must be a policy brief that meets the definition given above, (2) must focus on Nigeria, (3) must focus on the health of the population, and (4) must highlight recommendations relevant to health policy-making.
For the purpose of the PIP, an evaluation report was defined as a document that reports a systematic assessment of a health activity, project, programme, policy or institutional performance and that provides evidence-based information relevant to the decision-making processes, to allow an understanding of achievements or the lack thereof . To identify evaluation reports, another Google search was performed in August 2015, with keywords such as ‘evaluation reports’, ‘health policy’ and ‘Nigeria’; this search yielded 363 entries, of which 23 were selected. The selected documents fulfilled the following criteria: (1) must be an evaluation report meeting the definition given above, (2) must focus on Nigeria, (3) must focus on the health of the population, and (4) must highlight recommendations relevant to health policy-making.
For the purposes of the PIP, the grey literature was defined as documents produced by all levels of government, academics, business and industry, in print or electronic formats, but not controlled by commercial publishers [19, 21]. The websites of more than 30 health-related organisations were searched for policy-relevant grey literature in August 2015. The websites were hosted by organisations such as the Nigerian health-related government ministries, departments and agencies, non-governmental organisations (NGOs), civil society organisations, and health and professional associations. The materials obtained included policy documents, statistical publications, newsletters, bulletins, fact sheets, working papers, technical reports, conference proceedings, dissertations and multimedia content. The criteria used for the selection were as follows: (1) must be a grey literature document that fulfils the definition provided above, (2) must focus on Nigeria, (3) must focus on health of the population, and (4) must be relevant to health policy-making.
Critical appraisal of grey literature
Each document from the grey literature was critically appraised using a modified version of the Authority, Accuracy, Coverage, Objectivity, Date and Significance (AACODS) checklist  (Additional file 2). This tool considers the following criteria: (1) Authority: Is the document from a reputable organisation or individual author from a reputable organisation? (2) Accuracy: What are the aims of the document? Has it been peer-reviewed or edited? Is the basis for the document clear? Is the document well structured? (3) Coverage: Are any limits clearly stated? (4) Objectivity: Does the work seem to be balanced in presentation? (5) Date: Is the date of the document given? (6) Significance: In the researcher’s estimation, will the document be of interest? The AACODS checklist has been modified by the National Institute for Clinical and Health Excellence (NICE, United Kingdom) and included among that organisation’s checklists for evidence evaluation . We further refined the tool to include grading and a cut-off threshold of three points for inclusion or exclusion from the PIP. Notably, in this study, the modified AACODS checklist was used as a decision-making tool for inclusion or exclusion of grey documentation, but not to provide a scientific quality appraisal per se (e.g. low, moderate or high quality). Of 393 documents assessed, 201 (51.1%) met the threshold for inclusion.
Health policy documents
We searched for relevant policy documents at the websites of all Nigerian federal ministries that deal, directly or indirectly, with health, including Health and Social Services, Women Affairs and Social Development, Water Resources and Rural Development, Science and Technology, Finance and Economic Development, Environment, Education and Youth Development, and Agriculture and Natural Resources. We also searched the websites of all agencies and parastatals under the Ministry of Health, including the National Health Insurance Scheme and the National Primary Health Care Development Agency. A total of 85 health policy documents were identified and included in the PIP.
Promotion of PIP website in national and state health policy meetings
A presentation about the PIP was made at one national and two state health policy meetings. The national meeting was a stakeholder engagement event on MNCH organised by the West African Health Organization and held in October 2015 in Abuja; 92 participants were in attendance. The two state meetings took place at Ebonyi State University in Ebonyi State, Nigeria, with 32 participants in the November 2015 event and 35 in the April 2016 meeting. Attendees at all three meetings included policy-makers, researchers and other stakeholders. The engagement process involved interaction between policy-makers and researchers on issues related to the research-to-policy interface and the establishment of formal mechanisms for continuous partnership. The PIP was introduced to the participants as a platform that would continue to provide policy-relevant information; contact information was provided so that users could request additional relevant information to address future policy-making needs.
Evaluation of the PIP using stakeholder survey questionnaire
How has the PIP Nigeria benefited or served you in your work as a policy-maker or in research as a researcher?
What are the drivers or motivation in your demand for information, and engagement with the PIP Nigeria website?
How can we make sure that the resource is known by policy-makers and researchers and used to inform the different policy steps?
What do you think are the facilitators and impediments to your use of the platform?
How can we improve the PIP Nigeria?
The written responses to the evaluation survey were analysed according to Giorgi’s phenomenological approach [39, 40]. Qualitative data analysis was conducted by assessing narratives and textual information, identifying all comments that appeared significant, abstracting units of meaning, categorising and summarising the abstractions, and returning to the extracted text to ensure a good fit.
Outcome of consultation with policy-makers and identification of priority policy issues
Health policy priority areas identified by key Nigerian policy-makers for coverage in the Policy Information Platform
Disease control and prevention
Population health issues
(1) Infectious/communicable diseases
(2) Environmental health
(3) Vector control
(5) Disease prevention among
pregnant women and children under 5 years of age
(6) Non-communicable diseases
(1) Adolescent health
(2) Reproductive health
(3) Gender issues in health and
(4) Geriatric health
(5) Community participation and ownership
(1) Human resources for health
(2) Primary healthcare systems
(3) Leadership and governance
(4) Health financing
(5) National health management systems
(6) Partnerships for health
Outcome of stakeholder engagement workshop
Characteristics of participants in the health stakeholder engagement event, Abakaliki, Nigeria (September 2015)
Number (%) of participants
Type of organisation
Ministry of Health
Local Government Service Commission
Federal Teaching Hospital (Abakaliki, Nigeria)
Head of Department
Ordinary national diploma/higher national diploma
Summary of participants’ responses regarding areas of health policy information needs, capacity constraints and formats in which policy-relevant information could be made accessible
Areas of health policy information needs
Challenges and capacity constraints in accessing evidence for policy-making
How evidence is utilised in policy-making activities
Suggested ways and formats to make policy-relevant information easily available and accessible to policy-makers
(1) Public health law
(2) Personnel/health administration
(3) Monitoring and supervision of healthcare delivery
(4) Health economics, budget and resource management
(5) Health insurance
(6) Adolescent health
(7) Reproductive health
(8) Health policy advocacy
(9) Family planning
(10) Programme planning and implementation
(11) Primary healthcare
(1) Lack of credible, context-specific evidence
(2) Government’s poor attitude towards policy-relevant evidence
(3) Difficulty in finding suitable evidence
(4) Inadequate format for presentation of evidence
(5) Poor dissemination of evidence
(6) Lack of skill in accessing evidence and policy-relevant information
(7) Poor internet access at workplaces
(8) Inadequate ICT knowledge and skill
(9) Lack of capacity to understand evidence that is not presented plainly
(10) Lack of adequate capacity to transform evidence into policy
(1) Policy dialogue
(2) Proposal writing/memoranda
(3) Policy formulation
(4) Data analysis
(5) Monthly/annual planning meetings
(6) Monitoring/evaluation/impact assessment
(7) Resource management
(8) Development of future work plans
(9) Training of staff
(10) Programme implementation
(1) Systematic reviews
(2) Expert information
(3) Pilot studies
(4) Case studies
(5) National surveys
(6) Monthly/annual reports
(8) Policy briefs
(9) Executive summaries
(10) Policy advocacy
(11) Rapid response mechanism/timely information
(12) Involvement of policy-makers in research
Outcome of data and information extraction from policy-relevant publications for the PIP
Outcome of PIP evaluation survey
Of the 195 individuals contacted via email with the survey questionnaire, a total of 31 provided responses, which represented a 15.9% response rate. The respondents consisted of policy-makers (22.6%), researchers (35.5%), health practitioners (35.5%) and NGO executives (6.5%). Themes that emerged from analysis of respondents’ comments are highlighted below.
Access and efficiency
“What we look out for in engagement with PIP Nigeria website includes informational materials on health systems and support, health policy, health financing, advocacy and mobilisation for evidence-based and inclusive health policy and systems, links for engagement with wider networks for sustainable and resource saving systems.” (NGO Executive)
“The website helps to connect me to articles relevant for research, other materials and books not easy to come by. These articles and books are eye opening and very informative.” (Researcher)
“The PIP Nigeria indeed served [as] a-one-stop-shop when seeking for policy documents on health policy. Its organisation made searching easy. The collections of local researches made discussion in relation to our Nigerian context robust.” (Policy-maker)
Networking and collaboration
“The Policy Information Platform PIP Nigeria has served me as a researcher in Nigeria by providing information on critical areas and issues that require research intervention and networking for optimum results. As a Research Team Leader, the PIP has also been of benefit to me and my research team by providing the platform for collaboration between researchers and policy-makers to ensure that research outputs are made readily available to policy implementers.” (Researcher)
Strengthening capacities of policy-makers to use research
“Strengthened my capacity to apply health research in making policy decisions.” (Policy-maker)
“The PIP exposed me to health policies … it provided relevant resources for policy analysis … it provided scientific and evidence-based policies.” (Policy-maker)
“My motivations for demand are versatility, quality, scope of studies as well as ease of use.” (Researcher)
Facilitators and barriers
Respondents identified factors stimulating and impeding their use of the PIP in the context of the Nigerian health system. Facilitators included the open access feature and ongoing, iterative development and updating of content, as well as outreach features of the platform, such as prompts to users about specific evidence showcased by the PIP. Barriers included lack of knowledge about the platform and its content and limited capacities of end-users in accessing and using data included in the platform.
Regarding the facilitators, one researcher noted that “the website is created and maintained by University free for our use, thereby making information on global issues and updates available”.
Concerning the impediments, one policy-maker stated that ignorance of the existence and availability of the platform limited access to the information that it contains.
Supporting demand and sustainability
“This would be possible by periodically engaging the stakeholders in the health sector where this platform would be introduced and its practical use made in the formulation of policy. Such programmes should be sustained.” (Researcher)
“The resources can be made known to people by advertising it in journals, newspapers, referring friends to the site, encouraging users to cite it in their publications, and presenting it in government/stakeholder meetings in different ministries not only the health ministry.” (NGO executive)
“Visibility for the resource can be assured by any or a combination of the following options, creating a link with the websites of affiliated institutions and agencies, advertise via online newsletters and blogs, writing feature articles via mass media, in-house magazines and newsletters of Ebonyi State University and affiliated organisations, etc.” (NGO executive)
“Do more stakeholder engagement/sensitization for them to have the knowledge of PIP, the importance and the use of the website.” (Policy-maker)
The PIP as a ‘one-stop shop’ innovation
This study represents the first attempt in Nigeria to promote a mechanism (one-stop shop) for fostering access to and use of relevant knowledge in decision-making. The PIP was designed with consideration of policy-makers as the target key users. This one-stop shop is the only platform indexing Nigerian health policy documents, policy briefs, evaluation reports and grey literature that are available online from the Ministry of Health and other related ministries, NGOs, civil society organisations and development partners. Our study suggests that the PIP platform is perceived by policy-makers and stakeholders as a useful tool to inform various steps in the policy-making process. The acceptability and reported utility of the PIP platform in Nigeria corroborates experiences in high-income health system settings, where one-stop shops have been used to ensure that policy-makers have timely access to research evidence when pressing issues emerge, which increases the prospects that research will be used in decision-making [23, 41].
The outcome of our evaluation of the PIP showed strong acceptability and feasibility of the one-stop shop platform as a resource to inform policy and systems decisions in Nigeria, including but not limited to policy analysis. Our study also identified the PIP as an asset that might strengthen the interest of policy-makers in using evidence for policy change. Decision-makers in the present study expressed their willingness to use research, which corroborates previous experiences showing that decision-makers are likely to use research evidence for policy-making when the evidence is available and accessible in a timely and user-friendly fashion [2, 4, 5, 6]. In this regard, the PIP explicitly aims to address the main barriers to integrating research into policy and practice, including high costs of access, poor clarity, relevance and reliability of research findings, and complex format of research outputs [7, 8]. Our study further reinforces previous recommendations to improve reporting formats to enhance the intelligibility of policy-relevant research outputs [2, 6, 14]. It is pertinent to state that the PIP may not be able to entirely support the translation of evidence into policy and enhance the capacity to access evidence. We are, however, confident that the availability and accessibility of the evidence in this platform are stepping stones towards enhancing any effort by policy-makers in evidence uptake and improvement of their capacities to access evidence.
Lessons from policy-makers’ engagement
This study included active engagement of policy-makers in the process of developing the PIP platform. Interest in exchanges and engagement between researchers and policy-makers is gaining momentum worldwide, and such engagement seems better suited for the complex nature of policy-making processes than static ‘generalisable’ research (i.e. research initiated by the researchers without any interaction with policy-makers and not targeting the needs of policy-makers) [42, 43]. Interactions with policy-makers enabled the identification of key areas of policy priorities, as well as policy information needs, capacity constraints and formats in which policy-relevant information could be made available and easily accessible. The information that we obtained from the policy-makers aided in development of the platform, making it highly suitable to policy-makers’ needs and thereby facilitating the concept of ownership.
Previous evidence has suggested that, because of inadequate engagement of policy-makers, health policy needs neither drive nor determine the research priority-setting process, thereby resulting in lack of ownership of the health research agenda by policy-makers . It is important for both researchers and decision-makers to recognise the value of coming together in what is in fact a symbiotic relationship in which decision-makers generate feedback from the front lines, while researchers provide expertise in research methods needed for trustworthy studies . This suggests that engagement between policy-makers and researchers may ensure that knowledge generated is better aligned with the health needs of society [44, 45]. Interactions during the stakeholder engagement workshop likely led to differential use of the material in the platform, which might have increased participants’ willingness to use research or their access to the research.
Some vital lessons can be drawn from policy-makers’ involvement in the development of the PIP. One of these lessons is that policy-makers are knowledgeable about Nigerian health policy priority areas and are willing to offer helpful information to researchers. Previous studies in Nigeria have suggested that the engagement of policy-makers is crucial to the use of evidence in the planning and execution stages of health policies [17, 46]; the PIP experience corroborates this knowledge. The topics identified by policy-makers were consistent with the key health policy priority areas outlined in the most recent Nigerian National Strategic Health Development Plan (NSHDP) . The NSHDP has the following eight strategic priority areas: leadership and governance for health, health service delivery, human resources for health, financing for health, national health management information system, partnerships for health, community participation and ownership, and research for health . By increasing access to and usability of policy-relevant documentation, the PIP can serve as a tool to foster evidence-informed policy-making, directed towards the implementation of strategic health programmes in Nigeria, under the aegis of the NSHDP. Health policy documents, policy briefs and policy-relevant peer-reviewed articles covering these priority areas were among the publications included in the PIP.
Value of grey literature
A major innovation of the PIP repository was the inclusion of policy-relevant grey documentation, which accounted for the largest number of publications indexed in the platform. Policy-makers specifically requested the inclusion of policy-relevant information presented in easily available and accessible formats. Specific requests pertained to publications in the form of expert information, pilot studies, case studies, national surveys, monthly and annual reports, bulletins, executive summaries and policy advocacy documents. Policy-makers thus seemed to have an understanding of and confidence in the value of grey literature as a source of vital policy-making resources and tools.
According to Lawrence et al. , peer-reviewed journals might be considered “the most credible source of evidence”, but the reality is that policy-relevant evidence is found in many kinds of resources circulating in the public sphere, most of which fall into the category of grey literature. The importance of grey literature therefore lies in its ability to communicate complex information in simple terms and to disseminate results more quickly . Previous experience has shown that, in policy settings, information in the grey literature may be given greater emphasis than information from peer-reviewed journals because the language is more accessible; furthermore, more rapid and more flexible delivery of information can facilitate knowledge uptake in situations where decisions are based on competing factors (e.g. the pressures of political processes) [50, 51, 52, 53, 54].
It is pertinent to state that concern about the reliability of grey literature has generated some reservations. The lack of editorial control may also raise questions about the authenticity and reliability of documents in the grey literature . In addition, Adams et al.  noted that grey literature is not bound by the publishing conventions that characterise peer-reviewed literature, and that it comes in a variety of forms, posing challenges for data management, extraction and synthesis. To address these limitations, the PIP in Nigeria employed a modified AACODS checklist to critically appraise the identified grey literature and to ensure that the platform included only documents that respect standards of scientific quality and integrity . The modified AACODS checklist was used to include or exclude grey documentation from the platform, and there is a need to further study and validate the modified AACODS checklist before it can be used for traditional appraisal and grading of scientific quality. We also need additional empirical evidence through pilot testing and application of the modified and traditional AACODS approaches to study face validity and implementation determinants (e.g. appropriateness and feasibility).
This study had several limitations. One major limitation was our inability to critically appraise the scientific studies that were identified with PubMed and included in the PIP. Our intention is to continue to improve the quality of studies included in the PIP; to achieve this goal, we will conduct critical appraisal of scientific publications that are considered for future inclusion. Another limitation was our use of a threshold score from the AACODS checklist to determine the inclusion or exclusion of each document from the grey literature. In an earlier report, Jüni et al.  highlighted the problems associated with using summary scores, noting that the incorporation of quality scores as weights lack statistical or empirical justification, as previously pointed out by Detsky et al. . Our intention for the future is to individually assess the relevant methodological aspects of each document to be included in the PIP, as recommended by Jüni et al. . The rather low response rate to the email survey (15.9%) is yet another major limitation. Many decision-makers in Nigeria do not use email frequently, and email-based distribution is therefore the survey strategy that generates the least number of responses among policy-makers .
The need for constant improvement and sustenance of the PIP Nigeria cannot be overstated. Therefore, we will continue to update the contents of the PIP on a monthly basis. We will also continue to expand the scope of the PIP by identifying other priority areas of interest to the policy-makers. Although the PIP focused on evidence specifically produced in Nigeria, global evidence may well be relevant to Nigerian policy-makers, as may be research from neighbouring countries. As part of our efforts to improve the reliability of the PIP, we intend to link the platform to other sources of relevant evidence for policy-makers. Potential improvements to the platform include ongoing and iterative stakeholder engagement, additional dissemination and advertising of the platform through social media and scientific conferences, and integration into the platform of a ‘frequently asked questions’ section and further user-friendly navigation guidance.
An easily accessible one-stop shop of policy-relevant evidence can considerably improve policy-makers’ willingness to use evidence in policy-making and practice. The comments of some of the policy-makers suggest that they are willing to work with researchers to improve the evidence-informed policy-making process when appropriate opportunities are made available to them. It is therefore important for researchers to initiate processes to encourage the engagement of policy-makers. The PIP is one such initiative that can facilitate interaction between policy-makers and researchers; this innovation merits further consideration.
The authors wish to thank all of the policy-makers from the Federal Ministry of Health (Abuja) and the Ebonyi State Ministry of Health (Abakaliki) for their participation in the study. The authors also gratefully acknowledge the participation in this study of senior researchers from Ebonyi State University (Abakaliki) and other stakeholders in the Nigerian health sector.
This investigation received financial support from the Alliance for Health Policy and Systems Research, WHO (AHPSR/WHO) (Reference 2015/529432–0).
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request.
CJU, EVL and AG conceived the idea, and designed and developed the study. CJU, HCU and JC performed data extraction, acquisition and analysis, and developed the PIP. All authors contributed to data interpretation. CJU drafted the manuscript, and EVL and AG provided critical revision of the manuscript for important intellectual content. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Ethics approval for this study was provided by Ebonyi State University Research Ethics Committee (UREC). The study strictly adhered to the UREC guideline, which stipulates voluntary participation, obtaining of informed consent, maintenance of anonymity of participants and confidential treatment of all participant-related information. Informed consent was obtained from all participants who took part in the study during the workshop and via email.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 9.World Health Organization. Global Report for Research on Infectious Diseases of Poverty. 2012. http://www.who.int/tdr/publications/global_report/en/. Accessed 28 Feb 2018.Google Scholar
- 14.Feldman PH, Nadash P, Gursen M. Improving communication between researchers and policy makers in long-term care or “researchers are from Mars; policy makers are from Venus”. Policy Brief (Cent Home Care Policy Res). 2001;5:1–6.Google Scholar
- 19.Nineteenth International Conference on Grey Literature. Public Awareness and Access to Grey Literature. Rome: National Research Council of Italy, CNR; 2017. http://greyguide.isti.cnr.it/attachments/category/30/GL19_Conference_Proceedings.pdf. Accessed 28 Feb 2018.
- 20.Korjonen H, Ford J. Grey literature in public health: valuable evidence? UK Health Forum 2013. https://wachslibrary.health.wa.gov.au/ld.php?content_id=32308111. Accessed 28 Feb 2018.
- 21.Pisa Declaration on Policy Development for Grey Literature Resources. http://www.greynet.org/images/Pisa_Declaration,_May_2014.pdf. Accessed 28 Feb 2018.
- 22.World Health Organization. Changing mindsets: strategy on health policy and systems research. Geneva: WHO; 2012. http://www.who.int/alliance-hpsr/alliancehpsr_changingmindsets_strategyhpsr.pdf. Accessed 28 Feb 2018.
- 28.Evidence Alerts. 2018. https://plus.mcmaster.ca/EvidenceAlerts/. Accessed 28 Feb 2018.
- 31.PDQ-Evidence. 2018. http://www.pdq-evidence.org/. Accessed 28 Feb 2018.
- 32.Policy Information Platform Nigeria. 2018. www.pipnigeria.org. Accessed 28 Feb 2018.
- 35.International Development Research Centre, Swiss Agency for Development and Cooperation. Chapter 8: The Two-Pager: Writing a Policy Brief. In: The RM Knowledge Translation Toolkit: A Resource for Researchers. Ottawa, IDRC and SADC; 2008.Google Scholar
- 36.World Health Organization. WHO Reform, WHO Evaluation Policy. Executive Board EB130/5 Add. 8, 130th session 22. Provisional Agenda Item 5. 2011. http://apps.who.int/gb/ebwha/pdf_files/EB130/B130_5Add8-en.pdf?ua=1. Accessed 6 Nov 2018.Google Scholar
- 37.Tyndall J. The AACODS Checklist is Designed to Enable Evaluation and Critical Appraisal of Grey Literature: Flinders University; 2010. http://dspace.flinders.edu.au/jspui/bitstream/2328/3326/4/AACODS_Checklist.pdf. Accessed 28 Feb 2018.
- 38.National Institute for Health and Care Excellence (NICE). Interim Methods Guide for Developing Service Guidance. 2014. https://www.nice.org.uk/process/pmg8/chapter/appendix-2-checklists. Accessed 28 Feb 2018.Google Scholar
- 39.Giorgi A. Sketch of a Psychological Phenomenological Method. In: Giorgi A, editor. Phenomenology and Psychological Research: Essays. Pittsburgh: Duquesne University Press; 1985. p. 8–22.Google Scholar
- 44.Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide: Alliance for Health Policy and Systems Research, World Health Organization; 2013. http://www.who.int/alliance-hpsr/alliancehpsr_irpguide.pdf. Accessed 28 Feb 2018.
- 45.World Health Organization. Implementation research for the control of infectious diseases of poverty: strengthening the evidence base for the access and delivery of new and improved tools, strategies and interventions: World Health Organization on behalf of the Special Programme for Research and Training in Tropical Diseases; 2011. http://apps.who.int/iris/handle/10665/75216. Accessed 28 Feb 2018.
- 47.Federal Ministry of Health, Nigeria. National Strategic Health Development Plan 2010–2015. Abuja: MoH; 2010. https://www.healthresearchweb.org/files/NSHDP.pdf. Accessed 28 Feb 2018.
- 48.Lawrence A, Houghton J, Thomas J, Weldon P. Where is the Evidence: Realising the Value of Grey Literature for Public Policy and Practice. Melbourne: Swinburne Institute for Social Research; 2014. https://researchbank.swinburne.edu.au/file/8546f00c-f178-48e6-bf86-cd04d0907bef/1/PDF%20%28Published%20version%29.pdf. Accessed 28 Feb 2018.
- 49.National Library of Medicine. Health Services Research and Health Policy Grey Literature Project: Summary Report. 2006. https://www.nlm.nih.gov/nichsr/greylitreport_06.html. Accessed 28 Feb 2018.
- 50.MacDonald BH, Ross JD, Soomai SS, Wells PG. How information in grey literature informs policy and decision-making: a perspective on the need to understand the processes. Grey J. 2015;11(1):7–16.Google Scholar
- 55.Mathews B. Gray literature: resources for locating unpublished research. C&RL News. 2004;65:3.Google Scholar
- 59.Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Enhancing health policymakers’ capacity to use information and communication technology in Nigeria. J Health Inform Dev Countr. 2011;5(2):228–46.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.