Abstract
Background
Information about design, implementation, monitoring and evaluation is central to understand the impact of programmes within the field of sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH). Existing reporting guidelines do not orient on reporting of contextual and implementation issues in sufficient detail. We therefore developed Programme Reporting Standards (PRS) to be used by SRMNCAH programme implementers and researchers.
Methods
Building on the first step of the PRS development (a systematic review to identify reporting items), we conducted a three-round online Delphi consensus survey with experts. Consensus was defined a-priori as 80% agreement of items as essential. This was followed by a technical consultation with a group of experts to refine the items, definitions and their structuring. The revised PRS was piloted to assess its relevance to current SRMNCAH programme reports and identify key issues regarding the use of the PRS.
Results
Of the 81 participants invited to the Delphi survey, 20 responded to all three rounds. In the final round, 27 items received consensus as essential; three items were ranked as “borderline” essential; 20 items as supplementary. The items were subsequently revised, followed by a technical consultation with 29 experts to further review and refine the PRS. The feedback resulted in substantial changes to the structure and content of the PRS into 24 items across five domains: Programme overview; Programme components and implementation; Monitoring of Implementation; Evaluation and Results; and Synthesis. This version was used in a piloting exercise, where questions regarding how much information to report and how to comment on the quality of the information reported were addressed. All items were kept in the PRS following the pilot although minor changes were made to the flow and description of items.
Conclusions
The PRS 1.0 is the result of a structured, collaborative process, including methods to incorporate input from SRMNCAH stakeholders. The World Health Organization will develop a document that explains the items in greater detail, and will also apply the PRS to on-going initiatives. We welcome continuous input from the field, while it is being used, to improve its relevance and usefulness.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Background
There is evidence that high-quality, evidence-based programmes within the field of sexual, reproductive, maternal, child and adolescent health (SRMNCAH) can lead to substantial improvements in health outcomes. As highlighted in the Every Women Every Child Global Strategy for Women’s, Children’s and Adolescents’ health (2016–2030), we have more knowledge than ever before to end preventable deaths and improve the health and wellbeing of individuals across the life course through the use of evidence-based interventions [1].
In order to sustain the current progress and support the further implementation and scaling-up of identified evidence-based SRMNCAH interventions, programmes need to understand not only if an intervention worked (or not), but also how, why and for whom it was successful, as well as information regarding the context in which the intervention was taken forward [2, 3]. This requires adequate and transparent documentation about how programmes were designed, implemented, monitored and evaluated. However, the complex realities of many programmes including their implementation make it difficult to communicate such processes to an external audience. Indeed, the successful implementation and impact of many programmes - particularly those of social and behavioural nature - is very much tied to the “real world” context (e.g. sociocultural, socioeconomic, geographical, legal, political, health system), which may not be easily described [3, 4]. Insufficient detail on the context and implementation does not only hamper replication and scale-up, but contributes to the gap between research and practice [5]. A standardized way of reporting on the processes and contextual elements of programmes throughout the different phases of a programme cycle would allow for conceptualization of information needs from the start of a programme, and also facilitate better documentation and synthesis of elements critical to implementation and sustainability.
Over the last few years, a number of guidelines have been developed to improve the (often inadequate [6, 7]) description of interventions in research articles. For example, the Template for Intervention Description and Replication (TIDieR) provides a 12-items checklist for describing clinical interventions in “sufficient detail to allow replication” [8]. In addition, the on-going development of the UNTIDieR standards aims to provide a framework for the reporting of public health and social policy interventions [9]. Other relevant guidelines include the Standards for Reporting Implementation Studies of Complex Interventions (StaRI) [10], and the Reporting Guidelines for Implementation and Operational Research endorsed by the Bulletin of the World Health Organization to be used by authors of implementation science articles [2]. In addition to the existing standards for different research study designs (see Additional file 1), these guidelines all have in common that they strive to reduce the gap between what is learnt in the field and what is communicated through scientific publications. However, while relevant, they were mostly developed for research reporting and do not cover all relevant aspects of programme design, development, implementation and evaluation processes and outcomes. In addition, the traditional structure of a scientific research article may not be the most appropriate space for many programmes to report on contextual issues or to elaborate on the implementation of components in sufficient detail to allow others to learn from their experiences.
To address this gap and complement existing reporting frameworks, we developed Programme Reporting Standards (PRS) to be used by programme implementers and researchers in the field of SRMNCAH. The overarching goal of the PRS is to provide guidance for complete and accurate reporting on the design, implementation, monitoring and evaluation processes of SRMNCAH programmes. The initiative is a collaborative effort led by the World Health Organization (WHO) Department of Reproductive Health and Research, including the UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), and the Department of Maternal, Newborn, Child and Adolescent Health, in partnership with the Alliance for Health Policy and Systems Research hosted by the WHO.
Overview of PRS development
We developed the PRS in four steps in line with recommendations for the development of health research reporting guidelines [11]: 1) systematic review, 2) three-round Delphi exercise, 3) face-to-face consensus meeting, and 4) piloting through existing programmes.
In the first step we conducted a systematic review of existing reporting guidelines, checklists and other tools, applicable for reporting on programmes targeting sexual and reproductive health (SRH) outcomes. We identified a total of 50 reporting items (Additional file 2), which formed the basis for the subsequent steps of the PRS development. A detailed description of the review, including the methods (screening, data extraction and synthesis) and results have been described elsewhere [12]. While we initially set out to develop the PRS specific to SRH programmes, the systematic review captured reporting tools related to diverse health programmes. In fact, all of the identified items were applicable to programmes within the broader frame of SRMNCAH, and we therefore decided to expand the PRS to these areas.
The subsequent phases of development aimed to determine consensus among experts about which items should be included in a PRS through an initial Delphi exercise (step 2) followed by a technical consultation (step 3), and then a pilot-test of the PRS in order to assess its feasibility and relevance in relation to existing programmes in the field of SRMNCAH (step 4). In this paper, we present results from these steps and introduce the PRS version 1.0. We also summarize the next steps to ensure its uptake and use by programmes.
Methods
Delphi exercise to reach consensus on core PRS items
We undertook a three-round Delphi survey with experts in the field of SRMNCAH to refine and revise the list of 50 items generated in the systematic review [12] by identifying those of highest relevance for the PRS. The Delphi survey technique (Delphi for short) is a structured method to explore, seek consensus and corroborate judgements on a specific topic [13]. Originally developed by the RAND Corporation in the 1950s, the method seeks the opinion of experts through an iterative series of structured survey rounds. Throughout the process, the responses and feedback from participants is fed into the next round until consensus has been reached [14].
Delphi participants and procedure
We invited 81 experts in the areas of implementation, research and funding of SRMNCAH programmes and/or reporting guideline development to participate in an iterative Delphi survey. Using a literature review and global WHO networks, participants were selected because of their specific expertise within these areas, with the goal of having different organizational perspectives (e.g. non-governmental, governmental, donor, universities, UN-bodies) and geographical representation. The Delphi process consisted of three prospectively planned rounds, with the first round starting in September 2015 and the last round concluding in March 2016. Participants had about two to four weeks to respond to each survey round, and reminders were sent after two and three weeks, respectively. Participants were initially encouraged to complete all survey rounds, however invitations for the second and third rounds were restricted to those that responded to the previous round to ensure continuity in responses. For each round, participants received personalized emails containing a link to an on-line survey administered through the electronic instrument SurveyMonkey (www.surveymonkey.com).
Rounds 1 and 2
The purpose of the first scoring round was to introduce participants to the list of items generated from the systematic review, produce initial rankings of the relevance of each item for a PRS, and obtain suggestions for item revisions and their descriptions. Participants were asked to rate the relevance of the items identified in the systematic review using a Likert-type scale ranging from 1 (not important) to 9 (essentially important), and encouraged to suggest new items as well as modification to the structure and language of items.
Participants who completed round 1 were sent a summary of the results (ranking of each items and a revised list of items) and invited to rank the importance of the revised items using the same 9-point scale. Participants were also invited to comment on the structure or language of items and their descriptions.
Quantitative analysis of items scorings: For rounds 1 and 2, we calculated descriptive statistics including mean (SD), median (IQR), minimum and maximum scores for each item and across items. We allocated the scores of each item into three categories of importance: not important (mean scores 1–3); important or desirable but not essential (mean scores 4–6); and essentially important (mean scores 7–9). These categories were used to estimate the percent agreement between participants, defined as the proportion of participants rating an item in the same category of importance. A-priori level of consensus was set to 80% agreement of items as essentially important (mean scores ≥7 on a 1–9 scale).
Qualitative analysis of open-ended comments: We used thematic analysis to synthesize open-ended comments for rounds 1 and 2. Following open-ended coding of relevant text units of each comment, item suggestions were categorized into the PRS domains and sub-domains identified in the systematic review. New sub-domains were created when applicable. For each new item, we summarized the number of participants suggesting the item, provided an explanation of the item, and one or more examples of the participant’s original comment (verbatim). Additional items had to be suggested by at least two participants in order to be included in the subsequent survey round. Comments on the structure or wording of items were reviewed and clustered into similar categories or themes.
Round 3
The goal of the third and final Delphi round was to reach consensus on items to be included in the PRS as essential or supplementary. Scoring results were once more sent back to participants who completed round 2 together with an updated list of items. Based on the rankings from round 2 (during which no items were ranked as “not important”, as further described in the results), we assigned all items into one of two categories, and asked participants to indicate their agreement with this categorization of items as: essential (include in a PRS tool, ranked as essential by at least 80% of participants in round 2) or supplementary (received lower consensus, but were still ranked as important in round 2).
Responses from round 3 were used to estimate the percent agreement between participants as the proportion of participants rating an item in the same category of importance (e.g. the % consistently ranking an item as essentially important in rounds 2 and 3).
Technical consultation to refine and finalize draft PRS
In the next step of the PRS development, we convened a technical face-to-face consultation in July 2016 at the WHO headquarters in Geneva to review the results of the systematic review and the Delphi survey in order to further refine and develop the next version of the PRS. Twenty-nine experts attended the meeting; of these, three had participated in the Delphi exercise, six were staff members or consultants at the WHO, and 20 had not previously participated. While we invited all Delphi participants who completed the three rounds, most responded that they were unable to take part in the consultation due to prior commitments. Additional experts (n = 20) were therefore identified to ensure coverage of the different health areas of SRMNCAH, expertise in different types of health programmes (e.g. service delivery, advocacy, social and behavioural interventions), and representation of different organizations. Plenary discussions and group work were organized to allow for a more in-depth review and discussion of the different items than the online-Delphi had allowed for, with specific focus on the organization, wording and relevance (essential vs. supplementary) of the items. Throughout the meeting, we strived to reach consensus on which items should be included in the PRS, and to identify concrete suggestions for how to improve the structure and flow of the PRS so as to make it user-friendlier.
After the meeting, a revised version was distributed to the meeting participants to ensure that the new version had adequately captured the agreements reached in the meeting. Their feedback was in turn used to produce a penultimate version, which served as the basis for the piloting described below.
Piloting the draft PRS in existing programmes
In the final step of the PRS development, we pilot-tested the PRS during October and November 2016 to assess its relevance and fit to existing SRMNCAH programme reports, and to identify issues and questions regarding its use. In this phase the PRS was used to report on an already completed programmes. Partner organizations nominated programmes of which four were selected to represent different SRMNCAH topics and to create a balance in the programme type (e.g. service delivery, communication, prevention), duration and implementation scale. A programme staff or other representative with good knowledge of the programme activities were asked to gather reports and other forms of documentation relevant to understanding the development, implementation and evaluation processes, and to complete the PRS items by indicating the source and page numbers where the corresponding information could be located. The participants were also asked to provide their feedback related to the completeness, logical flow and formatting of the PRS, definitions and explanations, and other issues or questions. A second reviewer (AEK) verified the information so as to determine if the correct information was reported to the corresponding item. A virtual meeting was held with the team who participated in the pilot to discuss and clarify the feedback and determine final changes to be made. The results from the pilot exercise were used to update the PRS into the final version presented in this paper.
Results
Delphi results
Participant characteristics
Of the 81 experts initially invited, 59.3% (N = 48) responded to the first round of the Delphi exercise. Out of these, 66.7% (N = 32) completed the second survey round and 41.7% (N = 20) responded to all three rounds. Among those that completed all three rounds, 40% represented Universities and 25% non-governmental organisations (NGOs), and the remaining worked for donor agencies (15%), UN bodies (10%), governments (5%), or other organisations (5%). As was the case for previous rounds, most participants in round 3 reported a background in research/academia (55%) and/or programme planning/implementation (45%), followed by management (40%) and medical/clinical (35%) areas (multiple response options possible). Over two-thirds of participants in round 3 (70%) came from organizations that conduct global work, followed by regional representation from Africa and Southeast Asia (10% respectively), Western Pacific and North America (5% respectively) (Table 1).
Item rankings and suggestions
Table 2 presents an overview of the scoring and changes made to items in rounds 1 and 2. Most items in the first round received high ratings with mean scores ranging from 8.8 (Overall goal/objectives of programme) to 6.2 (Innovation). No items were ranked as “not important” (received a mean score below 4). Out of the 50 items identified in the systematic review, 23 items received high consensus (ranked as essentially important by 80% or more of the participants). Based on the open-ended comments received, we made several revisions to the wording and description of items, merged and added new items, resulting in 50 items for the second scoring round.
The ranking of most items remained high in round 2 and none were rejected. Based on participants’ comments we merged six items generating a revised list of 47 items, out of which 28 received high consensus as essentially important for a PRS tool. Two items were ranked as borderline essential (79% agreement).
In the third and final round, all but one (27 of 28) of the items ranked as essential (scores 7–9) in round 2 received high consensus (80%) that this was the correct categorization, and three items were borderline essential. No items were rejected and we did not make additional changes to the items prior to the technical consultation. A detailed overview of the Delphi item scores and suggestions is available in Additional file 2.
Overarching issues raised by the Delphi participants
Beyond the suggestions for additional items, we identified several general issues raised by the participants in the open-ended comment sections. These comments were grouped into four main categories across the three survey rounds: 1) Clarification of items, 2) Justification of rankings, 3) Purpose of PRS development and 4) Programme results.
Comments related to the clarification of items focused on the need to elaborate or better describe the meaning of specific items, or that its applicability might depend on the programme, as in these examples:
Theory/logic model. These things aren't meaningful to everyone. More background/rationale rather than an important component of implementation.
The definition of Sustainability is a little odd. Would it be clearer to say something along the lines of 'the ability to maintain the programme and its effects over time?
Some items were revised accordingly, while others will be further elaborated in a WHO document that will describe and explain how to use the PRS.
A number of participants also made comments to justify their rankings, noting the difficulty in ranking items as they “all are important and useful”.
The relevance of Fidelity depends on the circumstances. If one is conducting a study, you need to be able to determine if you have poorer than expected performance whether the problem is one of design or execution. So, in that setting, knowledge of fidelity is important.
Some participants posed questions about the purpose of the PRS and how it will be used; for example, whether items should be used to describe pilots or on-going programmes.
I've found it somewhat difficult responding to questions, not having a clear enough sense of what's meant by "program" and, when "reporting" is referred to, who is to be reporting to whom? In places, the language suggests that what's referred to is some kind of pilot effort. In some places, the language suggests some short-term "intervention" (like a training activity). In other places, it sounds more like on-going service delivery.
There was also feedback related to items missing to describe programme results, for example by asking where “anticipated/expected effects are reported” given that the evaluation domain “doesn't seem to include reporting of the impacts apart from unexpected/negative effects and differential effects”. While the original purpose of the PRS tool was to provide guidance for the reporting of programme development, implementation and evaluation processes rather than results, this issue was further discussed during the technical consultation as described below.
Technical consultation results
The technical consultation to further refine the PRS brought together experts representing 17 non-governmental, governmental, bilateral and multilateral organizations from 12 countries in different global regions (Table 3). The synthesized feedback from this interactive expert meeting resulted in a number of changes to the content and description of items. Most importantly, the groups’ feedback resulted in substantial improvements to the structure and user-friendliness of the PRS by re-organizing and merging items (sometimes with new headings), and rewording many of research-oriented items to make these more programme-oriented. A decision was also made to use sub-items (1a, 1b, 2a, 2b…) to further improve the flow and structure of the PRS rather than numbering each item. Almost all of the items that received high consensus as essential during the Delphi survey were kept in some form, but their structure and wording changed or they were merged with other items, and most supplementary items were integrated into the revised list. For example, the items Target population and area (essential) and Partners and stakeholder involvement (supplementary) were included as sub-items under a new item called Stakeholders. The item Components/activities (essential) was expanded to cover the items Timing (borderline essential), Setting (essential), Dose and intensity (essential), Provider/staff characteristics (supplementary), Provider/staff training (supplementary), Participants (essential), and Methods used to deliver activities (essential). Some supplementary items (e.g. Programme name, Dissemination plans, Appropriateness, Participant preparation) were deleted as the group perceived these to be redundant. As a result of the revised structure, all items were considered by the expert group to be essential, thus removing the need to distinguish between essential and supplementary items.
Much attention was given to the role of context in programme reporting, and as a result, the expert group decided that context should be added as a specific item as well as highlighted where applicable throughout the PRS. The group also discussed the need for the PRS to capture the dynamic nature of many programmes, including how and why activities change over time. Following up on the issue of results raised in the Delphi survey, the group further decided that Results should indeed be part of the PRS and reported together with the programme evaluation process.
Taken together, the group condensed the Delphi version of the PRS, including 27 essential and 20 supplementary items, into 24 items across five re-organized domains: Programme overview; Programme components and implementation; Monitoring of Implementation; Evaluation and Results; and Synthesis. This version was piloted as further described below. A detailed overview of the versions prior to and following the technical consultation is available in Additional file 2.
Piloting results
Key characteristics of the four programmes selected to participate in the pilot are summarized in Table 4. These included a media and communication programme (BBC Media Action); a maternal health programme (United Nations Population Fund [UNFPA]); a family planning service delivery programme (United States Agency for International Development [USAID]); and a SRH information and service delivery programme (Rutgers).
Overall, the reviewers were able to report on and provide the source of all items applicable to the programme. While the second reviewer verified most of the items, there was some disagreement on whether or not certain items were reported on, mainly due to differential interpretations of their meaning and how much information should be provided in order for an item to be considered “reported”. Indeed, reviewers raised the questions of how extensive the items should be reported, highlighting the tension between what is reported and the quality of such reporting.
I also found it hard not to want to give a qualitative assessment of the information provided on each item, i.e. where there was some information reported on an item but I didn’t feel the reporting was complete/of a high standard. Guidance on how to approach partial/weak reporting on certain items when completing the tool would be helpful.
In their comments, all reviewers highlighted the usefulness of the tool to both guide programme reporting as well as to strengthen up-front programme design. However, they also stressed that the PRS took longer to complete than they had anticipated. This was mainly due to the fact that the information provided was scattered between multiple different sources and rarely were all items reflected in one single report. Despite its length, none of the reviewers felt the need to remove items, but rather noted that the PRS was useful to help them organize different information sources, and that it might be easier to use in a prospective manner.
The reviewers further emphasized that the PRS may need to be applied differently to large-scale programmes consisting of multiple different components. As one reviewer noted:
If this list will become the norm of reporting, it means the reports will remain many and big for such a large programme, whereas this is not always considered useful. Currently, there is even a tendency from our donor to require smaller reports, mostly based on reflection. Therefore, it should be clear how reports based on this PRS tool will be used.
This issue was further discussed during the virtual meeting during which the reviewers agreed that larger programmes might need to be broken down into smaller components (e.g. country levels, or specific topics) as part of the reporting process.
None of the items were deleted as a result of the piloting; minor language edits were made to further clarify the description and flow of the 24 items; and decisions were made as to what information should be picked up in the WHO document to be developed where more in-depth explanations will be made available.
PRS version 1.0
Table 5 presents the PRS (version 1.0), which consists of 24 items across five domains: 1) programme overview, 2) programme components and implementation, 3) monitoring of implementation, 4) evaluation and results, and 5) synthesis. Below we provide a brief description of each domain.
Programme overview
The programme overview section consists of seven items, starting with the programme’s rationale and objectives (why it was initiated, goals, anticipated effects), start and end date, and the programme context (geographical setting and an overview sociocultural, political, historical, legal, health systems or other relevant contextual aspects). Additionally, this domain includes a description of the programme stakeholders and their roles, the funding source, use of a theory of change or logic model to guide the programme, and human rights perspectives (e.g. gender, equity, rights and ethical considerations) as well as the use of an accountability framework to define and follow up on such commitments.
Programme components and implementation
Four items make up the programme components and implementation section, starting with how activities were decided upon and why, any piloting of activities along with results, followed by a detailed description of each core component, including what was done, how, when, by whom, for whom, and any support materials used. This section also includes an overview of mechanisms for ensuring the quality of implementation such as support to personnel, refresher trainings and product quality checks, and any efforts to increase and sustain the participation of different stakeholders.
Monitoring of Implementation
The third section covers six items related to how the programme implementation was monitored (and indicators used to track progress), the coverage and reach of activities, any adaptations to the programme (whether it was delivered as intended, on-going adaptation of the activities to better fit the context), and reflections on whether the programme was considered to be reasonable and relevant among stakeholders. The section ends with a reflection on the feasibility of the programme (i.e. the extent to which the it could be carried out the context where it was implemented), and factors affecting implementation (key barriers and facilitators)..
Evaluation and results
The fourth section covers three items related to whether an evaluation was conducted (type of evaluation and specific methods), and a description of the programme results including key indicators (process, output, outcome), short, mid and long-term effects by key sociodemographic characteristics as well as potential unexpected effects, and an overview of the required resources for implementation and whether a cost analysis or cost-effectiveness analysis was conducted.
Synthesis
The final section of the PRS includes four items to describe the key lessons learnt, the sustainability (e.g. expected ability to maintain activities) as well as scalability of the programme, and a reflection on the context-dependence of the programme (whether it could be implemented in or adapted to other settings).
At the end of the PRS, a section is included to allow for supplemental information (e.g. references/links to relevant sources), and any additional comments that could help clarify the items reported.
Discussion
In this paper we present the PRS 1.0; a tool intended to guide, and thus aims to improve the reporting of SRMNCAH programme design, implementation, monitoring and evaluation processes. The PRS is the result of a structured, collaborative work process including interactive input from different stakeholders in the field of SRMNCAH. The 24 items included in the PRS reflect those identified to be most central to the adequate and transparent reporting of programmes. The overarching goal of the PRS is to facilitate exchange of information and evidence synthesis within and between different programmes and sectors working to improve the health and wellbeing of individuals across the SRMNCAH continuum. Our efforts are in line with the increased recognition to understand not only the evidence-base of outcomes but how and in what contexts successful outcomes were achieved [3, 4]; what worked and what did not work, challenges faced in the field and actions taken to overcome such challenges.
In particular, the PRS responds to the need for more accurate descriptions of the role of context in programme development and implementation [3,4,5]. While its interpretation and meaning vary greatly in health sciences, a recent review defined context as “a set of characteristics and circumstances that consist of active and unique factors that surround the implementation effort” [5]. Central to this definition is the fact that context is not just a “backdrop” of programme implementation but “interacts, influences, modifies and facilitates or constrains the intervention and the implementation effort” [5]. Indeed, the complex and multifaceted nature of context and how best to capture this as part of programme reporting was intensely debated throughout the PRS development. In particular, participants in the Delphi survey as well as technical consultation noted the tension between including a specific item to describe programme context versus integrating the role of context throughout the PRS. This deliberation resulted in the decision to do both; i.e. the PRS includes both a specific item that provides an overview of the programme context (item 3), but the role of contextual elements in shaping programme implementation is also highlighted throughout the PRS, for example in items 16 (feasibility), 17 (factors affecting implementation) and 24 (possibilities for adaptation in other settings).
A central challenge has been to develop PRS so that is broad enough to apply to a wide range of fields within the SRMNCAH continuum; yet specific enough so that people feel it is useful for and applies to their specific programme or topic area. Similarly to the WHO Bulletin Reporting guidelines for implementation and operational research [2], the PRS is not specific to a certain programme or study design, but rather integrates and builds upon existing tools. Given the many checklists have already been developed for reporting of specific research designs in the peer-reviewed journals, we feel that this does not need to be repeated in the PRS. Rather, programme staff and authors of reports and publications should use guidelines applicable to their selected design for the programme evaluation while reporting them for publications. Beyond the designs listed in Additional file 1, the Enhancing the QUAlity and Transparency of Health Research (EQUATOR) network provides an excellent overview of research reporting guidelines (http://www.equator-network.org/library/) according to different designs and topics.
An additional question raised is when the PRS should be used. While the piloting conducted as part of this paper was retrospective and provided a sense of whether the items could be located in existing programme reports, our intent is that the PRS will be used in a prospective manner to guide the continuous reporting of programmes. In addition, many of the items might be highly relevant to consider at the beginning of the programme; i.e. “beginning with the end in mind” [11]. As part of the verification process, we noted how the reported items were scattered between multiple different reports and other sources (e.g. proposals, evaluations, briefs, monitoring frameworks) and the goal is thus to provide support for how this information can be brought together. In this respect the PRS is conceptually different from other research reporting standards.
The PRS can be used to note where additional information can be located beyond what is written in any report or publication. Furthermore, because of its comprehensive nature and amount of detail covered by the items, larger programmes may break their reporting into smaller components. For example, a global programme could be broken into national level or to specific sub-topics or objectives in a way that does not increase but rather ease the “burden” of reporting. It is also important to note that reported information does not need to be organized in the exact same order as the tool is structured; the intent is to use PRS is a checklist to verify that the core items have been covered.
What, then, “counts” as an item being covered or reported on? A key issue highlighted by our respondents was how to judge whether the information provided is sufficient and of good enough quality to be considered reported. Importantly, the PRS is not a quality assessment framework, but a tool to guide the reporting of programmes with specific focus on context and implementation processes. While there is no gold standard for how much information to provide, a sufficient amount should however be provided to allow for someone not familiar with the programme to understand the context and rationale underlying its implementation. In this first version of the PRS, we have agreed on what items need to be reported. How much information is needed, and the quality of that information, will be further assessed through the continuous use of the tool where feedback from the field will be essential. The PRS will be available to the public through the WHO website as well as the website of key partners. In addition, the WHO will develop a guidance document that explains and elaborates on each item in greater detail. The WHO also intends to actively use this tool as part of on-going initiatives and implementation, for example by sharing country experiences in a structured way as part of the learning platform designed for improving quality of maternal and newborn care in health facilities [15], as well as to promote its use across implementing and donor organizations. Many of the key partners and donors who have been part of the PRS development have also committed to actively promoting its use with the programmes that they support. Our goal is to follow up with agencies using the PRS during 2018 to determine what modifications could be made for version 2.0.
Strengths and limitations
In line with recommendations for the development of reporting guidelines [11], we developed the PRS through a systematic approach that is grounded both in the synthesis of existing literature as well as expert opinions and a piloting process. A key strength was the use of a Delphi panel with experts from different organizations in the field of SRMNCAH to rank and reach consensus on core reporting items. While the Delphi process was useful for us to prioritize the 50 items identified in the systematic review and invite feedback from a larger group, the technical consultation with a smaller group of experts provided a more in-depth understanding and feedback, which was not feasible during the Delphi. This meeting was essential to the development of the PRS; in fact, it was during this process that most changes to the PRS were made as a result of the interactive discussions, informed by the Delphi rankings. The revisions following the technical consultation were further evaluated in the pilot, which allowed more specific input from staff and managers with regards to the feasibility and applicability of the PRS to their specific programmes.
The development of the PRS should also be considered in light of its limitations. First, the consultative process was limited to the people who responded to the Delphi and came to the technical consultation. Although the PRS may be applied more generically, we developed the tool with input from the SRMNCAH field. In addition, the draft PRS was only piloted retrospectively with four existing programmes and it is possible that we would have received more input related to the real-word utility and been able to address issues prior to the launch of the PRS had we been able to extend this initial piloting face to a larger number of interventions, and over time. As such, it will be important to continuously seek input and update the PRS to reflect input from a wider audience.
Conclusion
The starting point for the PRS development was the need for better reporting of why, how, when and under which circumstances programme activities are implemented, including lessons learnt from the field. We believe that the PRS will be a useful reference to programme implementers and researchers in the field of SRMNCAH. As implicit by its name – PRS 1.0 – this is the first version of the tool. The development of the PRS is an evolving process, and we look forward to input from the field to improve its relevance and usefulness.
Abbreviations
- EQUATOR:
-
Enhancing the QUAlity and Transparency of Health Research
- HRP:
-
UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction
- PRS:
-
Programme Reporting Standards
- SRH:
-
Sexual and Reproductive Health
- SRMNCAH:
-
Sexual, reproductive, maternal, newborn, child and adolescent health
- StaRI:
-
Standards for Reporting Implementation Studies of Complex Interventions
- TIDieR:
-
Template for Intervention Description and Replication
- UNFPA:
-
United Nations Population Fund
- USAID:
-
United States Agency for International Development
- WHO:
-
World Health Organization
References
Every Woman Every Child. The Global strategy for women’s, children’s and adolescents’ health (2016–2030). 2015.
Hales S, Lesher-Trevino A, Ford N, Maher D, Ramsay A, Tran N. Reporting guidelines for implementation and operational research. Bull World Health Organ. 2016;94:58–64.
Waters E, Hall BJ, Armstrong R, Doyle J, Pettman TL, de Silva-Sanigorski A. Essential components of public health evidence reviews: capturing intervention complexity, implementation, economics and equity. J Public Health (Oxf). 2011;33:462–5.
May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016;11:141.
Pfadenhauer LM, Mozygemba K, Gerhardus A, Hofmann B, Booth A, Lysdahl KB, et al. Context and implementation: A concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhsw. 2015;109:103–14.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–9.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267–76.
Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348.
EQUATOR network. UNTIDieR project summary. Available: http://www.equator-network.org/wp-content/uploads/2009/02/UNTIDieR-project-summary.pdf
Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7:e1000217.
Kågesten A, Tunçalp Ö, Ali M, Chandra-Mouli V, Tran N, Gulmezoglu AM. A Systematic Review of Reporting Tools Applicable to Sexual and Reproductive Health Programmes: Step 1 in Developing Programme Reporting Standards. PLoS One. 2015;10:e0138647.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32:1008–15.
RAND Corporation. Delphi Method. Available: http://www.rand.org/topics/delphi-method.html
Tuncalp Ö, Were WM, MacLennan C, Oladapo OT, Gulmezoglu AM, Bahl R, et al. Quality of care for pregnant women and newborns-the WHO vision. BJOG. 2015;122:1045–9.
Acknowledgements
The authors wish to thank all the participants of the Delphi survey, the technical consultation, and the programme staff that participated in the piloting of the PRS. Special thanks to Dr. Venkatraman Chandra-Mouli at the WHO Deparment of Reproductive Health and Research for his continuous input. We are grateful to the funders for their support.
Funding
Funding support was provided by the Alliance for Health Policy and Systems Research hosted by the WHO, and the United States Agency for International Development.
Availability of data and materials
Most data generated or analysed during this study are included in this article [and its supplementary information files]. Additional data are available from the corresponding author on reasonable request.
Author’s contributions
OT, AEK and AGP conceptualized and conducted the steps to develop the PRS with critical input from AMG, MA and NT. AEK wrote the first draft of the manuscript, which was further developed by OT and AGP. All co-authors reviewed and contributed to the final version of the manuscript.
Disclaimer
This report contains the collective views of an international group of experts, and does not necessarily represent the decisions or the stated policy of the World Health Organization.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional files
Additional file 1:
Reporting guidelines for research studies. (DOCX 108 kb)
Additional file 2:
Summary of results of the systematic review, three-step Delphi process and the draft PRS in preparation for the technical consultation. (DOCX 565 kb)
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Kågesten, A.E., Tunçalp, Ö., Portela, A. et al. Programme Reporting Standards (PRS) for improving the reporting of sexual, reproductive, maternal, newborn, child and adolescent health programmes. BMC Med Res Methodol 17, 117 (2017). https://doi.org/10.1186/s12874-017-0384-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12874-017-0384-7