Background

In 2012, the World Health Organization (WHO) published evidence-based recommendations regarding the use of optimization or “task-shifting” strategies for key, effective maternal and newborn interventions (OptimizeMNH) [1]. Typically, task-shifting involves training lower level health worker cadres to deliver tasks that are normally delivered by higher level health worker cadres. The main aim of this strategy is to increase access to care for people living in areas with a shortage of health workers.

While task-shifting strategies have the potential to increase access to care in settings with critical shortages of specialised health workers, there is also concern that the delivery of health care by health workers with less training could cause harm to people already in vulnerable situations. The effectiveness and safety of such interventions therefore needs to be rigorously evaluated. However, task-shifting strategies that have been assessed as effective and safe in the context of experimental studies may be challenging to implement at scale within a health system. The shifting of tasks from one group of health workers to another can involve social, behavioural and organisational change and, like other complex interventions, its success may depend not only on characteristics of the intervention itself but also on characteristics of the systems in which the intervention is implemented or on interactions between properties of the intervention and systems [2–4]. The success of task-shifting strategies is also likely to be influenced by factors such as the training, supervision and support provided to health workers as well as access to supplies and referral systems. The culture and organisation of healthcare delivery in a particular setting, as well as views and expectations among stakeholders such as service users, the providers involved in task-shifting and managers, may also play important roles in the success of these types of strategies [5–7]. Furthermore, the shifting of tasks can have serious implications for health worker motivation, for instance because of changes in working conditions and salaries, and for health service users’ experience of care [8, 9].

As these issues can represent important challenges to successful implementation, information about an intervention’s feasibility and acceptability can therefore be as important as information regarding intervention effectiveness and safety when recommendations are made. Initiatives that aim to improve the quality of guidelines, such as the AGREE II instrument, also emphasise the importance of seeking stakeholders’ views and preferences, and of discussing potential organisational barriers to the application of recommendations [10]. But despite their potential importance, questions about acceptability and feasibility are usually not addressed with the same rigour as questions about intervention effectiveness. Instead, it appears that these issues are often left to the opinions and experiences of guideline panel members. While this input is useful as part of a dialogue [11], it is limited by the composition of the group. For global guidelines such as OptimizeMNH, the input of panel members from specific geographic or clinical settings and with experience of particular population groups may not give sufficient insight into the values and experiences of service users and other stakeholders more broadly. In addition, the perspectives of some panel members may take precedence over other members because of perceived differences in social status within the guideline panel [12, 13].

As the technical team responsible for developing the OptimizeMNH guidance (Table 1), we saw a need to address these issues in the evidence-to-decision process. In this paper, we describe our use of innovative strategies to expand the evidence used as a basis for WHO recommendations and discuss our experiences with these strategies.

Table 1 Defining guidance
Full size table

Methods

In developing the OptimizeMNH guidance, we followed the standard procedures set out in the WHO’s Handbook for Guideline Development [14]. These procedures (which include the formulation of questions and outcomes; the retrieval, assessment and synthesis of evidence; the development and formulation of recommendations; and finally, the dissemination and implementation of these recommendations) are intended to ensure that WHO recommendations address key needs and are evidence-based [15]. However, we also elaborated on these standard procedures by adopting innovative approaches that could help us address our goal of producing evidence-informed recommendations that are informed by a broader set of evidence than data on effectiveness alone. These approaches were used at different stages of guidance development: when gathering the evidence to support the guidance, when assessing and synthesising this evidence, and when developing the recommendations (Table 2). The approaches included the following:

Table 2 Innovative strategies and resources available to guidance developers
Full size table

  • Broadening the range of evidence used in the guidance by including systematic reviews of qualitative research, multi-country case study syntheses and results from a primary qualitative study

  • Using the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach to assess our confidence in the evidence from systematic reviews of qualitative research

  • Using a structured health systems framework (the DECIDE framework) to help the guidance panel move from this wider range of evidence to recommendations

Results

Gathering the evidence—broadening the range of evidence included in the guidance

The WHO’s Handbook for Guideline Development describes not only several types of questions that can legitimately be addressed in a WHO guideline, including the effectiveness or efficacy of an intervention, but also the negative consequences of the intervention, its social acceptability and its cost-effectiveness. The Handbook describes how qualitative evidence synthesis can be used to explore contextual barriers as well as values and preferences [14]. The main focus of the Handbook is, however, on the use of evidence about effectiveness and, in practice, most WHO guidelines have focused almost exclusively on this question.

For OptimizeMNH, we systematically assessed the effectiveness and safety of the relevant task-shifting interventions through systematic reviews and the use of GRADE. In addition, we used an innovative approach in this stage of guidance development by systematically including other types of evidence. Specifically, we compiled evidence on the acceptability of the interventions among health workers, recipients of care, policy makers and other stakeholders and on the feasibility of these interventions, taking into account the organisational changes to health systems that each task-shifting intervention would entail.

While systematic reviews of randomised trials are seen as the most robust source of evidence when assessing intervention effectiveness, this approach is not well-suited for exploring issues surrounding intervention acceptability and feasibility, particularly for more complex health and social interventions [2, 16]. To answer these questions, we included systematic reviews of qualitative studies (sometimes called qualitative evidence syntheses) as our primary source of evidence [17]. Together with colleagues, we co-authored four qualitative evidence syntheses focusing on factors affecting the implementation of task-shifting among lay health workers, midwives, doctors and nurses [18–21].

The qualitative evidence syntheses provided us with information that was clearly relevant to the questions we had asked. However, most of the studies were of relatively small-scale or pilot programmes, often implemented in the context of research, and focused on factors unfolding at the level of programme delivery in communities and primary care facilities. We decided therefore to complement this evidence with two multi-country case study syntheses focusing on factors affecting the implementation of large-scale programmes. When selecting these programmes, we aimed for broad geographical representation and looked for countries in Africa, Asia and South America that had implemented large-scale programmes. To make the synthesis manageable, we limited ourselves to seven programmes and only selected programmes where a reasonable amount of English-language documentation was available. The multi-country case study syntheses aimed to identify “upstream”, system-level factors associated with programme policies, governance, financing, planning, management and organisation [1, 22]. These case studies synthesised evidence from a variety of sources, including peer-reviewed qualitative and quantitative studies, programme reports, web sites and information from personal communication with individuals familiar with the selected task-shifting programmes.

Another limitation of the evidence we identified through the qualitative evidence syntheses was that it focused primarily on the views and experiences of service users and healthcare providers but paid less attention to the viewpoints of programme managers, policy makers and other stakeholders. We attempted to address this imbalance by carrying out our own primary study. We undertook qualitative analyses of the views and experiences of those contributing to the “Health Information For All By 2015” electronic discussion list, which covers a diverse membership of programme managers, healthcare providers, policy makers, academics and others from 170 countries [23]. These analyses evaluated opinions about how the roles of healthcare providers could be optimised to improve maternal and newborn health in LMICs and the implications of such role optimization [1, 24].

When presenting the evidence from the qualitative evidence syntheses, the country case studies and the primary qualitative studies, we attempted to extract findings that were relevant to specific recommendations being considered by the guidance panel and that concerned specific tasks and groups of health workers. These findings were used to (a) support the guidance panel when reaching recommendations and (b) develop information for end users of the guidance regarding implementation considerations for each recommendation. Tables 3 and 4 provide examples of how these findings informed the development of recommendations in the OptimizeMNH guidance.

Table 3 Example of how the qualitative evidence informed the final recommendations in the OptimizeMNH guidance (1): provision of continuous support during labour by lay health workers, in the presence of a skilled birth attendant
Full size table
Table 4 Example of how the qualitative evidence informed the final recommendations in the OptimizeMNH guidance (2): provision of vasectomy by trained midwives
Full size table

We also wanted to offer those using the OptimizeMNH guidance more general information about the acceptability and feasibility of task-shifting strategies. We therefore carried out a “cross-cutting” analysis of findings from all of the sources described above. This cross-cutting analysis provided evidence regarding task-shifting in general where no direct evidence was available for specific task-shifting interventions. The cross-cutting analysis also contributed to a chapter in the WHO OptimizeMNH guidance on overarching implementation considerations.

Assessing and synthesising the evidence—introducing an approach to assess our confidence in findings from qualitative evidence syntheses

The WHO Guideline Handbook notes that guideline technical teams need to provide an assessment of the quality of the evidence included in a WHO guideline [14]. For evidence of effectiveness, the WHO uses the GRADE approach to assess quality [25]. GRADE is not, however, designed to be applied to qualitative research, and we were not aware of other systems for assessing quality or confidence in findings from qualitative evidence syntheses or for indicating these assessments to end users. Having made the decision to include qualitative evidence syntheses, we therefore needed an approach in transparently assessing and describing how much confidence to place in findings from these types of syntheses.

Work to develop this approach was carried out in collaboration with other qualitative researchers [26], informed by the principles of qualitative research. We were also influenced by the processes used by the GRADE Working Group, which has given considerable thought to how to assess confidence in evidence from systematic reviews. This work resulted in an approach which we named GRADE-CERQual [26]. Since its initial use in the OptimizeMNH guidance, CERQual is being developed further as part of the GRADE Working Group and is now being used in several other WHO guidance development processes [27].

Developing the recommendations: using a structured health systems framework

When assessing the different factors that influence recommendations, including effectiveness, acceptability, feasibility, and cost, the Guideline Handbook [14] suggests that the guideline panel makes use of evidence-to-recommendation tables. These tables can be used to lay out what we know about these different factors and can also be used to record the guideline panel’s judgements about these factors and how they contributed to the development of the recommendation [14]. For OptimizeMNH, we piloted an early paper version of a decision table developed by the DECIDE project [28] and the Grade Working Group [29]: the DECIDE Evidence to Decision framework [30, 31].

The DECIDE framework aims to help guidance panel members move from evidence to health system recommendations by informing judgements about the balance of consequences of each option. The framework is based on a review of relevant literature, brainstorming, feedback from stakeholders [32], and the application of the framework to examples.

For each guidance question, the framework presented, in a structured format, a summary of the evidence regarding the following considerations:

  • The benefits and harms of the intervention/s (sometimes referred to as “effectiveness and safety”)

  • Anticipated resource use

  • Acceptability of the intervention/s

  • Feasibility of the intervention/s

For each of these elements, the framework also included an assessment of the certainty of the evidence, using GRADE (for evidence of effectiveness) and CERQual (for evidence on acceptability and feasibility from qualitative evidence syntheses). The following were also included for each guidance question:

  • A judgement regarding the balance of desirable and undesirable consequences for the intervention/s

  • A recommendation and a justification for this

  • Implementation considerations for the intervention/s

  • Relevant monitoring and evaluation/research priorities in relation to the intervention/s

In addition to the evidence collated in the DECIDE frameworks, full evidence profiles for the reviews of the effectiveness of interventions, as well as summaries of findings for the qualitative evidence syntheses on the acceptability, feasibility and implementation of these interventions, were made available to the guidance panel. These summaries were, in turn, linked to full systematic reviews. The DECIDE Team also prepared a video to help guidance panel members understand the purpose of the framework and how to use it. This was made available prior to the guidance panel meeting and also shown at the start of the meeting.

Discussion

Broadening the range of evidence included in guidance: lessons learnt

The first innovative approach that we used in the development of the OptimizeMNH guidance was the systematic inclusion of additional types of evidence to complement the evidence of effectiveness (Table 5). This approach had a number of implications. On a practical level, broadening the scope of evidence to be considered in relation to each guidance question required more time and resources. It also required a wider set of synthesis, assessment and interpretation skills within the technical team preparing the documents for the guidance panel. In our case, these skills were represented within the technical team. Other technical teams using a similar guidance development approach may not have the skills to undertake qualitative evidence syntheses or country case study syntheses but will at least need skills in commissioning and critically appraising such evidence. Creating technical teams with these skills may be challenging as there are few groups with extensive experience in implementing them. However, the number of researchers with the appropriate skills is increasing rapidly, and guidance is now well-developed [33–35].

Table 5 Overview of the different types of evidence used in the OptimizeMNH guidance development process
Full size table

We had originally planned to gather evidence about stakeholder acceptability and feasibility through qualitative evidence syntheses only. As described above, these syntheses, which only included published qualitative studies, primarily offered evidence surrounding factors and stakeholders at the community or primary healthcare facility level. This led us to carry out the multi-country case studies, where we included a broader set of information sources, and also led us to carry out our own primary study among stakeholders poorly represented in the qualitative evidence syntheses. In retrospect, we think that the findings of this primary study should have been included in the qualitative evidence syntheses rather than being treated as a stand-alone product.

Although this innovation expanded the preparatory work needed for the guidance, we would argue that the inclusion of evidence from qualitative evidence syntheses and country case study syntheses offered relevant information for the guidance panel on the feasibility and acceptability of the interventions. When such evidence is not systematically assessed and included, acceptability and feasibility issues may either be overlooked altogether or incorporated into guideline decisions based only on anecdotal evidence. Our experience from the development of the OptimizeMNH guidance suggests that the inclusion of evidence from qualitative evidence syntheses and country case study syntheses reduced the use of anecdotal evidence by guidance panel members when assessing the balance of consequences for each guidance question. In addition, our use of such evidence appeared to address guideline panel members’ perceptions that the WHO often over-emphasises evidence from randomised trials at the expense of evidence from programme experience. One additional benefit of synthesising relevant qualitative evidence was that we were able to use this to develop implementation considerations for task-shifting in maternal and newborn health [1].

Introducing an approach to assess confidence in findings from qualitative evidence syntheses: lessons learnt

Our second innovation was the use of the GRADE-CERQual approach in order to systematically and transparently assess our confidence in findings from the qualitative evidence syntheses conducted to inform the guidance.

Our use of CERQual was received well by the guidance panel, in part because it addressed the need to systematically incorporate a wider range of evidence into the guidance process. The approach also had a number of advantages. When making recommendations, all guidance panels need to take into account how confident they can be in the underlying evidence. Had we not provided the OptimizeMNH panels with the CERQual assessments of our confidence in the findings from the qualitative evidence syntheses, they would have had to make their own judgments and may not have done this in a systematic way. The use of CERQual also allowed us to identify and highlight gaps in the evidence. For instance, where we assessed the confidence in the evidence for a qualitative finding to be low or very low, this was an indication that further research was needed in this area. These research gaps were also reflected in the information presented to the guidance panel and carried through to the recommendations.

We were initially concerned that using the CERQual approach would confuse guidance panel members as they would not have encountered this approach before. In practice, this did not appear to be a problem, probably because members were already familiar with GRADE for assessing findings regarding the effectiveness of interventions, which uses the same principles, and because the panel was briefed on the approach at the start of the meetings.

Some of the findings contributing to OptimizeMNH were based on a very wide range of evidence types. In particular, the country case studies included both qualitative and quantitative studies as well as programme reports, which often did not provide detailed descriptions of methods used. Using CERQual to assess our confidence in the findings from the country case study syntheses was not feasible as the CERQual approach is not designed, at present, to accommodate this range of evidence [26].

Using a structured health systems framework: lessons learnt

Our final innovation was the use of DECIDE evidence-to-decision frameworks [36] to present evidence to the guidance panel. There were several advantages in using this structured framework. Firstly, the framework helped panel members to think through each of the considerations that might be important in making a recommendation and ensured that key considerations were taken into account. The concise format also appeared to help focus and structure the discussions and may have improved the use of the limited time that the guidance panel had to weigh the balance of consequences and make judgements about complex questions regarding task-shifting. Because the framework prompted the guidance panel to justify all of the recommendations made, we believe that the transparency of the decision-making process was improved. Furthermore, end users of the guidance have access to a clear record of recommendation decisions and how these were made. The framework also contributes to closing the knowledge-to-action cycle by capturing implementation considerations and highlighting evidence gaps where more research is needed [37, 38].

The use of the evidence-to-decision frameworks also led to a number of challenges. First of all, preparation of the frameworks took considerable time in advance of the panel meeting, although this may have saved time later on when preparing the final guidance document. Secondly, because the DECIDE framework is a new tool, it required some explanation in advance of the panel meetings. However, panel members appeared to grasp the function and content quickly, and the informal feedback on the framework was positive.

Because the OptimizeMNH guidance included a very large number of recommendations (128 in total), the amount of time available in the panel meetings to discuss each evidence-to-decision framework was limited. The inclusion of additional types of evidence also added to the amount of information that panel members were expected to read. Nonetheless, our experience suggests that the structured format of the DECIDE framework allowed the technical team to present a large and complex body of evidence to panel members in a fairly straightforward and easy-to-assimilate way. Further work is now needed to explore guidance panels’ views of such evidence-to-decision frameworks and to examine how the presentation of evidence in these formats impacts on the deliberations and decisions of guidance panels and on the transparency to users of these decisions.

Conclusion

In this paper, we have described our use of a number of innovative strategies to expand the range of evidence used to develop WHO global recommendations and discussed our experiences with these strategies. Expanding the range of evidence contributing to a guidance process increases the amount of time and resources needed and the range of skills required within the technical team developing the guidance. However, our experience with the OptimizeMNH guidance suggests that such efforts may be valuable, particularly for recommendations on more complex health and social interventions. This view is supported by efforts within WHO to replicate the OptimizeMNH approach in other guidance processes [27]. When considering whether to adopt a similar approach, producers of guidance should assess the extent to which acceptability and feasibility issues are likely to be important enough to influence their recommendations. So far, the WHO is using this approach in recommendations in guidance on more complex health systems and behavioural interventions, but acceptability and feasibility issues may be equally relevant for clinical interventions. In fact, we would suggest that there are few situations where these issues are not likely to be relevant, and even in situations where guidance developers feel confident that they already have a sufficient overview of these issues and additional evidence is not required, a framework approach where this is made transparent can still be useful. We have developed a list of questions (Table 6) to help groups involved in developing guidance decide whether additional evidence syntheses are needed. Further research is needed to refine these questions and to develop an understanding of the typical time and resources required to expand the range of evidence sources used to develop guidance. Table 2 also provides links to resources available to guidance developers who are considering expanding the range of syntheses that will be used to inform the guidance.

Table 6 When to consider expanding the range of evidence syntheses undertaken to inform the development of a guideline or guidance
Full size table

As more qualitative evidence syntheses are produced, methods for synthesising this type of evidence are becoming more sophisticated and the number of people able to produce these types of syntheses is growing. These developments will make the process of including qualitative evidence in evidence-to-decision processes easier in the future. However, a number of challenges remain, as discussed above. Elsewhere, we have described how evidence from qualitative research can be used in developing guidelines [39], including to help shape a guideline’s key questions by informing the populations, interventions, comparators and outcomes on which each key question should focus and to understand how different stakeholders value different outcomes. We did not use these additional strategies in OptimizeMNH, but they have been part of the development for a forthcoming WHO guideline on antenatal care [40].

Our experience from OptimizeMNH suggests that relevant programmatic experience regarding “upstream” systems-level factors is often unexplored or undocumented using traditional research approaches. Researchers should explore these issues to a greater extent [22]. Alongside this, we need to develop better methods for identifying and synthesising information from programme reports and other grey literature as well as methods for assessing our confidence in syntheses based on this type of data [41, 42].

The OptimizeMNH experience suggests that the DECIDE evidence-to-decision frameworks are a useful way of guiding panels through a wide range of evidence and towards the development of transparent recommendations. We have learnt a great deal about how to populate these frameworks with evidence of the effectiveness, feasibility and acceptability of health interventions. However, more research is needed, in particular on how these kinds of evidence are used by guidance panels in their decision-making and how different kinds of evidence influence the final recommendations made. In addition, we need further worked examples of approaches for including evidence regarding resource use and cost-effectiveness. This is one area that those involved in the DECIDE work are exploring [36].

Finally, we need to explore how this type of health system guidance can best be disseminated to and utilised by end users, including national and regional policy makers and programme planners. For most health systems questions, evidence regarding effectiveness, acceptability, feasibility, and resource use is all likely to be context-specific to some degree. End users therefore need help in adapting the recommendations to their own contexts. The OptimizeMNH guidance includes a workbook that specifically aims to help end users contextualise the recommendations [43]. However, more work is needed on tools to contextualise global and national research and guidance for implementation in specific settings [44–47] and on ways of appraising such guidance [48].