Background

Health inequalities are avoidable and unjust differences in health between individuals or populations [1]. Given that we know that interventions shown to be effective in improving the health of a population may actually widen the health inequalities gap while others reduce it, it is imperative that systematic reviewers consider how the findings of their reviews may impact (reduce or increase) the health inequality gap [2–4]. Furthermore, the existence of social inequalities, defined as “systematic differences in health between different socioeconomic groups within a society” (1, p. 473, [5]), increases the argument for all reviewers, not just those with a focus on health inequalities (HI), to consider the potential for their findings to reduce or increase HI. This is echoed by Cochrane and Campbell Collaborations who have called for the effects of interventions on HI to be considered by systematic review authors [6, 7].

Incorporating considerations of how review findings impact on health inequalities also aims to overcome one of the major barriers in using systematic reviews to inform decision-making, policy-making and practice [8]. Moving away from simply assessing what works to considering how review findings impact on disadvantaged populations, by assessing for example differential effects by subgroup populations can improve the applicability of the review findings to the local population[9, 10] thus increasing their fit for purpose in supporting decision-making and practice. Going beyond simply does it work to examine under what circumstances it works for whom and why [11] holds even more resonance when considering the impact on HI. As O’Neill et al. ([12], p. 57) point out, “the intervention has to be accessible, acceptable, effective in, and used by the most disadvantaged group within that population to be truly effective at reducing inequities in health” (whilst health inequalities are defined as avoidable differences in health outcomes across individuals or between populations, the narrower but related term health equity is often referred to in the literature as health inequalities which are also considered unfair and unjust [13].)

Methodological research has highlighted an absence of evidence with regards to the extent to which systematic reviews taking into account issues of HI when analysing and making recommendations for further research and practice [3, 14, 15]. Furthermore, recent methodological studies of systematic reviews demonstrated that very few (<5%) addressed differential impacts across socio-economic groups [16, 17]. The extent to which systematic reviewers in the past have failed to consider how their review findings impact on HI, is due in part to the focus reviewers placed on the effectiveness of interventions and also due to a lack of relevant data reported in the primary literature to assess such differential effects [18]. In addition, the lack of guidance or awareness of the existence of such guidance in this area may also have worsened the situation. More importantly, however, it is also due to a failure on the part of review authors to even consider differential impacts in reviews where HI is not the focus [8, 17, 19].

This paper aims to review existing guidance on incorporating considerations of HI in systematic reviews to examine the extent to which they can help reviewers incorporate such considerations in systematic reviews.

Aim and objectives

The aim of this study was to undertake a mapping review of existing guidance documents currently provided to assist reviewers when determining whether and how to incorporate considerations of HI. A mapping review aims to map out and categorise the literature according to key features (e.g. study design) on a particular topic and to identify gaps in the research literature [20]. The objectives were (1) to provide an overview on the types of guidance, in particular the focus, scope and purpose of the guidance, (2) to explore how the guidance is defined by authors, (3) to describe the methods used to develop the guidance, (4) to examine the comprehensiveness, overlap and operationalisation of the guidance.

Search strategy

A systematic approach to identifying the literature was undertaken in a two-tiered approach. Firstly, more generic guidelines to evidence synthesis were located and searched to identify specific guidance relating to the incorporation of HI. A search of review guidance manuals prepared by international organisations engaged in undertaking evidence synthesis was undertaken (see Table 1). Publications listed on the Campbell and Cochrane Equity Group website were also scanned.

Table 1 International evidence synthesis organisation websites
Full size table

In addition to this, a search was undertaken in Medline, CINAHL and The Cochrane Methodology Register. Key terms searched on included thesaurus and textwords terms comprising of synonyms for health inequalities, evidence synthesis and methodology/guidance/tools (see Appendix 1). A pre-published search strategy designed to capture HI studies was reviewed and utilised [21]. A practical approach to developing a search strategy to identify different types of evidence synthesis was adopted. This approach was informed by published systematic review filters and related evidence synthesis terms (e.g. realist review, realist synthesis, integrative). Searches were undertaken in September 2015, and search alerts were set up to capture relevant articles added into the databases after this date. No restrictions by year were applied, but publications were limited to English language only studies.

Requests for guidance were made via relevant email discussion lists and contacting experts in the Campbell and Cochrane Equity Group. For guidance that had been updated, the most recent update was included. Where multiple publications discuss the same guidance they were considered together.

Inclusion criteria

Any type of study was included if it provided an overview or discussed the development and testing of a conceptual or practical framework, tool, model or guidance (advice or formal recommendations) for dealing with the incorporation of considerations of HI in evidence synthesis. For the purpose of this review, HI is defined according to Whitehead’s ([1], p. 473) definition in which inequalities in the British context—and increasingly also across Europe—carries the same connotations of unfairness and injustice as the term inequities. Both generic guidance (e.g. guidelines published by evidence synthesis organisations or collaborations) that incorporated considerations of HI and HI-specific guidance (e.g. scholarly methodological studies presenting specific guidance for incorporating considerations of HI in systematic reviews) were included. Studies that primarily offer a theoretical discussion or comments on if and why HI should be included in evidence synthesis, or frameworks and guidance for the incorporation of HI for purposes other than evidence synthesis, were excluded. For practical reasons, studies were limited to English language publications. Screening of studies was undertaken by the author.

Data extraction

Data were extracted by the author on targeted audience (e.g. for reviewers, users of evidence synthesis), purpose (e.g. tools are to be used for planning, conducting, reporting, disseminating or using systematic reviews), scope (e.g. to inform systematic reviews, other research), how the tool was developed, operationalisation of the tool (how reviewers are instructed to apply the guidance) and whether and how they define health inequalities. Included studies were also categorised as to their focus, i.e. whether they were HI, or had a generic focus but which accounted for HI.

Data synthesis

Results are summarised in narrative and tabular forms. Strengths and weaknesses of the guidance in assisting systematic reviewers were assessed based on methods of development including evaluation, accessibility and transparency in operationalisation.

Results

The results of the search are summarised in Fig. 1. Eight hundred and thirty-six references were identified of which 20 were included in the review. All guidance documents [12, 22–40] incorporating considerations of HI in systematic reviews were published between 2009 and 2016. Table 2 outlines the characteristics of the included studies. Doull et al. [22] report on three guidance documents within the same study [23–25]. Tugwell et al. [29] and Ueffing et al. [30] both discuss the Cochrane Equity Checklist, whilst Welch et al. [32, 33] and Burford et al. [34] report on the PRISMA-Equity 2012 Extension. The majority of the guidance has been produced with the involvement of members of the Campbell and Cochrane Equity Methods Group.

Fig. 1
figure 1

Flow of search results

Full size image
Table 2 Characteristics of included studies
Full size table

Focus, scope and purpose of guidance

Table 2 outlines the focus, scope and purpose of the guidance. The majority of the guidance documents have an HI focus; three focus on sex and gender [23–25], one on socio-economic status (SES) [26] and one on both sex and SES analysis in systematic reviews [35] All four generic guidance documents in which HI was considered, signposted reviewers onto HI-focused guidance produced by the Campbell and Cochrane Equity Group or the PROGRESS framework [12]. All guidance were produced for informing the production of systematic reviews with two guidance documents also applicable to other types of research [12, 26]. Guidance has been produced to inform considerations of HI at different stages of the systematic review process with the most guidance produced to support the conduct of systematic reviews.

Guidance definitions

The documents defined their guidance in different ways, for example, as briefing notes [22–25] equity lens [12, 26, 27, 29, 30], recommendations [29, 30, 36], plausibility algorithm [35], tool [28] framework [31, 39], guidance [37, 38, 40] and guidelines [32–34].

All but three guidance documents [31, 39, 40] define what is meant by HI, equity or inequity. Where definitions are recorded, they differed across the studies. Whitehead’s [13] definition of health equity and HI were the most commonly reported within the guidance documents, although a number of different authors were cited for HI [8, 13, 41, 42], health inequity [13, 41, 43] and health equity [13, 44–48]. Four of the five guidance documents with a sex and gender HI focus all define what it meant by sex and gender in the same way [22–25]. Neither of the two guidance documents with a socio-economic focus define socio-economic status [26, 35].

Development of guidance

Table 3 outlines the guidance development process of the included studies. The majority of the guidance documents were transparent in outlining how the guidance had been produced. Of those reporting methods of guidance development, all were informed by a literature review.

Table 3 Development of guidance
Full size table

The guidance development process for most involved seeking feedback and revision from people with a wide range of expertise (including researchers, HI experts, review methodologists, decision-makers, clinical epidemiologists, practitioners and journal editors) and systematic review experience, the majority of whom were either members of the Cochrane Collaboration or were attending Cochrane Workshops. Burford et al. [34] and Doull et al. [22] specifically report involving novice reviewers in the development of their guidance. The PRISMA-E 2012 reporting guidelines [32, 33] were also informed by consensus methods. Whilst all of the guidance documents were produced from an HI perspective, only five had theoretical underpinnings or followed established methods in developing their guidance [22, 27, 32, 33, 35].

Operationalisation of guidance

As a means of demonstrating what reviewers should consider in the application of the items, most of the guidance provide examples from published systematic reviews. For example, when asking reviewers to consider whether there are known or possible differences by sex/gender, Welch et al. [23] use the following example, “In a systematic review on quality of life after total hip and total knee arthroplasty, men appeared to benefit more from the intervention in the few studies that addressed this issue.”

Despite the different purposes and audiences, there is a considerable overlap in what users of the guidance are asked to consider at different stages of the review process. Table 4 highlights this using the example of whether to expect differential effects across SES population characteristics.

Table 4 Overlap of guidance items on anticipating differential effects across SES in relation to population characteristics
Full size table

The rationale for why HI should be considered was also provided in some guidance. For example, Oxman et al. ([28], “Questions to consider”, no.3) ask “are there likely to be different baseline conditions across groups or settings such that the absolute effectiveness of the option would be different, and the problem more or less important, for disadvantaged groups or settings?” They then outline the rationale, “Typically, baseline risks are larger in disadvantaged populations and a larger absolute effect could therefore be expected.” ([28], “Questions to consider”, no.3).

When addressing how reviewers can operationalise the items, the comprehensiveness and application with which this is detailed differs across the guidance. For example, the number of items reviewers are asked to consider in the guidance documents ranges from 3 up to 26. Many guidance documents recommend using a theory-based approach, using programme theory or logic models to understand the assumptions behind how and why the intervention may work differently across disadvantaged populations and the influence of context on the outcome [22–26, 29, 30, 32, 33, 35, 36]. However, there is a lack of detail on how this could be implemented in practice. Tugwell et al. ([29], 2.“Defining disadvantage”, para. 2), for example, state that “implications on inequities are dependent on context, so authors of HI orientated reviews must strive to understand and explore the mediating effect of context”, yet they do not define what is meant by context and what data could be collected to explore this. Welch et al. [32] suggest that one limitation of the guidance is the use of terminology such as logic model, analytic framework, context and process evaluation, terms that are not widely accepted. A lack of consistency was noted in applying such terms across the guidance with some referring instead to causal pathway analysis, program theory and mechanisms of action.

Few guidance documents discuss who should be involved in making decisions on if and how HIs matter in systematic reviews. Tugwell et al. [29] suggest that relevant stakeholders should be involved in defining the review question, whilst HIAT [26] recommend involving members of the public (e.g. service users, carers, people living in disadvantaged neighbourhoods) in the planning stages. Only Welch et al. [35] started to explore how people came to their decisions of whether to expect differential effects across sex/SES and found that these decisions were made based on theory, personal experience, empirical data and guesses, but called for further research to investigate how these are used to make judgements.

Evaluation of guidance

Formal methods of evaluation were reported in three studies [22, 34, 35] for six of the guidance documents [23–25, 32, 33, 35]. Burford et al. [34] surveyed 151 systematic review authors on the perceived utility of the PRISMA-E 2012 [32, 33]. Reported advantages of using PRISMA-E 2012 include improved awareness of HI considerations in systematic reviews and improved consistency of reporting of HI in systematic reviews. Barriers reported include time required to apply guidance, length of guidance, increase length and complexity of reviews and lack of data in primary studies to operationalise some of the guidance.

Doull et al. [22] undertook a workshop evaluation involving 19 participants including potential users (researchers, practitioners and policy-makers) with little or some knowledge of the concepts of sex/gender to evaluate the content, readability and comprehensiveness of their briefing notes [23–25]. Although respondents reported that the briefing notes “provided clear methodological guidance to address sex/gender in reviews” ([22], Results, para. 2) and rated all aspects highly, responses were mixed on the level of complexity within the methods section.

Finally, in Welch et al. [35] four clinical methodologists evaluated the face and construct validity of a plausibility algorithm in predicting the likelihood of differential effects across sex and SES. Thirty-five review users, methodologists and clinicians also assessed the inter-rater reliability of the algorithm against ten pre-selected systematic reviews. The results found a low to no agreement beyond chance between raters for each of the three questions across sex and socioeconomic considerations. The authors suggest several reasons for the poor agreement relating to the design of the algorithm (use of multi-component questions covering several factors, omission of a don’t know category for responses), individual characteristics of respondents and poor choice of proxy or gold standard set of reviews to test the algorithm against. Whether any of the guidance has resulted in an uptake of considerations of HI in systematic reviews is still to be determined [22, 33].

Discussion

This review identified 20 guidance documents for incorporating considerations of equity in systematic reviews spanning the whole spectrum of the review process, from planning, conduct and reporting through to considerations of applicability to disadvantaged populations and knowledge translation when using reviews to inform decision-making and policy. Many of the documents have been published in the last 3 years, highlighting the fact that methodological research in this field is an important, emerging and evolving area of interest. The majority of the guidance is published in open access format in the journal literature. Whilst this increases the accessibility of the guidance, omission from standard review guidance handbooks places a greater emphasis on guidance authors to raise awareness of the existence of such tools to encourage greater uptake. For example, Welch et al. [32, 33] recognise the importance of widespread dissemination amongst journal editors, funding bodies and ethics committees in order to encourage the adoption of the PRISMA-Equity 2012 Extension reporting guideline.

Increasing awareness of such guidance is even more important when reviewers are faced with the different terminology used by authors to describe it. Use of multiple terms such as algorithm, equity lens, tool, may make them harder to locate within the journal literature. The Campbell and Cochrane Equity Methods Group have started to collate guidance on HI considerations for review authors [49], yet the list does not cover all guidance identified in this review and focuses to a greater extent on the guidance developed to support Cochrane Review authors.

Whitehead’s [13] definition of HI as “Inequality in health is a term commonly used in some countries to indicate systematic, avoidable and important differences” and health inequity as “refers to differences in health which are not only unnecessary and avoidable but, in addition, are considered unfair and unjust” were most commonly applied in the guidance. However, rather than demonstrate an acceptance of a common definition for HI or health equity, most of the guidance citing Whitehead [13] were produced by the same group (The Campbell and Cochrane Equity Methods Group). The different definitions of HI and health equity used by the guidance documents supports the view by Tugwell et al. [29] that these terms are used in different ways by different authors, and that there is no agreed definition of health equity or HI. This stresses the importance therefore for HI guidance to define health equity and/or HI to help reviewers to operationalise the guidance.

This is particularly true for guidance incorporating considerations of SES in systematic reviews. Neither of the two guidance documents with a focus on SES [26, 35] defined the term. Several of the guidance documents refer reviewers to the PROGRESS framework [12, 50] when asking reviewers to consider disadvantage. PROGRESS although does not explicitly define SES, relates it to income, while considering education and occupation separately. Yet SES has been defined more broadly as “a composite measure that typically incorporates economic status, measured by income, social status, measured by education; and work status measured by occupation” ([51], pp. 30), [52, 53]. The classification of individuals by SES has implications for reviewers in relation to types of SES indicators to collect and therefore, the definition and classification of HI terms such as SES needs to be operationalised within the guidance. This is further supported by the findings of Runnels et al. [54] who in a survey examining the challenges of including sex/gender analysis in systematic reviews found that one of the significant challenges was clarifying the concepts of sex and gender.

Much of the guidance is written from the perspective of HI having already being identified as the focus of the review and from a whole HI perspective rather than focusing on a specific HI dimension. More recently, however, guidance is being tailored more towards specific HI domains, in particular, sex/gender [23–25, 35] and SES [26, 35]. This may reflect the interests of the groups involved in producing the guidance. For example, HIAT [26] was developed by the National Institute of Health Research Collaboration for Leadership in Applied Health Research and Care North West Coast (NIHR CLAHRC NWC) whose remit is to ensure that all of the research it produces has socio-economic relation to HI as its core focus (www.hiat.org.uk).

Guidance on incorporating sex/gender analysis in systematic reviews was produced by the Cochrane Sex/Gender Methods Group (a subgroup of the Campbell and Cochrane Equity Methods Group). However, the development of more HI-specific guidance may also suggest that once reviewers have identified which HI domain(s) to consider, they may have difficulty operationalising the more generic HI-focused tools and require more tailored guidance. Or it may be that given that it would be impossible for reviewers to incorporate considerations across all HI dimensions, that there is a debate developing that certain HI domains may be considered more important than others to incorporate in systematic reviews. Indeed, in the application of the PROGRESS framework to systematic reviews, O’Neill et al. [12, p. 62] state that the framework is “not intended to encourage data dredging but to identify the most important factors that drive inequities in health”.

The Campbell and Cochrane Equity Group have been instrumental in driving forward methodological advances in evidence synthesis for the incorporation of HI in systematic reviews, and although not specifically stated, most of the guidance produced appears most relevant to reviewers undertaking systematic reviews measuring the effectiveness of interventions or at users assessing the applicability of reviews of effectiveness. This is no doubt partly due to the focus of the Cochrane Collaboration on reviews of effectiveness, yet how well this guidance translates to other types of HI systematic reviews (e.g. qualitative) therefore is unclear, and further research investigating the usefulness of the guidance for other types of reviews is required.

One potential challenge facing reviewers in operationalising the guidance is that methodological approaches to incorporating considerations of HI are still in development, and even the guidance authors themselves recognise that one of the limitations of the guidance may relate to the terminology used, such as logic model, process evaluation, mechanisms of action, terms which may not be accepted [32] or understood [55]. Encouraging reviewers to consider what works for disadvantaged populations, why, how and under what circumstances not only requires as Tugwell et al. ([29], Conclusions, para. 2) suggest “a paradigm shift in the generation and synthesis of evidence”, but also an acceptance of the terminology along with an understanding of how these methods can be applied in practice. Part of the problem is that current review methods cannot necessarily be transferred across to consider more complex issues and HI [3, 11] and methods to incorporate such complexity in systematic reviews are only just emerging [9, 11, 56–61]. Given the potential complexity of the process therefore, further research examining the challenges and barriers in incorporating considerations of HI in systematic reviews is required to better support reviewers in undertaking HI-focused reviews.

There is also perhaps, a greater need to understand how guidance items can be operationalised. Most of the guidance is operationalised by means of descriptive examples from published systematic reviews, and there is evidence to indicate that reviewers find them useful. For example, Doull et al., [22] in testing their guidance on incorporating issues of sex/gender analysis in systematic reviews found that reviewers wanted even more examples.

Much of the guidance is written with an underlying assumption that reviewers will recognise if and how equity matters. Empirical evidence however suggests otherwise [35, 54]. A recent survey by Runnels et al. [54] conducted on the challenges of including sex/gender analysis in systematic reviews seems to support this view. They found that concerns were raised over the construction of knowledge and cites one respondent saying “the biggest challenges are much more fundamental and have to do with the way that we arrive at decisions as to what is important for us to study, why it is important for us to study, and how we determine the way to study” ([53], Conceptual challenges, para. 3).

Welch et al., [35] in developing their equity plausibility algorithm, start to explore the rationale behind reviewers’ decisions and found that empirical data, theory or personal experience were often used to explain their reasons but call for further research to enhance understanding of how this was used and the contribution of individual characteristics to the process. In order to help reviewers to operationalise the guidance therefore, it may be useful to explore the rationale behind how reviewers are making decisions when applying the guidance and the contribution made to those decisions by different individuals (e.g. stakeholders).

Furthermore, if, as Welch et al. ([35], Discussion, para. 1) suggest, reviewers “need to have a deep understanding of the content area” to make judgements about likely differential effects, then single examples drawn from topic-specific reviews may not be the best way to demonstrate guidance application. Without a comprehensive understanding of the different ways in which HI issues may contribute towards differential effects in health outcomes, it may be difficult for some reviewers, particularly those new to HI, to recognise the need to incorporate or operationalise such issues in systematic reviews. Building on this, research is currently underway by the author (MM) to identify a comprehensive set of considerations to help reviewers operationalise the influence of socio-economic contextual factors on how and why an intervention may work differently across disadvantaged populations.

Strengths of the guidance evaluation methods include the involvement of a wide range of expertise (e.g. reviewers, methodologists, decision-makers, HI experts), consensus methods and piloting of the guidance. However, the use of self-selecting samples may not necessarily be representative of the wider population expected to utilise the guidance. Assessment of face validity alone, i.e. a subjective assessment of the relevance of the questions [62] rather than evaluating how well the guidance works when applied prospectively may not identify problems operationalising items. Burford et al. [34] used a prospective design to assess the utility of the reporting guidelines but, largely due to the type of guidance, did not ask reviewers to discuss how they had operationalised the checklist items. In addition, evaluation of the guidance appears to be undertaken by individuals rather than reflecting the collaborative approach that a systematic review encourages. This may be a further reason why the Welch et al. [35] plausibility algorithm had poor inter-rater reliability. Evaluating guidance using a case study approach, may better capture how well the guidance is interpreted or operationalised by those it is designed to assist. Involving novice reviewers in the guidance evaluation may also identify challenges in interpreting and operationalising the guidance that may not necessarily be considered when piloting is undertaken by the guidance developers or expert reviewers alone.

Strengths and limitations of the review

This study is the first to summarise the range of guidance available on the incorporation of HI in systematic reviews. One of the strengths of this study is detailing what guidance is available for considering HI at various stages of the systematic review process. There is no validated search filter for HI, however, terms were based on those used in a Cochrane methodological review exploring how effects on HI are assessed in systematic reviews [21]. The review did not seek to critique the individual items/questions in the guidance or recommend one guidance over another, but rather offer an overview of guidance available to reviewers when incorporating considerations of HI at different stages of the review process.

A potential limitation of this review is that as this study is part of a PhD study, only one person was involved in the selection of studies, data extraction and synthesis. One limitation of this review is the focus on English language literature when it is acknowledged that other languages, such as Spanish, may offer extensive coverage of literature regarding inequalities. As this is not a systematic review, the search was restricted to a small number of key databases in health, further databases outside of health could have been searched. Instead, a targeted approach to the search was adopted using a number of different search approaches, including scanning of relevant systematic review organisational websites, reference checking and contacting known experts in the field. Given the diverse nature of the guidance documents included in this review, no formal quality appraisal was undertaken instead each guidance document was assessed as to whether or not it formally evaluated.

Conclusions

Given the recent growing interest in the incorporation of HI in systematic reviews, it is not surprising that methodological guidance exploring how considerations of HI can be incorporated into evidence synthesis is a relatively new and emerging area of research [7, 36]. Above all, the strength of all the guidance documents reviewed in this study is in highlighting the importance of incorporating considerations of HI in systematic reviews, yet due to the fairly recent introduction of the guidance there is little evidence on the guidance uptake or whether it has led to an improvement in considerations of HI in systematic reviews. It is clear, however, that operationalising the guidance will require more work for the reviewer but aside from the Runnels et al. [54] survey, there is limited evidence on the challenges facing reviewers when incorporating considerations of HI. Furthermore, understanding how reviewers can operationalise the guidance and the challenges in doing so have implications not only for understanding the usefulness and burden of the guidance [34], but also has implications for the uptake of guidance and its ultimate goal of improving HI considerations in systematic reviews. There is currently a gap in the evidence examining how reviewers can operationalise the guidance and the barriers and facilitators involved. The results of this review will be used to inform the development of a framework to help reviewers rationalise whether or not to incorporate considerations of HI in systematic reviews.