Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In the next three chapters, we develop core principles for evaluating efficacy and external validity. In this chapter we put forward proposals for gathering evidence of mechanisms. Then, in Chap. 6 we discuss how to evaluate this evidence. In Chap. 7, we explain how this evaluation can be combined with an evaluation of correlation in order to produce an overall evaluation of a causal claim.

In the case of efficacy, where clinical studies find a correlation between the putative cause and effect, the task is to determine whether this correlation is causal by looking for further evidence of mechanisms. In order to evaluate efficacy, it is necessary to determine the status of the general mechanistic claim, i.e., to ask whether the correlated putative cause and effect are also linked by a mechanism that can account for the extent of the observed correlation.

In the case of external validity, the existing evidence may establish causality in a study population that differs from the target population of interest. Here the relevant general mechanistic claim that needs to be evaluated is that mechanisms in the study and target population are sufficiently similar.

General mechanistic claim for efficacy. In formulating the general mechanistic claim for efficacy, the following questions should be addressed:

  • What is the relevant population?

  • What is the intervention or exposure level?

  • What is the outcome and how is it measured?

General mechanistic claim for external validity. In determining the general mechanistic claim concerning external validity, the following questions should be addressed:

  • What is the target population? What is the study population?

  • What is the intervention or exposure level in the target?

  • What is the outcome and how is it measured in the target?

  • What is the intervention or exposure level in the study?

  • What is the outcome and how is it measured in the study?

It may be that existing evidence from clinical studies together with already well-established mechanisms is enough to establish the general mechanistic claim. In other cases, the existing evidence fails to establish causality, and it is necessary to identify and evaluate mechanistic studies. To this end, this chapter presents the following five-step strategy for gathering evidence of mechanisms:

  1. 1.

    Identify: Identify a number of specific mechanism hypotheses.

  2. 2.

    Formulate: For each specific mechanism hypothesis, formulate a number of review questions.

  3. 3.

    Search: Use these review questions to search the literature.

  4. 4.

    Refine: Identify the evidence most relevant to the mechanism hypothesis by refining the results of this search.

  5. 5.

    Present. Present the evidence relevant to the mechanism hypothesis.

This strategy is intended to help overcome some of the practical difficulties with identifying evidence of mechanisms—difficulties which may prevent appraisers from considering all the relevant evidence. Once this evidence of mechanisms has been identified, it can then be evaluated alongside the existing evidence of correlation from clinical studies, as explained in Chaps. 6 and 7.

Fig. 5.1
figure 1

The overall approach to gathering evidence of mechanisms

Full size image

The overall approach of this chapter is illustrated in Fig. 5.1. The five steps outlined above are explained in detail in the following sections.

1 Identify Specific Mechanism Hypotheses

Efficacy. In order to evaluate the general mechanistic claim that there is a mechanism that can account for the observed correlation between a putative cause and effect in a study population, it is useful to identify key features of possible mechanisms of action. Each proposed mechanism of action, or partial description of proposed mechanism of action, is a specific mechanism hypothesis. But note that a specific mechanism hypothesis need not be a complete description of a mechanism.

Example: Specific mechanism hypotheses for determining efficacy.

Aspirin prevents heart disease via cyclooxygenase (COX) inhibition, and the mechanisms that underlie this prevention are established. However, aspirin also seems to reduce the incidence of some cancers. Here, the mechanisms are much less well understood. As Chan et al. (2011) write: “the mechanism of aspirin’s antineoplastic effect is less clear, with substantial evidence supporting both COX-dependent and COX-independent mechanisms. Moreover, data supporting the importance of COX-dependent mechanisms are not entirely consistent concerning the relative importance of the COX-1 and COX-2 isoforms in carcinogenesis”. In this quotation, the general mechanistic claim is that aspirin exhibits an antineoplastic effect. There are also a couple of more specific mechanism hypotheses, for example, that this antineoplastic effect is mediated by COX-dependent mechanisms. Evidence relating to these more specific mechanism hypotheses provides a way to determine the status of the general mechanistic claim.

External validity. In order to evaluate the general mechanistic claim that there is a mechanism in the target population sufficiently similar to the mechanism responsible for the correlation observed in the study population, specific mechanism hypotheses need to pertain to the mechanism of action. It is important to consider the possibility that the mechanism in the target population may contain further component mechanisms that counteract the mechanism of action in the study population and affect the extent of the correlation between the putative cause and effect. So one needs to ask, are there any masking mechanisms in the target population?

Example: Specific mechanism hypotheses for determining external validity.

According to NICE guidelines, treatment for hypertension should differ depending on ethnicity (NICE 2011). Although ACE-inhibitors have proved beneficial for hypertension in many study populations, there remains the question of whether they are the optimal treatment in some distinct target population, such as African or Caribbean populations. In this case, it is necessary to determine the status of the following general mechanistic claim: the relevant hypertensive mechanisms in the study populations are sufficiently similar to the mechanisms in African or Caribbean populations. This general mechanistic claim can be evaluated by evaluating a more specific mechanism hypothesis, namely that African and Caribbean populations have a lower renin state. As we shall see in Chap. 6, there is some good mechanistic evidence in favour of this specific mechanism hypothesis, and this undermines the general mechanistic claim. This is why, instead, calcium channel blockers are the recommended antihypertensive treatment in African and Caribbean populations (Clarke et al. 2014).

There are two main ways to identify a specific mechanism hypothesis.

First, a specific mechanism hypothesis may be proposed on the basis of published studies from the clinical study literature. If a clinical study establishes a correlation between a putative cause and effect, and the suggestion is that this correlation is causal, then the authors of such a study usually identify at least one possible mechanism hypothesis of the following form: It is plausible that mechanism with features F links the putative cause and effect in the study population. The study may also point out possible masking mechanisms (Illari 2011). Given this, the discussion section of a published paper that reports the results of a clinical study is a good place to look in order to locate a specific mechanism hypothesis.

Example: The discussion section of a recent paper on the effect of long-term aspirin use on the risk of cancer says: ‘[O]ur findings suggest that for the gastrointestinal tract, aspirin may influence additional mechanisms critical to early tumorigenesis that may explain the stronger association of aspirin with a lower incidence of gastrointestinal tract cancer. Such mechanisms include modulation of cyclo-oxygenase-2, the principal enzyme that produces proinflammatory prostaglandins, including prostaglandin E2, which increases cellular proliferation, promotes angiogenesis, and increases resistance to apoptosis. Aspirin may also play a role in Wnt signaling, nuclear factor B signaling, polyamine metabolism, and DNA repair’ (Cao et al. 2016). References are given for these specific mechanism hypotheses.

Second, a specific mechanism hypothesis may also be proposed on the basis of existing mechanistic studies or clinical expertise.

Example: Large goitres may make it difficult to breathe. It has recently been established that radiotherapy leads to a reduction in the size of large nodular goitres (Nielsen et al. 2006; Bonnema et al. 2007). Will reducing the size of goitres lead to improved respiratory function? Basic clinical experience suggests that there is a mechanism by which a reduction in the size of obstructions in the airway leads to an improvement in respiratory function. This was not established on the basis of clinical studies, but rather on very basic clinical experience. A proponent of this view may propose that this clinical experience supports the existence of a mechanism by which radiotherapy makes a positive difference to respiratory function in patients with large nodular goitres, since large nodular goitres are simply a type of obstruction in the airway that results from an enlargement of the thyroid. However, it may also be proposed that there is a possible masking mechanism. Radiotherapy to the throat might otherwise reduce respiratory function (by, say, causing scarring). A proponent of this view might propose this masking mechanism which may affect the extent of the correlation between radiotherapy and improved respiratory function (Bonnema et al. 2007).

It is important to bear in mind the following practical point. Many policy-makers require an expert evaluation of evidence in their process. For instance, expert evaluations routinely take place at the International Agency for Research on Cancer (IARC), the UK Medicines and Healthcare Products Regulatory Agency (MHRA), the UK National Institute for Health and Care Excellence (NICE), and the EU Committee for Medicinal Products for Human Use (CHMP). In such cases, it may be useful to provide a list of specific mechanism hypotheses to committee members before gathering evidence, in order to give them the opportunity to suggest alterations to the list well in advance of the committee actually meeting (Aronson et al. 2018). Identifying a set of specific mechanism hypotheses at the outset is a good way of proceeding in the face of a large number of mechanistic studies: it makes the process of gathering evidence more manageable by helping to restrict focus to only those published mechanistic studies potentially relevant to the mechanism hypotheses of interest.

2 Formulate the Review Questions

An effective method for carrying out a review of the literature begins with a well-formulated review question. The suggestion here is to use the specific mechanism hypotheses to help formulate a number of review questions.

Two points are important to keep in mind:

  1. 1.

    Some features of the proposed mechanism may already be established, so it would be unnecessary to look for further evidence in favour of them. Such features should not figure in the more specific review questions. Only the contentious key features of the proposed mechanism should figure in the review question.

  2. 2.

    The review questions may need to be updated in the course of the literature search. In particular, the search may suggest some more specific review questions about the entities, activities, and their organization in the proposed mechanism. Any changes to the review questions should be documented.

Example: A number of clinical studies establish that there is a correlation between exposure to benzo[a]pyrene and lung cancer, because exposure to benzo[a]pyrene is correlated with tobacco smoking, which is itself correlated with lung cancer (IARC 2009). But these studies alone were not sufficient to establish causation (IARC 2015). A number of specific mechanism hypotheses might explain the correlation between benzo[a]pyrene and cancer: e.g., (i) The diolepoxide mechanism; (ii) The radical-cation mechanism. These hypotheses lead to the following review questions concerning contentious key features of the respective mechanisms: (i) Do intermediate metabolites of benzo[a]pyrene react with DNA to form DNA adducts associated with tumorigenesis? (ii) Is benzo[a]pyrene oxidized in such a way that leads to free radical formation which may in turn form DNA adducts? These review questions can then be used to search the literature.

The review questions may be formulated according to the PICO framework. PICO stands for Population, Intervention, Comparator, and Outcome (for more information see O’Connor et al. (2011)).

Suppose we are interested in the following research question: Is there a mechanism in women over fifty linking regularly taking aspirin (rather than not regularly taking aspirin) to developing asthma? The PICO framework helps in a number of ways to answer this question, by emphasizing what are the most important parts of the research question. Specifically, it picks out the relevant population (women over fifty), the intervention in that population of interest (regularly taking aspirin), and the outcome (developing asthma). It will also identify the comparator (asthma prevalence in members of the same population not regularly taking aspirin). This has the effect of making clear the most important aspects of the intended research objective. In turn, this focuses the search on the most relevant literature, as well as assisting in the presentation of the literature that is obtained by the search.

The PICO framework may be adapted to the research objective at hand. In particular, the PECO framework has been developed for non-interventional studies: Population, Exposure, Comparator, and Outcome (Vandenberg et al. 2016). One can ask, for instance: is there a mechanism in human males (population) linking exposure to high levels of benzo[a]pyrene (exposure) rather than low levels of benzo[a]pyrene (comparator) to scrotal cancer (outcome)?

3 Search the Literature

A review question can then be used to search the literature for evidence for the contentious key features of a specific mechanism hypothesis. This should take place with the assistance of domain experts.

At this stage, decisions need to be made about which databases and other sources should be searched. These decisions should be documented in order to aid transparency and reproducibility. (See Appendix A for some examples of databases, Part II for tools to support the process of evidence appraisal, and Sect. 5.6 for a worked example of a literature search.)

One can identify research potentially relevant to the assessment of the specific mechanism hypothesis by looking at the relevant mechanistic study literature:

  1. 1.

    In the first instance, this may be done by following up the references from the discussion section of any clinical study report which proposes a mechanism as the best explanation of an observed correlation. Any other publicly available reports may be useful here also, e.g., government agency reports, doctoral theses, etc.

  2. 2.

    More systematically, a preferred method for searching the literature may be used, e.g., a PubMed search using appropriate Medical Subject Heading (MeSH) terms, including key terms from the hypothesized mechanisms.

Efforts to standardise terminology and indexing practices for publications reporting mechanistic studies are welcome, especially in order to facilitate text mining techniques, which are becoming increasingly widespread. It is also important that even the negative findings of mechanistic studies are published, to reduce publication bias.

4 Refine Results of the Search

Identifying evidence from the literature requires expert judgement, which is susceptible to bias. In order to guard against the effects of such biases, the details of the search procedure should be clearly presented (O’Connor et al. 2011). This protects against the effects of bias by providing a transparent and reproducible literature search strategy (Vandenberg et al. 2016).

A study flow diagram can be used to present the process of selecting studies for inclusion in the review (O’Connor et al. 2011). This can be made with reference to the guidance in the PRISMA framework (Moher et al. 2009). According to this guidance, a study flow diagram consists of four phases: Identification, Screening, Eligibility, and Inclusion. After identifying studies by searching databases with a review question, the studies are then screened for duplicates, and excluded studies are recorded. The eligibility of the studies is then determined, and any ineligible studies are recorded as excluded along with the reasons for their exclusion. This leaves the included studies.

A key question here is: Is any of this evidence not relevant?

  1. 1.

    Use preferred inclusion and exclusion criteria and expert knowledge to rule out irrelevant mechanistic studies (Kushman et al. 2013).

    • Does the publication include original data? A good rule of thumb: if it does not include original data, then exclude the publication.

  2. 2.

    It may be possible to exclude some studies by a review of the title and abstract. A full-text review may be necessary to exclude other studies.

    • All excluded studies should be documented, along with the reasons for exclusion.

  3. 3.

    There are content management tools available to help in identifying, screening, organizing, and summarizing the evidence.

An example study flow diagram for evidence of mechanisms is presented in Fig. 5.2 (Vandenberg et al. 2016).

Fig. 5.2
figure 2

An example study flow diagram reproduced from Vandenberg et al. (2016)

Full size image

5 Presenting the Evidence of Mechanisms

A clear summary of the identified evidence of mechanisms is an important precursor to evaluating that evidence. (Presenting the quality of evidence of mechanisms is a separate issue, for which guidance is provided in Sect. 6.4.) A summary of evidence of mechanisms should clearly state the general mechanistic claim that the mechanism in question is proposed to account for, that is, whether it is presented as evidence of the existence of a mechanism of action for efficacy, or as evidence of similarity of mechanisms between populations to account for external validity. This includes a clear statement of the cause A under investigation as well as the particular outcome B of interest. The presentation of evidence should also make clear the specific mechanism hypotheses under consideration, and present the evidence in favour of the contentious key features of the specific mechanism hypotheses.

Example: IARC’s overall process of gathering and presenting evidence of mechanisms.

In order to help identify and organise further evidence of mechanisms in the literature, the International Agency for Research on Cancer makes use of existing evidence of mechanisms in the form of ten key characteristics, one or more of which are frequently exhibited by known carcinogens (Smith et al. 2016). In our terminology, the ten key characteristics are key features of specific mechanism hypotheses, which are possible instantiations of the general mechanistic claim that there is a mechanism linking the considered exposure to cancer in the relevant sites in humans. The ten key characteristics are the ability of the putative carcinogen to:

  1. 1.

    Act as an electrophile either directly or after metabolic activation;

  2. 2.

    Be genotoxic;

  3. 3.

    Alter DNA repair or cause genomic instability;

  4. 4.

    Induce epigenetic alterations;

  5. 5.

    Induce oxidative stress;

  6. 6.

    Induce chronic inflammation;

  7. 7.

    Be immunosuppressive;

  8. 8.

    Modulate receptor-mediated effects;

  9. 9.

    Cause immortalization;

  10. 10.

    Alter cell proliferation, cell death, or nutrient supply.

For instance, a correlation between benzene and cancer in humans has been observed in many studies. In order to determine whether this correlation is causal, it is necessary to determine the status of the relevant general mechanistic claim, namely, that there exists a mechanism linking exposure to benzene to cancer in humans that can account for the extent of the observed correlation (IARC 2015). A first step is to propose specific mechanism hypotheses, with the help of the ten key characteristics. For example, the specific mechanism hypothesis might be that benzene induces certain chromosomal aberrations that are characteristic of carcinogens. This leads to review questions that help to identify evidence relevant to this specific mechanism hypothesis. In this case, there is mechanistic evidence that exposure to benzene causes chromosomal aberrations in vivo in bone marrow cells of mice and rats. There is also mechanistic evidence that benzene exposure also causes chromosomal aberrations and mutation in human cells in vitro. This mechanistic evidence should be listed alongside the specific mechanism hypothesis and will adjudicate on the contentious features of the proposed mechanism. The identified evidence may be sufficient to determine the status of the general mechanistic claim, but this would involve first evaluating the evidence of mechanisms, which is the topic of Chap. 6.

6 Worked Example on Probiotics and Dental Caries

This worked example shows how our general method for gathering evidence of mechanisms can be applied to a specific case dealing with the effectiveness of probiotics for dental caries.

Identify specific mechanism hypotheses for probiotics in preventing dental caries. Cagetti et al. (2013) conducted a review of the caries-prevention effect of probiotics in human. Three studies were found assessing caries lesion development as outcome, with a further 20 studies reporting only caries risk factors as interim outcomes. The authors concluded “...[t]he effect of probiotics on the development of caries lesion seems encouraging, but to date, RCTs on this topic are insufficient to provide scientific clinical evidence.”

More recently, a systematic review on probiotics and oral health (Seminario-Amez et al. 2017) reached similar conclusions on the effectiveness for the prevention of dental caries; laboratory data and the effect on interim outcomes is promising, but long-term clinical trials are needed.

In the review by Cagetti et al. (2013), the mechanisms of action of probiotics were described. These were:

  • adhesion

  • co-aggregation

  • competitive inhibition

  • production of organic acids

  • bacteriocin-like compounds

  • immune-modulation (Teughels et al. 2008)

Cagetti et al. (2013) did note that not all of these mechanisms were fully understood. Seminario-Amez et al. (2017) also noted that the mechanism of probiotics in the oral cavity is not clearly established. However, studies are cited to support the role of probiotics in reducing counts of cariogenic pathogens, inhibiting periodontal pathogens, modulating the inflammatory response and producing beneficial substances. The ability of probiotics to compete with pathogens for adhesion surfaces and nutrients, causing displacement of the latter ones, was also confirmed in laboratory studies.

Formulate the review questions and search the literature. In order to further explore how probiotics might work for the prevention of dental caries, we searched for review articles describing the mechanism of action. Four relevant articles were found (Bonifait et al. 2009; Caglar et al. 2005; Saha et al. 2012; Singh et al. 2013).

Refine results of the search. Bonifait et al. (2009) postulated that “[t]o have a beneficial effect in limiting or preventing dental caries, a probiotic must be able to adhere to dental surfaces and integrate into the bacterial communities making up the dental biofilm. It must also compete with and antagonize the cariogenic bacteria and thus prevent their proliferation. Finally, metabolism of food-grade sugars by the probiotic should result in low acid production.” Bonifait et al. (2009) cite a number of studies showing the different abilities of the probiotics, such as the ability to integrate with the biofilm, and conclude that probiotics can neutralize acidic conditions in the mouth and interfere with cariogenic bacteria. The same evidence is cited in Singh et al. (2013).

Present the evidence of mechanisms. The number of studies investigating the effectiveness of probiotics for the prevention of dental caries is limited. There is a body of evidence from laboratory studies and clinical trials that interim outcomes linked with reduced dental caries can be improved through the use of probiotics. Several specific mechanism hypotheses were found in this research, mainly dealing with local (rather than systemic) effects of probiotics. However, not all mechanisms are yet fully understood.

In this example, understanding how probiotics might work through the various mechanisms of action helps to interpret the limited evidence of effectiveness. Probiotics are likely to have a preventive effect on dental caries, effected through a range of known mechanisms. Probiotics are also very unlikely to have significant adverse effects (Borriello et al. 2003).

We did not undertake a systematic review of the evidence on how probiotics might work. However, there appears to be a consistent view of the underlying mechanisms between the publications reviewed here. In this case, where unintended consequences are likely to be minimal due to the already wide and safe use of probiotics, a systematic review may not be needed to generate evidence of mechanisms.