Generalizing experimental results by leveraging knowledge of mechanisms

Abstract

We show how experimental results can be generalized across diverse populations by leveraging knowledge of local mechanisms that produce the outcome of interest, only some of which may differ in the target domain. We use structural causal models and a refined version of selection diagrams to represent such knowledge, and to decide whether it entails the invariance of probabilities of causation across populations, which then enables generalization. We further provide: (i) bounds for the target effect when some of these conditions are violated; (ii) new identification results for probabilities of causation and the transported causal effect when trials from multiple source domains are available; as well as (iii) a Bayesian approach for estimating the transported causal effect from finite samples. We illustrate these methods both with simulated data and with a real example that transports the effects of Vitamin A supplementation on childhood mortality across different regions.

Appendix

An example with continuous treatment

Here we provide a simple example in which, although the treatment variable is continuous, the relevant dependencies among potential outcomes are still amenable to graphical representation. Suppose we have the same selection diagram as in Fig. 2b, but now let X, B, and H all be continuous variables. Next, consider the following functional specification for the structural equation of Y,

$$\begin{aligned} Y = I(H> 0) \vee I(X \times B > 0) \end{aligned}$$

(4)

where $I(\cdot )$ denotes the indicator function. Now note from Equation 4 we can derive the potential outcomes $Y_0 = I(H>0)$ for $x =0$, and, $Y_x = I(H>0) \vee I(xB>0) = Y_0 \vee I(xB>0)$, for $x \ne 0$. We can thus draw the same modified selection diagram as in Fig. 3, but now replacing $Y_1$ with $Y_x$, leading to the conclusion that $Y_{x}\perp \!\!\!\!\perp S \mid Y_{0}$, for all $x \ne 0$.

Proofs

Bounds with a single source population

Here we show how to obtain the bounds of Theorem 1. To simplify notation, let $P_{ij}:=P(Y_{i} = j)$, $P^*_{ij}:= P^*(Y_{i} = j)$, ${\text {PS}}_{10} := P^*(Y_1 = 0 |Y_0 = 1) = P(Y_1 = 0|Y_0 =1)$ and ${\text {PS}}_{01} = P^*(Y_1 = 1|Y_0 = 0)= P(Y_1 = 1|Y_0 = 0)$. The target function to be optimized is $P^*_{11}$, which can be written as,

$$\begin{aligned} P^*_{11} = (1-{\text {PS}}_{10})P^*_{01} + {\text {PS}}_{01}(1-P^*_{01}) \end{aligned}$$

(5)

Our goal is to pick ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ such that it maximizes (or minimizes) Eq. 5 subject to the following constraints: (i) ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ need to be between zero and one (since ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ need to be valid probabilities); and, (ii) ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ must conform to the observed results of the trial in the source domain, that is, $P_{11} = (1-{\text {PS}}_{10})P_{01} + {\text {PS}}_{01}(1-P_{01})$. Thus, our optimization problem is,

$$\begin{aligned} \max _{{\text {PS}}_{10}, {\text {PS}}_{01}}&\, P^*_{11} = (1-{\text {PS}}_{10})P^*_{01} + {\text {PS}}_{01}(1-P^*_{01})\\ \quad \text {s.t.}&\quad P_{11} = (1-{\text {PS}}_{10})P_{01} + {\text {PS}}_{01}(1-P_{01}) \\ \text {and}&\quad 0\le {\text {PS}}_{10}\le 1, \, 0 \le {\text {PS}}_{01} \le 1 \end{aligned}$$

To simplify the problem, we can use the equality constraint $P_{11} = (1-{\text {PS}}_{10})P_{01} + {\text {PS}}_{01}(1-P_{01})$ to eliminate one of the variables. For instance, writing ${\text {PS}}_{10}$ in terms of $PS_{01}$ gives us,

$$\begin{aligned} 1-{\text {PS}}_{10} = \frac{P_{11} - {\text {PS}}_{01}(1-P_{01})}{P_{01}} \end{aligned}$$

(6)

Which results in a new target function,

$$\begin{aligned} P^*_{11}&= (1-{\text {PS}}_{10})P^*_{01} + {\text {PS}}_{01}(1-P^*_{01}) \end{aligned}$$

(7)

$$\begin{aligned}&= \left( \frac{P_{11} - {\text {PS}}_{01}(1-P_{01})}{P_{01}} \right) P^*_{01} + {\text {PS}}_{01}(1-P^*_{01}) \end{aligned}$$

(8)

$$\begin{aligned}&= \left( \frac{P_{11}}{P_{01}}\right) P^*_{01} + \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}_{01} \end{aligned}$$

(9)

$$\begin{aligned}&= RR \times P^*_{01} + \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}_{01} \end{aligned}$$

(10)

where $RR = \frac{P_{11}}{P_{01}}$ is the causal risk-ratio in the trial of the source domain $\Pi$. Since $0 \le (1-{\text {PS}}_{10}) \le 1$, the substitution also results in additional constraints on ${\text {PS}}_{01}$,

$$\begin{aligned} \frac{P_{11} - P_{01}}{1 - P_{01}} \le {\text {PS}}_{01} \le \frac{P_{11}}{1 - P_{01}} \end{aligned}$$

(11)

Thus, define the lower and upper bounds on ${\text {PS}}_{01}$ as

$$\begin{aligned} {\text {PS}}^L_{01} = \max \left\{ 0, \frac{P_{11} - P_{01}}{1-P_{01}} \right\} , \qquad {\text {PS}}^U_{01} = \min \left\{ \frac{P_{11}}{1-P_{01}}, 1 \right\} \end{aligned}$$

Our new maximization problem can be written as,

$$\begin{aligned} \max _{{\text {PS}}_{01}} \,\, RR \times P^*_{01} + \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}_{01} \qquad \text {s.t.} \quad {\text {PS}}^L_{01} \le {\text {PS}}_{01} \le {\text {PS}}^U_{01} \end{aligned}$$

(12)

Since the target function is linear, the maximum occurs at the extreme points of ${\text {PS}}_{01}$. The same reasoning holds for the minimization problem. Thus, we have that,

$$\begin{aligned} P_{11}^{*L} \le P_{11}^{*} \le P_{11}^{*U} \end{aligned}$$

where

$$\begin{aligned} P_{11}^{*L} = RR \times P^*_{01} + \min \left\{ \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}^L_{01},\, \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}^U_{01} \right\} \end{aligned}$$

and

$$\begin{aligned} P_{11}^{*U} = RR \times P^*_{01} + \max \left\{ \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}^L_{01},\, \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}^U_{01} \right\} \end{aligned}$$

Informativeness of the bounds

We now derive the width of the bounds for $P^*_{11}$ for the case when the bounds for ${\text {PS}}_{01}$ do not reach 0 nor 1 (this will happen when both $P_{11} > P_{01}$ and $P_{11} < 1 - P_{01}$). Define the width W of the bounds as the difference between the upper and lower bound of $P^*_{11}$, that is,

$$\begin{aligned} W = P_{11}^{*U} - P_{11}^{*L} \end{aligned}$$

Expanding the terms we obtain,

$$\begin{aligned} W&= P_{11}^{*U} - P_{11}^{*L} \end{aligned}$$

(13)

$$\begin{aligned}&= \left| \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}^U_{01} - \left( \frac{P_{01} - P^*_{01}}{P_{01}}\right) {\text {PS}}^L_{01}\right| \end{aligned}$$

(14)

$$\begin{aligned}&= \frac{|P_{01} - P^*_{01}|}{P_{01}} \times \left( PS^U_{01} - {\text {PS}}^L_{01}\right) \end{aligned}$$

(15)

$$\begin{aligned}&= \frac{|P_{01} - P^*_{01}|}{P_{01}} \times \frac{P_{01}}{1 - P_{01}} \end{aligned}$$

(16)

$$\begin{aligned}&= \frac{|P_{01} - P^*_{01}|}{1 - P_{01}} \end{aligned}$$

(17)

Thus, when the bounds for ${\text {PS}}_{01}$ are “interior,” the informativeness of the bounds depend only on $P_{01}$ and $P^*_{01}$. Moreover, even if the bounds for ${\text {PS}}_{01}$ are “wide,” the bounds for $P^*_{11}$ may be “narrow,” provided the baseline risks of the source and target population are close enough.

Identification with multiple source domains

We now show how to obtain the identification results of Theorems 2 and 3. Consider two source populations $\Pi ^a$ and $\Pi ^b$. Again, to simplify notation, let $P^a_{ij}:=P^a(Y_i = j)$, $P^b_{ij}:=P^a(Y_i = j)$, ${\text {PS}}_{10}:= P^a(Y_1 = 0 |Y_0 = 1) = P^b(Y_1 = 0 |Y_0 = 1) = P^*(Y_1 = 0|Y_0 =1)$ and ${\text {PS}}_{01}:= P^a(Y_1 = 1|Y_0 = 0)= P^b(Y_1 = 1|Y_0 = 0) = P^*(Y_1 = 1|Y_0 = 0)$.

First note that ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ are identified from the experimental data in $\Pi ^a$ and $\Pi ^b$. Using the law of total probability for $P^a_{11}$ and $P^b_{11}$ write,

$$\begin{aligned} P^a_{11}&= (1-{\text {PS}}_{10}) \times P^a_{01} + {\text {PS}}_{01} \times P^a_{00} \end{aligned}$$

(18)

$$\begin{aligned} P^b_{11}&= (1-{\text {PS}}_{10}) \times P^b_{01} + {\text {PS}}_{01} \times P^b_{00} \end{aligned}$$

(19)

We thus have a system of two equations and two unknowns,

$$\begin{aligned} \begin{bmatrix} P^a_{01} &{} P^a_{00}\\ P^b_{01} &{} P^b_{00} \end{bmatrix} \begin{bmatrix} (1-{\text {PS}}_{10})\\ {\text {PS}}_{01} \end{bmatrix} = \begin{bmatrix} P^a_{11}\\ P^b_{11} \end{bmatrix} \end{aligned}$$

(20)

Yielding the solution,

$$\begin{aligned} \begin{bmatrix} (1-{\text {PS}}_{10})\\ {\text {PS}}_{01} \end{bmatrix} = \frac{1}{P^a_{01}P^b_{00} - P^b_{01}P^a_{00}} \times \begin{bmatrix} P^b_{00} &{} -P^a_{00}\\ -P^b_{01} &{} P^a_{01} \end{bmatrix} \begin{bmatrix} P^a_{11}\\ P^b_{11} \end{bmatrix} \end{aligned}$$

(21)

Which amounts to:

$$\begin{aligned} {\text {PS}}_{10}&= 1- \frac{P^a_{11}P^b_{00} - P^b_{11}P^a_{00}}{P^a_{01}P^b_{00} - P^b_{01}P^a_{00}} \end{aligned}$$

(22)

$$\begin{aligned} {\text {PS}}_{01}&= \frac{P^b_{11}P^a_{01} - P^a_{11}P^b_{01}}{P^a_{01}P^b_{00} - P^b_{01}P^a_{00}} \end{aligned}$$

(23)

All values of the RHS can be computed from the experimental data of $\Pi ^a$ and $\Pi ^b$. Note that, since ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ must be between 0 and 1, not all solutions are valid. Therefore, two domains already entail some testable implications—if either ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$ are not valid probabilities, this means that the assumption that the probabilities of sufficiency are invariant across domains is false. If we add a third or more source domains, it is easy to see that we will have three or more equations but still only two unknowns, and the system is thus over-identified.

Once in possession of ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$, we can transport the causal effect to the target population $\Pi ^*$ by appealing again to the law of total probability,

$$\begin{aligned} P^*_{11}&= (1-{\text {PS}}_{10}) \times P^*_{01} + {\text {PS}}_{01} \times P^*_{00} \end{aligned}$$

(24)

$$\begin{aligned}&= \frac{P^a_{11}P^b_{00} - P^b_{11}P^a_{00}}{P^a_{01}P^b_{00} - P^b_{01}P^a_{00}}\times P^*_{01} + \frac{P^b_{11}P^a_{01} - P^a_{11}P^b_{01}}{P^a_{01}P^b_{00} - P^b_{01}P^a_{00}}\times P^*_{00} \end{aligned}$$

(25)

Finally, we note that all probabilities of causation, as discussed in [26], are also identifiable in this setting. First, consider the probability of necessity and sufficiency, $\text {PNS} = P(Y_1 = 1, Y_0 = 0)$ for $\Pi ^a$. Using the chain rule, PNS can be written as,

$$\begin{aligned} P^a(Y_1 = 1, Y_0 = 0)&= P^a(Y_1 = 1 \mid Y_0 = 0)P^a(Y_0 = 0) \end{aligned}$$

(26)

$$\begin{aligned}&= {\text {PS}}_{01} \times P^a(Y_0 = 0) \end{aligned}$$

(27)

Note ${\text {PS}}_{01}$ was already identified, and $P^a(Y_0 = 0)$ is given by the trial data in $\Pi ^a$, thus rendering $\text {PNS}^a$ identifiable. Similar reasoning holds for $\Pi ^b$.

For the probability of necessity, define ${\text {PN}}_{01}:= P(Y_0 = 0 \mid Y_1 = 1)$. Due to the randomization of X, ${\text {PN}}_{01}$ coincides with Tian and Pear’s probability of necessity during the trial (not the observational PN), by the same argument we provide for PS in the main text. The final step is to note that,

$$\begin{aligned} P^a(Y_0 = 0 \mid Y_1 = 1) = \frac{P^a(Y_0 = 0 , Y_1 = 1)}{P^a(Y_1 = 1)} = \frac{{\text {PNS}}^a}{P^a(Y_1 = 1)} \end{aligned}$$

The numerator is simply the PNS, which we have already identified, and the denominator is given by the trial data in $\Pi ^a$. Again, analogous argument can be given for $\Pi ^b$.

Modeling functional constraints

To illustrate the usefulness of explicitly modeling functional constraints in a structural framework, we apply the same modeling strategy of the paper in an example described in Huitfeldt et al. [10, p. 11]:

Consider a team of investigators who are interested in the effect of antibiotic treatment on mortality in patients with a specific bacterial infection (...) the investigators believe that the response to this antibiotic is completely determined by an unmeasured bacterial gene, such that only those who are infected with a bacterial strain with this gene respond to treatment. The prevalence of this bacterial gene is equal between populations, because the populations share the same bacterial ecosystem (...) if the investigators further believe that the gene for susceptibility reduces the mortality in the presence of antibiotics, but has no effect in the absence of antibiotics, they will conclude that G may be equal between populations.

Here the conclusion that G may be equal between populations is equivalent to claiming $Y_1 \perp \!\!\!\!\perp S \mid Y_0$. But is the description above sufficient for substantiating this claim? Figure 9 shows two models compatible with the description, yet leading to two opposite conclusions.

Fig. 9

Two selection diagrams compatible with the verbal description of Huitfeldt et al. [10, page 11]. Yet, model a implies $Y_1 \perp \!\!\!\!\perp S \mid Y_0$, and model b implies the opposite; conditioning on $Y_0$ opens the colliding path $S \rightarrow U \leftrightarrow BG \rightarrow Y_1$

Let the variable A represent the binary treatment (antibiotic), Y represent the binary outcome (mortality), BG stand for the presence or absence of the “bacterial gene” and finally let U be a binary variable that summarizes all other factors that may cause death ($Y=1$). The description of the problem suggests the functional specification,

$$\begin{aligned} Y = U \wedge (\lnot A \vee \lnot BG) \end{aligned}$$

(28)

showing the antibiotics and the bacterial gene both helping to reduce mortality ($\lnot$ denotes the logical “not”). Equation 28 entails the potential outcomes $Y_{0} = U$ and $Y_{1} = U \wedge (\lnot BG) = Y_{0 }\wedge (\lnot BG)$, which are explicitly shown in both diagrams as dictated by the functional specification. Moreover, in both models the prevalence of the bacterial gene BG is equal between populations (i.e., $BG \perp \!\!\!\!\perp S$). In the model of Fig. 9a, as in our previous analysis, we indeed conclude that $Y_1 \perp \!\!\!\!\perp S \mid Y_0$, and that $P^*(Y_{1})$ is transportable. However, in the model of Fig. 9b, there is an unmeasured confounder between BG and U.¹⁵ Conditioning on $Y_0$ (a child of a collider) opens the colliding path $S \rightarrow U \leftrightarrow BG \rightarrow Y_1$, thus not licensing the independence $Y_1 \perp \!\!\!\!\perp S \mid Y_0$.

Bayesian estimation

Multiple source domains

In this section we show how to extend the probabilistic graphical model of the section “A Bayesian approach to estimation” to two or more sources. Let us start with two source populations $\Pi ^a$ and $\Pi ^b$, and one target domain $\Pi ^*$. The observed data is now ${\mathcal {D}}\,=\,\{n^a_0, n^a_1, n^*_0, n^b_0, n^b_1\}$, all with binomial distributions:

$$\begin{aligned} n^a_0&\sim {\text {Binomial}}(N^a_{0}, P^a_{01}) \end{aligned}$$

(29)

$$\begin{aligned} n^a_1&\sim {\text {Binomial}}(N^a_{1}, P^a_{11}) \end{aligned}$$

(30)

$$\begin{aligned} n^*_{0}&\sim {\text {Binomial}}(N^*_{0}, P^*_{01}) \end{aligned}$$

(31)

$$\begin{aligned} n^b_0&\sim {\text {Binomial}}(N^b_{0}, P^b_{01}) \end{aligned}$$

(32)

$$\begin{aligned} n^b_1&\sim {\text {Binomial}}(N^b_{1}, P^b_{11}) \end{aligned}$$

(33)

We also have the following deterministic relationships for $P^a_{11}$, $P^b_{11}$ and $P^*_{11}$:

$$\begin{aligned} P^a_{11} = (1-{\text {PS}}_{10})P^a_{01} + {\text {PS}}_{01}(1-P^a_{01}) \end{aligned}$$

(34)

$$\begin{aligned} P^b_{11} = (1-{\text {PS}}_{10})P^b_{01} + {\text {PS}}_{01}(1-P^b_{01}) \end{aligned}$$

(35)

$$\begin{aligned} P^*_{11} = (1-{\text {PS}}_{10})P^*_{01} + {\text {PS}}_{01}(1-P^*_{01}) \end{aligned}$$

(36)

The probabilistic graphical model for this case is shown in Fig. 10.

Fig. 10

Probabilistic graphical model with two source populations $\Pi ^a$, $\Pi ^b$ and one target population $\Pi ^*$. Gray nodes ($n^a_0$, $n^a_1$, $n_0^*$, $n^b_0$, $n^b_1$) denote observed variables. White notes denote latent parameters ($P^a_{01}$, $P^a_{11}$, ${\text {PS}}_{10}$, ${\text {PS}}_{01}$, $P^*_{11}$, $P^*_{01}$, $P^b_{11}$, $P^b_{01}$). Note that $P^a_{11}$, $P^*_{11}$ and $P^b_{11}$ share the parameters ${\text {PS}}_{10}$ and ${\text {PS}}_{01}$, which are invariant across populations

Thus, one needs to place priors on the parent nodes only, and then perform inference as before. The extension to more than two populations follows the same logic. It is worth noting that, as we have seen in the section “Building the structural model” with two or more source populations the model entails testable implications. Therefore, we advise researchers to check whether the data is compatible with the model [6].

Finally, similarly to the discussion in the section “A Bayesian approach to estimation” a simpler modeling alternative here is to place priors only on the parameters of the observed data directly, and make inferences using the posterior of the functionals of the observed data that identify the target quantities.

Replication code

R code to replicate the estimation examples using JAGS [22] and the package rjags [21] is provided in the online supplemental material.