Learning Functional Causal Models with Generative Neural Networks

Abstract

We introduce a new approach to functional causal modeling from observational data, called Causal Generative Neural Networks (CGNN). CGNN leverages the power of neural networks to learn a generative model of the joint distribution of the observed variables, by minimizing the Maximum Mean Discrepancy between generated and observed data. An approximate learning criterion is proposed to scale the computational cost of the approach to linear complexity in the number of observations. The performance of CGNN is studied throughout three experiments. Firstly, CGNN is applied to cause-effect inference, where the task is to identify the best causal hypothesis out of “X → Y ” and “Y → X”. Secondly, CGNN is applied to the problem of identifying v-structures and conditional independences. Thirdly, CGNN is applied to multivariate functional causal modeling: given a skeleton describing the direct dependences in a set of random variables X = [X ₁, …, X _d], CGNN orients the edges in the skeleton to uncover the directed acyclic causal graph describing the causal structure of the random variables. On all three tasks, CGNN is extensively assessed on both artificial and real-world data, comparing favorably to the state-of-the-art. Finally, CGNN is extended to handle the case of confounders, where latent variables are involved in the overall causal model.

Appendix

1.1 The Maximum Mean Discrepancy (MMD) Statistic

The Maximum Mean Discrepancy (MMD) statistic (Gretton et al. 2007) measures the distance between two probability distributions P and $\hat {P}$, defined over $\mathbb {R}^d$, as the real-valued quantity

$$\displaystyle \begin{aligned} \text{MMD}_k(P, \hat{P}) = \left\| \mu_k(P) - \mu_k(\hat{P}) \right\|{}_{\mathcal{H}_k}. \end{aligned}$$

Here, $\mu _k = \int k(x, \cdot ) \mathrm {d} P(x)$ is the kernel mean embedding of the distribution P, according to the real-valued symmetric kernel function $k(x, x') = \langle k(x, \cdot ), k(x', \cdot ) \rangle _{\mathcal {H}_k}$ with associated reproducing kernel Hilbert space $\mathcal {H}_k$. Therefore, μ _k summarizes P as the expected value of the features computed by k over samples drawn from P.

In practical applications, we do not have access to the distributions P and $\hat {P}$, but to their respective sets of samples $\mathcal {D}$ and $\hat {\mathcal {D}}$, defined in Sect. 4.2.1. In this case, we approximate the kernel mean embedding μ _k(P) by the empirical kernel mean embedding $\mu _k(\mathcal {D}) = \frac {1}{|\mathcal {D}|} \sum _{x \in \mathcal {D}} k(x, \cdot )$, and respectively for $\hat {P}$. Then, the empirical MMD statistic is

$$\displaystyle \begin{aligned} \widehat{\text{MMD}}_k(\mathcal{D}, \hat{\mathcal{D}}) &= \left\| \mu_k(\mathcal{D}) - \mu_k(\hat{\mathcal{D}}) \right\|{}_{\mathcal{H}_k} \\ &=\frac{1}{n^2} \sum_{i, j}^{n} k(x_i, x_j) + \frac{1}{n^2} \sum_{i, j}^{n} k(\hat{x}_i, \hat{x}_j) - \frac{2}{n^2} \sum_{i,j}^n k(x_i, \hat{x}_j). \end{aligned} $$

Importantly, the empirical MMD tends to zero as n →∞ if and only if $P = \hat {P}$, as long as k is a characteristic kernel (Gretton et al. 2007). This property makes the MMD an excellent choice to model how close the observational distribution P is to the estimated observational distribution $\hat {P}$. Throughout this paper, we will employ a particular characteristic kernel: the Gaussian kernel $k(x, x') = \exp (- \gamma \| x - x' \|{ }_2^2)$, where γ > 0 is a hyperparameter controlling the smoothness of the features.

In terms of computation, the evaluation of $\text{MMD}_k(\mathcal {D}, \hat {\mathcal {D}})$ takes O(n ²) time, which is prohibitive for large n. When using a shift-invariant kernel, such as the Gaussian kernel, one can invoke Bochner’s theorem (Edwards 1964) to obtain a linear-time approximation to the empirical MMD (Lopez-Paz et al. 2015), with form

$$\displaystyle \begin{aligned} \widehat{\text{MMD}}^m_k(\mathcal{D}, \hat{\mathcal{D}}) = \left\| \hat{\mu}_k(\mathcal{D}) - \hat{\mu}_k(\hat{\mathcal{D}}) \right\|{}_{\mathbb{R}^m} {} \end{aligned}$$

and O(mn) evaluation time. Here, the approximate empirical kernel mean embedding has form

$$\displaystyle \begin{aligned} \hat{\mu}_k(\mathcal{D}) = \sqrt{\frac{2}{m}} \frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \left[ \cos{}(\langle w_1, x \rangle + b_1), \ldots, \cos{}(\langle w_m, x \rangle + b_m) \right], \end{aligned}$$

where w _i is drawn from the normalized Fourier transform of k, and b _i ∼ U[0, 2π], for i = 1, …, m. In our experiments, we compare the performance and computation times of both $\widehat {\text{MMD}}_k$ and $\widehat {\text{MMD}}_k^m$.

1.2 Proofs

Proposition 1

Let X = [X ₁, …, X _d] denote a set of continuous random variables with joint distribution P, and further assume that the joint density function h of P is continuous and strictly positive on a compact and convex subset of $\mathbb {R}^{d}$ , and zero elsewhere. Letting $\mathcal {G}$ be a DAG such that P can be factorized along $\mathcal {G}$ ,

$$\displaystyle \begin{aligned}P(X) = \prod_i P(X_i | X_{\mathit{\text{Pa}}({i}; \mathcal{G})})\end{aligned}$$

there exists f = (f ₁, …, f _d) with f _i a continuous function with compact support in $\mathbb {R}^{|\mathit{\text{Pa}}({i}; \mathcal {G})|}\times [0,1]$ such that P(X) equals the generative model defined from FCM $({\mathcal {G}}, f, {\mathcal {E}})$ , with ${\mathcal {E}} = \mathcal {U}[0,1]$ the uniform distribution on [0, 1].

Proof

By induction on the topological order of $\mathcal {G}$. Let X _i be such that $|\text{Pa}({i}; \mathcal {G})|=0$ and consider the cumulative distribution F _i(x _i) defined over the domain of X _i (F _i(x _i) = Pr(X _i < x _i)). F _i is strictly monotonous as the joint density function is strictly positive therefore its inverse, the quantile function Q _i : [0, 1]↦dom(X _i) is defined and continuous. By construction, $Q_i(e_i) =F_i^{-1}(e_i)$ and setting Q _i = f _i yields the result.

Assume f _i be defined for all variables X _i with topological order less than m. Let X _j with topological order m and Z the vector of its parent variables. For any noise vector $e = (e_i, i \in \text{Pa}({j}; \mathcal {G}))$ let $z = (x_i, i \in \text{Pa}({j}; \mathcal {G}))$ be the value vector of variables in Z defined from e. The conditional cumulative distribution F _j(x _j|Z = z) = Pr(X _j < x _j|Z = z) is strictly continuous and monotonous wrt x _j, and can be inverted using the same argument as above. Then we can define $f_j(z,e_j) = F_j^{-1}(z,e_j)$.

Let K _j = dom(X _j) and $K_{\text{Pa}({j}; \mathcal {G})} = dom(Z)$. We will show now that the function f _j is continuous on $K_{\text{Pa}({j}; \mathcal {G})} \times [0,1]$, a compact subset of $\mathbb {R}^{|\text{Pa}({j}; \mathcal {G})|}\times [0,1]$.

By assumption, there exist $a_j \in \mathcal {R}$ such that, for $(x_j, z) \in K_j \times K_{\text{Pa}({j}; \mathcal {G})}$, $F(x_j|z) = \int _{a_j}^{x_j} \frac {h_j(u,z)}{h_j(z)} \mathrm {d}u$, with h _j a continuous and strictly positive density function. For $(a,b) \in K_j \times K_{\text{Pa}({j}; \mathcal {G})}$, as the function $(u, z) \rightarrow \frac {h_j(u,z)}{h_j(z)}$ is continuous on the compact $K_j \times K_{\text{Pa}({j}; \mathcal {G})}$, $\lim \limits _{\substack {x_j \rightarrow a}} F(x_j|z) = \int _{a_j}^{a} \frac {h_j(u,z)}{h_j(z)} \mathrm {d}u$ uniformly on $K_{\text{Pa}({j}; \mathcal {G})}$ and $\lim \limits _{\substack {z \rightarrow b}} F(x_j|z) = \int _{a_j}^{x_j} \frac {h_j(u,b)}{h_j(b)}$ on K _j, according to exchanging limits theorem, F is continuous on (a, b).

For any sequence z _n → z, we have that F(x _j|z _n) → F(x _j|z) uniformly in x _j. Let define two sequences u _n and x _j,n, respectively on [0, 1] and K _j, such that u _n → u and x _j,n → x _j. As F(x _j|z) = u has unique root x _j = f _j(z, u), the root of F(x _j|z _n) = u _n, that is, x _j,n = f _j(z _n, u _n) converge to x _j. Then the function (z, u) → f _j(z, u) is continuous on $K_{\text{Pa}({i}; \mathcal {G})} \times [0,1]$.

Proposition 2

For m ∈ [[1, d]], let Z _m denote the set of variables with topological order less than m and let d _m be its size. For any d _m -dimensional vector of noise values e ^(m) , let z _m(e ^(m)) (resp. $\widehat {z_m}(e^{(m)})$ ) be the vector of values computed in topological order from the FCM $({\mathcal {G}}, f, {\mathcal { E}})$ (resp. the CGNN $({\mathcal {G}}, \hat {f}, {\mathcal {E}})$ ). For any 𝜖 > 0, there exists a set of networks $\hat {f}$ with architecture $\mathcal {G}$ such that

$$\displaystyle \begin{aligned} \forall e^{(m)}, \|z_m(e^{(m)})- \widehat{z_m}(e^{(m)})\| < \epsilon {} \end{aligned} $$

(14)

Proof

By induction on the topological order of $\mathcal {G}$. Let X _i be such that $|\text{Pa}({i}; \mathcal {G})|=0$. Following the universal approximation theorem Cybenko (1989), as f _i is a continuous function over a compact of $\mathbb {R}$, there exists a neural net $\hat {f_{i}}$ such that $\|f_i - \hat {f_{i}}\|{ }_\infty < \epsilon /d_1$. Thus Eq. (14) holds for the set of networks $\hat {f_i}$ for i ranging over variables with topological order 0.

Let us assume that Proposition 2 holds up to m, and let us assume for brevity that there exists a single variable X _j with topological order m + 1. Letting $\hat {f_j}$ be such that $\|f_j - \hat {f_j}\|{ }_\infty < \epsilon /3$ (based on the universal approximation property), letting δ be such that for all u $\|\hat f_j(u) - \hat f_j(u+\delta )\|< \epsilon /3$ (by absolute continuity) and letting $\hat f_i$ satisfying Eq. (14) for i with topological order less than m for min(𝜖∕3, δ)∕d _m, it comes: $\|(z_m,f_j(z_m,e_j)) - (\hat z_m,\hat {f_j}(\hat {z_m}, e_j))\| \le \|z_m - \hat z_m\| + |f_j(z_m,e_j) - \hat {f_j}(z_m, e_j)| + | \hat {f_j}(z_m,e_j) - \hat {f_j}(\hat {z_m}, e_j)| < \epsilon /3 + \epsilon /3 + \epsilon /3$, which ends the proof.

Proposition 3

Let $\mathcal {D}$ be an infinite observational sample generated from $({\mathcal {G}}, f, {\mathcal {E}})$ . With same notations as in Proposition 2 , for every sequence 𝜖 _t such that 𝜖 _t > 0 goes to zero when t →∞, there exists a set $\widehat {f_t} = (\hat f^{t}_1 \ldots \hat f^{t}_d)$ such that $\widehat {\mathit{\text{MMD}}_k}$ between $\mathcal D$ and an infinite size sample $\widehat {\mathcal {D}}_{t}$ generated from the CGNN $(\mathcal {G},\widehat {f_{t}},\mathcal {E})$ is less than 𝜖 _t.

Proof

According to Proposition 2 and with same notations, letting 𝜖 _t > 0 go to 0 as t goes to infinity, consider ${\hat f}_t=(\hat f^{t}_1 \ldots \hat f^{t}_d)$ and $\hat {z_t}$ defined from ${\hat f}_t$ such that for all e ∈ [0, 1]^d, $\|z(e)- \widehat {z}_t(e)\| < \epsilon _t$.

Let $\{ \hat {\mathcal {D}_t} \}$ denote the infinite sample generated after $\hat {f_t}$. The score of the CGNN $(\mathcal {G},\hat {f_t},\mathcal {E})$ is $\widehat {\text{MMD}}_k(\mathcal {D}, \hat {\mathcal {D}_t}) = \mathbb {E}_{e,e'}[k(z(e),z(e')) - 2 k(z(e),\widehat {z}_t(e')) + k(\widehat {z}_t(e), \widehat {z}_t(e'))]$.

As $\hat {f_t}$ converges towards f on the compact [0, 1]^d, using the bounded convergence theorem on a compact subset of $\mathbb {R}^{d}$, $\widehat {z_t}(e) \rightarrow z(e)$ uniformly for t →∞, it follows from the Gaussian kernel function being bounded and continuous that $\widehat {\text{MMD}}_k(\mathcal {D}, \hat {\mathcal {D}_t}) \rightarrow 0$, when t →∞.

Proposition 4

Let X = [X ₁, …, X _d] denote a set of continuous random variables with joint distribution P, generated by a CGNN $\mathcal {C}_{\mathcal {G},f} = (\mathcal {G}, f, \mathcal {E})$ with $\mathcal {G}$ , a directed acyclic graph. And let $\mathcal {D}$ be an infinite observational sample generated from this CGNN. We assume that P is Markov and faithful to the graph $\mathcal {G}$ , and that every pair of variables (X _i, X _j) that are d-connected in the graph are not independent. We note $\widehat {\mathcal {D}}$ an infinite sample generated by a candidate CGNN, $\mathcal {C}_{\widehat {\mathcal {G}},\hat {f}} = (\widehat {\mathcal {G}}, \hat {f}, \mathcal {E})$ . Then,

• (i) If $\widehat {\mathcal {G}} = \mathcal {G}$ and $\hat {f} = f$ , then $\widehat {\mathit{\text{MMD}}}_k(\mathcal {D}, \widehat {\mathcal {D}}) = 0$.

• (ii) For any graph $\widehat {\mathcal {G}}$ characterized by the same adjacencies but not belonging to the Markov equivalence class of $\mathcal {G}$ , for all $\hat {f}$ , $\widehat {\mathit{\text{MMD}}}_k(\mathcal {D}, \widehat {\mathcal {D}}) \neq 0$.

Proof

The proof of (i) is obvious, as with $\widehat {\mathcal {G}} = \mathcal {G}$ and $\hat {f} = f$, the joint distribution $\hat {P}$ generated by $\mathcal {C}_{\widehat {\mathcal {G}},\hat {f}} = (\widehat {\mathcal {G}}, \hat {f}, \mathcal {E})$ is equal to P, thus we have $\widehat {\text{MMD}}_k(\mathcal {D}, \widehat {\mathcal {D}}) = 0$.

(ii) Let consider $\widehat {\mathcal {G}}$ a DAG characterized by the same adjacencies but that do not belong to the Markov equivalence class of $\mathcal {G}$. According to Verma and Pearl (1991), as the DAG $\mathcal {G}$ and $\widehat {\mathcal {G}}$ have the same adjacencies but are not Markov equivalent, there are not characterized by the same v-structures.

• a) First, we consider that a v-structure {X, Y, Z} exists in $\mathcal {G}$, but not in $\widehat {\mathcal {G}}$. As the distribution P is faithful to $\mathcal {G}$ and X and Z are not d-separated by Y  in $\mathcal {G}$, we have that in P. Now we consider the graph $\widehat {\mathcal {G}}$. Let $\hat {f}$ be a set of neural networks. We note $\hat {P}$ the distribution generated by the CGNN $\mathcal {C}_{\widehat {\mathcal {G}},\hat {f}}$. As $\widehat {\mathcal {G}}$ is a directed acyclic graph and the variables E _i are mutually independent, $\hat {P}$ is Markov with respect to $\widehat {\mathcal {G}}$. As {X, Y, Z} is not a v-structure in $\widehat {\mathcal {G}}$, X and Z are d-separated by Y . By using the causal Markov assumption, we obtain that (X⊥ ⊥ Z|Y ) in $\hat {P}$.

• b) Second, we consider that a v-structure {X, Y, Z} exists in $\widehat {\mathcal {G}}$, but not in $\mathcal {G}$. As {X, Y, Z} is not a v-structure in $\mathcal {G}$, there is an “unblocked path” between the variables X and Z, the variables X and Z are d-connected. By assumption, there do not exist a set D not containing Y  such that (X⊥ ⊥ Z|D) in P. In $\widehat {\mathcal {G}}$, as {X, Y, Z} is a v-structure, there exists a set D not containing Y  that d-separates X and Z. As for all CGNN $\mathcal {C}_{\widehat {\mathcal {G}},\hat {f}}$ generating a distribution $\hat {P}$, $\hat {P}$ is Markov with respect to $\widehat {\mathcal {G}}$, we have that X⊥ ⊥ Z|D in $\hat {P}$.

In the two cases a) and b) considered above, P and $\hat {P}$ do not encode the same conditional independence relations, thus are not equal. We have then $\widehat {\text{MMD}}_k(\mathcal {D}, \mathcal {D}') \neq 0$.

1.3 Table of Scores for the Experiments on Cause-Effect Pairs

See Table 3.

1.4 Table of Scores for the Experiments on Graphs

See Tables 4, 5 and 6.