Penalized estimation of directed acyclic graphs from discrete data

Abstract

Bayesian networks, with structure given by a directed acyclic graph (DAG), are a popular class of graphical models. However, learning Bayesian networks from discrete or categorical data is particularly challenging, due to the large parameter space and the difficulty in searching for a sparse structure. In this article, we develop a maximum penalized likelihood method to tackle this problem. Instead of the commonly used multinomial distribution, we model the conditional distribution of a node given its parents by multi-logit regression, in which an edge is parameterized by a set of coefficient vectors with dummy variables encoding the levels of a node. To obtain a sparse DAG, a group norm penalty is employed, and a blockwise coordinate descent algorithm is developed to maximize the penalized likelihood subject to the acyclicity constraint of a DAG. When interventional data are available, our method constructs a causal network, in which a directed edge represents a causal relation. We apply our method to various simulated and real data sets. The results show that our method is very competitive, compared to many existing methods, in DAG estimation from both interventional and high-dimensional observational data.

Appendix: Asymptotic theory

In this appendix, we establish asymptotic theory for the DAG estimator $\hat{\varvec{\beta }}_{\lambda }$ (12) assuming that p is fixed and $n\rightarrow \infty$. By rearranging and relabeling individual components, we rewrite $\varvec{\beta }$ as $\varvec{\phi }=(\varvec{\phi }_{(1)}, \varvec{\phi }_{(2)})$, where $\varvec{\phi }_{(1)} = \text {vec}( \varvec{\beta }_{1\cdot 1},\ldots , \varvec{\beta }_{1\cdot p}, \ldots , \varvec{\beta }_{p\cdot 1},\ldots , \varvec{\beta }_{p\cdot p})$ is the parameter vector of interest and $\varvec{\phi }_{(2)} = \text {vec}(\varvec{\beta }_{1\cdot 0}, \ldots ,\varvec{\beta }_{p\cdot 0})$ denotes the vector of intercepts. Hereafter, we denote by $\phi _j$ the jth group of $\varvec{\phi }$, such that $\phi _1 = \varvec{\beta }_{1\cdot 1}$, $\phi _2 = \varvec{\beta }_{1\cdot 2}, \ldots , \phi _{p^2}=\varvec{\beta }_{p\cdot p}$, and so on. We say $\varvec{\phi }$ is acyclic if the graph $\mathcal {G}_{\varvec{\phi }}$ induced by $\varvec{\phi }$ (or the corresponding $\varvec{\beta }$) is acyclic.

Define $\varvec{\phi }_{[k]}$ ($k\in \{1,\ldots ,p\}$) to be the parameter vector obtained from $\varvec{\phi }$ by setting $\varvec{\beta }_{k\cdot i} = \mathbf {0}$ for $i=1, \ldots , p$. In other words, the DAG $\mathcal {G}_{\varvec{\phi }_{[k]}}$ is obtained by deleting all edges pointing to the kth node in $\mathcal {G}_{\varvec{\phi }}$; see (10). We assume the data set $\mathcal {X}$ consists of $(p+1)$ blocks, denoted by $\mathcal {X}^j$ of size $n_j \times p$, $j=1,\ldots ,p+1$. The node $X_j$ is experimentally fixed in $\mathcal {X}^j$ for the first p blocks, while the last block contains purely observational data. Let $\mathcal {I}_j$ be the set of row indices of $\mathcal {X}^j$. As demonstrated by (2), we can model interventional data in the kth block of the data matrix $\mathcal {X}^k$ as i.i.d. observations from a joint distribution factorized according to $\mathcal {G}_{\varvec{\phi }_{[k]}}$. Denote the corresponding probability mass function by $p(\mathbf {x}|\varvec{\phi }_{[k]})$, where $\mathbf {x}=(x_1,\ldots ,x_p)$ and $x_j \in \{ 1, \ldots , r_j \}$ for $j=1,\ldots ,p$. To simplify our notation, denote the parameter for the $(p+1)$th block by $\varvec{\phi }_{[p+1]} = \varvec{\phi }$. Then the log-likelihood of $\mathcal {X}$ is

$$\begin{aligned} L(\varvec{\phi }) = \sum _{k=1}^{p+1} L_k(\varvec{\phi }_{[k]}) = \sum _{k=1}^{p+1} \log p(\mathcal {X}^k \mid \varvec{\phi }_{[k]}), \end{aligned}$$

(23)

where $\log p(\mathcal {X}^k | \varvec{\phi }_{[k]})=\sum _{h \in \mathcal {I}_k} \log (p(\mathcal {X}_{h\cdot }| \varvec{\phi }_{[k]}))$ and $\mathcal {X}_{h\cdot } =(\mathcal {X}_{h1},\ldots ,\mathcal {X}_{hp})$. The penalized log-likelihood function with a tuning parameter $\lambda _n>0$ is

$$\begin{aligned} R(\varvec{\phi })= & {} L(\varvec{\phi })-\lambda _n\sum _{j=1}^{p^2}||\phi _j||_2 \nonumber \\= & {} \sum _{k=1}^{p+1}L_k(\varvec{\phi }_{[k]})-\lambda _n\sum _{j=1}^{p^2}||\phi _j||_2, \end{aligned}$$

(24)

where the component group $\phi _j\,(j=1,\ldots ,p^2)$ represents the influence of one variable on another. Let $\Omega = \{\varvec{\phi }: \mathcal {G}_{\varvec{\phi }} \text { is a DAG}\}$ be the parameter space. A penalized estimator $\hat{\varvec{\phi }}$ is obtained by maximizing $R(\varvec{\phi })$ in $\Omega$.

Though interventional data help distinguish equivalent DAGs, the following notion of natural parameters is needed to completely establish identifiability of DAGs for the case where each variable has interventional data. We say that i is an ancestor of j in a DAG $\mathcal {G}$ if there exists at least one path from i to j. Denote the set of ancestors of j by $\text {an}(j)$.

Definition 1

(Natural parameters) We say that $\varvec{\phi } \in \Omega$ is natural if $i \in \text {an}(j) \text { in } \mathcal {G}_{\varvec{\phi }}$ implies that j is not independent of i under the joint distribution given by $\varvec{\phi }_{[i]}$ for all $i,j=1,\ldots ,p$.

For a causal DAG, a natural parameter implies that the effects along multiple causal paths connecting the same pair of nodes do not cancel. This is a reasonable assumption for many real-world problems and is much weaker than the faithfulness assumption. Under the faithfulness assumption, all conditional independence restrictions can be read off from d-separations in the DAG. If nodes i and j are independent in $\varvec{\phi }_{[i]}$, then by faithfulness the nodes i and j must be separated by empty set and thus $i \notin \text {an}(j)$ in $\mathcal {G}_{\varvec{\phi }_{[i]}}$. This implies that $i \notin \text {an}(j)$ in $\mathcal {G}_{\varvec{\phi }}$ as well, by the construction of $\mathcal {G}_{\varvec{\phi }_{[i]}}$. Indeed, we see that the faithfulness assumption implies the natural parameter assumption.

To establish asymptotic properties of our penalized likelihood estimator, we make the following assumptions:

• (A1) The true parameter $\varvec{\phi }^*$ is natural and an interior point of $\Omega$.

• (A2) The parameter $\varvec{\theta }_j$ of the conditional distribution $[X_j | \Uppi _j^{\mathcal {G}}; \varvec{\theta }_j]$ is identifiable for each $j=1,\ldots ,p$. The log-likelihood function $\ell _j(\varvec{\theta }_j) = \log p({x}_j|\Uppi _j^{\mathcal {G}}; \varvec{\theta }_j)$ is strictly concave and continuously three times differentiable for any interior point.

Recall that the kth block of our data, $\mathcal {X}^k$, can be regarded as an i.i.d. sample of size $n_k$ from the distribution $p(\mathbf {x}|\varvec{\phi }_{[k]}^*)$ for all k, while we define $\varvec{\phi }_{[p+1]}^*=\varvec{\phi }^*$ for the last block of observational data.

Theorem 1

Assume (A1) and (A2). If $p(\mathbf {x}|\varvec{\phi }_{[k]})=p(\mathbf {x}|\varvec{\phi }_{[k]}^*)$ for all possible $\mathbf {x}$ and all $k=1,\ldots ,p$, then $\varvec{\phi }=\varvec{\phi }^*$. Furthermore, if $n_k\gg \sqrt{n}$ for all $k=1,\ldots ,p$, then for any $\varvec{\phi } \ne \varvec{\phi }^*$,

$$\begin{aligned} P(L(\varvec{\phi }^*)>L(\varvec{\phi })) \rightarrow 1 \quad \text { as } n \rightarrow \infty . \end{aligned}$$

(25)

Theorem 2

Assume (A1) and (A2). If $\lambda _n/\sqrt{n}\rightarrow 0$ and $n_k\gg \sqrt{n}$ for all $k=1,\ldots ,p$, then there exists a global maximizer $\hat{\varvec{\phi }}$ of $R(\varvec{\phi })$ such that $||\hat{\varvec{\phi }}-\varvec{\phi }^*||_2=O_p(n^{-1/2})$.

Proofs of the two theorems are relegated to Supplemental Material. Theorem 1 confirms that the causal DAG model is identifiable with interventional data assuming a natural parameter. Theorem 2 implies that there is a $\sqrt{n}$-consistent global maximizer of $R(\varvec{\phi })$ with the group norm penalty. Note that Assumption (A2) does not specify a particular choice of model for the conditional distribution $[X_j | \Uppi _j^{\mathcal {G}}]$, and thus, these theoretical results apply to a large class of DAG models for discrete data. In particular, the multi-logit regression model (4) satisfies (A2).

Remark 2

The assumption on the sample size of interventional data, $n_k\gg \sqrt{n}$, imposes a lower bound on how fast the fraction $\alpha _k=n_k/n\gg n^{-1/2}$ can approach zero for $k=1,\ldots ,p$. Although this allows the observational data to dominate when $\alpha _k\rightarrow 0$, the fractions of interventional data must be larger than the typical order $O_p(n^{-1/2})$ of statistical errors so that (25) can hold to establish identifiability of the true causal DAG parameter $\varvec{\phi }^*$. This guarantees that the global maximizer $\hat{\varvec{\phi }}$ will locate in a neighborhood of $\varvec{\phi }^*$ with high probability. Once in this neighborhood, the convergence rate of $\hat{\varvec{\phi }}$ then depends on the size n of all data, both interventional and observational. Therefore, increasing the size of observation data will lead to more accurate estimate $\hat{\varvec{\phi }}$ as long as we keep $\alpha _k\gg n^{-1/2}$ for $k=1,\ldots ,p$.

Remark 3

It is interesting to generalize the above asymptotic results to the case where $p=p_n$ grows with the sample size n, say, by developing nonasymptotic bounds on the $\ell _2$ estimation error $\Vert \hat{\varvec{\phi }}-\varvec{\phi }^*\Vert _2$. However, in order to estimate the causal network consistently, sufficient interventional data are needed for each node, i.e., $n_k$ must approach infinity, and thus, $p/ n \rightarrow 0$ as $n\rightarrow \infty$. This limits us to the low-dimensional setting with $p<n$. Suppose we have a large network with $p\gg n$. One may first apply some regularization method on observational data to screen out independent nodes and to partition the network into small subgraphs that are disconnected to one another. Then for each small subgraph, we can afford to generate enough interventional data for every node and apply the method in this paper to infer the causal structure. Our asymptotic theory provides useful guidance for the analysis in the second step.

For purely observational data, the theory becomes more complicated due to the existence of equivalent DAGs and parameterizations. It is left as future work to establish the consistency of a global maximizer for high-dimensional observational data.