Learning Bayesian Networks: A Copula Approach for Mixed-Type Data

Abstract

Estimating dependence relationships between variables is a crucial issue in many applied domains and in particular psychology. When several variables are entertained, these can be organized into a network which encodes their set of conditional dependence relations. Typically however, the underlying network structure is completely unknown or can be partially drawn only; accordingly it should be learned from the available data, a process known as structure learning. In addition, data arising from social and psychological studies are often of different types, as they can include categorical, discrete and continuous measurements. In this paper, we develop a novel Bayesian methodology for structure learning of directed networks which applies to mixed data, i.e., possibly containing continuous, discrete, ordinal and binary variables simultaneously. Whenever available, our method can easily incorporate known dependence structures among variables represented by paths or edge directions that can be postulated in advance based on the specific problem under consideration. We evaluate the proposed method through extensive simulation studies, with appreciable performances in comparison with current state-of-the-art alternative methods. Finally, we apply our methodology to well-being data from a social survey promoted by the United Nations, and mental health data collected from a cohort of medical students. R code implementing the proposed methodology is available at https://github.com/FedeCastelletti/bayes_networks_mixed_data.

Appendix

1.1 Comparison with Copula PC

In this section, we compare our methodology with the benchmark Copula PC method of Cui et al. (2016); see also Cui et al. (2018). Copula PC is a two-step approach which can be applied to mixed data comprising categorical (binary and ordinal), discrete and continuous variables. It first estimates a correlation matrix in the space of latent variables (each associated with one of the observed variables) which is then used to test conditional independencies as in the standard PC algorithm. For the first step, the same Gibbs sampling scheme introduced by Hoff (2007) and based on data augmentation with latent observations is adopted. Moreover, conditional independence tests are implemented at significance level $\alpha$ which we vary in $\{0.01,0.05,0.10\}$; lower values of $\alpha$ imply a higher expected level of sparsity in the estimated graph. We refer to the three benchmarks as Copula PC 0.01, 0.05 and 0.10, respectively. Output of Copula PC is a completed partially directed acyclic graph (CPDAG) representing the estimated equivalence class. With regard to our method, we also consider as a single graph estimate summarizing our MCMC output the CPDAG representing the equivalence class of the estimated median probability DAG model. Each model estimate is finally compared with the true CPDAG by means of the SHD between the two graphs.

Results for Scenario Free, type of variables Binary, Ordinal, Count, Mixed and each sample size $n\in \{100,200,500,1000,2000\}$ are summarized in Fig. 9 which reports the distribution across $N=40$ simulations of the SHD. It first appears that all methods improve their performances as the sample size n increases. In addition, structure learning is more difficult in the Binary case, while easier in general in the case of Ordinal and Count data. Moreover, Copula PC 0.01 (light grey) performs better than Copula PC 0.05 and 0.10 (middle and dark gray, respectively). Our method clearly outperforms the three benchmarks in the Binary scenario, a behavior which is more evident for large sample sizes. In addition, it performs better than Copula PC 0.05 and 0.10 most of the time under the remaining settings and remains highly competitive with Copula PC 0.01, with an overall better performance in terms of average SHD under almost all sample sizes for Scenarios Ordinal and Count.

Fig. 9

Simulations. Distribution (across $N=40$ simulated datasets) of the structural Hamming distance (SHD) between estimated and true CPDAG for type of variables Binary, Ordinal, Count, Mixed (b), sample size $n\in \{100,200,500,1000,2000\}$ (c) and type of DAG structure Free. Methods under comparison are: our Bayesian Copula DAG model (light blue) and the Copula PC method with independence tests implemented at significance level $\alpha \in \{0.01,0.0,0.10\}$ (from light to dark gray).

1.2 Comparison with Bayesian Parametric Strategy

Our methodology is based on a semi-parametric strategy which models separately the dependence parameter, corresponding to a DAG-dependent covariance matrix, and the marginal distributions of the observed variables, which are estimated using a rank-based nonparametric approach.

Alternatively, one can adopt suitable parametric families for modeling the various mixed types of variables, as in a generalized linear model (glm) framework. To implement this parametric strategy, we generalize the latent Gaussian DAG-model in (1) to accommodate a nonzero marginal mean for the latent variables. Specifically, we assume

$$\begin{aligned} Z_1,\ldots ,Z_q \,\vert \,\varvec{\mu },\varvec{\Omega }, \mathcal {D}\sim \mathcal {N}_q(\varvec{\mu },\varvec{\Omega }^{-1}), \end{aligned}$$

(21)

with $\varvec{\mu }\in \mathbb {R}^{q}$ and $\varvec{\Omega }\in \mathcal {P}_{\mathcal {D}}$, the space of all s.p.d. precision matrices Markov w.r.t. DAG $\mathcal {D}$. The allied structural equation model (SEM) representation of such model is given by $\varvec{\eta }+\varvec{L}^\top \varvec{z}= \varvec{\varepsilon }, \varvec{\varepsilon }\sim \mathcal {N}_q(\varvec{0},\varvec{D})$, or equivalently, in terms of node-distributions

$$\begin{aligned} Z_j = \eta _j - \varvec{L}_{\prec j \, ]}^\top \varvec{z}_{\text {pa}_{\mathcal {D}}(j)} + \varepsilon _j,\quad \varepsilon _j{\mathop {\sim }\limits ^{\text {ind}}} \mathcal {N}(0,\varvec{D}_{jj}), \end{aligned}$$

(22)

for each $j=1,\ldots ,q$ with $\varvec{D}_{jj} = \varvec{\Sigma }_{j\,\vert \,\text {pa}_{\mathcal {D}}(j)}, \varvec{L}_{\prec j \, ]} = -\varvec{\Sigma }^{-1}_{\prec j \succ }\varvec{\Sigma }_{\prec j \, ]}, \eta _j = \mu _j + \varvec{L}^{\top }_{\prec j \, ]} \varvec{\mu }_{\text {pa}_{\mathcal {D}}(j)}.$ Importantly, each equation in (22) now resembles the structure of a linear “regression” model with a nonzero intercept term $\eta _j$. A Normal-DAG-Wishart prior can be then assigned to $(\varvec{\eta },\varvec{D},\varvec{L})$; see Castelletti and Consonni (2023, Supplement, Section 1) for full details. Under such prior, the posterior distribution of $(\varvec{\eta },\varvec{D},\varvec{L})$ given independent (latent) Gaussian data $\varvec{Z}$ is still Normal-DAG-Wishart and also a marginal data distribution is available in closed-from expression. Therefore, we can adapt the MCMC scheme of Sect. 4 to this more general framework and specifically with the update of $(\varvec{D},\varvec{L},\mathcal {D})$ in Sect. 4.1 replaced by $(\varvec{\eta }, \varvec{D},\varvec{L},\mathcal {D})$.

Consider now the observed variables $X_1,\ldots ,X_q$, where each $X_j\sim F_j(\cdot )$, a suitably specified parametric family for $X_j$, e.g., Bernoulli, Poisson, or Binomial; see also Sect. 4.1. As in a glm framework, we assume that

$$\begin{aligned} \mathbb {E}(X_j\,\vert \,\varvec{z}_{\text {pa}_{\mathcal {D}}(j)}) = h^{-1}(\eta _j - \varvec{L}_{\prec j \, ]}^\top \varvec{z}_{\text {pa}_{\mathcal {D}}(j)}) \end{aligned}$$

(23)

where $h^{-1}(\cdot )$ is a suitable inverse-link function and it appears that $\eta _j - \varvec{L}_{\prec j \, ]}^\top \varvec{z}_{\text {pa}_{\mathcal {D}}(j)}$ plays the role of the linear predictor in the glm model for $X_j$. Specifically, we take $h(\cdot )=\text {logit}(\cdot )$ and $h(\cdot )=\log (\cdot )$ for $X_j \sim \text {Bern}(\pi _j)$ and $X_j\sim \text {Pois}(\lambda _j)$, respectively. Moreover, for $X_j \sim \text {Bin}(n_j,\pi _j)$ we take $h(\pi _j)=\text {logit}(\pi _j)$ while fix $n_j=\max \{x_{i,j}, i=1,\ldots ,n\}$. From (5) we then have $Z_j = \Phi ^{-1}\left\{ F_j(X_j\,\vert \,\varvec{z}_{\text {pa}_{\mathcal {D}}(j)})\right\} ,$ with $\Phi (\cdot )$ the standard normal c.d.f. and with $F_j$ implicitly depending on DAG parameters $(\eta _j,\varvec{L}_{\prec j \, ]})$ through (23). The update of $\varvec{Z}$ in Sect. 4.2 conditionally on the DAG parameters is then replaced by computing $z_{i,j}=\Phi ^{-1}\left\{ F_j(x_{i,j}\,\vert \,\varvec{z}_{\text {pa}_{\mathcal {D}}(j)})\right\}$ iteratively for each $i=1,\ldots ,n$ and $j=1,\ldots ,q$.

We consider the same simulation settings as in the Balanced Scenario, with the four different types of variables and with class of DAG structure Free; see Sect. 5.1. We compare the performance of the parametric strategy introduced above with our original method. Specifically, from the MCMC output provided by each method we first recover a CPDAG estimate and compare true and estimated graphs in terms of structural Hamming distance (SHD); see also Sect. 5.2 for details.

Results are summarized in the box-plots of Fig. 10, representing the distribution of SHD (across the 40 independent simulations) obtained from our original method (light blue) and its parametric version (dark blue) under the various scenarios. It appears that the parametric “version” of our method outperforms our original semi-parametric model in the Binary Scenario, while it is clearly outperformed under all the other scenarios for small-to-moderate sample sizes; however, the two approaches tend to perform similarly as the sample size n increases.

Fig. 10

Simulations. Distribution (across $N=40$ simulated datasets) of the structural Hamming distance (SHD) between estimated and true CPDAG for type of variables Binary, Ordinal, Count, Mixed (b), sample size $n\in \{100,200,500,1000,2000\}$ (c) and type of DAG structure Free. Methods under comparison are: our original semi-parametric Bayesian Copula DAG model (light blue) and its modified version based on parametric assumptions (dark blue).

1.3 MCMC Diagnostics of Convergence and Computational Time

Our methodology relies on Markov Chain Monte Carlo (MCMC) methods to approximate the posterior distribution of the parameters. Accordingly, diagnostics of convergence of the resulting MCMC output to the target distribution should be implemented before posterior analysis. In the following, we include a few results relative to the application of our method to the well-being data presented in Sect. 6.1.

As a first diagnostic tool, we monitor the behavior of the estimated posterior expectation of each correlation coefficient across iterations. Each quantity is computed at MCMC iteration s using the sampled values collected up to step s, for $s=1,\ldots ,$25,000. According to the results, reported for selected variables $(X_u,X_v)$ in Fig. 11, we discard the initial $B=5000$ draws that are therefore used as a burnin period. The behavior of each traceplot suggests for each parameter an appreciable degree of convergence to the posterior mean.

Fig. 11

Well-being data. Trace plots of the posterior mean of four correlation coefficients (for randomly selected variables $X_u,X_v$) estimated from the MCMC output up to iteration s, for $s=1,\ldots ,$25,000.

As a further diagnostic, we run two independent MCMC chains of length $S=$ 25,000, again including a burnin period of $B=5000$ runs, and with randomly chosen DAGs for the MCMC initialization. Results in terms of estimated posterior probabilities of edge inclusion computed from the two MCMC chains are reported in the heatmaps of Fig. 12 and suggest a visible agreement between the two outputs.

Fig. 12

Well-being data. Estimated correlation matrices obtained under two independent MCMC chains.

Finally, we investigate the computational time of our algorithm as a function of the number of variables q and sample size n. Plots in Fig. 13 summarize the behavior of the running time (averaged over 40 replicates) per iteration, as a function of $q\in \{5,10,20,50,100\}$ for $n = 500$, and as a function of $n\in \{50,100,200,500,1000\}$ for $q = 20$. Results were obtained on a PC Intel(R) Core(TM) i7-8550U 1,80 GHz.

Fig. 13

Computational time (in seconds) per iteration, as a function of the number of variables q for fixed $n = 500$ (upper plot) and as a function of the sample size n for fixed $q = 20$ (lower plot), averaged over 40 simulated datasets.