Scoring Bayesian networks of mixed variables

Abstract

In this paper we outline two novel scoring methods for learning Bayesian networks in the presence of both continuous and discrete variables, that is, mixed variables. While much work has been done in the domain of automated Bayesian network learning, few studies have investigated this task in the presence of both continuous and discrete variables while focusing on scalability. Our goal is to provide two novel and scalable scoring functions capable of handling mixed variables. The first method, the Conditional Gaussian (CG) score, provides a highly efficient option. The second method, the Mixed Variable Polynomial (MVP) score, allows for a wider range of modeled relationships, including nonlinearity, but it is slower than CG. Both methods calculate log likelihood and degrees of freedom terms, which are incorporated into a Bayesian Information Criterion (BIC) score. Additionally, we introduce a structure prior for efficient learning of large networks and a simplification in scoring the discrete case which performs well empirically. While the core of this work focuses on applications in the search and score paradigm, we also show how the introduced scoring functions may be readily adapted as conditional independence tests for constraint-based Bayesian network learning algorithms. Lastly, we describe ways to simulate networks of mixed variable types and evaluate our proposed methods on such simulations.

Appendices

Appendix A

Proposition 1

The Conditional Gaussian Score is score equivalent.

Proof

Let $\mathcal {G}_{1}$ and $\mathcal {G}_{2}$ be directed acyclic graphs with Conditional Gaussian scores $\mathcal {S}_{1}$ and $\mathcal {S}_{2}$, respectively. Further, let $\mathcal {G}_{1} \ne \mathcal {G}_{2}$, but $\mathcal {G}_{1}$ and $\mathcal {G}_{2}$ in the same Markov equivalence class. $\square$

Remove all shared local components between $\mathcal {S}_{1}$ and $\mathcal {S}_{2}$ and the corresponding edges in $\mathcal {G}_{1}$ and $\mathcal {G}_{2}$. Call the newly pruned scores $\mathcal {S}_{1}'$ and $\mathcal {S}_{2}'$ and the newly pruned graphs $\mathcal {G}_{1}'$ and $\mathcal {G}_{2}'$, respectively. Note that we have removed all initial unshielded colliders common to both $\mathcal {G}_{1}$ and $\mathcal {G}_{2}$. Additionally, it follows from Meek’s rules that $\mathcal {G}_{1}'$ and $\mathcal {G}_{2}'$ must contain no unshielded colliders since any component which could have become an unshielded collider is necessarily shared between graphs $\mathcal {G}_{1}$ and $\mathcal {G}_{2}$ and thus pruned [14].

Since $\mathcal {G}_{1}'$ and $\mathcal {G}_{2}'$ are acyclic graphs without any unshielded colliders, we can represent them both as a join tree of cliques. Further, they share the same skeleton because they come from the same Markov equivalence class, and thus, they can be represented by the same join tree of cliques.

It follows from Pearl, Probabilistic Reasoning in Intelligent Systems 3.2.4, Theorem 8, that the distribution encoded by $\mathcal {G}_{1}'$ and $\mathcal {G}_{2}'$ can be written as a product of the distributions of the cliques of $\mathcal {G}_{1}'$ and $\mathcal {G}_{2}'$ divided by a product of the distributions of their intersections [16]. Therefore, when we calculate $\mathcal {S}_{1}'$ and $\mathcal {S}_{2}'$, we can use the same ratio of joint distributions to obtain the log likelihood and degrees of freedom terms. Hence $\mathcal {S}_{1}' = \mathcal {S}_{2}'$ and therefore, after adding back the shared local components, we have $\mathcal {S}_{1} = \mathcal {S}_{2}$.

Proposition 2

If the approximating polynomial from Weierstrass Approximation Theorem for the data generating function is a polynomial of degree d, and the maximum-degree polynomial used by MVP is at least d, then the MVP score will be consistent in the large sample limit.

Proof

Let $\varvec{p}$ be the approximating polynomial(s) from Weierstrass Approximation Theorem [11]. Assuming the least squares estimate(s) contains the same polynomials degrees as $\varvec{p}$, the least squares estimate will converge to $\varvec{p}$ as the number of samples $n \rightarrow \infty$ [1]. Therefore, by Weierstrass Approximation Theorem, the least squares estimate(s) will converge to the true data generating function(s). Accordingly, the log likelihood term of MVP will be maximal for any model that has either been correctly specified or over specified, where by over over specified we are referring to a model containing all the true parameters and more. $\square$

However for any over specified model, the parameter penalty will necessarily be larger and hence the MVP score in that case will be lower for the over specified model.

Additionally, in the case of an underspecified model, note that the parameter penalty term is of order $O(\log n)$ while the likelihood term is of order O(n). This means that in the large sample limit, when comparing an underspecified model to any model containing the correctly specified model, the MVP score will be lower for the under specified model since the log likelihood is not maximal while in the other case it is.

Therefore, the MVP score is consistent.

Proposition 3

The approximated values from least squares regression sum to one.

Proof

Let the terms in the below equation be defined according to Sect. 4.3.

$$\begin{aligned} \sum _{h=1}^{d} \varvec{X}_p \hat{\varvec{\beta }}_{h}&= \varvec{X}_p (\varvec{X}_p^{T} \varvec{X}_p)^{-1} \varvec{X}_p^{T} \sum _{h=1}^{d} \varvec{1}_{\{\varvec{y}_p = h\}} \\&= \varvec{X}_p (\varvec{X}_p^{T} \varvec{X}_p)^{-1} \varvec{X}_p^{T} \varvec{1}_p \\&= \varvec{1}_p \end{aligned}$$

For the last step, we use that $\varvec{1}_p$ is in the column space of ${\varvec{X}}_{p}$ and is thus projected to itself. $\square$

Proposition 4

If the approximating polynomial from Weierstrass Approximation Theorem for the data generating function is a polynomial of degree d, and the maximum-degree polynomial used by MVP is at least d, then the least squares approximations for probability mass functions will be strictly nonnegative in the large sample limit.

Proof

Let f be a generating component of a conditional probability mass function. Since the Weierstrass Approximation Theorem is satisfied by the assumptions of MVP, there must exists a polynomial p such that for every $\epsilon > 0$ and all $x \in [a,b]$, we have $|f(x) - p(x)| < \epsilon$ [11]. $\square$

For $x \in [a,b]$ where $p(x) \ge f(x)$, p(x) is trivially nonnegative since $f(x) > 0$.

For $x \in [a,b]$ where $p(x) < f(x)$, let $m = f(x)$ and choose $\epsilon = \frac{m}{2}$. Then,

$$\begin{aligned} |f(x) - p(x)|&< \epsilon \\ f(x) - p(x)&< \epsilon \\ p(x)&> f(x) - \epsilon \\ p(x)&> m - \frac{m}{2} \\ p(x)&> 0 \end{aligned}$$

since $m > 0$.

Assuming the least squares estimate(s) contains the same polynomials degrees as p, the least squares estimate will converge to p as the number of samples $n \rightarrow \infty$ [1]. Thus, as the number of samples $n \rightarrow \infty$, the least squares approximations are strictly nonnegative.

Appendix B

In this appendix, we detail the parameters used for simulation of the data. Each parameter will be followed by the values we used in simulation and a short description. We split the parameters into 3 groups: general parameters used across all simulations, parameters specific to linear simulation, and parameters specific to nonlinear simulation.

1.1 General Parameters

• numRuns: 10 - number of runs

• numMeasures: 100, 500 - number of measured variables

• avgDegree: 2, 4 - average degree of graph

• sampleSize: 200, 1000 - sample size

• minCategories: 2 - minimum number of categories

• maxCategories: 5 - maximum number of categories

• percentDiscrete: 50 - percentage of discrete variables (0 - 100) for mixed data

• differentGraphs: true - true if a different graph should be used for each run

• maxDegree: 5 - maximum degree of the graph

• maxIndegree: 5 - maximum indegree of graph

• maxOutdegree: 5 - maximum outdegree of graph

• coefSymmetric: true - true if negative coefficient values should be considered

1.2 Linear Parameters

• varLow: 1 - low end of variance range

• varHigh: 3 - high end of variance range

• coefLow: 0.05 - low end of coefficient range

• coefHigh: 1.5 - high end of coefficient range

• meanLow: -1 - low end of mean range

• meanHigh: 1 - high end of mean range

1.3 Nonlinear Parameters

• dirichlet: 0.5 - alpha parameter for Dirichlet to draw multinomials

• interceptLow: 1 - low end of intercept range

• interceptHigh: 2 - high end of intercept range

• linearLow: 1.0 - low end of linear coefficient range

• linearHigh: 2.0 - high end of linear coefficient range

• quadraticLow: 0.5 - low end quadratic coefficient range

• quadraticHigh: 1.0 - high end of quadratic coefficient range

• cubicLow: 0.2 - low end of cubic coefficient range

• cubicHigh: 0.3 - high end of cubic coefficient range

• varLow: 0.5 - low end of variance range

• varHigh: 0.5 - high end of variance range