Graphical Models and Message-Passing Algorithms: Some Introductory Lectures

Abstract

Graphical models provide a framework for describing statistical dependencies in (possibly large) collections of random variables. At their core lie various correspondences between the conditional independence properties of a random vector, and the structure of an underlying graph used to represent its distribution. They have been used and studied within many sub-disciplines of statistics, applied mathematics, electrical engineering and computer science, including statistical machine learning and artificial intelligence, communication and information theory, statistical physics, network control theory, computational biology, statistical signal processing, natural language processing and computer vision among others.

Appendix: Triangulation and Equivalent Graph-Theoretic Properties

In this Appendix, we prove Theorem 3 as part of a more general discussion of triangulation and related graph-theoretic properties. Having already defined the notions of triangulations and junction tree, let us now define the closely related notions of decompsable and recursively simplicial. The following notion serves to formalize the “divide-and-conquer” nature of efficient algorithms:

Definition 9

A graph $\mathcal{G} = (\mathcal{V},\mathcal{E})$ is decomposable if either it is complete, or its vertex set $\mathcal{V}$ can be split into the disjoint union of three sets $A \cup B \cup S$ such that (a) $A$ and $B$ are non-empty; (b) the set $S$ separates $A$ and $B$ in $\mathcal{G}$, and is complete (i.e., $(s,t) \in \mathcal{E}$ for all $s,t \in S$); and (c) $A \cup S$ and $B \cup S$ are also decomposable.

Recall from our discussion of the elimination algorithm in Sect. 3.1 that when a vertex is removed from the graph, the algorithm always connects together all of its neighbors, thereby creating additional edges in the reduced graph. The following property characterizes when there is an elimination ordering such that no edges are added by the elimination algorithm.

Definition 10

A vertex is simplicial if its neighbors form a complete subgraph. A non-empty graph is recursively simplicial if it contains a simplicial vertex, and when s is removed, any graph that remains is recursively simplicial.

It should be intuitively clear that these four properties—namely, triangulated, decomposable, recursively simplicial, and having a junction tree—are related. We now show that all four properties are actually equivalent:

Theorem 4

The following properties of an undirected graph $\mathcal{G}$ are all equivalent:

Property (T)::

$\mathcal{G}$ is triangulated.

Property (D)::

$\mathcal{G}$ is decomposable.

Property (R)::

$\mathcal{G}$ is recursively simplicial.

Property (J)::

$\mathcal{G}$ has a junction tree.

We prove the sequence of implications $(T) \Rightarrow (D) \Rightarrow (R) \Rightarrow (J) \Rightarrow (T)$.

(T) $\Rightarrow$ (D): :

We proceed via induction on the graph size $N$. The claim is trivial for $N = 1$, so let us assume it for all graphs with $N$ vertices, and prove that it also holds for any graph $\mathcal{G}$ with $N + 1$ vertices. If $\mathcal{G}$ is complete, then it is certainly decomposable. Moreover, if $\mathcal{G}$ has more than one connected component, each of which is complete, then it is also decomposable. Otherwise, we may assume that at least one connected component of $\mathcal{G}$ is not complete. (Without loss of generality in the argument to follow, we assume that $\mathcal{G}$ has a single connected component which is not complete.) Since $\mathcal{G}$ is not complete, it contains two non-adjacent vertices $a,b$. Let $S$ be a minimal set that separates $a$ and $b$; the set $S$ must be non-empty since $\mathcal{G}$ has a single connected component. Define $A$ as the set of all vertices connected to a in $\mathcal{V}\setminus S$, and set $B:= \mathcal{V}\setminus (A \cup S)$. Clearly, $S$ separates $A$ from $B$ in $\mathcal{G}$.

Now we need to show that $S$ is complete. If $\vert S\vert = 1$, the claim is trivial. Otherwise, for any two distinct vertices $s,t \in \mathcal{S}$, there exist paths $(s,a_{1},\ldots,a_{i},t)$ and $(s,b_{1},\ldots,b_{j},t)$ where $a_{k} \in A$, $b_{k} \in B$ and i, j ≥ 1. (This claim relies on the minimality of $S$: if there did not exist a path from a to s, then vertex s could be removed from $\mathcal{S}$. Similar reasoning applies to establish a path from a to t, and also the paths involving $B$.)

We claim that s and t are joined by an edge. If not, take the path from s to t through $A$ with minimal length, and similarly for $B$. This pair of paths forms a cycle of length at least four, which must have a chord. The chord cannot be in $A$ or $B$, since this would contradict minimality. It cannot be between vertices in $A$ and $B$ since $S$ separates these two sets. Therefore, s and t are joined, and $S$ is complete.

Finally, we need to show that $A \cup S$ and $B \cup S$ are also decomposable. But they must be triangulated, since otherwise $\mathcal{G}$ would not be triangulated, and they have cardinality strictly smaller than $N + 1$, so the result follows by induction.

(D) $\Rightarrow$ (R): :

Proof by induction on graph size $N$. Trivial for $N = 1$. To complete the induction step, we require the following lemma:

Lemma 6

Every decomposable graph with at least two vertices has at least two simplicial vertices. If the graph is not complete, these vertices can be chosen to be non-adjacent.

Proof

Proof by induction on graph size $N$. Trivial for $N = 2$. Consider a decomposable graph with $N + 1$ vertices. If the graph is complete, all vertices are simplicial. Otherwise, decompose the graph into disjoint sets $A$, $B$ and $S$. The subgraphs $A \cup S$ and $B \cup S$ are also chordless, and hence we have two simplicial vertices in $A \cup S$. If $A \cup S$ is not complete, these can be chosen to be non-adjacent. Given that $S$ is complete, one of the vertices can be taken in $A$. Otherwise, if $A \cup S$ is complete, choose any node in $A$. Proceed in a symmetric fashion for $B$. The simplicial vertices thus chosen will not be connected, since $S$ separates $A$ and $B$. □ 

Thus, given a decomposable graph, we can find some simplicial vertex s to remove. We need to show that the remaining graph is also decomposable, so as to apply the induction hypothesis. In particular, we prove that $\mathcal{G}$ decomposable implies that any vertex-induced subgraph $\mathcal{G}[U]$ is also decomposable. We prove this induction on | U | . Trivial for | U |  = 1. Trivially true if $\mathcal{G}$ is complete; otherwise, break into $A \cup S \cup B$. Removing a node from $S$ leaves $S\setminus \{s\}$ complete, and $A \cup S$ and $B \cup S$ decomposable by the induction hypothesis. Removing a node from $A$ does not change $B \cup S$, and either leaves $A$ empty (in which case remainder $B \cup S$ is decomposable), or leaves $A \cup S$ decomposable by induction.

(R) $\Rightarrow$ (J): :

Proof by induction on graph size $N$. Trivial for $N = 1$. Let s be a simplicial vertex, and consider subgraph $\mathcal{G}'$ obtained by removing s. By induction, $\mathcal{G}'$ has a junction tree $\mathcal{T}'$, which we will extend to a junction tree $\mathcal{T}$ for $\mathcal{G}$. Let C′ be a maximal clique in $\mathcal{T}'$ that contains all the neighbors of s; this must exist since s is simplicial. If C′ is precisely the neighbors of s, then we can add s to C′ so as to obtain $\mathcal{T}$, which is a junction tree for $\mathcal{G}$.

If not (i.e., if C′ contains the neighbors of s as a proper subset), then we can add a new clique containing s and its neighbors to $\mathcal{T}'$, with an edge to C′. Since s is in no other clique of $\mathcal{T}$ and $C\setminus \{s\}$ is a subset of $C'$, the tree $\mathcal{T}'$ is a junction tree for $\mathcal{G}$¿

(J) $\Rightarrow$ (T): :

Proof by induction on number of vertices M in junction tree. For M = 1, $\mathcal{G}$ is complete and hence triangulated. Consider a junction tree $\mathcal{T}$ with M + 1 vertices. For a fixed leaf C of $\mathcal{T}$, let C′ be the unique neighbor of C in $\mathcal{T}$, and let $\mathcal{T}'$ be the tree that remains when C is removed.

Step 1::

If C ⊆ C′, then $\mathcal{T}'$ is a junction tree for $\mathcal{G}$, and result follows by induction.

Step 2::

If $C \cap C' \subset C$ (in a strict sense), then consider the subgraph $\mathcal{G}'$ formed by removing the non-empty set $R:= C\setminus C'$ from $\mathcal{V}$. We claim that it is chordal. First, observe that R has an empty intersection with every clique in $\mathcal{T}'$ (using junction tree property). (That is, say $R \cap D\neq 0$ for some clique node D in $\mathcal{T}'$. Then there exists s ∈ C ∩ D, but $s\notin C'$, with violates running intersection.) Follows that $\mathcal{T}'$ is a junction tree for $\mathcal{G}'$, and so $\mathcal{G}'$ is chordal (by applying induction hypothesis).

Step 3::

Now claim that $\mathcal{G}$ is chordal. Any cycle entirely contained in $\mathcal{G}'$ is chordless by induction. If the cycle is entirely within the complete subgraph $\mathcal{G}[C]$, it is also chordless. Any other cycle must intersect R, $C \cap C'$ and $\mathcal{V}\setminus C$. In particular, it must cross $C \cap C'$ twice, and since this set is complete, it has a chord.