Appendix A: Maximum Entropy
Let us define Q(A,D
0,D
1,…,D
n
) the joint probability distribution maximizing its entropy \(H(Q) = -\sum_{A \in{\mathcal{A}}} Q(D_{0},D_{1},\dots,D_{n})(A) \ln Q(D_{0},D_{1},\dots,D_{n}) (A)\) subject to the following constraints.
-
1.
Q(A,D
0)=Q(A∣D
0)Q(D
0)∝P
0(A), for all \(A \in {\mathcal{A}}\).
-
2.
Q(A,D
0,D
i
)=Q(A∣D
i
)Q(D
i
)Q(D
0)∝P
i
(A), for all \(A \in{\mathcal{A}}\) and all i=1,…,n.
We will first show that
$$Q(A,D_0,D_1,\dots,D_n) \propto P_0(A)^{1-n} \prod_{i=1}^{n}P_i(A), $$
from which the conditional probability
is immediately derived. For ease of notation, we will use ∑
A
as a short notation for \(\sum_{A \in{\mathcal{A}}} \).
Proof
The adequate approach is to use the Lagrange multiplier technique on the objective function
where μ
A
and λ
A,i
are Lagrange multipliers. For finding the solution Q optimizing the constrained problem, we set all partial derivatives to 0. This leads to the system of equations
From Eqs. (54) and (55), we get
$$Q(A,D_0) = e^{-1}\prod_A e^{\mu_A} \propto P_0(A). $$
Similarly, from Eqs. (54) and (56), we get
$$Q(A,D_0,D_i) = Q(A,D_0) \prod_A e^{\lambda_{A,i}} \propto P_i(A),\quad \hbox{for}\ i=1,\dots,n, $$
from which we find
$$\prod_A e^{\lambda_{A,i}} \propto P_i(A)/P_0(A),\quad \hbox{for}\ i=1,\dots,n. $$
Plugging this in Eq. (54) yields
$$Q(A,D_0,D_1,\dots,D_n) \propto P_0(A) \prod_{i=1}^n \frac{P_i(A)}{P_0(A)}. $$
Hence,
□
Appendix B: Conditional Probabilities for the Trinary Event Example
1. Let us first compute the conditional probability
where \(G^{2}_{2}(t,t;\rho)\) is the bivariate cpf of a (0,1) bi-Gaussian random vector with correlation ρ. For symmetry reasons, one has P(I(s′)=2∣I(s)=1)=P(I(s′)=3∣I(s)=1), from which it follows immediately
2. We consider now
3. The picture is slightly more complicated for P(I(s′)=2∣I(s)=2)
There is no closed-form expression for the double integral which must be evaluated numerically. Then P(I(s′)=3∣I(s)=2) is computed as the complement to 1.
4. The conditional probabilities of I(s′) given that I(s)=3 are then obtained by symmetry.