Information Gain and Approaching True Belief

Abstract

Recent years have seen a renewed interest in the philosophical study of information. In this paper a two-part analysis of information gain—objective and subjective—in the context of doxastic change is presented and discussed. Objective information gain is analyzed in terms of doxastic movement towards true belief, while subjective information gain is analyzed as an agent’s expectation value of her objective information gain for a given doxastic change. The resulting expression for subjective information gain turns out to be a familiar one with well-known formal properties: the Kullback–Leibler divergence. The two notions of information are discussed and the suggested measure of subjective information gain is then compared with the widely held view that information gain equals uncertainty reduction.

Appendix: Proof of Theorem 1

Below I show that there is a simple functional (1) that satisfies the requirements I1 through I5 (though I attempt no uniqueness proof), if the following conventions are assumed:

Convention 1

log₂(0/0) = 0

Convention 2

log₂(a/0) = ∞, a > 0

Convention 3

log₂(0/a) = log₂0 = −∞, a > 0

$$ {\text{I}}_{\text{H}} \left( {p,q} \right) = { \log }_{ 2} \left( {q\left( t \right)/p\left( t \right)} \right) $$

(7)

Theorem 1

Assuming Conventions 1, 2 and 3, the functional (1) satisfies the requirements I1–I5.

Satisfaction of I1

For p(t) > 0, the functional log₂(q(t)/p(t)) is greater than 0 if and only if q(t)/p(t) is greater than 1, and this is the case if and only if p(t) < q(t). When p(t) = 0, log₂(q(t)/p(t)) is greater than 0 if and only if p(t) < q(t) (according to Conventions 1 and 2). We can therefore conclude that log₂(q(t)/p(t)) > 0 iff p(t) < q(t).

Satisfaction of I2

The functional log₂(q(t)/p(t)) is less than 0 if and only if either (1) p(t) > q(t) = 0 (Convention 3) or (2) p(t) > q(t) > 0. We can conclude that log₂(q(t)/p(t)) < 0 iff p(t) > q(t).

Satisfaction of I3

The functional log₂(q(t)/p(t)) equals 0 if and only if either (1) q(t)/p(t) equals 1, which is the case if and only if p(t) = q(t) > 0 or (2) p(t) = q(t) = 0 (according to Convention 1). So log₂(q(t)/p(t)) equals 0 if and only if p(t) = q(t).

Satisfaction of I4

Let t _A be the true member of A and t _B the same for B. We then also have <t _A, t _B> as the true member of A × B. By assumption we have that $p^{\prime\prime} \left( {t_{A} ,t_{B} } \right) = p\left( {t_{A} } \right)p^\prime \left( {t_{B} } \right){\text{ and}}\quad q^{\prime \prime} \left( {t_{A} ,t_{B} } \right) = q\left( {t_{A} } \right)q^\prime \left( {t_{B} } \right)$. The following equality then holds when ${\text{I}}_{\text{H}} \left( {p,q} \right) = \log_{2} \left( {q\left( t \right)/p\left( t \right)} \right)$:

$$ {\text{I}}_{\text{A}} \left( {p,q} \right) + {\text{I}}_{\text{B}} \left( {p^\prime ,q^\prime } \right) = { \log }_{ 2} \left( {q\left( {t_{\text{A}} } \right)/p\left( {t_{\text{A}} } \right)} \right) + { \log }_{ 2} \left( {q^\prime \left( {t_{\text{B}} } \right)/p^\prime \left( {t_{\text{B}} } \right)} \right) = { \log }_{ 2} \left( {q^{\prime \prime} \left( {t_{\text{A}} ,t_{\text{B}} } \right)/p^{\prime \prime} \left( {t_{\text{A}} ,t_{\text{B}} } \right)} \right) = {\text{I}}_{{{\text{A}} \times {\text{B}}}} \left( {p^{\prime \prime} ,q^{\prime \prime} } \right) $$

Satisfaction of I5

The proof is trivial. Inserting q(t) = 1 into log₂(q(t)/p(t)) gives log₂(1/p(t)), or −log₂ p(t).

This concludes the proof.