Some Universal Insights on Divergences for Statistics, Machine Learning and Artificial Intelligence

Abstract

Dissimilarity quantifiers such as divergences (e.g. Kullback–Leibler information, relative entropy) and distances between probability distributions are widely used in statistics, machine learning, information theory and adjacent artificial intelligence (AI). Within these fields, in contrast, some applications deal with divergences between other-type real-valued functions and vectors. For a broad readership, we present a correspondingly unifying framework which – by its nature as a “structure on structures” – also qualifies as a basis for similarity-based multistage AI and more humanlike (robustly generalizing) machine learning. Furthermore, we discuss some specificalities, subtleties as well as pitfalls when e.g. one “moves away” from the probability context. Several subcases and examples are given, including a new approach to obtain parameter estimators in continuous models which is based on noisy divergence minimization.

Appendix: Proofs

Proof of Theorem 4. Assertion (1) and the “if-part” of (2) follow immediately from Theorem 1 which uses less restrictive assumptions. In order to show the “only-if” part of (2) (and the “if-part” of (2) in an alternative way), one can use the straightforwardly provable fact that the Assumption 2 implies

$$\begin{aligned}& \overline{\mathbbm {w}_{3} \cdot \psi _{\phi ,c}}(x,s,t) \ = \ 0 \qquad \text {if and only if} \qquad s \ = \ t \end{aligned}$$

(136)

for all $s \in \mathscr {R}\big (\frac{P}{M_{1}}\big )$, all $t \in \mathscr {R}\big (\frac{Q}{M_{2}}\big )$ and $\lambda$-a.a. $x \in \mathscr {X}$. To proceed, assume that $D^{c}_{\phi ,M_{1},M_{2},\mathbbm {M}_{3},\lambda }(P,Q) = 0$, which by the non-negativity of $\overline{\mathbbm {w}_{3} \cdot \psi _{\phi ,c}}(\cdot ,\cdot )$ implies that $\overline{\mathbbm {w}_{3} \cdot \psi _{\phi ,c}}\big (\frac{p(x)}{m_{1}(x)},\frac{q(x)}{m_{2}(x)}\big ) = 0$ for $\lambda$-a.a. $x \in \mathscr {X}$. From this and the “only-if” part of (136), we obtain the identity $\frac{p(x)}{m_1(x)}=\frac{q(x)}{m_2(x)} \ \text {for}\, \lambda \text {-a.a.}\, x \in \mathscr {X}$.    $\square$

Proof of Theorem 5. Consistently with Theorem 1 (and our adaptions) the “if-part” follows from (51). By our above investigations on the adaptions of the Assumptions 2 to the current context, it remains to investigate the “only-if” part (2) for the following four cases (recall that $\phi$ is strictly convex at $t=1$): (ia) $\phi$ is differentiable at $t=1$ (hence, c is obsolete and $\phi _{+,c}^{\prime }(1)$ collapses to $\phi ^{\prime }(1)$) and the function $\phi$ is affine linear on [1, s] for some $s \in \mathscr {R}\big (\frac{P}{Q}\big )\backslash [a,1]$; (ib) $\phi$ is differentiable at $t=1$, and the function $\phi$ is affine linear on [s, 1] for some $s \in \mathscr {R}\big (\frac{P}{Q}\big )\backslash [1,b]$; (ii) $\phi$ is not differentiable at $t=1$, $c=1$, and the function $\phi$ is affine linear on [1, s] for some $s \in \mathscr {R}\big (\frac{P}{Q}\big )\backslash [a,1]$; (iii) $\phi$ is not differentiable at $t=1$, $c=0$, and the function $\phi$ is affine linear on [s, 1] for some $s \in \mathscr {R}\big (\frac{P}{Q}\big )\backslash [1,b]$. It is easy to see from the strict convexity at 1 that for (ii) one has $\phi (0) + \phi _{+,1}^{\prime }(1) - \phi (1) >0$, whereas for (iii) one gets $\phi ^{*}(0) -\phi _{+,0}^{\prime }(1) >0$; furthermore, for (ia) there holds $\phi (0) + \phi ^{\prime }(1) - \phi (1) >0$ and for (ib) $\phi ^{*}(0) -\phi ^{\prime }(1) >0$. Let us first examine the situations (ia) respectively (ii) under the assumptive constraint $D^{c}_{\phi ,\mathbbm {Q},\mathbbm {Q},\mathbbm {R}\cdot \mathbbm {Q},\lambda }(\mathbbm {P},\mathbbm {Q})= 0$ with $c=1$ respectively (in case of differentiability) obsolete c, for which we can deduce from (51)

$$\begin{aligned}& \textstyle \textstyle 0 = D^{c}_{\phi ,\mathbbm {Q},\mathbbm {Q},\mathbbm {R}\cdot \mathbbm {Q},\lambda }(\mathbbm {P},\mathbbm {Q}) \nonumber \\& \textstyle \geqslant \int _{{\mathscr {X}}} \mathbbm {r}(x) \cdot \big [ \mathbbm {q}(x) \cdot \phi \big ( { \frac{\mathbbm {p}(x)}{\mathbbm {q}(x)}}\big ) - \mathbbm {q}(x) \cdot \phi \big ( 1 \big ) - \phi _{+,c}^{\prime } \big ( 1 \big ) \cdot \big ( \mathbbm {p}(x) - \mathbbm {q}(x) \big ) \big ] \, \nonumber \\[-0.1cm]&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \cdot \varvec{1}_{]0,\infty [}\big (\mathbbm {p}(x) \big ) \cdot \varvec{1}_{]\mathbbm {p}(x),\infty [}\big (\mathbbm {q}(x) \big ) \, \mathrm {d}\lambda (x) \nonumber \\& \textstyle + \big [ \phi (0) + \phi _{+,c}^{\prime }(1) - \phi (1) \big ] \cdot \int _{{\mathscr {X}}} \mathbbm {r}(x) \cdot \mathbbm {q}(x) \cdot \varvec{1}_{\{0\}}\big (\mathbbm {p}(x)\big ) \cdot \varvec{1}_{]\mathbbm {p}(x),\infty [}\big (\mathbbm {q}(x)\big ) \, \mathrm {d}\lambda (x) \, \geqslant 0 , \nonumber \end{aligned}$$

and hence $\int _{{\mathscr {X}}} \varvec{1}_{]\mathbbm {p}(x),\infty [}\big (\mathbbm {q}(x)\big ) \cdot \mathbbm {r}(x) \, \mathrm {d}\lambda (x) \, = 0$. From this and (55) we obtain

$$\begin{aligned}& \textstyle 0 \, = \, \int _{{\mathscr {X}}} \, \big ( \mathbbm {p}(x) \, - \, \mathbbm {q}(x) \big ) \cdot \mathbbm {r}(x) \, \mathrm {d}\lambda (x) \, = \, \int _{{\mathscr {X}}} \, \big ( \mathbbm {p}(x) - \mathbbm {q}(x) \big ) \, \cdot \,\varvec{1}_{]\mathbbm {q}(x),\infty [}\big (\mathbbm {p}(x)\big ) \, \cdot \, \mathbbm {r}(x) \, \mathrm {d}\lambda (x) \qquad \ \ \nonumber \end{aligned}$$

and therefore $\int _{{\mathscr {X}}} \varvec{1}_{]\mathbbm {q}(x),\infty [}\big (\mathbbm {p}(x)\big ) \cdot \mathbbm {r}(x) \, \mathrm {d}\lambda (x) \, = 0$. Since for $\lambda$-a.a. $x\in \mathscr {X}$ we have $\mathbbm {r}(x) >0$, we arrive at $\mathbbm {p}(x) =\mathbbm {q}(x)$ for $\lambda$-a.a. $x\in \mathscr {X}$. The remaining cases (ib) respectively (iii) can be treated analogously.        $\square$