Histogram-based embedding for learning on statistical manifolds

Abstract

A novel binning and learning framework is presented for analyzing and applying large data sets that have no explicit knowledge of distribution parameterizations, and can only be assumed generated by the underlying probability density functions (PDFs) lying on a nonparametric statistical manifold. For models’ discretization, the uniform sampling-based data space partition is used to bin flat-distributed data sets, while the quantile-based binning is adopted for complex distributed data sets to reduce the number of under-smoothed bins in histograms on average. The compactified histogram embedding is designed so that the Fisher–Riemannian structured multinomial manifold is compatible to the intrinsic geometry of nonparametric statistical manifold, providing a computationally efficient model space for information distance calculation between binned distributions. In particular, without considering histogramming in optimal bin number, we utilize multiple random partitions on data space to embed the associated data sets onto a product multinomial manifold to integrate the complementary bin information with an information metric designed by factor geodesic distances, further alleviating the effect of over-smoothing problem. Using the equipped metric on the embedded submanifold, we improve classical manifold learning and dimension estimation algorithms in metric-adaptive versions to facilitate lower-dimensional Euclidean embedding. The effectiveness of our method is verified by visualization of data sets drawn from known manifolds, visualization and recognition on a subset of ALOI object database, and Gabor feature-based face recognition on the FERET database.

Appendices

Appendix A

Proof of Lemma 1. For any s, we need to prove

$$\hbox{max} \left\{ {{\mathbb{E}}\left( {\left\| {\varOmega_{{k_{s} }}^{s} } \right\|} \right),s = 1, \ldots ,d;1 \le k_{s} \le n + 1} \right\} \to 0\;\text{as} \, n \to \infty ,$$

(29)

where $\left\| {\varOmega_{{k_{s} }}^{s} } \right\| = \left| {Y_{{\left( {k_{s} + 1} \right)}}^{s} - Y_{{\left( {k_{s} } \right)}}^{s} } \right|$, and $0 = Y_{(0)}^{s} , \ldots ,Y_{(k)}^{s} ,Y_{(k + 1)}^{s} , \ldots ,Y_{(n)}^{s} ,Y_{(n + 1)}^{s} = 1$ are the order statistics of $0 { = }Y_{0}^{s} ,Y_{1}^{s} , \ldots ,Y_{n}^{s} ,Y_{n + 1}^{s} = 1.$ Since, the density function of k-th order statistic $Y_{(k)}^{s}$ is

$$f_{k} \left( y \right) = \frac{n!}{(k - 1)!(n - k)!} \cdot y^{k - 1} \left( {1 - y} \right)^{n - k} ,\;\;\left( {0 \le y \le 1,}\quad \right.\left. {1 \le k \le n} \right),$$

(30)

one can obtain

$${\mathbb{E}}\left( {Y_{(k)}^{s} } \right) = \frac{{\varGamma \left( {n + 1} \right) \cdot \varGamma \left( {k + 1} \right)}}{{\varGamma (k) \cdot \varGamma \left( {n + 2} \right)}} = \frac{k}{n + 1},$$

(31)

where $1 \le k \le n,$ and $\varGamma ( \cdot )$ is the Gamma function. So ${\mathbb{E}}\left| {\left( {Y_{(k)}^{s} } \right) - \left( {Y_{(k - 1)}^{s} } \right)} \right| = {\mathbb{E}}\left( {Y_{(k)}^{s} } \right) - {\mathbb{E}}\left( {Y_{(k - 1)}^{s} } \right) = \frac{1}{n + 1}$, and $\mathop {\sup }\limits_{1 < k \le n} {\mathbb{E}}\left| {\left( {Y_{(k)}^{s} } \right) - \left( {Y_{(k - 1)}^{s} } \right)} \right| \to 0,\;\;\left( {n \to \infty } \right),$ that is,

$$\left\| {\varOmega_{{k_{s} }}^{s} } \right\| = \left| {\left( {Y_{{(k_{s} )}}^{s} } \right) - \left( {Y_{{(k_{s} - 1)}}^{s} } \right)} \right|\mathop{\longrightarrow}\limits_{}^{L_{1}}0\;\;\left( {n \to \infty } \right),\quad{\text{as}}\;1 < k_{s} \le n.$$

(32)

In addition, we can obtain

$${\mathbb{E}}\left| {\left( {Y_{(1)}^{s} } \right) - \left( {Y_{(0)}^{s} } \right)} \right| = {\mathbb{E}}\left| {\left( {Y_{(1)}^{s} } \right)} \right| = \frac{1}{n + 1} \to 0\;\;\left( {n \to \infty } \right),$$

(33)

and

$${\mathbb{E}}\left| {\left( {Y_{(n + 1)}^{s} } \right) - \left( {Y_{(n)}^{s} } \right)} \right|{ = {\mathbb{E}}}\left( {Y_{(n + 1)}^{s} } \right) - {\mathbb{E}}\left( {Y_{(n)}^{s} } \right) = 1 - \frac{n}{n + 1} \to 0,\quad \left( {n \to \infty } \right).$$

(34)

These results show that for any fixed s,

$$\left\| {\varOmega_{{k_{s} }}^{s} } \right\|\mathop{\longrightarrow}\limits_{}^{L_{1}}0\;\; \quad \left( {n \to \infty } \right),k_{s} = 1, \ldots ,n + 1,$$

(35)

namely,

$$\mathop {\hbox{max} }\limits_{s} \left\{ {\left\| {\varOmega_{{k_{s} }}^{i} } \right\|,k_{s} = 1, \ldots ,n + 1} \right\}\mathop{\longrightarrow}\limits_{}^{L_{1}}0\; \quad \left( {n \to \infty } \right).$$

(36)

It completes the proof. □

Appendix B

Proof of Lemma 2. For $\forall x \in {\mathcal{X}}$, let $C\left( {x,\varvec{Y}_{n} } \right) \subset \aleph \left( {\varvec{Y}_{n} } \right)$ be the minimum covering of $\left[ {{\mathbf{0}},x} \right]$, and $\delta_{ + } \left( {x,\varvec{Y}_{n} } \right)$ be the relative complement set of $\left[ {{\mathbf{0}},x} \right]$ in $C\left( {x,\varvec{Y}_{n} } \right)$, that is,

$$\delta_{ + } \left( {x,\varvec{Y}_{n} } \right) = \left\{ {z:z \in C\left( {x,\varvec{Y}_{n} } \right) \wedge z \notin \left[ {{\mathbf{0}},x} \right]} \right\}.$$

(37)

By the definitions of $F_{{H_{i} \left( {\varvec{Y}_{n} } \right)}} \left( x \right)$ and $F_{{X_{i} }} \left( x \right)$, we can obtain

$$F_{{H_{i} \left( {\varvec{Y}_{n} } \right)}} \left( x \right) - F_{{X_{i} }} \left( x \right) = \sum\limits_{j = 1}^{{n_{i} }} {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right\}}} } ,$$

(38)

and then

$${\mathbb{E}}\left( {\left| {F_{{H_{i} \left( {\varvec{Y}_{n} } \right)}} \left( x \right) - F_{{X_{i} }} \left( x \right)} \right|} \right) = {\mathbb{E}}\left( {\sum\limits_{j = 1}^{{n_{i} }} {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right\}}} } } \right).$$

(39)

Because $x_{i1} , \ldots ,x_{{in_{i} }}$ are i.i.d., the expectation of the right side of above equation can be expressed as

$$\sum\limits_{j = 1}^{{n_{i} }} {\left( {{\mathbb{E}}\left( {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right\}}} } \right)} \right)} = n_{i} \cdot \left( {{\mathbb{E}}\left( {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right\}}} } \right)} \right)$$

(40)

Applying the Fubini theorem [48], we can get

$${\mathbb{E}}\left( {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right\}}} } \right) = \int\limits_{{(\mathop {\mathcal{X}}\limits^{o} )^{n} }} {\left( {{\mathbb{E}}_{i} \left( {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{y}_{n} } \right)} \right\}}} } \right)} \right)\prod\limits_{j = 1}^{n} {\text{d}y_{j} } } .$$

(41)

For $p_{i} \left( x \right)$ is continuous on ${\mathcal{X}}$, so exist $M_{i} \in \left( {0,1} \right]$, making $0 \le p_{i} \left( x \right) \le M_{i}$ for all $x$, and then

$$\begin{gathered} {\mathbb{E}}\left( {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right\}}} } \right) = \int\limits_{{(\mathop {\mathcal{X}}\limits^{o} )^{n} }} {\left( {{\mathbb{E}}_{i} \left( {1_{{\left\{ {x_{ij} \in \delta_{ + } \left( {x,\varvec{y}_{n} } \right)} \right\}}} } \right)} \right)\prod\limits_{j = 1}^{n} {\text{d}y_{j} } } \hfill \\ \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \le \int\limits_{{(\mathop {\mathcal{X}}\limits^{o} )^{n} }} {\left( {M_{i} \cdot \int\limits_{{\delta_{ + } \left( {x,\varvec{y}_{n} } \right)}} {\text{dz}} } \right)} \prod\limits_{j = 1}^{n} {\text{d}y_{j} } \hfill \\ \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; = M_{i} \cdot {\mathbb{E}}_{{\varvec{Y}_{n} }} \left( {\text{Vol}\left( {\delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right)} \right) \hfill \\ \end{gathered}$$

(42)

where ${\mathbb{E}}_{i}$,${\mathbb{E}}_{{\varvec{Y}_{n} }}$ and ${\mathbb{E}}$are the expectation operators that correspond to the distributions determined by PDFs $p_{i} \left( x \right)$, $p_{{\varvec{Y}_{n} }} \left( {\varvec{y}_{n} } \right) = 1$ and $p_{{\varvec{x}_{i1} \text{,}\varvec{Y}_{n} }} \left( {x\text{,}\varvec{y}_{n} } \right)$ respectively; $\text{Vol}\left( \cdot \right)$ is the volume operator for some area of ${\mathcal{X}}$. For any $x \in {\mathcal{X}}$and $\varvec{Y}_{n} \in U_{n} \left( {\mathcal{X}} \right)$, using the above definition, we can find

$$\delta_{ + } \left( {x,\varvec{Y}_{n} } \right) \subset \left[ {x^{1} ,x^{1} + \Delta \left( {\varvec{Y}_{n} } \right)} \right] \times \ldots \times \left[ {x^{d} ,x^{d} + \Delta \left( {\varvec{Y}_{n} } \right)} \right] \cap {\mathcal{X}}\;\left( {{\text{a}}.{\text{e}}.} \right).$$

(43)

So $\text{Vol}\left( {\delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right) \le \left[ {\Delta \left( {\varvec{Y}_{n} } \right)} \right]^{d}$ (a.e.). According to lemma 1, we can obtain

$$\text{Vol}\left( {\delta_{ + } \left( {x,\varvec{Y}_{n} } \right)} \right)\mathop{\longrightarrow}\limits_{}^{L_{1}}0\;\; \quad \left( {n \to \infty } \right).$$

(44)

It follows the conclusion. □

Appendix C

Proof of Proposition 1. According to the Glivenko–Cantelli theorem [48], one can get

$$\sup_{x} \left| {F_{{X_{i} }} \left( x \right) - F_{i} \left( x \right)} \right|\mathop{\longrightarrow}\limits_{}^{a.s.}0\; \quad \left( {n_{i} \to \infty } \right),$$

(45)

which implies $F_{{X_{i} }} \left( x \right)$ converges to $F_{i} \left( x \right)$ in probability as $n_{i} \to \infty$. Lemma 2 shows $F_{{H_{i} \left( {\varvec{Y}_{n} } \right)}} \left( x \right)$ converges to $F_{{X_{i} }} \left( x \right)$ in probability as $n \to \infty$. So the conclusion holds □

Appendix D

Proof of Proposition 2. We first take $\varvec{y} = \left( {y_{1} , \ldots ,y_{n} } \right)$ as n real variables in $(\mathop {\mathcal{X}}\limits^{o} )^{n} = \left( {{\mathbf{0}}\text{,}{\mathbf{1}}} \right)^{n}$. Based on previous definitions, for data set $X_{i}$ of X, $n_{{k_{1} , \ldots ,k_{d} }}^{i}$ is continuous almost everywhere (a.e.) w.r.t. $\varvec{y}$ for fixed $k_{1} , \ldots ,k_{d}$. Further $H_{i} \left( \varvec{y} \right)$ defined by Eq. (16). is also a.e. continuous in $(\mathop {\mathcal{X}}\limits^{o} )^{n}$ w.r.t. $\varvec{y}$ for all $k_{1} , \ldots ,k_{d}$. Since the function $\text{dist}_{g} \left( { \cdot , \cdot } \right)$ described by Eq. (6) is continuous on${\bar{\mathcal{P}}}_{m} \times {\bar{\mathcal{P}}}_{m}$, we can get $\text{dist}_{g} \left( {H_{i} \left( \varvec{y} \right),H_{j} \left( \varvec{y} \right)} \right)$ a.e. continuous w.r.t. y when another data set $X_{j}$ of X is also considered. For $\varvec{Y}_{n}^{(1)} , \ldots ,\varvec{Y}_{n}^{(T)}$ are i.i.d., so $\text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n}^{(1)} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n}^{(1)} }} \left( {X_{j} } \right)} \right), \ldots ,\text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n}^{(T)} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n}^{(T)} }} \left( {X_{j} } \right)} \right)$ are i.i.d. almost surely, in addition, we can obtain

$${\mathbb{E}}\left( {\text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{j} } \right)} \right)} \right)\varvec{ = }{\mathbb{E}}\left( {\text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n} }} \left( {X_{j} } \right)} \right)} \right),\;\;t = 1, \ldots ,T,$$

(46)

where $\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( \cdot \right)$ is defined in Eq. (17). The strong law of large numbers [48] implies

$$T^{ - 1} \cdot \sum\limits_{t = 1}^{T} {\text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{j} } \right)} \right)} \mathop{\longrightarrow}\limits_{}^{\text{a}\text{.s}}{\mathbb{E}}\left( {\text{dist}_{g} \left( {H_{i} \left( {\varvec{Y}_{n} } \right),H_{j} \left( {\varvec{Y}_{n} } \right)} \right)} \right)\;\;\left( {T \to \infty } \right).$$

(47)

Due to the continuity of $\text{dist}_{g} \left( { \cdot , \cdot } \right)$, we can obtain

$$\text{dist}_{g} \left( {H_{i}^{\varepsilon } \left( {\varvec{Y}_{n}^{(t)} } \right),H_{j}^{\varepsilon } \left( {\varvec{Y}_{n}^{(t)} } \right)} \right) \to \text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{j} } \right)} \right)\;\;\;\left( {\varepsilon \to 0} \right),\;t = 1, \ldots ,T.$$

(48)

Thus, we can get $\text{Diss}\left( {\bar{H}_{i,n,T}^{\varepsilon } ,\bar{H}_{j,n,T}^{\varepsilon } } \right) \to T^{ - 1} \cdot \sum\nolimits_{t = 1}^{T} {\text{dist}_{g} \left( {\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{i} } \right),\varphi_{{\varvec{Y}_{n}^{(t)} }} \left( {X_{j} } \right)} \right)}$as $\varepsilon \to 0$, which follows the conclusion. □