Density estimation with distribution element trees

Abstract

The estimation of probability densities based on available data is a central task in many statistical applications. Especially in the case of large ensembles with many samples or high-dimensional sample spaces, computationally efficient methods are needed. We propose a new method that is based on a decomposition of the unknown distribution in terms of so-called distribution elements (DEs). These elements enable an adaptive and hierarchical discretization of the sample space with small or large elements in regions with smoothly or highly variable densities, respectively. The novel refinement strategy that we propose is based on statistical goodness-of-fit and pairwise (as an approximation to mutual) independence tests that evaluate the local approximation of the distribution in terms of DEs. The capabilities of our new method are inspected based on several examples of different dimensionality and successfully compared with other state-of-the-art density estimators.

Appendix: Derivation of MMSE slope estimator

Writing without loss of generality the linear marginal PDF (4) in a simpler form with $x_i\in [0,1]$ and the subscripts skipped, we obtain

$$\begin{aligned} p(x|\theta ) = \left( x - {\textstyle \frac{1}{2}}\right) \theta + 1. \end{aligned}$$

By calculating the mean of random variable X based on this PDF, we obtain $\langle X\rangle = \frac{1}{12}(6 + \theta )$, and therefore, can express the slope parameter in terms of this mean as

$$\begin{aligned} \theta = 6(2\langle X\rangle - 1). \end{aligned}$$

(20)

In the case of a finite ensemble, we estimate the mean with $\langle X\rangle _n = \frac{1}{n}\sum _{j = 1}^n x_j$ and the slope by

$$\begin{aligned} \hat{\theta } = 6c(2\langle X\rangle _n - 1). \end{aligned}$$

Here, c is a correction factor that is determined by minimizing the mean square error (MSE) expressed as

$$\begin{aligned} \langle (\hat{\theta } - \theta )^2\rangle= & {} \left\langle \left[ 6c\left( \frac{2}{n}\sum _{j = 1}^n x_j - 1\right) - \theta \right] ^2\right\rangle \\= & {} \left\langle 36c^2\left( \frac{2}{n}\sum _{j = 1}^n x_j - 1\right) ^2 \right. \nonumber \\&\left. - 12c\left( \frac{2}{n}\sum _{j = 1}^n x_j - 1\right) \theta + \theta ^2\right\rangle \\= & {} 36c^2\left\langle \frac{4}{n^2}\sum _{j = 1}^n\sum _{k = 1}^n x_j x_k - \frac{4}{n}\sum _{j = 1}^n x_j + 1\right\rangle \\&- 12c(2\langle X\rangle - 1)\theta + \theta ^2 \\= & {} 36c^2\left( \frac{4}{n^2}\left\langle \sum _{j = 1}^n\sum _{k = 1}^n x_j x_k\right\rangle - 4\langle X\rangle + 1\right) \\&- 12c(2\langle X\rangle - 1)\theta + \theta ^2 \\= & {} 36c^2\left( \frac{4}{n^2}\left\langle \sum _{j = 1}^n\sum _{\begin{array}{c} k = 1\\ k \ne j \end{array}}^n x_j x_k + \sum _{j = 1}^n x_j^2\right\rangle - 4\langle X\rangle + 1\right) \\&- 12c(2\langle X\rangle - 1)\theta + \theta ^2 \\= & {} 36c^2\left[ \frac{4(n-1)}{n}\langle X\rangle ^2 + \frac{4}{n}\langle X^2\rangle - 4\langle X\rangle + 1\right] \\&- 12c(2\langle X\rangle - 1)\theta + \theta ^2. \end{aligned}$$

To determine the minimum MSE, we set

$$\begin{aligned} \frac{\mathrm {d} }{\mathrm {d} c}\langle (\hat{\theta } - \theta )^2\rangle= & {} 72c\left( \frac{4(n-1)}{n}\langle X\rangle ^2 + \frac{4}{n}\langle X^2\rangle \right. \\&\left. - 4\langle X\rangle + 1\right) \\&- 12(2\langle X\rangle - 1)\theta = 0, \end{aligned}$$

which leads for the correction factor to

$$\begin{aligned} c= & {} \frac{(2\langle X\rangle - 1)n\theta }{6[4(n-1)\langle X\rangle ^2 + 4\langle X^2\rangle - 4n\langle X\rangle + n]} \nonumber \\{}= & {} \frac{(2\langle X\rangle - 1)n\theta }{6(4n\langle X\rangle ^2 - 4\langle X\rangle ^2 + 4\langle X^2\rangle - 4n\langle X\rangle + n])} \nonumber \\{}= & {} \frac{(2\langle X\rangle - 1)n\theta }{6[n(2\langle X\rangle - 1)^2 + 4\langle X^{\prime 2}\rangle ]} \nonumber \\{}= & {} \frac{6n(2\langle X\rangle - 1)^2}{6[n(2\langle X\rangle - 1)^2 + 4\langle X^{\prime 2}\rangle ]}. \end{aligned}$$

(21)

For $n\rightarrow \infty$ the correction factor c goes to one.