Basic statistics for distributional symbolic variables: a new metric-based approach

Abstract

In data mining it is usual to describe a group of measurements using summary statistics or through empirical distribution functions. Symbolic data analysis (SDA) aims at the treatment of such kinds of data, allowing the description and the analysis of conceptual data or of macrodata summarizing classical data. In the conceptual framework of SDA, the paper aims at presenting new basic statistics for distribution-valued variables, i.e., variables whose realizations are distributions. The proposed measures extend some classical univariate (mean, variance, standard deviation) and bivariate (covariance and correlation) basic statistics to distribution-valued variables, taking into account the nature and the variability of such data. The novel statistics are based on a distance between distributions: the $\ell _2$ Wasserstein distance. A comparison with other univariate and bivariate statistics presented in the literature points out some relevant properties of the proposed ones. An application on a clinic dataset shows the main differences in terms of interpretation of results.

Appendix A: Proof of the decomposition of the \(\ell _2\) squared Wasserstein distance

Let $\phi _i$ and $\phi _{i'}$ be two density functions having finite the first two moments. The $\phi _i$ density function is in a one-to-one correspondence with the cumulative distribution function $\varvec{\varPhi }_i$ and the quantile function $\varvec{\varPhi }_i^{-1}$ (the inverse of the distribution function). The expected value of $\phi _i$ is denoted by $\mu _i$ and the standard deviations with $\sigma _i$. In this appendix we prove the result in Eq. (15).

First of all we note that

$$\begin{aligned} {\mu _i} = \int \limits _{ - \infty }^{ + \infty } {y\cdot {\phi _i}(y)dy} = \int \limits _{ - \infty }^{ + \infty } {yd{\varvec{\varPhi } _i}(y)} = \int \limits _{ - \infty }^{ + \infty } {\varvec{\varPhi } _i^{ - 1}({\varvec{\varPhi } _i}(y))d{\varvec{\varPhi } _i}(y)} = \int \limits _0^1 {\varvec{\varPhi } _i^{ - 1}(t)dt},\nonumber \\ \end{aligned}$$

(38)

where $t = \varvec{\varPhi }(y)$, $\varvec{\varPhi }(-\infty )=0$, $\varvec{\varPhi }(+\infty )=1$ and $y = \varvec{\varPhi }^{ - 1} (\varvec{\varPhi }(y)) = \varvec{\varPhi }^{ - 1} (t)$. Analogously, for $\sigma ^2$ we have that:

$$\begin{aligned} {\sigma _i}^2&= \int \limits _{ - \infty }^{ + \infty } {{y^2}{\phi _i}(y)dy - \mu _i^2} = \int \limits _{ - \infty }^{ + \infty } {{{\left[ {\varvec{\varPhi } _i^{ - 1}({\varvec{\varPhi } _i}(y))} \right] }^2}d{\varvec{\varPhi } _i}(y)} - \mu _i^2\nonumber \\&= \int \limits _0^1 {{{\left[ {\varvec{\varPhi } _i^{ - 1}(t)} \right] }^2}dt - \mu _i^2} . \end{aligned}$$

(39)

We develop the squared term of the distance, and using Eqs. (38) and (39) we obtain:

$$\begin{aligned}&d_W^2\left( {{\phi _i}(y),{\phi _{i'}}(y)} \right) = \displaystyle \int \limits _0^1 {{{\left[ {\varvec{\varPhi } _i^{ - 1}(t) - \varvec{\varPhi } _{i'}^{ - 1}(t)} \right] }^2}dt} = \int \limits _0^1 {{{\left[ {\varvec{\varPhi } _i^{ - 1}(t)} \right] }^2}dt} + \int \limits _0^1 {{{\left[ {\varvec{\varPhi } _{i'}^{ - 1}(t)} \right] }^2}dt} \nonumber \\&\quad - 2\displaystyle \int \limits _0^1 {\varvec{\varPhi } _i^{ - 1}(t)\cdot \varvec{\varPhi } _{i'}^{ - 1}(t)dt} = \sigma _i^2 + \mu _i^2 + \sigma _{i'}^2 + \mu _{i'}^2 - 2\int \limits _0^1 {\varvec{\varPhi } _i^{ - 1}(t)\cdot \varvec{\varPhi } _{i'}^{ - 1}(t)dt} \end{aligned}$$

(40)

Now we introduce the following quantity:

$$\begin{aligned} \begin{array}{c} {\rho _{i,i'}} = \frac{{\int \nolimits _0^1 {\left( {\varvec{\varPhi } _i^{ - 1}(t) - {\mu _i}} \right) \left( {\varvec{\varPhi } _{i'}^{ - 1}(t) - {\mu _{i'}}} \right) dt} }}{{\sqrt{\left[ {\int \nolimits _0^1 {{{\left( {\varvec{\varPhi } _i^{ - 1}(t) - {\mu _i}} \right) }^2}dt} } \right] \left[ {\int \nolimits _0^1 {{{\left( {\varvec{\varPhi } _{i'}^{ - 1}(t) - {\mu _{i'}}} \right) }^2}dt} } \right] } }} = \frac{{\int \nolimits _0^1 {\varvec{\varPhi } _i^{ - 1}(t)\varvec{\varPhi } _{i'}^{ - 1}(t)dt} - {\mu _i}{\mu _{i'}}}}{{{\sigma _i}{\sigma _{i'}}}} \end{array} \end{aligned}$$

(41)

that is the correlation of two series of data where each couple of observations is represented respectively by the $t$th quantile of the first distribution and the $t-th$ quantile of the second. In this sense we may consider it as the correlation between quantile functions represented by the curve of the infinite quantile points in a Q–Q plot. It is worth noting that, if $\sigma _i$ and $\sigma _{i'}$ are positive, $0 < \rho _{i,i'} \le 1$ and is equal to 1 when the two standardized series of quantiles are the same, or, in other words, when the two distributions are identical except for the means and the standard deviations. Using the last term of $\rho _{i,i'}$ in Eq. (41), we observe that

$$\begin{aligned} {\int \limits _0^1 {\varvec{\varPhi } _i^{ - 1}(t)\cdot \varvec{\varPhi }_{i'}^{ - 1}(t)dt} }=\rho _{i,i'}\sigma _i\sigma _{i'} + {\mu _i}{\mu _{i'}}. \end{aligned}$$

Thus, we continue developing Eq.(40) as follows

$$\begin{aligned} d_W^2\left( {{\phi _i},{\phi _{i'}}} \right)&= \sigma _i^2 + \mu _i^2 + \sigma _{i'}^2 + \mu _{i'}^2 - 2\left[ {{\rho _{i,i'}}{\sigma _i}{\sigma _{i'}} + {\mu _i}{\mu _{i'}}} \right] \nonumber \\&= \left( {\mu _i^2 + \mu _{i'}^2 - 2{\mu _i}{\mu _{i'}}} \right) + \sigma _i^2 + \sigma _{i'}^2 - 2{\rho _{i,i'}}{\sigma _i}{\sigma _{i'}}. \end{aligned}$$

(42)

Finally, adding and subtracting $2\sigma _i \sigma _{i'}$ we obtain Eq. (15):

$$\begin{aligned} d_W^2\left( {{\phi _i},{\phi _{i'}}} \right)&= \left( {\mu _i^2 + \mu _{i'}^2 - 2{\mu _i}{\mu _{i'}}} \right) + \sigma _i^2 + \sigma _{i'}^2 - 2{\rho _{i,i'}}{\sigma _i}{\sigma _{i'}} + 2{\sigma _i}{\sigma _{i'}} - 2{\sigma _i}{\sigma _{i'}} \nonumber \\&= {\left( {{\mu _i} - {\mu _{i'}}} \right) ^2} + \left( {\sigma _i^2 + \sigma _{i'}^2 - 2{\sigma _i}{\sigma _{i'}}} \right) + 2{\sigma _i}{\sigma _{i'}} - 2{\rho _{i,i'}}{\sigma _i}{\sigma _{i'}} \nonumber \\&= {\left( {{\mu _i} - {\mu _{i'}}} \right) ^2} + {\left( {{\sigma _i} - {\sigma _{i'}}} \right) ^2} + 2{\sigma _i}{\sigma _{i'}}\left( {1 - {\rho _{i,i'}}} \right) . \end{aligned}$$

(43)