Association-Based Optimal Subpopulation Selection for Multivariate Data

Abstract

In the analysis of multivariate data, a useful problem is to identify a subset of observations for which the variables are strongly associated. One example is in driving safety analytics, where we may wish to identify a subset of drivers with a strong association among their driving behavior characteristics. Other interesting domains include finance, health care, marketing, etc. Existing approaches, such as the Top-k method or the tau-path approach, primarily relate to bivariate data and/or invoke the normality assumption. Directly adapting these methods to the multivariate framework is cumbersome. In this work, we propose a semiparametric statistical approach for the optimal subpopulation selection based on the patterns of associations in multivariate data. The proposed method leverages the concept of general correlation coefficients to enable the optimal selection of a subpopulation for a variety of association patterns. We develop efficient algorithms consisting of sequential inclusion of cases into the subpopulation. We illustrate the performance of the proposed method using simulated data and an interesting real data.

Appendix

Proof of Proposition 1

For (c), if $u^{(x)}_{ij} = x_{i} - x_{j}$ and $u^{(y)}_{ij} = y_{i} - y_{j}$, it is easy to see that

$$\begin{aligned} \sum _{i, j} u^{(x)}_{ij} u^{(y)}_{ij}&= \sum _{i, j} (x_{i} - x_{j}) (y_{i} - y_{j}) = 2 \sum _{i, j} x_{i}y_{i} - 2 \sum _{i, j} x_{i}y_{j} \\&= 2n \sum _{i} x_{i}y_{i} - 2n^{2} \bar{x} \bar{y} \\&= 2n(\sum _{i} x_{i}y_{i} - n \bar{x} \bar{y}) \\&= 2n \sum _{i} (x_{i}- \bar{x}) (y_{i} - \bar{y}). \end{aligned}$$

Similarly, we can have

$$\begin{aligned} \sum _{i, j} (u^{(x)}_{ij})^{2}&= \sum _{i, j} (x_{i} - x_{j})^{2} = 2 \sum _{i, j} x_{i}^{2} - 2 \sum _{i, j} x_{i}x_{j} \\&= 2n \sum _{i} x_{i}^{2} - 2n^{2} \bar{x}^{2} \\&= 2n(\sum _{i} x_{i}^{2} - n \bar{x}^{2}) \\&= 2n \sum _{i} (x_{i}- \bar{x})^{2}. \end{aligned}$$

Then the expression in (2) can be written as

$$\begin{aligned} \tau (x, y)&= \frac{\sum _{i, j} u^{(x)}_{ij} u^{(y)}_{ij}}{\sqrt{ \sum _{i, j} (u^{(x)}_{ij})^{2} \sum _{i,j} (u^{(y)}_{ij})^{2} } } = \frac{ \sum _{i, j} (x_{i} - x_{j}) (y_{i} - y_{j}) }{\sqrt{ \sum _{i, j} (x_{i} - x_{j})^{2} \sum _{i,j} (y_{i} - y_{j}) } } \\&= \frac{ 2n \sum _{i} (x_{i}- \bar{x}) (y_{i} - \bar{y}) }{\sqrt{ 2n \sum _{i} (x_{i}- \bar{x})^{2} 2n \sum _{i} (y_{i}- \bar{y})^{2} } } \\&= \frac{\sum _{i} (x_{i}- \bar{x}) (y_{i} - \bar{y}) }{\sqrt{ \sum _{i} (x_{i}- \bar{x})^{2} \sum _{i} (y_{i}- \bar{y})^{2} } }. \end{aligned}$$

Proof of Corollary 1

For binary variables $x \in \{0, 1\}$ and $y \in \{0, 1\}$ and data points $(x_{i}, y_{i})$, $i=1,\ldots , n$, we can first check the Pearson correlation coefficient. First, we have

$$\begin{aligned} \sum _{i} (x_{i}- \bar{x}) (y_{i} - \bar{y})&= \sum _{i} x_{i}y_{i} - n \bar{x} \bar{y} = n_{11} - n \frac{n_{1\cdot }}{n}\frac{n_{\cdot 1}}{n} \\&= \frac{1}{n} \big [ n_{11} n - n_{1\cdot }n_{\cdot 1}\big ] \\&= \frac{1}{n} \big [ n_{11} (n_{11}+n_{10}+n_{01}+n_{00}) - (n_{10}+n_{11})(n_{01}+n_{11})\big ] \\&= \frac{1}{n} \big [ n_{11}n_{00} - n_{10}n_{01} \big ]. \end{aligned}$$

Second, we can get

$$\begin{aligned} \sum _{i} (x_{i}- \bar{x})^{2}&= \sum _{i} x_{i}^{2} - n \bar{x}^{2} = n_{1\cdot } - n (\frac{n_{1\cdot }}{n})^{2} \\&= \frac{1}{n} \big [ n_{1\cdot }n -n_{1\cdot }^{2} \big ] \\&= \frac{1}{n} \big [ n_{1\cdot } n_{0\cdot } \big ]. \end{aligned}$$

Similarly, we obtain $\sum _{i} (y_{i}- \bar{y})^{2} = \frac{1}{n} \big [ n_{\cdot 1} n_{\cdot 0} \big ]$. Thus,

$$\begin{aligned} \tau (x, y)&= \frac{\sum _{i} (x_{i}- \bar{x}) (y_{i} - \bar{y}) }{\sqrt{ \sum _{i} (x_{i}- \bar{x})^{2} \sum _{i} (y_{i}- \bar{y})^{2} } } \\&= \frac{n_{11}n_{00} - n_{10}n_{01}}{\sqrt{n_{1\cdot }n_{0\cdot }n_{\cdot 1}n_{\cdot 0}}}. \end{aligned}$$