Fast exhaustive subgroup discovery with numerical target concepts

Abstract

Subgroup discovery is a key data mining method that aims at identifying descriptions of subsets of the data that show an interesting distribution with respect to a pre-defined target concept. For practical applications the integration of numerical data is crucial. Therefore, a wide variety of interestingness measures has been proposed in literature that use a numerical attribute as the target concept. However, efficient mining in this setting is still an open issue. In this paper, we present novel techniques for fast exhaustive subgroup discovery with a numerical target concept. We initially survey previously proposed measures in this setting. Then, we explore options for pruning the search space using optimistic estimate bounds. Specifically, we introduce novel bounds in closed form and ordering-based bounds as a new technique to derive estimates for several types of interestingness measures with no previously known bounds. In addition, we investigate efficient data structures, namely adapted FP-trees and bitset-based data representations, and discuss their interdependencies to interestingness measures and pruning schemes. The presented techniques are incorporated into two novel algorithms. Finally, the benefits of the proposed pruning bounds and algorithms are assessed and compared in an extensive experimental evaluation on 24 publicly available datasets. The novel algorithms reduce runtimes consistently by more than one order of magnitude.

Appendix

Lemma 1

Using the notations of Theorem 9, the function $f^a (x) (n+x)^a \cdot \left( \frac{\sigma + x \cdot \theta }{n+x}-\mu _\emptyset \right)$ has no local maxima inside its domain of definition:

$$\begin{aligned} f^a(x) \le max (f(0), f(x_{max})) \end{aligned}$$

Proof

We distinguish three cases by the parameter a of the applied generic mean interestingness measure:

first, for $a = 1$, it holds that

$$\begin{aligned} f^1 (x)&= (n+x)^1 \cdot \left( \frac{\sigma + x \cdot \theta }{n+x}-\mu _\emptyset \right) \\&= \sigma + \theta x - \mu _\emptyset n - \mu _\emptyset x \\&= \left( \theta - \mu _\emptyset \right) \cdot x + \sigma - \mu _\emptyset n \end{aligned}$$

As this is a linear function in x, the function $f^1(x)$ is strictly increasing for $\theta > \mu _\emptyset$ and strictly decreasing otherwise. Thus, the theorem holds for $a=1$.

Second, we consider the case $(a \ne 1) \wedge (\sigma = \theta n)$, that is, the first n instances all had the same target value. In this case, the function $f^a(x)$ is given by $f^a(x) = (n+x)^a (\theta -\mu _\emptyset )$. This is strictly monotone since $n > 0, x > 0$. Thus, again $f^a(x)$ has no local maximum.

Third, the case $(a \ne 1) \wedge (\sigma \ne \theta n)$ is considered in detail: since $\sigma$ was computed as a sum of n values that are at least as large as $\theta$ it can be assumed that $\theta \cdot n < \sigma$. In the following, the maxima of $f^a(x)$ is determined by deriving this function twice.

$$\begin{aligned} f^a\,'(x)&= \frac{d}{d x} f^a(x) = (n+x)^a \cdot \left( \frac{\sigma + x \cdot \theta }{n+x}-\mu _\emptyset \right) \\&= (n+x)^a \left( \frac{d}{dx} \left( \frac{\theta x+\sigma }{n+x}-\mu _\emptyset \right) \right) + \left( \frac{\theta x +\sigma }{n+x}-\mu _\emptyset \right) \cdot \left( \frac{d}{dx}(n+x)^a\right) \\&= (n+x)^a \left( \frac{d}{dx} \left( \frac{\theta x+\sigma }{n+x} \right) \right) + \left( \frac{\theta x+\sigma }{n+x}-\mu _\emptyset \right) \cdot a (n+x)^{a-1} \\&= (n+x)^a \left( \frac{\theta }{n+x} - \frac{\theta x+\sigma }{(n+x)^2}\right) + \left( \frac{\theta x+\sigma }{n+x}-\mu _\emptyset \right) \cdot a(n+x)^{a-1} \\&= (n+x)^{a-2} \left( (\theta (n+x)- (\theta x+\sigma )\right) + a\left( \theta x+\sigma - \mu _\emptyset (n+x))\right) \\&= (n+x)^{a-2} (\theta n - \sigma + a\theta x+ a\sigma - a \mu _\emptyset n - a \mu _\emptyset x))\\&= (n+x)^{a-2} \left( (x (a\theta -a \mu _\emptyset ) + a\sigma -a \mu _\emptyset n + \theta n - \sigma \right) \end{aligned}$$

In line 2, the product rule is used. In line 3 the chain rule is applied, substituting (n+x). $\mu _\emptyset$ can be omitted, as it is constant with respect to x. In line 4 the quotient rule is used. Finally, in line 5 $(n+x)^{a-2}$ is factored out.

Since $x > 0, n > 0$ by definition, the first factor is obviously greater than zero for any valid x. For $a = 0$ or $\theta = \mu _\emptyset$, the second factor of this function is independent from x, so it has no root, thus f(x) has no maxima except the definition boundaries in this case. Otherwise the root of this function and therefore the only candidate for a maximum of $f^a(x)$ is given at the point

$$\begin{aligned} x^* = \frac{-a \sigma + an \mu _\emptyset - \theta n + \sigma }{a(\theta - \mu _\emptyset )}. \end{aligned}$$

In the following, it is shown that $x^*$ can not be a maximum value in our setting. For that purpose, the second derivative of f(x) is computed at the point $x^*$:

$$\begin{aligned} f^a\,''(x)&= \frac{d}{d x} f'(x) \\&= (n+x)^{a-3}(a-2)(x(a\theta -a \mu _\emptyset )\\&\quad + a \sigma -an +\theta n-\sigma )+ (a\theta -a \mu _\emptyset )(n+x)^{a-2} \\&= (n+x)^{a-3} ((a-2) (x(a\theta -a \mu _\emptyset ) \\&\quad + a \sigma -an \mu _\emptyset +\theta n-\sigma ) + (a\theta -a \mu _\emptyset ) (n+x)) \\&= (n+x)^{a-3} (a^2x\theta -a^2x \mu _\emptyset + a^2\sigma -a^2 \mu _\emptyset n + a\theta n-a\sigma - 2xa\theta \\&\quad + 2ax \mu _\emptyset -2a\sigma + 2an \mu _\emptyset - 2\theta n + 2\sigma + a\theta n - an \mu _\emptyset + a\theta x - ax \mu _\emptyset ) \\&= (n+x)^{a-3} (a-1) (a\theta x+a\sigma -an\mu _\emptyset -ax\mu _\emptyset +2\theta n-2\sigma ) \\&= (n+x)^{a-3} (a-1) (x(a\theta -a\mu _\emptyset ) +a\sigma -an\mu _\emptyset +2\theta n-2\sigma ) \end{aligned}$$

We now can determine the second derivative of f in $x^*$:

$$\begin{aligned} f^a\,''(x^*)&= (n+x^*)^{a-3} (a-1) (x^*(a\theta -a \mu _\emptyset ) +a\sigma - an\mu _\emptyset +2\theta n-2\sigma )\\&= (n+x^*)^{a-3} (a-1) \\&\quad \left( \frac{-a \sigma + an \mu _\emptyset - \theta n + \sigma }{a(\theta - \mu _\emptyset )}(a\theta -a \mu _\emptyset ) +a\sigma -an \mu _\emptyset +2\theta n-2\sigma \right) \\&= (n+x^*)^{a-3} (a-1) (-a \sigma + an \mu _\emptyset - \theta n + \sigma +a \sigma -an \mu _\emptyset + 2 \theta n-2 \sigma )\\&= (n+x^*)^{a-3} (a-1) (\theta n - \sigma )\\&= (n+x^*)^{a-3} (a-1) (\theta n - \sigma ) \end{aligned}$$

Since $f^a(x)$ is defined only for positive x, the first factor is always positive. Since by premise $a < 1$ and $\theta n < \sigma$, the second derivative at point $x^*$ is always positive. Thus, if $x^*$ is an extreme value of f(x), then it is a local minimum. Since it was shown above that f(x) has no other candidates for extreme values besides $x^*$, this proves the lemma.

Lemma 2

The generic mean-based measures $q_{mean}^a$ are convex for $a=1$ in the $(\sum T(c), i_P)$ space. They are not convex for arbitrary a.

Fig. 3

A surface plot of the mean test interestingness measure $q_{mean}^{0.5}$ for $\mu _\emptyset =0$ shows the non-convexity of this measure

Proof

For $a=1$, the interestingness measure is given by $q_{mean}^1 (P) = i_P \cdot (\mu _P - \mu _\emptyset ) = i_P \cdot (\frac{\sum _{c \in P} T(c)}{i_P} - \mu _\emptyset ) = \sum _{c \in P} T(c) - i_P \mu _\emptyset$. This function is linear in both $\sum T(C)$ and $i_P$. Since linear functions are known to be convex, $q_{mean}^1$ is convex.

To show that generic mean-based measures are not convex in general, we show an example where the definition of convex for a function f(x), that is, $\forall x,y, \lambda \in (0,1): f((1-\lambda ) x + \lambda y) \le (1-\lambda ) f (x) + \lambda f (y)$, is violated. In our case, x and y are each two-dimensional points in the $(\sum T(c), i_P)$ space. In that regard, we consider a dataset with $\mu _\emptyset = 0$ and the mean test interestingness measure $q_{mean}^{0.5}$. Then, the considered interestingness measure is given by $q_{mean}^{0.5} = i_P^{0.5} \cdot (\mu _P - \mu _\emptyset ) = \frac{\sum _{c \in P} T(c)}{\sqrt{i_P}} := f(x)$. As two points in the $(\sum T(c), i_P)$ space for which the convexity condition is violated we choose $x=(-100, 2)$ and $y=(-100, 10)$. Additionally, we choose $\lambda =0.5$. Then, the convexity inequality is violated:

$$\begin{aligned} f((1-\lambda ) x + \lambda y)&\le (1-\lambda ) f (x) + \lambda f (y) \\ f((0.5 x + 0.5 y)&\le 0.5\cdot f (x) + 0.5 \cdot f (y) \\ f((-100,6))&\le 0.5 \cdot f ((-100,2)) + 0.5 \cdot f (-100,10) \\ \frac{-100}{\sqrt{6}}&\le 0.5 \cdot \frac{-100}{\sqrt{2}} + 0.5 \cdot \frac{-100}{\sqrt{10}} \\ \approx -40.82&\le ~\approx -51.17 \end{aligned}$$

Since the definition of convexity is violated in at least one example, the mean test interestingness measure $q_{mean}^{0.5}$ is not convex.

The non-convexity of $q_{mean}^{0.5}$ is also evident by a surface plot of the function for $\mu _\emptyset =0$, see Fig. 3. $\square$