Taking the Big Picture: representative skylines based on significance and diversity

Abstract

The skyline is a popular operator to extract records from a database when a record scoring function is not available. However, the result of a skyline query can be very large. The problem addressed in this paper is the automatic selection of a small number $(k)$ of representative skyline records. Existing approaches have only focused on partial aspects of this problem. Some try to identify sets of diverse records giving an overall approximation of the skyline. These techniques, however, are sensitive to the scaling of attributes or to the insertion of non-skyline records into the database. Others exploit some knowledge of the record scoring function to identify the most significant record, but not sets of records representative of the whole skyline. In this paper, we introduce a novel approach taking both the significance of all the records and their diversity into account, adapting to available knowledge of the scoring function, but also working under complete ignorance. We show the intractability of the problem and present approximate algorithms. We experimentally show that our approach is efficient, scalable and that it improves existing works in terms of the significance and diversity of the results.

Appendix: Proofs

Proof (Proof of Proposition 1)

To prove stability, we simply notice that div does not use non-skyline records in their definitions. To prove scale invariance, we notice that in case of complete ignorance, $E(\sigma (p)) = \frac{1}{2}$ and $\delta (p, q)$ is not affected by attribute re-scaling because the actual values are not used in its definition. Therefore, the value of the objective function does not change after re-scaling. $\square$

Proof (Proof of Proposition 2)

We show that $\delta (r,t) + \delta (t,s)$ $\ge \delta (r,s)$. This holds when $t=r$ because $\delta (r,t) + \delta (t,s)$ reduces to $\delta (r,r) + \delta (r,s) = \delta (r,s)$ and $\delta (r,r) = 0$. Similarly, it holds when $t=s$. When $t \ne r,s$ we verify that for each dimension $i$ we have:

$$\begin{aligned}&\frac{|\{ o \in {\mathrm {sky}}(\mathrm {R}) \ | \ (r.i \ge o.i \ge s.i) \vee (s.i \ge o.i \ge r.i)\}|}{|{\mathrm {sky}}(\mathrm {R})|-1} \\&\quad \le \frac{|\{ o \in {\mathrm {sky}}(\mathrm {R}) \ | \ (r.i \ge o.i \ge t.i) \vee (t.i \ge o.i \ge r.i)\}|}{|{\mathrm {sky}}(\mathrm {R})|-1} \\&\qquad + \frac{|\{ o \in {\mathrm {sky}}(\mathrm {R}) \ | \ (t.i \ge o.i \ge s.i) \vee (s.i \ge o.i \ge t.i)\}|}{|{\mathrm {sky}}(\mathrm {R})|-1} \end{aligned}$$

This is done by noting that if a record $o$ belongs to the left-hand side expression, it also belongs to at least one of the two sets in the right-hand side expression. $\square$

Proof (Proof of Proposition 3)

In general, the expected value of a function $g$ where the probability density of $x$ is $f(x)$ is given by the integral $\int g(x) f(x) \mathrm {d}x$ computed between $-\infty$ and $\infty$. In the specific case of sigmoid functions, having a probability density function for the thresholds (let us call it $f$), we can precisely write the expected value $E(\sigma (r))$ of the significance of a record $r$ as follows (in the following equations we omit the normalization factor for simplicity):⁷

$$\begin{aligned} \frac{\sum _{i \in [1,d]} \lim \limits _{a.i\rightarrow -\infty , b.i\rightarrow \infty } \int _{a.i}^{b.i} \frac{1}{1+e^{-r.i+t.i}} f(t.i) \,\mathrm {d}t.i}{d} \end{aligned}$$

Starting from this, we can compute the expected value according to our actual knowledge. If we do not know anything about $f$, we may use a uniform probability distribution:

$$\begin{aligned} \frac{\sum _{i \in [1,d]} \lim \limits _{a.i\rightarrow -\infty , b.i\rightarrow \infty } \int _{a.i}^{b.i} \frac{1}{1+e^{-r.i+t.i}} \frac{1}{b.i-a.i} \,\mathrm {d}t.i}{d} \end{aligned}$$

and if we are able to further restrict the possible thresholds inside $d$ intervals $[a.i,b.i], i \in [1,d]$, we can express the expected significance as:

$$\begin{aligned} \frac{\sum _{i \in [1,d]} \int _{a.i}^{b.i} \frac{1}{1+e^{-r.i+t.i}} \frac{1}{b.i-a.i} \,\mathrm {d}t.i}{d} \end{aligned}$$

This expression can be treated analytically: The terms $\frac{1}{b.i-a.i}$ do not depend on $r.i$, and the logistic function $\frac{1}{1+e^{-x}}$ has indefinite integral $\ln (1+e^x)$. Therefore, we can move $\frac{1}{b.i-a.i}$ out of the integral and solve it by substituting $x = - t.i + r.i$. We obtain:

$$\begin{aligned} \frac{\sum _{i \in [1,d]} \frac{1}{b.i-a.i} (-\ln (1+e^{(r.i-b.i)}) + \ln (1+e^{(r.i-a.i)}))}{d} \end{aligned}$$

$\square$

Proof (of Theorem 1)

We want to prove that the problem in Definition 2, that we call here Rep-Skyline(R, k, $\lambda , \delta , \sigma$), is NP-hard. For convenience, we repeat the definition of Rep-Skyline: Let $R$ be a relation, $k$ an integer constant, $\lambda$ a real number in $[0,1], \delta$ a function assessing the diversity of two records and $\sigma$ a function assessing the significance of a record. We notate the skyline of $\mathrm {R}$ as ${\mathrm {sky}}(\mathrm {R})$ and its subsets of size $k$ as $\mathcal {P}_k({\mathrm {sky}}(\mathrm {R}))$. We want to find:

$$\begin{aligned} {\mathop {\hbox {arg max}}\limits _{S \in \mathcal {P}_k({\mathrm {sky}}(\mathrm {R}))}} \ \lambda \sum _{r \in S} \min _{s \in S \setminus \{r\}} \delta (r, s) + (1-\lambda ) \sum _{r \in S} E(\sigma (r)) \end{aligned}$$

To prove NP-hardness, we provide a polynomial time reduction of the problem known as Remote- Pseudoforest(R, k, $\overline{\delta }$) to RepSkyline. The Remote- Pseudoforest problem is the following: Let R be a set of $d$-dimensional points, $\overline{\delta }$ be a distance function and $\mathcal {P}_k(R)$ indicate the set of subsets of R of size $k$. We want to find:

$$\begin{aligned} {\mathop {\hbox {arg max}}\limits _{S \in \mathcal {P}_k(R)}} \ \sum _{r \in S} \min _{s \in S \setminus \{r\}} \overline{\delta }(r, s) \end{aligned}$$

The Remote-Pseudoforest problem has been proven to be NP-complete [8].

To solve Remote-Pseudoforest(R, k, $\overline{\delta }$), first we map $R$ into a new set $R'$ where for every $d$-dimensional record $x = \langle x_1, \ldots , x_d\rangle$ in $R$ we have a $2d$-dimensional record $x' = \langle x_1, \ldots , x_d, -x_1, \ldots , -x_d\rangle$ in $R'$. We also define a new distance function $\overline{\delta }': R' \rightarrow [0,1]$ such that $\overline{\delta }'(x') = \overline{\delta }(x)$, i.e., $\overline{\delta }$ is computed on the first $d$ dimensions of $x'$. This transformation guarantees that all points in $R'$ are in the $2d$-dimensional skyline and preserves the distance between all pairs of points $x, y$ before and after the transformation ($\overline{\delta }(x,y) = \overline{\delta }'(x',y')$).

Now we can solve the problem of Remote- Pseudoforest($R, k, \overline{\delta }$) by solving Rep-Skyline $(R', k, 1, \overline{\delta }', 0)$ ⁸ and mapping the result of Rep-Skyline back to the $d$-dimensional space. In fact, if we set the values $R',1,\gamma ,\delta$ and $0$ in the Rep-Skyline we obtain:

$$\begin{aligned} {\mathop {\hbox {arg max}}\limits _{S \in \mathcal {P}_k({\mathrm {sky}}(R'))}} \ \sum _{r \in S} \min _{s \in S \setminus \{r\}} \overline{\delta }'(r, s) \end{aligned}$$

where ${\mathrm {sky}}(R') = R'$.

As a result, as we are solving Remote-Pseudoforest by calling Rep-Skyline as a sub-procedure and only applying a transformation of the input that is linear on the number of records and dimensions, we can state that Remote-Pseudoforest is not more difficult ($\le _p$) than Rep-Skyline. Because Remote-Pseudoforest is NP-complete, we can conclude that Rep-Skyline is an NP-hard problem. $\square$