Continuous similarity search for evolving queries

Abstract

In this paper, we study a novel problem of continuous similarity search for evolving queries. Given a set of objects, each being a set or multiset of items, and a data stream, we want to continuously maintain the top-k most similar objects using the last n items in the stream as an evolving query. We show that the problem has several important applications. At the same time, the problem is challenging. We develop a filtering-based method and a hashing-based method. Our experimental results on both real data sets and synthetic data sets show that our methods are effective and efficient.

Appendix: Other similarity measures and their upper bounds for pruning method

In this section, we extend the upper bounds for pruning method to weighted overlap, weighted cosine, and weighted dice similarity. Similar to the case of weighted Jaccard similarity, they all have monotonicity with respect to number of updates u.

Property 1

(A progressive upper bound for weighted overlap similarity). We first define the weighted overlap similarity as follows.

$$\begin{aligned} sim_{over}(X, Y) = sim_{over}(\vec {X}, \vec {Y}) = \sum _{i=1}^{|\Psi |} \min (x_i, y_i) \end{aligned}$$

(8)

Let X, Y be two multisets and $Y'$ be the multiset with u updates on Y. Given |X|, |Y|, and the weighted overlap similarity score $sim_{over}(X, Y)$ between X and Y, without the knowledge of the updated elements in $Y'$, we have an upper bound for $sim_{over}(X, Y')$.

$$\begin{aligned} sim_{over}(X, Y') \le sim_{over}(X, Y) + u \end{aligned}$$

(9)

Proof

By definition,

$$\begin{aligned} sim_{over}(X, Y) = \sum _{i=1}^{|\Psi |} \min (x_i, y_i) \end{aligned}$$

Obviously, the maximum possible increase after u updates is u. $\square$

Property 2

(A progressive upper bound for weighted cosine similarity) We first define the weighted cosine similarity as follows.

$$\begin{aligned} sim_{cos}(X, Y) = sim_{cos}(\vec {X}, \vec {Y}) = \frac{\sum _{i=1}^{|\Psi |} \min (x_i, y_i)}{\sqrt{|X||Y|}} \end{aligned}$$

(10)

Let X, Y be two multisets and $Y'$ be the multiset with u updates on Y. Given |X|, |Y|, and the weighted cosine similarity score $sim_{cos}(X, Y)$ between X and Y, without the knowledge of the updated elements in $Y'$, we have an upper bound for $sim_{cos}(X, Y')$.

$$\begin{aligned} sim_{cos}(X, Y') \le sim_{cos}(X, Y) + \frac{u}{\sqrt{|X||Y|}} \end{aligned}$$

(11)

Proof

In our scenario, $|Y| = |Y'|$. Thus, $\sqrt{|X||Y|} = \sqrt{|X||Y'|}$. By Property 1, we have

$$\begin{aligned} sim_{cos}(X, Y') \le \frac{\sum _{i=1}^{|\Psi |} \min (x_i, y_i) + u}{\sqrt{|X||Y'|}} = sim_{cos}(X, Y) + \frac{u}{\sqrt{|X||Y|}} \end{aligned}$$

$\square$

Property 3

(A progressive upper bound for weighted dice similarity) We first define the weighted dice similarity as follows.

$$\begin{aligned} sim_{dice}(X, Y) = sim_{dice}(\vec {X}, \vec {Y}) = \frac{2\sum _{i=1}^{|\Psi |} \min (x_i, y_i)}{|X| + |Y|} \end{aligned}$$

(12)

Let X, Y be two multisets and $Y'$ be the multiset with u updates on Y. Given |X|, |Y|, and the weighted dice similarity score $sim_{dice}(X, Y)$ between X and Y, without the knowledge of the updated elements in $Y'$, we have an upper bound for $sim_{dice}(X, Y')$.

$$\begin{aligned} sim_{dice}(X, Y') \le sim_{dice}(X, Y) + \frac{2u}{|X| + |Y|} \end{aligned}$$

(13)

Proof

In our scenario, $|Y| = |Y'|$. Thus, $|X|+|Y| = |X|+|Y'|$. By Property 1, we have

$$\begin{aligned} sim_{dice}(X, Y') \le \frac{2\sum _{i=1}^{|\Psi |} \min (x_i, y_i) + 2u}{|X|+|Y'|} = sim_{dice}(X, Y) + \frac{2u}{|X|+|Y|} \end{aligned}$$

$\square$