On measuring similarity for sequences of itemsets

Abstract

Computing the similarity between sequences is a very important challenge for many different data mining tasks. There is a plethora of similarity measures for sequences in the literature, most of them being designed for sequences of items. In this work, we study the problem of measuring the similarity between sequences of itemsets. We focus on the notion of common subsequences as a way to measure similarity between a pair of sequences composed of a list of itemsets. We present new combinatorial results for efficiently counting distinct and common subsequences. These theoretical results are the cornerstone of an effective dynamic programming approach to deal with this problem. In addition, we propose an approximate method to speed up the computation process for long sequences. We have applied our method to various data sets: healthcare trajectories, online handwritten characters and synthetic data. Our results confirm that our measure of similarity produces competitive scores and indicate that our method is relevant for large scale sequential data analysis.

Appendix

1.1 Proof of Lemma 1

Let $T=\left\langle T_1,\ldots ,T_m\right\rangle$ be a sequence that is counted multiple times; i.e., $T\in (\varphi (S)\circ \mathcal {P}_{\ge 1}(Y))\cap \varphi (S)$. Clearly $T_m\in \mathcal {P}_{\ge 1}(Y)$ as otherwise $T$ would not have been in $\varphi (S)\circ \mathcal {P}_{\ge 1}(Y)$. Let $\ell$ denote $\max \{j|T_m \subseteq S[j]\}$. Since $T\in \varphi (S)$, such $\ell$ must exist. Then, $\ell \in L(S,Y)$, since $\ell$ is the largest index for which $S[\ell ]\cap Y$ includes $T_m$. Therefore, $T\in S^{\ell -1}\circ (\mathcal {P}_{\ge 1}(S[\ell ]\cap Y))$ for a $\ell \in L(S,Y)$. $\Box$

1.2 Proof of Theorem 1

The proof is a simple application of the inclusion–exclusion principle to compute the cardinality of the union of Lemma 1:

$$\begin{aligned} R(S,Y)&= \left| \bigcup _{\ell \in L} \left\{ \varphi (S^{\ell -1})\circ \mathcal {P}_{\ge 1}(S[\ell ]\cap Y) \right\} \right| \\&= \sum _{K\subseteq L}(-1)^{|K|+1} \left| \bigcap _{\ell \in K} \left\{ \varphi (S^{\ell -1})\circ \mathcal {P}_{\ge 1}(S[\ell ]\cap Y) \right\} \right| \end{aligned}$$

The proof is completed by the following two observations:

$$\begin{aligned} set _K:=\bigcap _{\ell \in K} \left\{ \varphi (S^{\ell -1})\circ \mathcal {P}_{\ge 1}(S[\ell ]\cap Y) \right\} =\varphi (S^{\min (K)-1})\circ \mathcal {P}_{\ge 1}(\left( \cap _{k\in K}S[k]\right) \cap Y) \end{aligned}$$

Indeed; any sequence of length $m$ in $set _K$ has $T^{m-1}\in S^{\min (K)-1}$, and $T_m\in \mathcal {P}_{\ge 1}(S[k]\cap Y)$, for all $k\in K$ and

$$\begin{aligned} \left| \varphi (S^{\min (K)-1})\circ \mathcal {P}_{\ge 1}(\left( \cap _{k\in K}S[k]\right) \cap Y)\right| =\left| \phi (S^{\min (K)-1})\right| \cdot \left( 2^{\left| \left( \cap _{k\in K}S[k]\right) \cap Y\right| }-1\right) \end{aligned}$$

$\Box$

1.3 Proof of Lemma 2

Let $Z=\left\langle Z_1,\ldots ,Z_m\right\rangle$ be a new subsequence that is added to $\varphi (S ,T)$ after concatenating sequence S with itemset Y; i.e., $Z \in \varphi (S \circ Y ,T) \setminus \varphi (S,T)$. Clearly $Z_m\in \mathcal {P}_{\ge 1}(Y)$ as otherwise $Z$ would not have been added to $\varphi (S\circ Y,T)$. Let $\ell ^{'}=\max \{j|Z_m \subseteq T[j]\}$. Since $Z \in \varphi (S \circ Y,T)$, with $Z \preceq T$, then such $\ell ^{'}\in L(T,Y)$ must exist. $\ell ^{'}$ is the largest index for which $T[\ell ^{'}]\cap Y$ includes $Z_m$. Therefore, $Z\in \varphi (S,T^{\ell ^{'}-1})\circ (\mathcal {P}_{\ge 1}(T[\ell ^{'}]\cap Y))$ for a $\ell ^{'} \in L(T,Y)$.

Let $W=\left\langle W_1,\ldots ,W_m\right\rangle$ be a sequence that is counted multiple times; i.e., $W \in (\varphi (S,T^{\ell ^{'}-1})\circ \mathcal {P}_{\ge 1}(T[\ell ^{'}] \cap Y))\cap \varphi (S,T)$ where $\ell ^{'}\in L(T,Y)$. Clearly $W_m\in \mathcal {P}_{\ge 1}(T[\ell ^{'}] \cap Y)$ as otherwise $W$ would not have been in $\varphi (S,T^{\ell ^{'}-1})\circ \mathcal {P}_{\ge 1}(T[\ell ^{'}] \cap Y)$. Let $\ell =\max \{j|W_m \subseteq S[j]\}$. Since $W\in \varphi (S,T)$, such $\ell \in L(S,Y)$ must exist, since $\ell$ is the largest index for which $S[\ell ]\cap Y$ includes $Z_m$. Therefore, $Z \in \varphi (S^{\ell -1},T^{\ell ^{'} -1})\circ (\mathcal {P}_{\ge 1}(S[\ell ] \cap T[\ell ^{'}] \cap Y))$ for $\ell \in L(S,Y)$ and $\ell ^{'} \in L(T,Y)$.   $\square$

1.4 Proof of Theorem 2

• Case 1: No items in $Y$ appear in any itemset of $S$ and $T$, in this case the set of all common distinct subsequences between $S \circ Y$ and $T$ is exactly the same set of all common distinct subsequences between $S$ and $T$. Hence, $\phi (S \circ Y, T)=\phi (S, T)$.

• Case 2: If at least an item in $Y$ appears in either one of the sequences $S$ or $T$ (or both), then $\varphi (S \circ Y, T)$ is expressed as the union of the set of all common distinct subsequences between $S$ and $T$ with the set of added sequences $\mathcal {A}$ without the set of repeated sequences $\mathcal {R}$. Formally,

  $$\begin{aligned} \varphi (S \circ Y, T) = \varphi (S, T) \cup \mathcal {A} \backslash \mathcal {R} \end{aligned}$$

  (8)

  with

  $$\begin{aligned} \mathcal {A}=\left\{ \displaystyle \bigcup _{\ell ^{'} \in L(T,Y)}\varphi (S,T^{\ell ^{'}-1}) \circ \mathcal {P}_{\ge 1}(T[\ell ^{'}] \cap Y)\right\} \end{aligned}$$
  $$\begin{aligned} \mathcal {R}=\left\{ \displaystyle \bigcup _{\ell \in L(S,Y)} \left\{ \displaystyle \bigcup _{\ell ^{'} \in L(T,Y)} \varphi (S^{\ell -1},T^{ \ell ^{'}-1}) \circ \mathcal {P}_{\ge 1}(S[\ell ] \cap T[\ell ^{'}] \cap Y) \right\} \right\} \end{aligned}$$

  Notice that because these three sets are disjoint, the cardinality of $\varphi (S \circ Y, T)$ can be simply expressed as $|\varphi (S \circ Y, T)| = |\varphi (S, T)| + |\mathcal {A}| - |\mathcal {R}|$. Using the inclusion–exclusion principle, $|\mathcal {A}|$, denoted as $A(S,T,Y)$ can be written as,

  $$\begin{aligned} A(S,T,Y)&= \left| \left\{ \displaystyle \bigcup _{\ell \in L(T,Y) } \varphi (S,T^{\ell -1})\circ \mathcal {P}_{\ge 1}(T[\ell ] \cap Y) \right\} \right| \nonumber \\&= \sum _{K\subseteq L(T,Y)} (-1)^{|K|+1} \left| \bigcap _{\ell \in K} \left\{ \varphi (S,T^{\ell -1})\circ \mathcal {P}_{\ge 1}(T[\ell ] \cap Y)\right\} \right| \end{aligned}$$

  (9)

  $A(S,T,Y)$ is completed by the following two observations:

  $$\begin{aligned} set _K&:= \bigcap _{\ell \in K} \left\{ \varphi (S,T^{\ell -1})\circ \mathcal {P}_{\ge 1}(T[\ell ] \cap Y)\right\} \\&= \varphi (S,T^{\min (K)-1})\circ \mathcal {P}_{\ge 1}(\left( \cap _{k\in K}T[k]\right) \cap Y) \end{aligned}$$

  And, the second observation:

  $$\begin{aligned} \left| set _K\right| =\phi (S,T^{\min (K)-1})\cdot \left( 2^{\left| \left( \cap _{k\in K}T[k]\right) \cap Y\right| }-1\right) \end{aligned}$$

  $A(S,T,Y)$ can be written as,

  $$\begin{aligned} A(S,T,Y) = \displaystyle \sum _{K\subseteq L(T,Y)} (-1)^{|K|+1} \left( \phi (S,T^{\min (K)-1})\cdot \left( 2^{|\left( \bigcap _{j\in K}T[j]\right) \cap Y|} - 1 \right) \right) \end{aligned}$$

  The same inclusion–exclusion reasoning applies to the cardinality of $\mathcal {R}$, denoted $R(S,T,Y)$

  $$\begin{aligned} R(S,T,Y)=\left| \left\{ \displaystyle \bigcup _{\ell \in L(S,Y)}\! \left\{ \displaystyle \bigcup _{\ell ^{'} \in L(T,Y)}\! \varphi (S^{\ell -1},T^{ \ell ^{'}-1})\circ \mathcal {P}_{\ge 1}(S[\ell ] \cap T[\ell ^{'}] \cap Y) \right\} \! \right\} \right| \end{aligned}$$
  $$\begin{aligned}&=\sum _{K\subseteq L(S,Y)} (-1)^{|K|+1}\left( \sum _{K^{'}\subseteq L(T,Y)} (-1)^{|K^{'}|+1}\left| \bigcap _{\ell \in K}\bigcap _{\ell ^{'}\in K^{'}} \left\{ \varphi (S^{\ell -1},T^{\ell ^{'}-1})\circ \right. \right. \right. \\&\qquad \left. \left. \left. \mathcal {P}_{\ge 1}(S[\ell ]\cap T[\ell ^{'}]\cap Y) \right\} \right| \right) \end{aligned}$$

  The final result follows after noticing that,

  $$\begin{aligned} set _{K,K^{'}}&= \bigcap _{\ell \in K}\bigcap _{\ell ^{'}\in K^{'}} \varphi (S^{\ell -1},T^{\ell ^{'}-1})\circ \mathcal {P}_{\ge 1}(S[\ell ]\cap T[\ell ^{'}]\cap Y) \\ set _{K,K^{'}}&= \varphi (S^{\min (K)-1},T^{\min (K^{'})-1})\circ \mathcal {P}_{\ge 1}(\left( \cap _{k\in K}S[k]\right) \cap \left( \cap _{k^{'}\in K^{'}}T[k^{'}]\right) \cap Y) \end{aligned}$$

  $R(S,T,Y)$ can be written as,

  $$\begin{aligned} R(S,T,Y) = \displaystyle \sum _{K\subseteq L(S,Y)} (-1)^{|K|+1} \left( \sum _{K^{'}\subseteq L(T,Y)} (-1)^{|K^{'}|+1} \cdot f(K,K^{'}) \right) \end{aligned}$$

  where:

  $$\begin{aligned} f(K,K^{'})=\phi (S^{\min (K)-1},T^{\min (K^{'})-1}) \cdot \left( 2^{| \left( \bigcap _{j\in K}S[j] \right) \cap \left( \bigcap _{j^{'}\in K^{'}}T[{j^{'}]} \right) \cap Y|} - 1 \right) \end{aligned}$$

$\square$

1.5 Details of Linial–Nisan approximation for Example 7

To do the Linial–Nisan approximation, remark that $N=|L|=9$ and $K=\lceil \sqrt{N} \rceil =3$. The vector of Linial–Nisan coefficients is defined as $\overrightarrow{\alpha }=(\alpha _1^{3,9} , \alpha _2^{3,9},\alpha _3^{3,9}) = \overrightarrow{t} \cdot \mathcal {M}^{-1}$ where $\mathcal {A}$ is the matrix whose $(i,j)$ entry is ${j \atopwithdelims ()i}$. The inverse matrix $\mathcal {A}^{-1}(i,j)$ is defined as $(-1)^{i+j}{j \atopwithdelims ()i}$. In our example,

$$\begin{aligned} \mathcal {M}^{-1}&= \begin{pmatrix} 1 &{} -2 &{} 3 \\ 0 &{} 1 &{} -3 \\ 0 &{} 0 &{} 1 \\ \end{pmatrix}\\ \end{aligned}$$

$\overrightarrow{t}=(q_{K,N}(1),q_{K,N}(2),\dots ,q_{K,N}(K))$ is the vector of linearly transformed Chebyshev polynomials and is computed using the polynomial $T_K(x)$ as follows:

$$\begin{aligned} q_{3,9}(1)&= 1- \frac{T_{3}(\frac{2-(9+1)}{9-1})}{T_{3}(\frac{-(9+1)}{9-1})} =1-\frac{T_{3}(-1)}{T_{3}(-\frac{10}{8})}=1- \frac{-1}{-4,06}=0.75\\ q_{3,9}(2)&= 1- \frac{T_{3}(\frac{4-(9+1)}{9-1})}{T_{3}(\frac{-(9+1)}{9-1})}= 1-\frac{T_{3}(-\frac{6}{8})}{T_{3}(-\frac{10}{8})}=1- \frac{0.56}{-4,06}=1.13\\ q_{3,9}(3)&= 1- \frac{T_{3}(\frac{6-(9+1)}{9-1})}{T_{3}(\frac{-(9+1)}{9-1})}= 1-\frac{T_{3}(-\frac{4}{10})}{T_{3}(-\frac{10}{8})}=1- \frac{1}{-4,06}=1.24\\ \end{aligned}$$

This vector is necessary to solve the system of linear equations:

$$\begin{aligned} \overrightarrow{\alpha }&= (\alpha _1^{3,9} , \alpha _2^{3,9},\alpha _3^{3,9})=\overrightarrow{t} \cdot \mathcal {M}^{-1}\\&= \begin{pmatrix} 0.75 &{} 1.13 &{} 1.24 \\ \end{pmatrix}.\begin{pmatrix} 1 &{} -2 &{} 3 \\ 0 &{} 1 &{} -3 \\ 0 &{} 0 &{} 1 \\ \end{pmatrix}\\&= \begin{pmatrix} 0.75 &{} -0.36 &{} 0.1 \\ \end{pmatrix} \end{aligned}$$

A solution to the system above is given by $\alpha _1^{3,9}=0.75;~\alpha _2^{3,9}=-0.36;~\alpha _3^{3,9}=0.1$. The real numbers $\alpha _k^{3,9}$ can now be used to approximate the inclusion–exclusion formula in the correction term as follows:

$$\begin{aligned} R_{LN}(S^{9},S[10])&= \displaystyle \sum _{k=1}^{3} \alpha _k^{3,9} \displaystyle \sum _{\begin{array}{c} O \subseteq L(S,Y) \\ |O|=k \end{array}} \phi _{LN}(S^{\min (O)-1})\cdot \left( 2^{|\left( \bigcap _{j\in O}S[j]\right) \cap S[10]|} - 1 \right) \\&= 9,298\,1279.10^26 \end{aligned}$$

Notice here that the formula contains only $\sum _{i=1}^{3}{9 \atopwithdelims ()i}$ terms, which is already a significant computation gain in comparison with the $\sum _{i=1}^{9}{9 \atopwithdelims ()i}$ terms in the classical approach. Finally, the approximated number of distinct subsequences for sequence $S$ is $\phi _{LN}(S^{10})=2^{|\{a,b,c,d,e,f,g,h,i,j,,k\}|} \cdot \phi _{LN}(S^{9})-R_{LN}(S^{9},S[10])=2,524\,4956.10^30$.