Sequence graph transform (SGT): a feature embedding function for sequence data mining

Abstract

Sequence feature embedding is a challenging task due to the unstructuredness of sequences—arbitrary strings of arbitrary length. Existing methods are efficient in extracting short-term dependencies but typically suffer from computation issues for the long-term. Sequence Graph Transform (SGT), a feature embedding function, that can extract a varying amount of short- to long-term dependencies without increasing the computation is proposed. SGT’s properties are analytically proved for interpretation under normal and uniform distribution assumptions. SGT features yield significantly superior results in sequence clustering and classification with higher accuracy and lower computation as compared to the existing methods, including the state-of-the-art sequence/string Kernels and LSTM.

Appendices

Appendices

Mean and variance of $\psi _{uv}$

Consider an arbitrary sequence s, where the sequence has an ordered list of symbols. These symbols belong to a finite set ${\mathcal {V}}$. SGT embedding works by finding the dependencies between every pair of symbols $(u,v); u, v \in {\mathcal {V}}$.

To easily denote various (u, v) pairs in s, we use a term, mth neighboring pair, where an mth neighbor pair for (u, v) will have $m-1$ other u’s in between. A first neighbor is thus the immediate (u, v) neighboring pairs, while the 2nd-neighbor has one other u in between, and so on (see Fig. 4 for illustration). The immediate neighbor mentioned in the assumption in Sect. 2.4.1 is the same as the first neighbor defined here.

Based on the sequence patterns assumption in Sect. 2.4.1 and assuming u, v occur uniformly in the sequence with probability p, the expected number of first-neighbor (u, v) pairs is given as $M=pL$. Consequently, it is easy to show that the expected number of mth neighboring (u, v) pairs is $(M-m+1)$, i.e., the second neighboring (u, v) pairs will be $(M-1)$, $(M-2)$ for the third, so on and so forth, till one instance for the Mth neighbor. The gap distance for an mth neighbor is given as $Z_{1}=X;Z_{m}=X+\sum _{i=2}^{m}Y_{i},m=2,\ldots ,M$.

Besides, the total number of (u, v) pair instances will be $\sum _{m=1}^{M}m=\frac{M(M+1)}{2}$($=|\varLambda _{uv}|$, by definition). Suppose we define a set that contains distances for each possible (u, v) pairs as ${\mathcal {Z}}=\{Z_{m}^{i},i=1,\ldots ,(M-m+1);m=1,\ldots ,M\}$. Also, since $Z_{m}\sim N(\mu _{\alpha }+(m-1)\mu _{\beta },\sigma _{\alpha }^{2}+(m-1)\sigma _{\beta }^{2})$, $\phi _{\kappa }(Z_{m})$ becomes a lognormal distribution. Thus,

$$\begin{aligned} E[\phi _{\kappa }(Z_{m})]= & {} e^{-{\tilde{\mu }}_{\alpha }-(m-1){\tilde{\mu }}_{\beta }} \end{aligned}$$

(13)

$$\begin{aligned} \text {var}[\phi _{\kappa }(Z_{m})]= & {} e^{-2{\tilde{\mu }}_{\alpha }'-2(m-1){\tilde{\mu }}_{\beta }'}-e^{-2{\tilde{\mu }}_{\alpha }-2(m-1){\tilde{\mu }}_{\beta }} \end{aligned}$$

(14)

where,

$$\begin{aligned} {\tilde{\mu }}_{\alpha }=\kappa \mu _{\alpha }-\frac{\kappa ^{2}}{2}\sigma _{\alpha }^{2}&;&{\tilde{\mu }}'_{\alpha }=\kappa \mu _{\alpha }-\kappa ^{2}\sigma _{\alpha }^{2}\nonumber \\ {\tilde{\mu }}_{\beta }=\kappa \mu _{\beta }-\frac{\kappa ^{2}}{2}\sigma _{\beta }^{2}&;&{\tilde{\mu }}'_{\beta }=\kappa \mu _{\beta }-\kappa ^{2}\sigma _{\beta }^{2} \end{aligned}$$

(15)

Besides, the feature, $\psi _{uv}$, in Eq. (3a) can be expressed and further derived using Sect. A.1 as,

$$\begin{aligned} E[\psi _{uv}]= & {} \frac{\sum _{Z\in {\mathcal {Z}}}E[\phi _{\kappa }(Z)]}{M(M+1)/2}\nonumber \\= & {} \frac{\sum _{m=1}^{M}(M-(m-1))e^{-{\tilde{\mu }}_{\alpha }-(m-1){\tilde{\mu }}_{\beta }}}{M(M+1)/2}\nonumber \\= & {} \frac{2}{pL+1}\frac{e^{-{\tilde{\mu }}_{\alpha }}}{\underbrace{\left| \left( 1-e^{-{\tilde{\mu }}_{\beta }}\right) \left[ 1-\frac{1-e^{-pL{\tilde{\mu }}_{\beta }}}{pL(e^{{\tilde{\mu }}_{\beta }}-1)}\right] \right| }_{\gamma }} \end{aligned}$$

(16)

This yields to the expectation expression in Eq. (4). Besides, the variances will be

$$\begin{aligned} \text {var}(\psi _{uv})= & {} \left( \frac{1}{pL(pL+1)/2}\right) ^{2}\left[ \left\{ \frac{e^{-2{\tilde{\mu }}'_{\alpha }}}{1-e^{-2{\tilde{\mu }}'_{\beta }}}\left( pL\right. \right. \right. \\&\left. \left. -e^{-2{\tilde{\mu }}'_{\beta }}\left( \frac{1-e^{-2pL{\tilde{\mu }}'_{\beta }}}{1-e^{-2{\tilde{\mu }}'_{\beta }}}\right) \right) \right\} \\&-\underbrace{\left. \left\{ \frac{e^{-2{\tilde{\mu }}{}_{\alpha }}}{1-e^{-2{\tilde{\mu }}{}_{\beta }}}\left( pL-e^{-2{\tilde{\mu }}_{\beta }}\left( \frac{1-e^{-2pL{\tilde{\mu }}{}_{\beta }}}{1-e^{-2{\tilde{\mu }}{}_{\beta }}}\right) \right) \right\} \right] }_{\pi }\\ \text {var}(\psi _{uv})= & {} {\left\{ \begin{array}{ll} \left( \frac{1}{pL(pL+1)/2}\right) ^{2}\pi ; &{} \text {length sensitive}\\ \left( \frac{1}{p(pL+1)/2}\right) ^{2}\pi ; &{} \text {length insensitive} \end{array}\right. } \end{aligned}$$

1.1 Arithmetico-geometric series

The sum of a series, where the kth term for $k\ge 1$ can be expressed as,

$$\begin{aligned} t_{k}= & {} \left( a+(k-1)d\right) br^{k-1} \end{aligned}$$

is called an arithmetico-geometric because of a combination of arithmetic series term $(a+(k-1)d)$, where a is the initial term and common difference d, and geometric $br^{k-1}$, where b is the initial value and common ratio being r.

Suppose the sum of the series till n terms is denoted as,

$$\begin{aligned} S_{n}= & {} \sum _{k=1}^{n}\left( a+(k-1)d\right) br^{k-1} \end{aligned}$$

(17)

Without loss of generality we can assume $b=1$ for deriving the expression for $S_{n}$ (the sum for any other value of b can be easily obtained by multiplying the expression for $S_{n}$ with b). Expanding Eq. (17),

$$\begin{aligned} S_{n}= & {} a+(a+d)r+\ldots +(a+(n-1)d)r^{n-1} \end{aligned}$$

(18)

Now multiplying $S_{n}$ with r,

$$\begin{aligned} rS_{n}= & {} ar+(a+d)r^{2}+\ldots +(a+(n-1)d)r^{n} \end{aligned}$$

(19)

Subtracting Eq. (19) from (18), if $|r|<1$, else we subtract the latter from the former, we get,

Therefore,

$$\begin{aligned} S_{n}= & {} \left| \frac{1}{1-r}\left[ a+\frac{dr(1-r^{n-1})}{1-r}-\left( a+(n-1)d\right) r^{n}\right] \right| \end{aligned}$$

(20)

or, for any value of b,

$$\begin{aligned} S_{n}= & {} b\left| \frac{1}{1-r}\left[ a+\frac{dr(1-r^{n-1})}{1-r}-\left( a+(n-1)d\right) r^{n}\right] \right| \end{aligned}$$

(21)

${\mathbf {W}}^{(\kappa )}$ independent with respect to $\kappa$

Proof

We have ${\mathbf {W}}^{(\kappa )} = [W^{(\kappa )}_{uv}],\,u,v\in {\mathcal {V}}$ where

$$\begin{aligned} W^{(\kappa )}_{uv} = \sum _{\forall (l,m) \in \varLambda _{uv}(s)}e^{-\kappa |m-l|} \end{aligned}$$

(22)

To prove the independence of ${\mathbf {W}}^{(\kappa )}$ with respect to $\kappa$, we need to show ${\mathbf {w}}_{u\cdot }^{(\kappa )} \perp \!\!\! \perp {\mathbf {w}}_{u\cdot }^{(\kappa + \delta )}$ where ${\mathbf {w}}_{u\cdot }$ is a column in ${\mathbf {W}}$.

Without loss of generality assume $\varLambda _{uv}(s) = 1$ and replacing $|m-l|$ with $x, x>0$ in Eq. (22) the column ${\mathbf {w}}_{u\cdot }$ for $\kappa$ and $\kappa + \delta$ will be, ${\mathbf {w}}_{u\cdot }^{(\kappa )} = [e^{-\kappa x_i}]$ and ${\mathbf {w}}_{u\cdot }^{(\kappa + \delta )} = [e^{-\kappa x_i}]$ where $i = 1, \ldots , |{\mathcal {V}}|$.

${\mathbf {w}}_{u\cdot }^{(\kappa )}$ and ${\mathbf {w}}_{u\cdot }^{(\kappa +\delta )}$ will be dependent iff there is a nontrivial solution for a, b in Eq. (23).

$$\begin{aligned} a{\mathbf {w}}_{u\cdot }^{(\kappa )} - b{\mathbf {w}}_{u\cdot }^{(\kappa + \delta )} = 0 \end{aligned}$$

(23)

Solving this for a and b by taking any element i in ${\mathbf {w}}_{u\cdot }$.

$$\begin{aligned}&a w_{ui}^{(\kappa )} - b w_{ui}^{(\kappa +\delta )}&=0\\ \implies&a e^{-\kappa x_i} - b e^{-(\kappa +\delta ) x_i}&=0\\ \implies&e^{-\kappa x_i}(a - b e^{-\delta x_i})&=0\\ \end{aligned}$$

Since $e^{-\kappa x_i}\ne 0$, we have $a = b e^{-\delta x_i}$. Plugging this in the equation for another element j in ${\mathbf {w}}_{u\cdot }$ we get,

$$\begin{aligned}&a w_{uj}^{(\kappa )} - b w_{uj}^{(\kappa +\delta )}&=0\\ \implies&b e^{-\delta x_i} e^{-\kappa x_j} - b e^{-(\kappa +\delta ) x_j}&=0\\ \implies&b e^{-\kappa x_j}(e^{-\delta x_i} - e^{-\delta x_j})&=0\\ \end{aligned}$$

Since $e^{-\delta x_i} \ne e^{-\delta x_j} \forall i,j$, and $e^{-\kappa x_j} \ne 0$ we have $b=0$. Consequently, $a=0$.

Therefore, Eq. (23) has a trivial solution $a,b=0$. Thus, ${\mathbf {w}}_{u\cdot }^{(\kappa )}$ is independent with respect to $\kappa$ implying independence of ${\mathbf {W}}^{(\kappa )}$. $\square$

Proof for symbol clustering

We have, $\frac{\partial \varDelta }{\partial \kappa }=\frac{\partial }{\partial \kappa }E[\phi _{\kappa }(X)-\phi _{\kappa }(Y)]=E[\frac{\partial }{\partial \kappa }\phi _{\kappa }(X)-\frac{\partial }{\partial \kappa }\phi _{\kappa }(Y)]$. For $E[X]<E[Y]$, we want, $\frac{\partial \varDelta }{\partial \kappa }>0$, in turn, $\frac{\partial }{\partial \kappa }\phi _{\kappa }(X)>\frac{\partial }{\partial \kappa }\phi _{\kappa }(Y)$. This will hold if $\frac{\partial ^{2}}{\partial d\partial \kappa }\phi _{\kappa }(d)>0$, that is, slope, $\frac{\partial }{\partial \kappa }\phi _{\kappa }(d)$ is increasing with d. For an exponential expression for $\phi$ (Eq. 1), the above condition holds true if $\kappa d>1$. Hence, under these conditions, the separation increases as we increase the tuning parameter, $\kappa$.

Sequence simulation

In this section, we explain the sequence generation for Exp 1–2 in Sect. 4.

A sequence is generated by randomly selecting a string from the motif set and placing them in random order between arbitrary strings. These interspersed arbitrary symbols are the noise in a sequence. Figure 17 shows an example of a sequence generated from a set of seed motifs. About 50% of the sequence is noise—arbitrary strings. Note that due to the random motif selection, a simulated sequence does not necessarily contain all seed motifs.

1.1 Generating sequence clusters

Suppose we have to generate K sequence clusters. We first randomly simulate K sets of motifs. In each set, the motifs are of random lengths (between 2 and 8 in our simulations). The size of a set is also randomly chosen (between 6 and 11 in this paper). Sequences are then generated from each motif set as described above.

Fig. 17

A simulated sequence from seed motifs

Fig. 18

Motif sets of overlapping clusters

1.2 What are overlapping clusters?

In our experiments, we have to test the efficacy of clustering methods when the clusters are difficult to separate. This is the case when the clusters overlap. In traditional multidimensional data in Euclidean space, it means the centroids of the clusters are close. In our problem, overlapping clusters imply that they have some common seed motifs. In other words, the intersection of the clusters’ motif sets is not null. Thus, 0% overlap implies the intersection of motif sets is null, and a 100% overlap implies the intersection is equal to the union. Figure 18 shows an example of overlapping motif sets of two clusters.

Code repository and data sets

The source code and data sets are available on GitHub as https://github.com/cran2367/sgt. Its python package is also provided at https://pypi.org/project/sgt/.

The python package can be installed as follows.