Applying temporal dependence to detect changes in streaming data

Abstract

Detection of changes in streaming data is an important mining task, with a wide range of real-life applications. Numerous algorithms have been proposed to efficiently detect changes in streaming data. However, the limitation of existing algorithms is that they assume that data are generated independently. In particular, temporal dependencies of data in a stream are still not thoroughly studied. Motivated by this, in this work we propose a new efficient method to detect changes in streaming data by exploring the temporal dependencies of data in the stream. As part of this, we introduce a new statistical model called the Candidate Change Point (CCP) model, with which the main idea is to compute the probabilities of finding change points in the stream. The computed probabilities are used to generate a distribution, which is, in turn, used in statistical hypothesis tests to determine the candidate changes. We use the CCP model to develop a new algorithm called Candidate Change Point Detector (CCPD), which detects change points in linear time, and is thus applicable for real-time applications. Our extensive experimental evaluation demonstrates the efficiency and the feasibility of our approach.

Appendix: Proof

A.1 Proof of property 1

Proof

Equation 7 can be easily proved by induction. When $t_2 = t_1$, we have:

$$\begin{array}{@{}rcl@{}} \prod\limits_{t = t_{1}}^{t_{2}} (1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}}) & = & 1_{\{t_{1}\neq t_{1}\}}\times ch_{t_{1}} + 1_{\{t_{1}=t_{1}\}} \\ & = & 0 \times ch_{t_{1}} + 1 = 1\\ & = & ct(t_{1}, t_{1}) = ct(t_{2}, t_{1}). \end{array} $$

When $t_2 = t_1 + 1$, we have:

$$\begin{array}{@{}rcl@{}} \prod\limits_{t = t_{1}}^{t_{2}} 1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}} & = & \prod\limits_{t = t_{1}}^{t_{1}+ 1}1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}} \\ & = & ch_{t_{1}+ 1} = ct(t_{1}+ 1,t_{1}) \\ & = & ct\left( t_{2},t_{1}\right). \end{array} $$

Assume that Eq. 7 is satisfied when $t_2 = t_1 + m$, with $m \in \mathbb {N}, m > 0$. We prove that Eq. 7 is also satisfied with $t_2 = t_1 + m + 1$. We have:

$$\begin{array}{@{}rcl@{}} ct(t_{2}, t_{1}) & = & ct\left( t_{1} + m + 1, t_{1}\right) \\ & = & ct\left( t_{1} + m + 1,t_{1} + m\right)\times ct\left( t_{1} + m,t_{1}\right) \\ & = & ch_{t_{1} + m + 1}\times\prod\limits_{t = t_{1}}^{t_{1}+m} (1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}}) \\ & = & \prod\limits_{t = t_{1}}^{t_{1}+m + 1} \left( 1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}}\right). \end{array} $$

□

A.2 Proof of proposition 1

Proof

We have:

$$\begin{array}{@{}rcl@{}} &&ctsum(t-1)\times ch_{t} + ct(t,t) \\ & = & ct(t,t) + ch_{t} \times \sum\limits_{i = 1}^{t-1} ct(t-1,i) \\ & = & ct(t,t) + \sum\limits_{i = 1}^{t-1} ch_{t} \times ct(t-1,i) \end{array} $$

$$\begin{array}{@{}rcl@{}} & = & ct(t,t) + \sum\limits_{i = 1}^{t-1} ct{(t,t-1)} \times ct(t-1,i) \\ & = & ct(t,t) + \sum\limits_{i = 1}^{t-1} ct{(t,i)} = \sum\limits_{i = 1}^{t} ct{(t,i)} = ctsum(t). \end{array} $$

□

A.3 Proof of proposition 2

Proof

We have:

$$\begin{array}{@{}rcl@{}} cp(t) \!&=&\! \sum\limits_{i = 1}^{t} v_{i} \!\times\! ct(t,i) = \sum\limits_{i = 1}^{t} v_{i} \prod\limits_{j = i}^{t} (1_{\{j\neq i\}} \!\times\! ch_{j} + 1_{\{j=i\}} ) \\ \!&=&\! \sum\limits_{i = 1}^{t-1} v_{i} \prod\limits_{j = i}^{t} (1_{\{j\neq i\}} \times ch_{j} + 1_{\{j=i\}} ) \\ && \!+ v_{t} \prod\limits_{j = t}^{t} (1_{\{j\neq t\}} \times ch_{j} + 1_{\{j=t\}} ) \\ \!&=&\! v_{t} + \sum\limits_{i = 1}^{t-1} v_{i} (1_{\{t\neq i\}} \times ch_{t} + 1_{\{t=i\}} ) \\ &&\!\times \prod\limits_{j = i}^{t-1} (1_{\{j\neq i\}} \times ch_{j} + 1_{\{j=i\}} ) \\ \!&=&\! v_{t} + ch_{t} \sum\limits_{i = 1}^{t-1} v_{i} \prod\limits_{j = i}^{t-1} (1_{\{j\neq i\}} \times ch_{j} + 1_{\{j=i\}} ) \\ \!&=&\! v_{t} + ch_{t} \times cp(t-1). \end{array} $$

□