Applying temporal dependence to detect changes in streaming data

Abstract

Detection of changes in streaming data is an important mining task, with a wide range of real-life applications. Numerous algorithms have been proposed to efficiently detect changes in streaming data. However, the limitation of existing algorithms is that they assume that data are generated independently. In particular, temporal dependencies of data in a stream are still not thoroughly studied. Motivated by this, in this work we propose a new efficient method to detect changes in streaming data by exploring the temporal dependencies of data in the stream. As part of this, we introduce a new statistical model called the Candidate Change Point (CCP) model, with which the main idea is to compute the probabilities of finding change points in the stream. The computed probabilities are used to generate a distribution, which is, in turn, used in statistical hypothesis tests to determine the candidate changes. We use the CCP model to develop a new algorithm called Candidate Change Point Detector (CCPD), which detects change points in linear time, and is thus applicable for real-time applications. Our extensive experimental evaluation demonstrates the efficiency and the feasibility of our approach.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    Version 4.0.0, June 2017.

  2. 2.

    https://moa.cms.waikato.ac.nz/datasets/

  3. 3.

    http://archive.ics.uci.edu/ml/datasets.html

  4. 4.

    Source code of the FHDDM is provided by the authors of the algorithm

  5. 5.

    Source code of the RDDM is obtained from the authors personal website

References

  1. 1.

    Adä I, Berthold MR (2013) EVE: a framework for event detection. Evolving Systems 4(1):61–70

    Article  Google Scholar 

  2. 2.

    Adhikari U, Morris T, Pan S (2017) Applying Hoeffding adaptive trees for real-time cyber-power event and intrusion classification. IEEE Transactions on Smart Grid PP(99):1–12

    Google Scholar 

  3. 3.

    Anagnostopoulos C, Tasoulis DK, Adams NM, Pavlidis NG, Hand DJ (2012) Online linear and quadratic discriminant analysis with adaptive forgetting for streaming classification. Statistical Analysis and Data Mining 5(2):139–166

    MathSciNet  Article  Google Scholar 

  4. 4.

    Baena-García M, del Campo-Ȧvila J, Fidalgo R, Bifet A, Gavaldȧ R, Morales-Bueno R (2006) Early drift detection method. In: The 4th international workshop on knowledge discovery from data streams

  5. 5.

    Barros RS, Cabral DR, Gonçalves PM, Santos SG (2017) RDDM: Reactive drift detection method. Expert Syst Appl 90(Supplement C):344–355

    Article  Google Scholar 

  6. 6.

    Bifet A (2017) Classifier concept drift detection and the illusion of progress. In: Artificial intelligence and soft computing. Springer International Publishing, Cham, pp 715–725

    Google Scholar 

  7. 7.

    Bifet A, Gavaldà R (2007) Learning from time-changing data with adaptive windowing. In: Proceedings of the 2007 SIAM international conference on data mining, pp 443–448

    Google Scholar 

  8. 8.

    Bifet A, Gavaldà R (2009) Adaptive learning from evolving data streams. In: Proceedings of the 8th international symposium on intelligent data analysis, pp 249–260

    Google Scholar 

  9. 9.

    Bifet A, Holmes G, Kirkby R, Pfahringer B (2010) MOA: Massive online analysis. J Mach Learn Res 11:1601–1604

    Google Scholar 

  10. 10.

    Bifet A, Read J, žliobaitė I, Pfahringer B, Holmes G (2013) Pitfalls in benchmarking data stream classification and how to avoid them. In: Proceedings of the european conference on machine learning and knowledge discovery in databases, ECML PKDD, pp 465–479

  11. 11.

    Bodenham DA, Adams NM (2017) Continuous monitoring for changepoints in data streams using adaptive estimation. Stat Comput 27(5):1257–1270

    MathSciNet  Article  Google Scholar 

  12. 12.

    Bryc W (2002) A uniform approximation to the right normal tail integral. Appl Math Comput 127(2):365–374

    MathSciNet  MATH  Google Scholar 

  13. 13.

    Chattopadhyay S, Murthy C, Pal SK (2014) Fitting truncated geometric distributions in large scale real world networks. Theor Comput Sci 551:22–38

    MathSciNet  Article  Google Scholar 

  14. 14.

    Condat L (2016) Fast projection onto the simplex and the and the \(\ell _{1}\) ball. Math Program 158(1):575–585

    MathSciNet  Article  Google Scholar 

  15. 15.

    Demšar J (2006) Statistical Comparisons of Classifiers over Multiple Data Sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  16. 16.

    Duchi J, Shalev-Shwartz S, Singer Y, Chandra T (2008) Efficient projections onto the \(\ell _{1}\)-ball for learning in high dimensions. In: Proceedings of the 25th international conference on machine learning, ICML, pp 272–279

  17. 17.

    Frías-Blanco II, del Campo-Ávila J, Ramos-Jiménez G, Carvalho ACPLF, Díaz AAO, Morales-Bueno R (2016) Online adaptive decision trees based on concentration inequalities. Knowl-Based Syst 104:179–194

    Article  Google Scholar 

  18. 18.

    Frías-Blanco II, del Campo-Ávila J, Ramos-Jiménez G, Morales-Bueno R, Ortiz-Díaz AA, Caballero-Mota Y (2015) Online and non-parametric drift detection methods based on Hoeffding’s bounds. IEEE Trans Knowl Data Eng 27(3):810–823

    Article  Google Scholar 

  19. 19.

    Gama J, Medas P, Castillo G, Rodrigues PP (2004) Learning with drift detection. In: Proceedings of brazilian symposium on artificial intelligence, pp 286–295

    Google Scholar 

  20. 20.

    Gama J, Sebastião R, Rodrigues PP (2013) On evaluating stream learning algorithms. Mach Learn 90 (3):317–346

    MathSciNet  Article  Google Scholar 

  21. 21.

    Gama JA, žliobaitė I, Bifet A, Pechenizkiy M, Bouchachia A (2014) A survey on concept drift adaptation. ACM Comput Surv 46(4):1–37

    Article  Google Scholar 

  22. 22.

    Gomes HM, Barddal JP, Enembreck F, Bifet A (2017) A survey on ensemble learning for data stream classification. ACM Computing Surveys 50(2):23:1–23:36

    Article  Google Scholar 

  23. 23.

    Gomes HM, Bifet A, Read J, Barddal JP, Enembreck F, Pfharinger B, Holmes G, Abdessalem T (2017) Adaptive random forests for evolving data stream classification. Mach Learn 106(9):1469–1495

    MathSciNet  Article  Google Scholar 

  24. 24.

    Harries MB, Sammut C, Horn K (1998) Extracting hidden context. Mach Learn 32(2):101–126

    Article  Google Scholar 

  25. 25.

    Hoeffding W (1963) Probability inequalities for sums of bounded random variables. J Am Stat Assoc 58 (301):13–30

    MathSciNet  Article  Google Scholar 

  26. 26.

    Kifer D, Ben-David S, Gehrke J (2004) Detecting change in data streams

    Google Scholar 

  27. 27.

    Kolter JZ, Maloof MA (2007) Dynamic weighted majority: an ensemble method for drifting concepts. J Mach Learn Res 8:2755–2790

    MATH  Google Scholar 

  28. 28.

    Kumar R, Raghu M, Sarlós T, Tomkins A (2017) Linear additive markov processes. In: Proceedings of the 26th international conference on World Wide Web, pp 411–419

  29. 29.

    Li P, Wu X, Hu X (2010) Mining recurring concept drifts with limited labeled streaming data. In: Proceedings of the 2nd Asian conference on machine learning. PMLR, vol 13, pp 241–252

  30. 30.

    Liu J, Ye J (2009) Efficient euclidean projections in linear time. In: Proceedings of the 26th annual international conference on machine learning. ICML, pp 657–664

  31. 31.

    Markov A (1971) Extension of the Limit Theorems of Probability Theory to a Sum of Variables Connected in a Chain. In: Appendix B, dynamic probabilistic systems (Volume I: Markov models), pp 552–577

  32. 32.

    Page ES (1954) Continuous Inspection Schemes. Biometrika 41(1/2):100–115

    MathSciNet  Article  Google Scholar 

  33. 33.

    Pears R, Sakthithasan S, Koh YS (2014) Detecting concept change in dynamic data streams. Mach Learn 97(3):259–293

    MathSciNet  Article  Google Scholar 

  34. 34.

    Pesaranghader A, Viktor H, Paquet E (2017) McDiarmid drift detection methods for evolving data streams. CoRR arXiv:1710.02030

  35. 35.

    Pesaranghader A, Viktor H, Paquet E (2017) Reservoir of diverse adaptive learners and stacking fast Hoeffding drift detection methods for evolving data streams. CoRR arXiv:1709.02457

  36. 36.

    Pesaranghader A, Viktor HL (2016) Fast Hoeffding Drift Detection Method for Evolving Data Streams. In: Proceedings of the 2016 machine learning and knowledge discovery in databases. ECML PKDD, pp 96–111

  37. 37.

    Roberts SW (1959) Control chart tests based on geometric moving averages. Technometrics 1(3):239–250

    Article  Google Scholar 

  38. 38.

    Rösler O, Suendermann D (2013) A first step towards eye state prediction using EEG. In: Proceedings of the international conference on applied informatics for health and life sciences (AIHLS 2013)

  39. 39.

    Ross GJ, Adams NM, Tasoulis DK, Hand DJ (2012) Exponentially weighted moving average charts for detecting concept drift. Pattern Recogn Lett 33(2):191–198

    Article  Google Scholar 

  40. 40.

    Schlimmer JC, Granger RH (1986) Incremental learning from noisy data. Mach Learn 1(3):317–354

    Google Scholar 

  41. 41.

    Sebastião R, Gama J, Mendonça T (2017) Fading histograms in detecting distribution and concept changes. International Journal of Data Science and Analytics 3(3):183–212

    Article  Google Scholar 

  42. 42.

    Tibshirani RJ, Taylor J, Lockhart R, Tibshirani R (2016) Exact Post-Selection inference for sequential regression procedures. J Am Stat Assoc 111(514):600–620

    MathSciNet  Article  Google Scholar 

  43. 43.

    Weissman T, Ordentlich E, Seroussi G, Verdu S, Weinberger MJ (2003) Inequalities for the \(\ell _{1}\) Deviation of the Empirical Distribution. Technical report, Hewlett-Packard Labs

  44. 44.

    Wu T, Gleich DF (2017) Retrospective higher-order markov processes for user trails. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. KDD, pp 1185–1194

  45. 45.

    žliobaitė I, Bifet A, Read J, Pfahringer B, Holmes G (2015) Evaluation methods and decision theory for classification of streaming data with temporal dependence. Mach Learn 98(3):455–482

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

This research is funded by the Norwegian University of Science and Technology (NTNU) through the MUSED project.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Quang-Huy Duong.

Appendix: Proof

Appendix: Proof

A.1 Proof of property 1

Proof

Equation 7 can be easily proved by induction. When \(t_2 = t_1\), we have:

$$\begin{array}{@{}rcl@{}} \prod\limits_{t = t_{1}}^{t_{2}} (1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}}) & = & 1_{\{t_{1}\neq t_{1}\}}\times ch_{t_{1}} + 1_{\{t_{1}=t_{1}\}} \\ & = & 0 \times ch_{t_{1}} + 1 = 1\\ & = & ct(t_{1}, t_{1}) = ct(t_{2}, t_{1}). \end{array} $$

When \(t_2 = t_1 + 1 \), we have:

$$\begin{array}{@{}rcl@{}} \prod\limits_{t = t_{1}}^{t_{2}} 1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}} & = & \prod\limits_{t = t_{1}}^{t_{1}+ 1}1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}} \\ & = & ch_{t_{1}+ 1} = ct(t_{1}+ 1,t_{1}) \\ & = & ct\left( t_{2},t_{1}\right). \end{array} $$

Assume that Eq. 7 is satisfied when \(t_2 = t_1 + m\), with \(m \in \mathbb {N}, m > 0\). We prove that Eq. 7 is also satisfied with \(t_2 = t_1 + m + 1\). We have:

$$\begin{array}{@{}rcl@{}} ct(t_{2}, t_{1}) & = & ct\left( t_{1} + m + 1, t_{1}\right) \\ & = & ct\left( t_{1} + m + 1,t_{1} + m\right)\times ct\left( t_{1} + m,t_{1}\right) \\ & = & ch_{t_{1} + m + 1}\times\prod\limits_{t = t_{1}}^{t_{1}+m} (1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}}) \\ & = & \prod\limits_{t = t_{1}}^{t_{1}+m + 1} \left( 1_{\{t\neq t_{1}\}}\times ch_{t} + 1_{\{t=t_{1}\}}\right). \end{array} $$

A.2 Proof of proposition 1

Proof

We have:

$$\begin{array}{@{}rcl@{}} &&ctsum(t-1)\times ch_{t} + ct(t,t) \\ & = & ct(t,t) + ch_{t} \times \sum\limits_{i = 1}^{t-1} ct(t-1,i) \\ & = & ct(t,t) + \sum\limits_{i = 1}^{t-1} ch_{t} \times ct(t-1,i) \end{array} $$
$$\begin{array}{@{}rcl@{}} & = & ct(t,t) + \sum\limits_{i = 1}^{t-1} ct{(t,t-1)} \times ct(t-1,i) \\ & = & ct(t,t) + \sum\limits_{i = 1}^{t-1} ct{(t,i)} = \sum\limits_{i = 1}^{t} ct{(t,i)} = ctsum(t). \end{array} $$

A.3 Proof of proposition 2

Proof

We have:

$$\begin{array}{@{}rcl@{}} cp(t) \!&=&\! \sum\limits_{i = 1}^{t} v_{i} \!\times\! ct(t,i) = \sum\limits_{i = 1}^{t} v_{i} \prod\limits_{j = i}^{t} (1_{\{j\neq i\}} \!\times\! ch_{j} + 1_{\{j=i\}} ) \\ \!&=&\! \sum\limits_{i = 1}^{t-1} v_{i} \prod\limits_{j = i}^{t} (1_{\{j\neq i\}} \times ch_{j} + 1_{\{j=i\}} ) \\ && \!+ v_{t} \prod\limits_{j = t}^{t} (1_{\{j\neq t\}} \times ch_{j} + 1_{\{j=t\}} ) \\ \!&=&\! v_{t} + \sum\limits_{i = 1}^{t-1} v_{i} (1_{\{t\neq i\}} \times ch_{t} + 1_{\{t=i\}} ) \\ &&\!\times \prod\limits_{j = i}^{t-1} (1_{\{j\neq i\}} \times ch_{j} + 1_{\{j=i\}} ) \\ \!&=&\! v_{t} + ch_{t} \sum\limits_{i = 1}^{t-1} v_{i} \prod\limits_{j = i}^{t-1} (1_{\{j\neq i\}} \times ch_{j} + 1_{\{j=i\}} ) \\ \!&=&\! v_{t} + ch_{t} \times cp(t-1). \end{array} $$

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Duong, QH., Ramampiaro, H. & Nørvåg, K. Applying temporal dependence to detect changes in streaming data. Appl Intell 48, 4805–4823 (2018). https://doi.org/10.1007/s10489-018-1254-7

Download citation

Keywords

  • Data streams
  • Change detection
  • Temporal dependence
  • Adaptive estimation