Advertisement

Evolving Systems

, Volume 4, Issue 1, pp 13–25 | Cite as

Drift detection using uncertainty distribution divergence

  • Patrick LindstromEmail author
  • Brian Mac Namee
  • Sarah Jane Delany
Original Paper

Abstract

Data generated from naturally occurring processes tends to be non-stationary. For example, seasonal and gradual changes in climate data and sudden changes in financial data. In machine learning the degradation in classifier performance due to such changes in the data is known as concept drift and there are many approaches to detecting and handling it. Most approaches to detecting concept drift, however, make the assumption that true classes for test examples will be available at no cost shortly after classification and base the detection of concept drift on measures relying on these labels. The high labelling cost in many domains provides a strong motivation to reduce the number of labelled instances required to detect and handle concept drift. Triggered detection approaches that do not require labelled instances to detect concept drift show great promise for achieving this. In this paper we present Confidence Distribution Batch Detection, an approach that provides a signal correlated to changes in concept without using labelled data. This signal combined with a trigger and a rebuild policy can maintain classifier accuracy which, in most cases, matches the accuracy achieved using classification error based detection techniques but using only a limited amount of labelled data.

Keywords

Concept drift Explicit drift detection Labelling cost Classifier confidence 

References

  1. Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Mach Learn 15(2):201–221Google Scholar
  2. Delany SJ, Cunningham P, Tsymbal A, Coyle L (2005) A case-based technique for tracking concept drift in spam filtering. Knowl Based Syst 18(4–5):187–195CrossRefGoogle Scholar
  3. Fan W, Huang Y, Wang H, Yu PS (2004) Active mining of data streams. Proc Fourth SIAM Int Conf Data Min 35(4):457–461MathSciNetGoogle Scholar
  4. Gama J, Medas P, Castillo G, Rodrigues P (2004) Learning with drift detection. In: Bazzan A, Labidi S (eds) Advances in artificial intelligence SBIA 2004. Lecture notes in computer science, vol 3171. Springer, Berlin, pp 66–112Google Scholar
  5. Gao J, Fan W, Han J (2007) On appropriate assumptions to mine data streams: analysis and practice. In: Seventh IEEE international conference on data mining, 2007. ICDM 2007, pp 143–152Google Scholar
  6. Hsiao W, Chang T (2008) An incremental cluster-based approach to spam filtering. Expert Syst Appl 34(3):1599–1608CrossRefGoogle Scholar
  7. Huang S, Dong Y (2007) An active learning system for mining time-changing data streams. Intell Data Anal 11(4):401–419Google Scholar
  8. Kifer D, Ben-David S, Gehrke J (2004) Detecting change in data streams. In: Proceedings of the thirtieth international conference on very large data bases. VLDB Endowment, vol 30, pp 180–191Google Scholar
  9. Klinkenberg R (2004) Learning drifting concepts: example selection vs. example weighting. Intell Data Anal 8(3):281–300Google Scholar
  10. Klinkenberg R, Renz I (1998) Adaptive information filtering: learning in the presence of concept drifts. In: Workshop notes of the ICML/AAAI-98 workshop learning for text categorization. AAAI Press, Menlo Park, pp 33–40Google Scholar
  11. Kolter J, Maloof M (2003) Dynamic weighted majority: a new ensemble method for tracking concept drift. In: Third IEEE international conference on data mining, 2003. ICDM 2003. IEEE, New York, pp 123–130Google Scholar
  12. Kubat M (1989) Floating approximation in time-varying knowledge bases. Pattern Recognit Lett 10(4):223–227zbMATHCrossRefGoogle Scholar
  13. Kuncheva LI (2009) Using control charts for detecting concept change in streaming data. Tech. Rep. BCS-TR-001-2009, School of Computer Science, Bangor University, UKGoogle Scholar
  14. Lanquillon C (1999) Information filtering in changing domains. In: Proceedings of the 16th international joint conference on artificial intelligence, pp 41–48Google Scholar
  15. Lewis D (1995) Evaluating and optimizing autonomous text classification systems. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 246–254Google Scholar
  16. Lindstrom P, Mac Namee B, Delany SJ (2010) Handling concept drift in a text data stream constrained by high labelling cost. In: Guesgen HW, Murray RC (eds) Proceedings of the twenty-third international Florida artificial intelligence research society conference. AAAI Press, Menlo ParkGoogle Scholar
  17. Lindstrom P, Mac Namee B, Delany SJ (2011) Drift detection using uncertainty distribution divergence. In: 2nd International workshop on handling concept drift in adaptive information systems (HaCDAIS). IEEE Computer Society, New York, pp 604–608Google Scholar
  18. Masud M, Gao J, Khan L, Han J, Thuraisingham B (2008) A practical approach to classify evolving data streams: training with limited amount of labeled data. In: Eighth IEEE international conference on data mining, 2008. ICDM ’08, pp 929–934Google Scholar
  19. Montgomery DC (2004) Introduction to statistical quality control. Wiley, New YorkGoogle Scholar
  20. Nishida K, Yamauchi K (2007) Detecting concept drift using statistical testing. In: Corruble V, Takeda M, Suzuki E (eds) Discovery science. Lecture notes in computer science, vol 4755. Springer, Berlin, pp 264–269Google Scholar
  21. Schlimmer JC, Granger RH (1986) Incremental learning from noisy data. Mach Learn 1:317–354Google Scholar
  22. Sebastio R, Gama J (2007) Change detection in learning histograms from data streams. In: Neves J, Santos M, Machado J (eds) Progress in artificial intelligence. Lecture notes in computer science, vol 4874. Springer, Berlin, pp 112–123Google Scholar
  23. Spinosa EJ, de Leon AP, Gama J (2007) OLINDDA: a cluster-based approach for detecting novelty and concept drift in data streams. In: Proceedings of the 2007 ACM symposium on applied computing, SAC ’07. ACM, New York, pp 448–452Google Scholar
  24. Swan R, Allan J (1999) Extracting significant time varying features from text. In: Proceedings of the eighth international conference on information and knowledge management, CIKM ’99. ACM, New York, pp 38–45Google Scholar
  25. Vapnik VN (1995) The nature of statistical learning theory. Springer, New YorkzbMATHGoogle Scholar
  26. Woolam C, Masud M, Khan L (2009) Lacking labels in the stream: classifying evolving stream data with few labels. In: Rauch J, Ras Z, Berka P, Elomaa T (eds) Foundations of intelligent systems. Lecture notes in computer science, vol 5722. Springer, Berlin, pp 552–562Google Scholar
  27. Yang Y, Liu X (1999) A re-examination of text categorization methods. In: Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval, SIGIR ’99. ACM, New York, pp 42–49Google Scholar
  28. Zhu X, Zhang P, Lin X, Shi Y (2007) Active learning from data streams. In: Proceedings of the 2007 seventh IEEE international conference on data mining, ICDM ’07. IEEE Computer Society, Washington, DC, pp 757–762Google Scholar
  29. Žliobaite I (2010) Change with delayed labeling: when is it detectable? In: Proceedings of the 2010 IEEE international conference on data mining workshops, ICDMW ’10. IEEE Computer Society, Washington, DC, pp 843–850Google Scholar
  30. Žliobaite I, Bifet A, Pfahringer B, Holmes G (2011) Active learning with evolving streaming data. In: Gunopulos D, Vazirgiannis M, Malerba D, Hofmann T (eds) Proceedings of the 2011 European conference on machine learning and knowledge discovery in databases, ECML PKDD’11, vol Part III. Springer, Berlin, pp 597–612Google Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Patrick Lindstrom
    • 1
    Email author
  • Brian Mac Namee
    • 1
  • Sarah Jane Delany
    • 1
  1. 1.School of ComputingDublin Institute of TechnologyDublinIreland

Personalised recommendations