Evolving Systems

, Volume 4, Issue 1, pp 43–60 | Cite as

Using a classifier pool in accuracy based tracking of recurring concepts in data stream classification

  • Mohammad Javad Hosseini
  • Zahra Ahmadi
  • Hamid Beigy
Original Article

Abstract

Data streams have some unique properties which make them applicable in precise modeling of many real data mining applications. The most challenging property of data streams is the occurrence of “concept drift”. Recurring concepts is a type of concept drift which can be seen in most of real world problems. Detecting recurring concepts makes it possible to exploit previous knowledge obtained in the learning process. This leads to quick adaptation of the learner whenever a concept reappears. In this paper, we propose a learning algorithm called Pool and Accuracy based Stream Classification with some variations, which takes the advantage of maintaining a pool of classifiers to track recurring concepts. Each classifier is used to describe an existing concept. Consecutive batches of instances are first classified by the pool of classifiers. Two approaches are presented for this task: active classifier and weighted classifiers methods. Then the true labels are revealed and the pool is updated at the end of the batch. Updating the pool is done using one of the following methods: exact Bayesian, Bayesian and Heuristic. As the algorithm may assign multiple classifiers to a single concept, a classifier merging process is used to resolve this problem. Experimental results on real and artificial datasets show the effectiveness of weighted classifiers method while dealing with sudden concept drifting datasets. In addition, the proposed updating methods outperform the existing algorithms in datasets with arbitrary attributes. Finally some performed experiments represent superiority of using merging process in large datasets.

Keywords

Recurring concepts Concept drift Stream mining Ensemble learning 

References

  1. Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66Google Scholar
  2. Baena-García M et al (2006) Early drift detection method. In: ECML PKDD workshop on knowledge discovery from data streams, pp 77–86Google Scholar
  3. Bifet A et al (2010) Accurate ensembles for data streams: combining restricted Hoeffding trees using stacking. In: 2nd Asian conference on machine learning, Tokyo, Japan: JMLR, pp 225–240Google Scholar
  4. Bifet A (2009) Adaptive learning and mining for data streams and frequent patterns, PhD Thesis in Departament de Llenguatges i Sistemes Informatics. Universitat Politecnica de CatalunyaGoogle Scholar
  5. Bifet A, Gavaldà R (2007) Learning from time-changing data with adaptive windowing. In: SIAM international conference on data mining (SDM’07). Minneapolis, Minnesota, pp 443–448Google Scholar
  6. Castillo G (2006) Adaptive learning algorithms for Bayesian network classifiers, PhD thesis in mathematics. Aveiro UniversityGoogle Scholar
  7. Domingos P, Hulten G (2000), Mining high-speed data streams. In: Proceedings of the 6th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, Boston, pp 71–80Google Scholar
  8. Elwell R, Polikar R (2011) Incremental learning of concept drift in nonstationary environments. IEEE Trans Neural Netw 99:1517–1531CrossRefGoogle Scholar
  9. Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed Jan 2012
  10. Freund Y, Schapire RE (1996) Game theory, on-line prediction and boosting. In: Proceedings of the ninth annual conference on Computational learning theory. ACM, New York, pp 325–332Google Scholar
  11. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139MathSciNetMATHCrossRefGoogle Scholar
  12. Gama J, Castillo G (2006) Learning with local drift detection. In: The proceedings of the advanced data mining and applications. Springer, Berlin, pp 42–55Google Scholar
  13. Gama J, Kosina P (2009) Tracking recurring concepts with meta-learners. In: Proceedings of the 14th Portuguese conference on artificial intelligence: progress in artificial intelligence. Springer, Aveiro, pp 423–434Google Scholar
  14. Gama J, Medas P, Rocha R (2004) Forest trees for on-line data. In: Proceedings of the 2004 ACM symposium on applied computing. ACM, Nicosia, pp 632–636Google Scholar
  15. Gama J, Fernandes R, Rocha R (2006) Decision trees for mining data streams. Intell Data Anal 10(1):23–45Google Scholar
  16. Gao J, Fan W, Han J (2007) On appropriate assumptions to mine data streams: analysis and practice. In: Proceedings of the 2007 seventh IEEE international conference on data mining, pp 143–159Google Scholar
  17. Gao J et al (2008) Classifying data streams with skewed class distributions and concept drifts. IEEE Internet Comput 12(6):37–49CrossRefGoogle Scholar
  18. Garnett R (2010) learning from data streams with concept drift, PhD Thesis in engineering science. University of OxfordGoogle Scholar
  19. Gomes JB, Menasalvas E, Sousa PAC (2011) Learning recurring concepts from data streams with a context-aware ensemble. In: Proceedings of the 2011 ACM symposium on applied computing, pp 994–999Google Scholar
  20. Hosseini MJ, Ahmadi Z, Beigy H (2011) Pool and accuracy based stream classification: a new ensemble algorithm on data stream classification using recurring concepts detection. In: Proceedings of the 2011 IEEE eleventh international conference on data mining workshops (ICDMW). IEEE, Vancouver, pp 588–595Google Scholar
  21. Hulten G, Spencer L, Domingos P (2001) Mining time-changing data streams. In: Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, San Francisco, pp 97–106Google Scholar
  22. Ikonomovska E, Gama J, S. Deroski., (2011) Learning model trees from evolving data streams. Data Mining Knowl Discov 23(1): 128–168Google Scholar
  23. Katakis I, Tsoumakas G, Vlahavas I (2009) Tracking recurring contexts using ensemble classifiers: an application to email filtering. Knowl Inf Syst 22(3):371–391CrossRefGoogle Scholar
  24. Klinkenberg R (2004) Learning drifting concepts: example selection vs. example weighting. Intell Data Anal 8(3):281–300Google Scholar
  25. Kolter JZ, Maloof MA (2005) Using additive expert ensembles to cope with concept drift. In: Proceedings of the 22nd international conference on Machine learning. ACM, Bonn, pp 449–456Google Scholar
  26. Kolter JZ, Maloof MA (2007) Dynamic weighted majority: an ensemble method for drifting concepts. J Mach Learn Res 8:2755–2790MATHGoogle Scholar
  27. Kuncheva LI, Žliobaitė I (2009) On the window size for classification in changing environments. Intell Data Anal 13(6):861–872Google Scholar
  28. Lazarescu MM (2005) A multi-resolution learning approach to tracking concept drift and recurrent concepts. In: 5th IAPR workshop on pattern recognition in information systems (PRIS). Miami, USA, pp 52–61Google Scholar
  29. Littlestone N (1988) Learning quickly when irrelevant attributes abound: a new linear-threshold algorithm. Mach Learn 2(4):285–318Google Scholar
  30. Littlestone N, Warmuth MK (1994) The weighted majority algorithm. Inf Comput 108(2):212–261MathSciNetMATHCrossRefGoogle Scholar
  31. Minku L, Yao X (2012) DDD: a new ensemble approach for dealing with concept drift. IEEE Trans Knowl Data Eng 99:619–633CrossRefGoogle Scholar
  32. Morshedlou H, Barforoush AA (2009) A new history based method to handle the recurring concept shifts in data streams. World Acad Sci Eng Technol 58:917–922Google Scholar
  33. Nishida K (2008) Learning and detecting concept drift. PhD Thesis in information science and technology. Hokkaido University, HokkaidoGoogle Scholar
  34. Padovitz A, Loke SW, Zaslavsky A (2004) Towards a theory of context spaces. In: Proceedings of the 2nd IEEE annual conference on pervasive computing and communications workshops, pp 38–42Google Scholar
  35. Ramamurthy S, Bhatnagar R (2007) Tracking recurrent concept drift in streaming data using ensemble classifiers. In: Proceedings of the sixth international conference on machine learning and applications, pp 404–409Google Scholar
  36. Schlimmer JC, Richard J, Granger H (1986) Incremental learning from noisy data. Mach Learn 1(3):317–354Google Scholar
  37. Scholz M, Klinkenberg R (2005) An ensemble classifier for drifting concepts. In: Proceedings of the 2nd international workshop on knowledge discovery in data streams, Porto, Portugal, pp 53–64Google Scholar
  38. Stanley KO (2003) Learning concept drift with a committee of decision trees, Technical Report UT-AI-TR-03-302, Computer Sciences Department, University of TexasGoogle Scholar
  39. Street WN, Kim Y (2001) A streaming ensemble algorithm (SEA) for large-scale classification. In: Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, San Francisco, pp 377–382Google Scholar
  40. Tsymbal A (2004) The problem of concept drift: definitions and related work, Technical Report TCD-CS-2004-15, Computer Science Department, Trinity College DublinGoogle Scholar
  41. Wang H et al (2003) Mining concept-drifting data streams using ensemble classifiers. In: Proceedings of the 9th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, Washington, pp 226–235Google Scholar
  42. Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23(1):69–101Google Scholar
  43. Widmer G, Kubat M (1998) Special issue on context sensitivity and concept drift—introduction. Mach Learn 32(2):83–84CrossRefGoogle Scholar
  44. Woolam C, Masud MM, Khan L (2009) Lacking labels in the stream: classifying evolving stream data with few labels. In: Proceedings of the 18th international symposium on foundations of intelligent systems. Springer, Prague, pp 552–562Google Scholar
  45. Zhu X (2010) Stream data mining repository. http://www.cse.fau.edu/~xqzhu/stream.html. Accessed Jan 2012
  46. Zliobaite I (2010) Learning under concept drift: an overview. Vilnius University, Technical reportGoogle Scholar
  47. Zliobaite I (2010) Adaptive training set formation, PhD Thesis in physical sciences and informatics, Vilnius UniversityGoogle Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Mohammad Javad Hosseini
    • 1
  • Zahra Ahmadi
    • 1
  • Hamid Beigy
    • 1
  1. 1.Department of Computer EngineeringSharif University of TechnologyTehranIran

Personalised recommendations