Data Mining and Knowledge Discovery

, Volume 19, Issue 1, pp 95–131 | Cite as

Flexible decision tree for data stream classification in the presence of concept change, noise and missing values

  • Sattar Hashemi
  • Ying YangEmail author


In recent years, classification learning for data streams has become an important and active research topic. A major challenge posed by data streams is that their underlying concepts can change over time, which requires current classifiers to be revised accordingly and timely. To detect concept change, a common methodology is to observe the online classification accuracy. If accuracy drops below some threshold value, a concept change is deemed to have taken place. An implicit assumption behind this methodology is that any drop in classification accuracy can be interpreted as a symptom of concept change. Unfortunately however, this assumption is often violated in the real world where data streams carry noise that can also introduce a significant reduction in classification accuracy. To compound this problem, traditional noise cleansing methods are incompetent for data streams. Those methods normally need to scan data multiple times whereas learning for data streams can only afford one-pass scan because of data’s high speed and huge volume. Another open problem in data stream classification is how to deal with missing values. When new instances containing missing values arrive, how a learning model classifies them and how the learning model updates itself according to them is an issue whose solution is far from being explored. To solve these problems, this paper proposes a novel classification algorithm, flexible decision tree (FlexDT), which extends fuzzy logic to data stream classification. The advantages are three-fold. First, FlexDT offers a flexible structure to effectively and efficiently handle concept change. Second, FlexDT is robust to noise. Hence it can prevent noise from interfering with classification accuracy, and accuracy drop can be safely attributed to concept change. Third, it deals with missing values in an elegant way. Extensive evaluations are conducted to compare FlexDT with representative existing data stream classification algorithms using a large suite of data streams and various statistical tests. Experimental results suggest that FlexDT offers a significant benefit to data stream classification in real-world scenarios where concept change, noise and missing values coexist.


Classification learning Data stream classification Decision tree learning Fuzzy learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Basak J (2006) Online adaptive decision trees: pattern classification and function approximation. Neural Comput 18(9): 2062–2101zbMATHCrossRefMathSciNetGoogle Scholar
  2. Bhatt RB, Gopal M (2006) Neuro-fuzzy decision trees. Int J Neural Syst 16(1): 63–78CrossRefGoogle Scholar
  3. Chan P, Dunn OJ (1972) The treatment of missing values in discriminant analysis. J Am Stat Assoc (6):473–477Google Scholar
  4. Cohen W (1995) Fast effective rule induction. In: Proceedings of the 12th international conference on machine learning (ICML), pp 115–123Google Scholar
  5. Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7: 1–30MathSciNetGoogle Scholar
  6. Domingos P, Hulten G (2000) Mining high speed data streams. In: Proceedings of the 6th ACM SIGKDD international conference on knowledge discovery and data mining (SIGKDD), pp 71–80Google Scholar
  7. Fan W (2004) Systematic data selection to mine concept-drifting data streams. In: Proceedings of the 10th ACM international conference on knowledge discovery and data mining (SIGKDD), pp 128–137Google Scholar
  8. Fayyad U, Irani K (1993) Multi-interval discretization of continuous-valued attributes for classification learning. In: 13th international joint conference of artificial intelligence, pp 1022–1027Google Scholar
  9. Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32: 675–701CrossRefGoogle Scholar
  10. Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. Ann Math Stat 11: 86–92zbMATHCrossRefGoogle Scholar
  11. Hashemi S, Yang Y, Pourkashani M, Kangavari M (2007) To better handle concept change and noise: a cellular automata approach to data stream classification. In: Australian joint conference on artificial intelligence, pp 669–674Google Scholar
  12. Haykin S (1994) Neural networks: a comprehensive foundation. Prentice Hall PTR, Upper Saddle River, NJ, USAzbMATHGoogle Scholar
  13. Ho SS (2005) A martingale framework for concept change detection in time-varying data streams. In: Proceedings of the 22nd international conference on machine learning (ICML), pp 321–327Google Scholar
  14. Hulten G, Spencer L, Domingos P (2001) Mining time-changing data streams. In: Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining (SIGKDD), pp 97–106Google Scholar
  15. Jang J-SR (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cybern 23: 665–684CrossRefGoogle Scholar
  16. Janikow CZ (1998) Fuzzy decision trees: issues and methods. IEEE Trans Syst Man Cybern B Cybern 28(1): 1–14CrossRefGoogle Scholar
  17. Janikow CZ, Kawa K (2005) Fuzzy decision tree fid. In: Annual meeting of the north American fuzzy information processing society, IEEE, pp 379–384Google Scholar
  18. Kolter JZ, Maloof MA (2003) Dynamic weighted majority: a new ensemble method for tracking concept drift. In: Proceedings of the 3rd IEEE international conference on data mining (ICDM), p 123Google Scholar
  19. Maher PE, Clair DCS (1993) Uncertain reasoning in an id3 machine learning framework. In: 2nd IEEE international conference on fuzzy systems, pp 7–12Google Scholar
  20. Mitchell TM (1997) Machine learning. McGraw HillGoogle Scholar
  21. Mitra S, Konwar KM, Pal SK (2002) Fuzzy decision tree, linguistic rules and fuzzy knowledge-based network: generation and evaluation. IEEE Trans Syst Man Cybern C Appl Rev 32(4): 328–339CrossRefGoogle Scholar
  22. Mundfrom DJ, Whitcomb A (1998) Imputing missing values: The effect on the accuracy of classification. Multiple Linear Regre Viewp 25(1): 13–19Google Scholar
  23. Newman DJ, Hettich S, Blake C, Merz C (1998) UCI repository of machine learning databasesGoogle Scholar
  24. Olaru C, Wehenkel L (2003) A complete fuzzy decision tree technique. Fuzzy Sets Syst 138: 221–254CrossRefMathSciNetGoogle Scholar
  25. Quinlan JR (1993) C4.5: Programs for machine learning. Morgan Kaufmann PublishersGoogle Scholar
  26. Quinlan JR (1993) Induction of decision trees, pp 349–361Google Scholar
  27. Saar-Tsechansky M, Provost F (2007) Handling missing values when applying classification models. J Mach Learn Res 8: 1625–1657Google Scholar
  28. Street WN, Kim Y (2001) A streaming ensemble algorithm (sea) for large-scale classification. In: Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining (SIGKDD), pp 377–382Google Scholar
  29. Tsymbal A (2004) The problem of concept drift: definitions and related work. Technical report TCD-CS-2004-15, Computer Science Department, Trinity College Dublin, IrelandGoogle Scholar
  30. Umanol M, Okamoto H, Hatono I, Tamura H, Kawachi F, Umedzu S, Kinoshita J (1994) Fuzzy decision trees by fuzzy id3 algorithm and its application to diagnosis systems. In: IEEE world congress on computational intelligence, pp 2113–2118Google Scholar
  31. Wang H, Fan W, Yu PS, Han J (2003) Mining concept drifting data streams using ensemble classifiers. In: Proceedings of the 9th ACM SIGKDD international conference on knowledge discovery and data mining (SIGKDD), pp 226–235Google Scholar
  32. Wang P, Wang H, Wu X, Wang W, Shi B (2005) On reducing classifier granularity in mining concept-drifting data streams. In: Proceedings of the 5th IEEE international conference on data mining (ICDM), pp 474–481Google Scholar
  33. Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Machine Learn 23: 69–101Google Scholar
  34. Yang Y, Wu X, Zhu X (2005) Combining proactive and reactive predictions for data streams. In: Proceedings of the 11th ACM international conference on knowledge discovery and data mining (SIGKDD), ACM Press, pp 710–715Google Scholar
  35. Yang Y, Wu X, Zhu X (2006) Mining in anticipation for concept change: proactive-reactive prediction in data streams. Data Mining Knowl Discov 13(3): 261–289CrossRefMathSciNetGoogle Scholar
  36. Zhu X, Wu X (2004) Class noise vs. attribute noise: a quantitative study. Artif Intell Rev 22(3): 177–210zbMATHCrossRefMathSciNetGoogle Scholar
  37. Zhu X, Wu X, Chen Q (2003) Eliminating class noise in large datasets. In: Proceedings of the 20th internationl conference in machine learning (ICML), pp 920–927Google Scholar
  38. Zhu X, Wu X, Yang Y (2004) Dynamic classifier selection for effective mining from noisy data streams. In: Proceedings of the 4th IEEE international conference on data mining (ICDM), pp 305–312Google Scholar
  39. Zhu X, Wu X, Yang Y (2006) Effective classification of noisy data streams with attribute-oriented dynamic classifier selection. Knowl Inf Syst 9(3): 339–363CrossRefMathSciNetGoogle Scholar
  40. Zimmermann HJ (2001) Fuzzy set theory and its applications. Kluwer Academic Publishers, Dordrecht, BostonGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.School of Electrical Engineering and Computer SciencesShiraz UniversityShirazIran
  2. 2.Australian Taxation OfficeMelbourneAustralia

Personalised recommendations