Soft Computing

, Volume 15, Issue 12, pp 2389–2414 | Cite as

Fuzzy knowledge representation study for incremental learning in data streams and classification problems

  • Albert Orriols-Puig
  • Jorge Casillas


The extraction of models from data streams has become a hot topic in data mining due to the proliferation of problems in which data are made available online. This has led to the design of several systems that create data models online. A novel approach to online learning of data streams can be found in Fuzzy-UCS, a young Michigan-style fuzzy-classifier system that has recently demonstrated to be highly competitive in extracting classification models from complex domains. Despite the promising results reported for Fuzzy-UCS, there still remain some hot issues that need to be analyzed in detail. This paper carefully studies two key aspects in Fuzzy-UCS: the ability of the system to learn models from data streams where concepts change over time and the behavior of different fuzzy representations. Four fuzzy representations that move through the dimensions of flexibility and interpretability are included in the system. The behavior of the different representations on a problem with concept changes is studied and compared to other machine learning techniques prepared to deal with these types of problems. Thereafter, the comparison is extended to a large collection of real-world problems, and a close examination of which problem characteristics benefit or affect the different representations is conducted. The overall results show that Fuzzy-UCS can effectively deal with problems with concept changes and lead to different interesting conclusions on the particular behavior of each representation.


Fuzzy rule-based representation Genetic algorithms Learning classifier systems Genetic fuzzy systems Data streams Concept drift 



The authors would like to warmly thank Marlon Núñez, Raúl Fidalgo, and Rafael Morales from the Universidad de Málaga for providing us with the experimental results of OnlineTree2. The authors thank the support of Ministerio de Ciencia y Tecnología under projects TIN2008-06681-C06-01 and TIN2008-06681-C06-05, Generalitat de Catalunya under Grant 2005SGR-00302, and Andalusian Government under grant P07-TIC-3185.


  1. Abbass HA, Bacardit J, Butz MV, Llorà X (2004) Online adaptation in learning classifier systems: stream data mining. IlliGAL report 2004031. Illinois Genetic Algorithms Laboratory, University of Illinois at Urbana-ChampaignGoogle Scholar
  2. Aggarwal C (ed) (2007) Data streams: models and algorithms. Springer, BerlinGoogle Scholar
  3. Aha D, Kibler D, Albert M (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66Google Scholar
  4. Alcalá R, Casillas J, Cordón O, Herrera F (2001) Building fuzzy graphs: features and taxonomy of learning for non-grid-oriented fuzzy rule-based systems. J Intell Fuzzy Syst 11(3–4):99–119Google Scholar
  5. Angelov P, Lughofer E, Zhou X (2008) Evolving fuzzy classifiers using different model architectures. Fuzzy Sets Syst 159(23):3160–3182MathSciNetzbMATHCrossRefGoogle Scholar
  6. Asuncion A, Newman DJ (2007) UCI Machine Learning Repository. University of California.∼mlearn/MLRepository.html
  7. Bacardit J, Butz MV (2004) Data mining in learning classifier systems: comparing XCS with GAssist. In: Proceedings of the 7th international workshop on learning classifier systems. SpringerGoogle Scholar
  8. Botta A, Lazzerini B, Marcelloni F, Stefanescu DC (2009) Context adaptation of fuzzy systems through a multi-objective evolutionary approach based on a novel interpretability index. Soft Comput 13(5):437–449CrossRefGoogle Scholar
  9. Butz MV, Sastry K, Goldberg DE (2005) Strong, stable, and reliable fitness pressure in XCS due to tournament selection. Genet Program Evolvable Mach 6(1):53–77CrossRefGoogle Scholar
  10. Carse B, Fogarty TC, Munro A (1996) Evolving fuzzy rule based controllers using genetic algorithms. Fuzzy Sets Syst 80:273–294CrossRefGoogle Scholar
  11. Casillas J, Cordón O, del Jesus MJ, Herrera F (2005) Genetic tuning of fuzzy rule deep structures preserving interpretability and its interaction with fuzzy rule set reduction. IEEE Trans Fuzzy Syst 13(1):13–29CrossRefGoogle Scholar
  12. Choi JN, Oh SK, Pedrycz W (2008) Identification of fuzzy models using a successive tuning method with a variant identification ratio. Fuzzy Sets Syst 159(21):2873–2889MathSciNetzbMATHCrossRefGoogle Scholar
  13. Cooper MG, Vidal JJ (1994) Genetic design of fuzzy controllers: the cart and jointed pole problem. In: Proceedings of the 3rd IEEE international conference on fuzzy systems, Piscataway, NJ, USA, pp 1332–1337Google Scholar
  14. Cordón O, Herrera F (1997) A three-stage evolutionary process for learning descriptive and approximate fuzzy logic controller knowledge bases from examples. Int J Approx Reason 17(4):369–407zbMATHCrossRefGoogle Scholar
  15. Cordón O, del Jesus MJ, Herrera F (1999) A proposal on reasoning methods in fuzzy rule-based classification systems. Int J Approx Reason 20(1):21–45Google Scholar
  16. del Campo-Ávila J, Ramos-Jiménez G, Gama J, Morales-Bueno R (2008) Improving the performance of an incremental algorithm driven by error margins. Intell Data Anal 12(3):305–318Google Scholar
  17. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
  18. Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10(7):1895–1924CrossRefGoogle Scholar
  19. Domingos P, Hulten G (2000) Mining high-speed data streams. In: KDD’00: Proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, pp 71–80Google Scholar
  20. Fernández A, del Jesus M, Herrera F (2009) Hierarchical fuzzy rule based classification systems with genetic rule selection for imbalanced data-sets. Int J Approx Reason 50(3):561–577zbMATHCrossRefGoogle Scholar
  21. Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32:675–701CrossRefGoogle Scholar
  22. Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. Ann Math Stat 11:86–92zbMATHCrossRefGoogle Scholar
  23. Fritzke B (1997) Incremental neuro-fuzzy systems. In: Proceedings of the international society for optical engineering: applications of soft computing, vol 3165, pp 86–97Google Scholar
  24. Gacto M, Alcalá R, Herrera F (2009) Adaptation and application of multi-objective evolutionary algorithms for rule reduction and parameter tuning of fuzzy rule-based systems. Soft Comput 13(5):419–436CrossRefGoogle Scholar
  25. Gama J, Gaber MM (eds) (2007) Learning from data streams. Springer, HeidelbergGoogle Scholar
  26. Gama J, Rocha R, Medas P (2003) Accurate decision trees for mining high-speed data streams. In: KDD ’03: Proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining, ACM, New York, NY, USA, pp 523–528Google Scholar
  27. Gama J, Medas P, Castillo G, Rodrigues P (2004) Learning with drift detection. In: Proceedings of the 17th Brasilian symposium on artificial intelligence, pp 286–295Google Scholar
  28. García S, Herrera F (2008) An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J Mach Learn Res 9:2677–2694Google Scholar
  29. Goldberg DE (1989) Genetic algorithms in search, optimization & machine learning, 1st edn. Addison Wesley, Reading, MAGoogle Scholar
  30. Goldberg DE (2002) The design of innovation: lessons from and for competent genetic algorithms, 1st edn. Kluwer Academic Publishers, Boston, MAGoogle Scholar
  31. González A, Pérez R (1998) Completeness and consistency conditions for learning fuzzy rules. Fuzzy Sets Syst 96(1):37–51CrossRefGoogle Scholar
  32. Ho TK, Basu M (2002) Complexity measures of supervised classification problems. IEEE Trans Pattern Anal Mach Intell 24(3):289–300CrossRefGoogle Scholar
  33. Ho TK, Basu M, Law M (2006) Measures of geometrical complexity in classification problems. In: Basu M, Ho TK (eds) Data complexity in pattern recognition. Springer, London, pp 1–23Google Scholar
  34. Holland JH (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbor, MIGoogle Scholar
  35. Ishibuchi H, Nakashima T (2001) Effect of rule weights in fuzzy rule-based classification systems. IEEE Trans Fuzzy Syst 9(4):506–515CrossRefGoogle Scholar
  36. Ishibuchi H, Nojima Y (2007) Analysis of interpretability-accuracy tradeoff of fuzzy systems by multiobjective fuzzy genetics-based machine learning. Int J Approx Reason 44(1):4–31MathSciNetzbMATHCrossRefGoogle Scholar
  37. Ishibuchi H, Yamamoto T (2005) Rule weight specification in fuzzy rule-based classification systems. IEEE Trans Fuzzy Syst 13(4):428–435CrossRefGoogle Scholar
  38. Ishibuchi H, Nozaki K, Tanaka H (1992) Distributed representation of fuzzy rules and its application to pattern classification. Fuzzy Sets Syst 52(1):21–32CrossRefGoogle Scholar
  39. Ishibuchi H, Murata T, Türkşen IB (1997) Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems. Fuzzy Sets Syst 89(2):135–150CrossRefGoogle Scholar
  40. Ishibuchi H, Nakashima T, Murata T (1999) Performance evaluation of fuzzy classifier systems for multidimensional pattern classification problems. IEEE Trans Syst Man Cybern Part B Cybern 29(5):601–618CrossRefGoogle Scholar
  41. Ishibuchi H, Yamamoto T, Nakashima T (2005) Hybridization of fuzzy GBML approaches for pattern classification problems. IEEE Trans Syst Man Cybern Part B Cybern 35(2):359–365CrossRefGoogle Scholar
  42. Ishibuchi H, Kaisho Y, Nojima Y (2009) Complexity, interpretability and explanation capability of fuzzy rule-based classifiers. In: Proceedings of the IEEE international conference on fuzzy systems, Jeju, Korea, pp 1730–1735Google Scholar
  43. Jong KAD, Spears WM, Gordon D (1993) Using genetic algorithms for concept learning. Genet algorithms Mach Learn (Special Issue of Machine Learning) 13(2–3):161–188Google Scholar
  44. Llorà X, Sastry K (2006) Fast rule matching for learning classifier systems via vector instructions. In: GECCO ’06: Proceedings of the 8th annual conference on genetic and evolutionary computation. ACM, New York, NY, USA, pp 1513–1520Google Scholar
  45. Luengo J, Herrera F (2010) Domains of competence of fuzzy rule based classification systems with data complexity measures: a case of study using a fuzzy hybrid genetic based machine learning method. Fuzzy Sets Syst 161(1):3–19MathSciNetCrossRefGoogle Scholar
  46. Maloof MA, Michalski RS (2004) Incremental learning with partial instance memory. Artif Intell 154(1–2):95–126MathSciNetzbMATHCrossRefGoogle Scholar
  47. Michalski R (1983) A theory and methodology of inductive learning. Artif Intell 20:111–161MathSciNetCrossRefGoogle Scholar
  48. Nemenyi PB (1963) Distribution-free multiple comparisons. PhD thesis. Princeton University, New Jersey, USAGoogle Scholar
  49. Núñez M, Fidalgo R, Morales R (2007) Learning in environments with unknown dynamics: towards more robust concept learners. J Mach Learn Res 8:2595–2628MathSciNetzbMATHGoogle Scholar
  50. Orriols-Puig A, Casillas J, Bernadó-Mansilla E (2008) Evolving fuzzy rules with UCS. In: Advances at the frontier of LCSs. Springer, LNCS, vol 4998, pp 57–76Google Scholar
  51. Orriols-Puig A, Casillas J, Bernadó-Mansilla E (2009) Fuzzy-UCS: a Michigan-style learning fuzzy-classifier system for supervised learning. IEEE Trans Evolut Comput 13(2):1093–1119CrossRefGoogle Scholar
  52. Pulkkinen P, Koivisto H (2010) A dynamically constrained multiobjective genetic fuzzy system for regression problems. IEEE Trans Fuzzy Syst 18(1):161–177CrossRefGoogle Scholar
  53. Quinlan JR (1993) C4.5: Programs for machine learning. Morgan Kaufmann Publishers, San Mateo, CAGoogle Scholar
  54. Sheskin D (2000) Handbook of parametric and nonparametric statistical procedures. Chapman & Hall, Boca RatonGoogle Scholar
  55. Street WN, Kim Y (2001) A streaming ensemble algorithm (sea) for large-scale classification. In: KDD ’01: Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, pp 377–382Google Scholar
  56. Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23(1):69–101Google Scholar
  57. Wilson SW (1998) Generalization in the XCS classifier system. In: Proceedings of the 3rd annual conference on genetic programming, Morgan Kaufmann, pp 665–674Google Scholar
  58. Witten I, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San FranciscoGoogle Scholar
  59. Wu X, Kumar V, Quinlan JR, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS, Zhou ZH, Steinbach M, Hand DJ, Steinberg D (2007) Top 10 algorithms in data mining. Knowl Inform Syst 14(1):1–37CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  1. 1.Grup de Recerca en Sistemes Intel·ligentsLa Salle-Universitat Ramon LlullBarcelonaSpain
  2. 2.Department of Computer Science and Artificial Intelligence, Research Center on Communication and Information Technology (CITIC-UGR)University of GranadaGranadaSpain

Personalised recommendations