# Fuzzy knowledge representation study for incremental learning in data streams and classification problems

- 236 Downloads
- 13 Citations

## Abstract

The extraction of models from data streams has become a hot topic in data mining due to the proliferation of problems in which data are made available online. This has led to the design of several systems that create data models online. A novel approach to online learning of data streams can be found in Fuzzy-UCS, a young Michigan-style fuzzy-classifier system that has recently demonstrated to be highly competitive in extracting classification models from complex domains. Despite the promising results reported for Fuzzy-UCS, there still remain some hot issues that need to be analyzed in detail. This paper carefully studies two key aspects in Fuzzy-UCS: the ability of the system to learn models from data streams where concepts change over time and the behavior of different fuzzy representations. Four fuzzy representations that move through the dimensions of flexibility and interpretability are included in the system. The behavior of the different representations on a problem with concept changes is studied and compared to other machine learning techniques prepared to deal with these types of problems. Thereafter, the comparison is extended to a large collection of real-world problems, and a close examination of which problem characteristics benefit or affect the different representations is conducted. The overall results show that Fuzzy-UCS can effectively deal with problems with concept changes and lead to different interesting conclusions on the particular behavior of each representation.

## Keywords

Fuzzy rule-based representation Genetic algorithms Learning classifier systems Genetic fuzzy systems Data streams Concept drift## Notes

### Acknowledgments

The authors would like to warmly thank Marlon Núñez, Raúl Fidalgo, and Rafael Morales from the *Universidad de Málaga* for providing us with the experimental results of OnlineTree2. The authors thank the support of *Ministerio de Ciencia y Tecnología* under projects TIN2008-06681-C06-01 and TIN2008-06681-C06-05, *Generalitat de Catalunya* under Grant 2005SGR-00302, and *Andalusian Government* under grant P07-TIC-3185.

## References

- Abbass HA, Bacardit J, Butz MV, Llorà X (2004) Online adaptation in learning classifier systems: stream data mining. IlliGAL report 2004031. Illinois Genetic Algorithms Laboratory, University of Illinois at Urbana-ChampaignGoogle Scholar
- Aggarwal C (ed) (2007) Data streams: models and algorithms. Springer, BerlinGoogle Scholar
- Aha D, Kibler D, Albert M (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66Google Scholar
- Alcalá R, Casillas J, Cordón O, Herrera F (2001) Building fuzzy graphs: features and taxonomy of learning for non-grid-oriented fuzzy rule-based systems. J Intell Fuzzy Syst 11(3–4):99–119Google Scholar
- Angelov P, Lughofer E, Zhou X (2008) Evolving fuzzy classifiers using different model architectures. Fuzzy Sets Syst 159(23):3160–3182MathSciNetzbMATHCrossRefGoogle Scholar
- Asuncion A, Newman DJ (2007) UCI Machine Learning Repository. University of California. http://www.ics.uci.edu/∼mlearn/MLRepository.html
- Bacardit J, Butz MV (2004) Data mining in learning classifier systems: comparing XCS with GAssist. In: Proceedings of the 7th international workshop on learning classifier systems. SpringerGoogle Scholar
- Botta A, Lazzerini B, Marcelloni F, Stefanescu DC (2009) Context adaptation of fuzzy systems through a multi-objective evolutionary approach based on a novel interpretability index. Soft Comput 13(5):437–449CrossRefGoogle Scholar
- Butz MV, Sastry K, Goldberg DE (2005) Strong, stable, and reliable fitness pressure in XCS due to tournament selection. Genet Program Evolvable Mach 6(1):53–77CrossRefGoogle Scholar
- Carse B, Fogarty TC, Munro A (1996) Evolving fuzzy rule based controllers using genetic algorithms. Fuzzy Sets Syst 80:273–294CrossRefGoogle Scholar
- Casillas J, Cordón O, del Jesus MJ, Herrera F (2005) Genetic tuning of fuzzy rule deep structures preserving interpretability and its interaction with fuzzy rule set reduction. IEEE Trans Fuzzy Syst 13(1):13–29CrossRefGoogle Scholar
- Choi JN, Oh SK, Pedrycz W (2008) Identification of fuzzy models using a successive tuning method with a variant identification ratio. Fuzzy Sets Syst 159(21):2873–2889MathSciNetzbMATHCrossRefGoogle Scholar
- Cooper MG, Vidal JJ (1994) Genetic design of fuzzy controllers: the cart and jointed pole problem. In: Proceedings of the 3rd IEEE international conference on fuzzy systems, Piscataway, NJ, USA, pp 1332–1337Google Scholar
- Cordón O, Herrera F (1997) A three-stage evolutionary process for learning descriptive and approximate fuzzy logic controller knowledge bases from examples. Int J Approx Reason 17(4):369–407zbMATHCrossRefGoogle Scholar
- Cordón O, del Jesus MJ, Herrera F (1999) A proposal on reasoning methods in fuzzy rule-based classification systems. Int J Approx Reason 20(1):21–45Google Scholar
- del Campo-Ávila J, Ramos-Jiménez G, Gama J, Morales-Bueno R (2008) Improving the performance of an incremental algorithm driven by error margins. Intell Data Anal 12(3):305–318Google Scholar
- Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
- Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10(7):1895–1924CrossRefGoogle Scholar
- Domingos P, Hulten G (2000) Mining high-speed data streams. In: KDD’00: Proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, pp 71–80Google Scholar
- Fernández A, del Jesus M, Herrera F (2009) Hierarchical fuzzy rule based classification systems with genetic rule selection for imbalanced data-sets. Int J Approx Reason 50(3):561–577zbMATHCrossRefGoogle Scholar
- Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32:675–701CrossRefGoogle Scholar
- Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. Ann Math Stat 11:86–92zbMATHCrossRefGoogle Scholar
- Fritzke B (1997) Incremental neuro-fuzzy systems. In: Proceedings of the international society for optical engineering: applications of soft computing, vol 3165, pp 86–97Google Scholar
- Gacto M, Alcalá R, Herrera F (2009) Adaptation and application of multi-objective evolutionary algorithms for rule reduction and parameter tuning of fuzzy rule-based systems. Soft Comput 13(5):419–436CrossRefGoogle Scholar
- Gama J, Gaber MM (eds) (2007) Learning from data streams. Springer, HeidelbergGoogle Scholar
- Gama J, Rocha R, Medas P (2003) Accurate decision trees for mining high-speed data streams. In: KDD ’03: Proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining, ACM, New York, NY, USA, pp 523–528Google Scholar
- Gama J, Medas P, Castillo G, Rodrigues P (2004) Learning with drift detection. In: Proceedings of the 17th Brasilian symposium on artificial intelligence, pp 286–295Google Scholar
- García S, Herrera F (2008) An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J Mach Learn Res 9:2677–2694Google Scholar
- Goldberg DE (1989) Genetic algorithms in search, optimization & machine learning, 1st edn. Addison Wesley, Reading, MAGoogle Scholar
- Goldberg DE (2002) The design of innovation: lessons from and for competent genetic algorithms, 1st edn. Kluwer Academic Publishers, Boston, MAGoogle Scholar
- González A, Pérez R (1998) Completeness and consistency conditions for learning fuzzy rules. Fuzzy Sets Syst 96(1):37–51CrossRefGoogle Scholar
- Ho TK, Basu M (2002) Complexity measures of supervised classification problems. IEEE Trans Pattern Anal Mach Intell 24(3):289–300CrossRefGoogle Scholar
- Ho TK, Basu M, Law M (2006) Measures of geometrical complexity in classification problems. In: Basu M, Ho TK (eds) Data complexity in pattern recognition. Springer, London, pp 1–23Google Scholar
- Holland JH (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbor, MIGoogle Scholar
- Ishibuchi H, Nakashima T (2001) Effect of rule weights in fuzzy rule-based classification systems. IEEE Trans Fuzzy Syst 9(4):506–515CrossRefGoogle Scholar
- Ishibuchi H, Nojima Y (2007) Analysis of interpretability-accuracy tradeoff of fuzzy systems by multiobjective fuzzy genetics-based machine learning. Int J Approx Reason 44(1):4–31MathSciNetzbMATHCrossRefGoogle Scholar
- Ishibuchi H, Yamamoto T (2005) Rule weight specification in fuzzy rule-based classification systems. IEEE Trans Fuzzy Syst 13(4):428–435CrossRefGoogle Scholar
- Ishibuchi H, Nozaki K, Tanaka H (1992) Distributed representation of fuzzy rules and its application to pattern classification. Fuzzy Sets Syst 52(1):21–32CrossRefGoogle Scholar
- Ishibuchi H, Murata T, Türkşen IB (1997) Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems. Fuzzy Sets Syst 89(2):135–150CrossRefGoogle Scholar
- Ishibuchi H, Nakashima T, Murata T (1999) Performance evaluation of fuzzy classifier systems for multidimensional pattern classification problems. IEEE Trans Syst Man Cybern Part B Cybern 29(5):601–618CrossRefGoogle Scholar
- Ishibuchi H, Yamamoto T, Nakashima T (2005) Hybridization of fuzzy GBML approaches for pattern classification problems. IEEE Trans Syst Man Cybern Part B Cybern 35(2):359–365CrossRefGoogle Scholar
- Ishibuchi H, Kaisho Y, Nojima Y (2009) Complexity, interpretability and explanation capability of fuzzy rule-based classifiers. In: Proceedings of the IEEE international conference on fuzzy systems, Jeju, Korea, pp 1730–1735Google Scholar
- Jong KAD, Spears WM, Gordon D (1993) Using genetic algorithms for concept learning. Genet algorithms Mach Learn (Special Issue of Machine Learning) 13(2–3):161–188Google Scholar
- Llorà X, Sastry K (2006) Fast rule matching for learning classifier systems via vector instructions. In: GECCO ’06: Proceedings of the 8th annual conference on genetic and evolutionary computation. ACM, New York, NY, USA, pp 1513–1520Google Scholar
- Luengo J, Herrera F (2010) Domains of competence of fuzzy rule based classification systems with data complexity measures: a case of study using a fuzzy hybrid genetic based machine learning method. Fuzzy Sets Syst 161(1):3–19MathSciNetCrossRefGoogle Scholar
- Maloof MA, Michalski RS (2004) Incremental learning with partial instance memory. Artif Intell 154(1–2):95–126MathSciNetzbMATHCrossRefGoogle Scholar
- Michalski R (1983) A theory and methodology of inductive learning. Artif Intell 20:111–161MathSciNetCrossRefGoogle Scholar
- Nemenyi PB (1963) Distribution-free multiple comparisons. PhD thesis. Princeton University, New Jersey, USAGoogle Scholar
- Núñez M, Fidalgo R, Morales R (2007) Learning in environments with unknown dynamics: towards more robust concept learners. J Mach Learn Res 8:2595–2628MathSciNetzbMATHGoogle Scholar
- Orriols-Puig A, Casillas J, Bernadó-Mansilla E (2008) Evolving fuzzy rules with UCS. In: Advances at the frontier of LCSs. Springer, LNCS, vol 4998, pp 57–76Google Scholar
- Orriols-Puig A, Casillas J, Bernadó-Mansilla E (2009) Fuzzy-UCS: a Michigan-style learning fuzzy-classifier system for supervised learning. IEEE Trans Evolut Comput 13(2):1093–1119CrossRefGoogle Scholar
- Pulkkinen P, Koivisto H (2010) A dynamically constrained multiobjective genetic fuzzy system for regression problems. IEEE Trans Fuzzy Syst 18(1):161–177CrossRefGoogle Scholar
- Quinlan JR (1993) C4.5: Programs for machine learning. Morgan Kaufmann Publishers, San Mateo, CAGoogle Scholar
- Sheskin D (2000) Handbook of parametric and nonparametric statistical procedures. Chapman & Hall, Boca RatonGoogle Scholar
- Street WN, Kim Y (2001) A streaming ensemble algorithm (sea) for large-scale classification. In: KDD ’01: Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, pp 377–382Google Scholar
- Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23(1):69–101Google Scholar
- Wilson SW (1998) Generalization in the XCS classifier system. In: Proceedings of the 3rd annual conference on genetic programming, Morgan Kaufmann, pp 665–674Google Scholar
- Witten I, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San FranciscoGoogle Scholar
- Wu X, Kumar V, Quinlan JR, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS, Zhou ZH, Steinbach M, Hand DJ, Steinberg D (2007) Top 10 algorithms in data mining. Knowl Inform Syst 14(1):1–37CrossRefGoogle Scholar