# On-line elimination of local redundancies in evolving fuzzy systems

## Abstract

In this paper, we examine approaches for reducing the complexity of evolving fuzzy systems (EFSs) by eliminating local redundancies during training, evolving the models on on-line data streams. Thus, the complexity reduction steps should support fast incremental single-pass processing steps. In EFSs, such reduction steps are important due to several reasons: (1) originally distinct rules representing distinct local regions in the input/output data space may move together over time and get significantly over-lapping as data samples are filling up the gaps in-between these, (2) two or several fuzzy sets in the fuzzy partitions may become redundant because of projecting high-dimensional clusters onto the single axes, (3) they can be also seen as a first step towards a better readability and interpretability of fuzzy systems, as unnecessary information is discarded and the models are made more transparent. One technique is performing a new rule merging approach directly in the product cluster space using a novel concept for calculating the similarity degree between an updated rule and the remaining ones. Inconsistent rules elicited by comparing the similarity of two redundant rule antecedent parts with the similarity of their consequents are specifically handled in the merging procedure. The second one is operating directly in the fuzzy partition space, where redundant fuzzy sets are merged based on their joint α-cut levels. Redundancy is measured by a novel kernel-based similarity measure. The complexity reduction approaches are evaluated based on high-dimensional noisy real-world measurements and an artificially generated data stream containing 1.2 million samples. Based on this empirical comparison, it will be shown that the novel techniques are (1) fast enough in order to cope with on-line demands and (2) produce fuzzy systems with less structural components while at the same time achieving accuracies similar to EFS not integrating any reduction steps.

## Keywords

Evolving fuzzy systems Redundancy elimination Similarity Complexity reduction On-line rule Fuzzy set merging## Notes

### Acknowledgments

This work was funded by the Austrian and German fund for promoting scientific research (FWF and DFG, contract number I328-N23, acronym IREFS). It reflects only the authors’ views. The residential premise data set was provided by Bogdan and Krysztof Trawinski. The authors also acknowledge Eyke Hüllermeier for providing valuable comments and suggestions on the manuscript.

## References

- Abraham W, Robins A (2005) Memory retention—the synaptic stability versus plasticity dilemma. Trends Neurosci 28(2):73–78CrossRefGoogle Scholar
- Angelov P (2010) Evolving takagi-sugeno fuzzy systems from streaming data, eTS+. In: Angelov P, Filev D, Kasabov N (eds) Evolving intelligent systems: methodology and applications. Wiley, New York, pp 21–50Google Scholar
- Angelov P, Filev D (2005) Simpl_eTS: A simplified method for learning evolving Takagi-Sugeno fuzzy models. In: Proceedings of FUZZ-IEEE 2005. Reno, Nevada, USA, pp 1068–1073Google Scholar
- Angelov P, Kordon A (2010) Evolving inferential sensors in the chemical process industry. In: Angelov P, Filev D, Kasabov N (eds) Evolving intelligent systems: methodology and applications. Wiley, New York, pp 313–336Google Scholar
- Babuska R (1998) Fuzzy modeling for control. Kluwer, NorwellGoogle Scholar
- Baturone I, Moreno-Velo F, Gersnoviez A (2006) A cad approach to simplify fuzzy system descriptions. In: Proceedings of the 2006 IEEE international conference on fuzzy systems. Vancouver, pp 2392–2399Google Scholar
- Bifet A, Holmes G, Kirkby R, Pfahringer B (2010) MOA: massive online analysis. J Mach Learning Res 11:1601–1604Google Scholar
- Breiman L, Friedman J, Stone C, Olshen R (1993) Classification and regression trees. Chapman & Hall, Boca RatonGoogle Scholar
- Burger M, Haslinger J, Bodenhofer U, Engl HW (2002) Regularized data-driven construction of fuzzy controllers. J Inverse Ill Posed Probl 10(4):319–344MathSciNetzbMATHGoogle Scholar
- Casillas J, Cordon O, Herrera F, Magdalena L (2003) Interpretability issues in fuzzy modeling. Springer, BerlinGoogle Scholar
- Chen M, Linkens D (2004) Rule-base self-generation and simplification for data-driven fuzzy models. Fuzzy Sets Syst 142(2):243–265MathSciNetzbMATHCrossRefGoogle Scholar
- Destercke S, Guillaume S, Charnomordic B (2007) Building an interpretable fuzzy rule base from data using orthogonal least squares—application to a depollution problem. Fuzzy Sets Syst 158(18):2078–2094MathSciNetzbMATHCrossRefGoogle Scholar
- Domingos P, Hulten G (2000) Mining high-speed data streams. In: Proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining. Boston, MA, pp 71–80Google Scholar
- Espinosa J, Vandewalle J (2000) Constructing fuzzy models with linguistic intergrity from numerical data—AFRELI algorithm. IEEE Trans Fuzzy Syst 8(5):591–600CrossRefGoogle Scholar
- Groißböock W, Lughofer E, Klement E (2004) A comparison of variable selection methods with the main focus on orthogonalization. In: Lopéz-Díaz M, Gil M, Grzegorzewski P, Hryniewicz O, Lawry J (eds) Soft methodology and random information systems. Advances in soft computing. Springer, Berlin, pp 479–486Google Scholar
- Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference and prediction, 2nd edn. Springer, BerlinGoogle Scholar
- Hathaway R, Bezdek J (1993) Switching regression models and fuzzy clustering. IEEE Trans Fuzzy Syst 1(3):195–204CrossRefGoogle Scholar
- Jimenez F, Gomez-Skarmeta AF, Sanchez G, Roubos H, Babuska R (2003) Accurate, transparent and compact fuzzy models by multi-objective evolutionary algorithms. In: Casillas J, Cordón O, Herrera F, Magdalena L (eds) Interpretability issues in fuzzy modeling. Studies in fuzziness and soft computing, vol 128. Springer, Berlin, pp 431–451Google Scholar
- Kasabov N (2007) Evolving connectionist systems: the knowledge engineering approach, 2nd edn. Springer, LondonGoogle Scholar
- Kasabov NK, Song Q (2002) DENFIS: dynamic evolving neural-fuzzy inference system and its application for time-series prediction. IEEE Trans Fuzzy Syst 10(2):144–154CrossRefGoogle Scholar
- Klement E, Mesiar R, Pap E (2000) Triangular norms. Kluwer, DordrechtGoogle Scholar
- Kordon A, Smits G, Kalos A, Jordaan E (2003) Robust soft sensor development using genetic programming. In: Leardi R (ed) Nature-inspired methods in chemometrics, pp 69–108Google Scholar
- Kurzhanskiy AA, Varaiya P (2006) Ellipsoidal toolbox. Tech. rep. UCB/EECS-2006-46, EECS Department, University of California, Berkeley. http://code.google.com/p/ellipsoids
- Lima E, Hell M, Ballini R, Gomide F (2010) Evolving fuzzy modeling using participatory learning. In: Angelov P, Filev D, Kasabov N (eds) Evolving intelligent systems: methodology and applications. Wiley, New York, pp 67–86Google Scholar
- Lughofer E (2006) Process safety enhancements for data-driven evolving fuzzy models. In: Proceedings of 2nd symposium on evolving fuzzy systems. Lake District, UK, pp 42–48Google Scholar
- Lughofer E (2008a) Extensions of vector quantization for incremental clustering. Pattern Recogn 41(3):995–1011zbMATHCrossRefGoogle Scholar
- Lughofer E (2008b) FLEXFIS: a robust incremental learning approach for evolving TS fuzzy models. IEEE Trans Fuzzy Syst 16(6):1393–1410CrossRefGoogle Scholar
- Lughofer E (2010) Towards robust evolving fuzzy systems. In: Angelov P, Filev D, Kasabov N (eds) Evolving intelligent systems: methodology and applications. Wiley, New York, pp 87–126Google Scholar
- Lughofer E (2011) Evolving fuzzy systems-Methodologies, advanced concepts and applications. Springer, Berlin. ISBN: 978-3-642-18086-6Google Scholar
- Lughofer E, Angelov P (2011) Handling drifts and shifts in on-line data streams with evolving fuzzy systems. Applied Soft Computing 11(2):2057–2068CrossRefGoogle Scholar
- Lughofer E, Hüllermeier E, Klement E (2005) Improving the interpretability of data-driven evolving fuzzy systems. In: Proceedings of EUSFLAT 2005. Barcelona, Spain, pp 28–33Google Scholar
- Lughofer E, Macian V, Guardiola C, Klement E (2010) Data-driven design of takagi-sugeno fuzzy systems for predicting NOx emissions. In: Hüllermeier E, Kruse R, Hoffmann F (eds) Proc. of the 13th international conference on information processing and management of uncertainty, IPMU 2010, part II (applications), CCIS, vol 81. Springer, Dortmund, pp 1–10Google Scholar
- Mikut R, Mäkel J, Gröll L (2005) Interpretability issues in data-based learning of fuzzy systems. Fuzzy Sets Syst 150(2):179–197zbMATHCrossRefGoogle Scholar
- Moser B A similarity measure for images and volumetric data based on Hermann Weyl’s discrepancy. IEEE Trans Pattern Anal Mach Intell (2010). doi: 10.1109/TPAMI.2009.50
- Nelles O (2001) Nonlinear system identification. Springer, BerlinGoogle Scholar
- Oliveira JVD (1999) Semantic constraints for membership function optimization. IEEE Trans Syst Man Cybern Part A Syst Hum 29(1):128–138CrossRefGoogle Scholar
- Pang S, Ozawa S, Kasabov N (2005) Incremental linear discriminant analysis for classification of data streams. IEEE Trans Syst Man Cybern Part B Cybern 35(5):905–914CrossRefGoogle Scholar
- Qin S, Li W, Yue H (2000) Recursive PCA for adaptive process monitoring. J Process Control 10:471–486CrossRefGoogle Scholar
- Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann, San FranciscoGoogle Scholar
- Ramos J, Dourado A (2006) Pruning for interpretability of large spanned eTS. In: Proceedings of the 2006 international symposium on evolving fuzzy systems. Ambleside, UK, pp 55–60Google Scholar
- Rong HJ, Sundararajan N, Huang GB, Saratchandran P (2006) Sequential adaptive fuzzy inference system (SAFIS) for nonlinear system identification and prediction. Fuzzy Sets Syst 157(9):1260–1275MathSciNetzbMATHCrossRefGoogle Scholar
- Rong HJ, Sundararajan N, Huang GB, Zhao GS (2011) Extended sequential adaptive fuzzy inference system for classification problems. Evol Syst (in press). doi: 10.1007/s12530-010-9023-9
- Ros L, Sabater A, Thomas F (2002) An ellipsoidal calculus based on propagation and fusion. IEEE Trans Syst Man Cybern Part B Cybern 32(4):430–442CrossRefGoogle Scholar
- Setnes M (2003) Simplification and reduction of fuzzy rules. In: Casillas J, Cordón O, Herrera F, Magdalena L (eds) Interpretability issues in fuzzy modeling. Studies in fuzziness and soft computing, vol 128. Springer, Berlin, pp 278–302Google Scholar
- Setnes M, Babuska R, Kaymak U, Lemke H (1998a) Similarity measures in fuzzy rule base simplification. IEEE Trans Syst Man Cybern Part B Cybern 28(3):376–386CrossRefGoogle Scholar
- Setnes M, Babuska R, Verbruggen H (1998b) Complexity reduction in fuzzy modeling. Math Comput Simul 46(5-6):509–518CrossRefGoogle Scholar
- Shaker A, Senge R, Hüllermeier E (2011) Evolving fuzzy pattern trees for binary classification on data streams. Inf Sci (in press)Google Scholar
- Stone M (1974) Cross-validatory choice and assessment of statistical predictions. J R Stat Soc 36:111–147zbMATHGoogle Scholar
- Takagi T, Sugeno M (1985) Fuzzy identification of systems and its applications to modeling and control. IEEE Trans Syst Man Cybern 15(1):116–132zbMATHGoogle Scholar
- Wang L (1992) Fuzzy systems are universal approximators. In: Proceedings of the 1st IEEE conf. fuzzy systems. San Diego, CA, pp 1163–1169Google Scholar
- Wang L, Mendel J (1992) Fuzzy basis functions, universal approximation and orthogonal least-squares learning. IEEE Trans Neural Netw 3(5):807–814CrossRefGoogle Scholar
- Yager RR (1990) A model of participatory learning. IEEE Trans Syst Man Cybern 20:1229–1234MathSciNetCrossRefGoogle Scholar
- Yen J, Wang L, Gillespie C (1998) Improving the interpretability of TSK fuzzy models by combining global learning and local learning. IEEE Trans Fuzzy Syst 6(4):530–537CrossRefGoogle Scholar
- Zhou S, Gan J (2008) Low-level interpretability and high-level interpretability: a unified view of data-driven interpretable fuzzy systems modelling. Fuzzy Seta Syst 159(23):3091–3131MathSciNetCrossRefGoogle Scholar