International Journal of Fuzzy Systems

, Volume 19, Issue 5, pp 1645–1658 | Cite as

Top–Down Sparse Fuzzy Regression Modeling from Data with Improved Coverage

  • Edwin Lughofer
  • Stefan Kindermann
  • Mahardhika Pratama
  • Jose de Jesus Rubio


We propose a new fuzzy modeling algorithm from data for regression problems. It acts in a top–down manner by allowing the user to specify an upper number of allowed rules in the rule base which is sparsed out with the usage of an iterative constrained numerical optimization procedure. It is based on the combination of the least squares error and the sum of rule weights over all rules to achieve minimal error with lowest possible number of significantly active rules. Two major novel concepts are integrated into the optimization process: the first respects a minimal coverage degree of the sample space in order to approach \(\epsilon \)-completeness of the rule base (an important interpretability criterion) and the second optimizes the positioning and ranges of influence of the rules, which is done synchronously to the optimization of the rule weights within an intervened, homogeneous procedure. Based on empirical results achieved for several high-dimensional (partially noisy) data sets, it can be shown that our advanced, intervened optimization yields fuzzy systems with a better coverage and a higher degree of \(\epsilon \)-completeness compared to the fuzzy models achieved by related data-driven fuzzy modeling methods. This is even achieved with a significantly lower or at least equal number of rules and with a similar model error on separate validation data.


Top–down fuzzy system modeling Rule base out-sparseing Constrained numerical optimization problem Penalty term Coverage \(\epsilon \)-completeness Intervened Homogeneous optimization 



The first author acknowledges the support of the Austrian COMET-K2 programme of the Linz Center of Mechatronics (LCM), funded by the Austrian federal government and the federal state of Upper Austria, and the support of the COMET Project ’Heuristic Optimization in Production and Logistics’ (HOPL), #843532 funded by the Austrian Research Promotion Agency (FFG). This publication reflects only the authors’ views.


  1. 1.
    Akerkar, R., Sajja, P.: Knowledge-Based Systems. Jones & Bartlett Learning, Sudbury (2009)Google Scholar
  2. 2.
    Babuska, R.: Fuzzy Modeling for Control. Kluwer Academic Publishers, Norwell (1998)CrossRefGoogle Scholar
  3. 3.
    Castro, J., Delgado, M.: Fuzzy systems with defuzzification are universal approximators. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 26(1), 149–152 (1996)CrossRefGoogle Scholar
  4. 4.
    Celikyilmaz, A., Türksen, I.: Modeling Uncertainty with Fuzzy Logic: With Recent Theory and Applications. Springer, Berlin (2009)CrossRefzbMATHGoogle Scholar
  5. 5.
    Cernuda, C., Lughofer, E., Röder, T., Märzinger, W., Reischer, T., Pawliczek, M., Brandstetter, M.: Self-adaptive non-linear methods for improved multivariate calibration in chemical processes. Lenzing. Ber. 92, 12–32 (2015)Google Scholar
  6. 6.
    Chiu, S.: Fuzzy model identification based on cluster estimation. J. Intell. Fuzzy Syst. 2(3), 267–278 (1994)Google Scholar
  7. 7.
    Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best \(k\)-term approximation. J. Am. Math. Soc. 22(1), 211–231 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Cordon, O., Gomide, F., Herrera, F., Hoffmann, F., Magdalena, L.: Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Sets Syst. 141(1), 5–31 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Daubechies, I., Defrise, M., Mol, C.D.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Dennis, J.E., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall Series in Computational Mathematics, Englewood Cliffs (1983)Google Scholar
  11. 11.
    Fletcher, R.: Practical Methods of Optimization. Wiley, New York (2000)CrossRefzbMATHGoogle Scholar
  12. 12.
    Gacto, M., Alcala, R., Herrera, F.: Interpretability of linguistic fuzzy rule-based systems: an overview of interpretability measures. Inf. Sci. 181(20), 4340–4360 (2011)CrossRefGoogle Scholar
  13. 13.
    Gray, R.: Vector quantization. IEEE ASSP Mag. 1(2), 4–29 (1984)CrossRefGoogle Scholar
  14. 14.
    Gustafson, D., Kessel, W.: Fuzzy clustering with a fuzzy covariance matrix. In: Proceedings of the IEEE CDC Conference, pp. 761–766. San Diego, CA (1979)Google Scholar
  15. 15.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2nd edn. Springer, New York, Berlin Heidelberg (2009)CrossRefzbMATHGoogle Scholar
  16. 16.
    Hensel, A., Spittel, T.: Kraft- und Arbeitsbedarf bildsamer Formgebungsverfahren. VEB Deutscher Verlag für Grundstoffindustrie (1978)Google Scholar
  17. 17.
    Iglesias, J., Angelov, P., Ledezma, A., Sanchis, A.: Evolving classification of agent’s behaviors: a general approach. Evol. Syst. 1(3), 161–172 (2010)CrossRefGoogle Scholar
  18. 18.
    Iglesias, J., Tiemblo, A., Ledezma, A., Sanchis, A.: Web news mining in an evolving framework. Inf. Fusion 28, 90–98 (2016)CrossRefGoogle Scholar
  19. 19.
    J. Casillas, F.H., Pereza, R., Jesus, M.D., Villar, P.: Special issue on genetic fuzzy systems and the interpretability-accuracy trade-off. Int. J. Approx. Reason. 44(1), 1–3 (2007)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Klement, E., Mesiar, R., Pap, E.: Triangular Norms. Kluwer Academic Publishers, Dordrecht, Norwell, New York, London (2000)CrossRefzbMATHGoogle Scholar
  21. 21.
    Lughofer, E.: FLEXFIS: a robust incremental learning approach for evolving TS fuzzy models. IEEE Trans. Fuzzy Syst. 16(6), 1393–1410 (2008)CrossRefGoogle Scholar
  22. 22.
    Lughofer, E.: Evolving Fuzzy Systems—Methodologies, Advanced Concepts and Applications. Springer, Berlin Heidelberg (2011)CrossRefzbMATHGoogle Scholar
  23. 23.
    Lughofer, E.: On-line assurance of interpretability criteria in evolving fuzzy systems—achievements, new concepts and open issues. Inf. Sci. 251, 22–46 (2013)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Lughofer, E., Cernuda, C., Kindermann, S., Pratama, M.: Generalized smart evolving fuzzy systems. Evol. Syst. 6(4), 269–292 (2015)CrossRefGoogle Scholar
  25. 25.
    Lughofer, E., Kindermann, S.: SparseFIS: data-driven learning of fuzzy systems with sparsity constraints. IEEE Trans. Fuzzy Syst. 18(2), 396–411 (2010)Google Scholar
  26. 26.
    Lughofer, E., Macian, V., Guardiola, C., Klement, E.: Identifying static and dynamic prediction models for nox emissions with evolving fuzzy systems. Appl. Soft Comput. 11(2), 2487–2500 (2011)CrossRefGoogle Scholar
  27. 27.
    Lughofer, E., Smith, J.E., Caleb-Solly, P., Tahir, M., Eitzinger, C., Sannen, D., Nuttin, M.: Human-machine interaction issues in quality control based on on-line image classification. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans 39(5), 960–971 (2009)CrossRefGoogle Scholar
  28. 28.
    Lughofer, E., Trawinski, B., Trawinski, K., Kempa, O., Lasota, T.: On employing fuzzy modeling algorithms for the valuation of residential premises. Inf. Sci. 181(23), 5123–5142 (2011)CrossRefGoogle Scholar
  29. 29.
    Lughofer, E., Weigl, E., Heidl, W., Eitzinger, C., Radauer, T.: Integrating new classes on the fly in evolving fuzzy classifier designs and its application in visual inspection. Appl. Soft Comput. 35, 558–582 (2015)CrossRefGoogle Scholar
  30. 30.
    Marquardt, D.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11(2), 431–441 (1963)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Nelles, O.: Nonlinear Syst. Identif. Springer, Berlin (2001)CrossRefGoogle Scholar
  32. 32.
    Nguyen, H., Sugeno, M., Tong, R., Yager, R.: Theor. Asp. Fuzzy Control. Wiley, New York (1995)Google Scholar
  33. 33.
    Oliveira, J.V.D., Pedrycz, W.: Advances in Fuzzy Clustering and its Applications. Wiley, Hoboken (2007)CrossRefGoogle Scholar
  34. 34.
    Pal, N., Chakraborty, D.: Mountain and subtractive clustering method: improvement and generalizations. Int. J. Intell. Syst. 15(4), 329–341 (2000)CrossRefzbMATHGoogle Scholar
  35. 35.
    Pedrycz, W., Gomide, F.: Fuzzy Systems Engineering: Toward Human-Centric Computing. Wiley, Hoboken (2007)CrossRefGoogle Scholar
  36. 36.
    Pratama, M., Anavatti, S., Angelov, P., Lughofer, E.: PANFIS: a novel incremental learning machine. IEEE Trans. Neural Netw. Learn. Syst. 25(1), 55–68 (2014)CrossRefGoogle Scholar
  37. 37.
    Pratama, M., Anavatti, S., Garret, M., Lughofer, E.: Online identification of complex multi-input-multi-output system based on generic evolving neuro-fuzzy inference system. In: Proceedings of the IEEE EAIS 2013 workshop (SSCI 2013 conference), pp. 106–113. Singapore (2013)Google Scholar
  38. 38.
    Rong, H.J., Sundararajan, N., Huang, G.B., Saratchandran, P.: Sequential adaptive fuzzy inference system (SAFIS) for nonlinear system identification and prediction. Fuzzy Sets Syst. 157(9), 1260–1275 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Schaffer, C.: Overfitting avoidance as bias. Mach. Learn. 10(2), 153–178 (1993)Google Scholar
  40. 40.
    Serdio, F., Lughofer, E., Pichler, K., Pichler, M., Buchegger, T., Efendic, H.: Fault detection in multi-sensor networks based on multivariate time-series models and orthogonal transformations. Inf. Fusion 20, 272–291 (2014)CrossRefGoogle Scholar
  41. 41.
    Siler, W., Buckley, J.: Fuzzy Expert Systems and Fuzzy Reasoning: Theory and Applications. Wiley, Chichester, West Sussex (2005)zbMATHGoogle Scholar
  42. 42.
    Takagi, T., Sugeno, M.: Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. Syst. Man Cybern. 15(1), 116–132 (1985)CrossRefzbMATHGoogle Scholar
  43. 43.
    Tikhonov, A., Arsenin, V.: Solutions of Ill-Posed Problems. Winston & Sons, Washington (1977)zbMATHGoogle Scholar
  44. 44.
    Vetterlein, T., Ciabattoni, A.: On the (fuzzy) logical content of cadiag-2. Fuzzy Sets and Syst. 161, 1941–1958 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Vetterlein, T., Mandl, H., Adlassnig, K.P.: Fuzzy arden syntax: a fuzzy programming language for medicine. Artif. Intell. Med. 49, 1–10 (2010)CrossRefGoogle Scholar
  46. 46.
    Zhou, S., Gan, J.: Low-level interpretability and high-level interpretability: a unified view of data-driven interpretable fuzzy systems modelling. Fuzzy Sets Syst. 159(23), 3091–3131 (2008)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. Royal Stat. Soc, Series B 301–320 (2005)Google Scholar

Copyright information

© Taiwan Fuzzy Systems Association and Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Edwin Lughofer
    • 1
  • Stefan Kindermann
    • 2
  • Mahardhika Pratama
    • 3
  • Jose de Jesus Rubio
    • 4
  1. 1.Department of Knowledge-Based Mathematical SystemsJohannes Kepler University LinzLinzAustria
  2. 2.Industrial Mathematics InstituteJohannes Kepler University LinzLinzAustria
  3. 3.Latrobe UniversityMelbourneAustralia
  4. 4.Seccion de Estudios de Posgrado e Investigacion, Esime Azcapotzalco Instituto Politecnico NacionalMexico CityMexico

Personalised recommendations