EEFR-R: extracting effective fuzzy rules for regression problems, through the cooperation of association rule mining concepts and evolutionary algorithms

Abstract

Fuzzy rule-based systems, due to their simplicity and comprehensibility, are widely used to solve regression problems. Fuzzy rules can be generated by learning from data examples. However, this strategy may result in high numbers of rules that most of them are redundant and/or weak, and they affect the systems’ interpretability. Hence, in this paper, a new rule learning method, EEFR-R, is proposed to extract the effective fuzzy rules from regression data samples. This method is formed through the cooperation of association rule mining concepts and evolutionary algorithms in the three stages. Indeed, the components of a Mamdani fuzzy rule-based system are generated during the first two stages, and then, they will be refined through some modifications in the last stage. In EEFR-R, fuzzy rules are extracted from numerical data using the idea of Wang and Mendel’s method and utilizing the concepts of Support and Confidence; furthermore, a new rule pruning method is presented to refine these rules. By employing this method, non-effective rules can be pruned in three different modes as the preferences of a decision maker. The proposed model and its stages were validated using 19 real-world regression datasets. The experimental results and the conducted statistical tests confirmed the effectiveness of EEFR-R in terms of complexity and accuracy and in comparison with the three state-of-the-art regression solutions.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    These advantages are general and may be variant depending on the application.

  2. 2.

    In mathematics, the number of elements of a set is called the cardinality of that set.

References

  1. Alcalá R, Alcalá-Fdez J, Herrera F (2007) A proposal for the genetic lateral tuning of linguistic fuzzy systems and its interaction with rule selection. IEEE Trans Fuzzy Syst 15(4):616–635

    MATH  Article  Google Scholar 

  2. Alcalá R, Gacto MJ, Herrera F (2011) A fast and scalable multiobjective genetic fuzzy system for linguistic fuzzy modeling in high-dimensional regression problems. IEEE Trans Fuzzy Syst 19(4):666–681

    Article  Google Scholar 

  3. Alcalá-Fdez J, Alcala R, Herrera F (2011a) A fuzzy association rule-based classification model for high-dimensional problems with genetic rule selection and lateral tuning. IEEE Trans Fuzzy Syst 19(5):857–872

    Article  Google Scholar 

  4. Alcalá-Fdez J, Fernández A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2011b) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. J Mult-Valued Log Soft Comput 17(2–3):255–287

    Google Scholar 

  5. Alonso JM, Castiello C, Mencar C (2015) Interpretability of fuzzy systems: current research trends and prospects. In: Springer handbook of computational intelligence. Springer, pp 219–237

  6. Antonelli M, Bernardo D, Hagras H, Marcelloni F (2017) Multiobjective evolutionary optimization of type-2 fuzzy rule-based systems for financial data classification. IEEE Trans Fuzzy Syst 25(2):249–264

    Article  Google Scholar 

  7. Batbarai A, Naidu D (2014) Survey for rule pruning in association rule mining for removing redundancy. Int J Innov Res Sci Eng Technol 3(4):11313–11315

    Google Scholar 

  8. Bhargava N, Shukla M (2016) Survey of interestingness measures for association rules mining: data mining, data science for business perspective. Analysis 6(2):2249–9555

    Google Scholar 

  9. Blej M, Azizi M (2016) Comparison of mamdani-type and sugeno-type fuzzy inference systems for fuzzy real time scheduling. Int J Appl Eng Res 11(22):11071–11075

    Google Scholar 

  10. Cheng R, Jin Y (2015) A social learning particle swarm optimization algorithm for scalable optimization. Inf Sci 291:43–60

    MathSciNet  MATH  Article  Google Scholar 

  11. Cordón O, Herrera F, Sánchez L (1999) Solving electrical distribution problems using hybrid evolutionary data analysis techniques. Appl Intell 10(1):5–24

    Article  Google Scholar 

  12. Dash R, Paramguru RL, Dash R (2011) Comparative analysis of supervised and unsupervised discretization techniques. Int J Adv Sci Technol 2(3):29–37

    Google Scholar 

  13. de Sá CR, Soares C, Knobbe A (2016) Entropy-based discretization methods for ranking data. Inf Sci 329:921–936

    Article  Google Scholar 

  14. Debie E, Shafi K, Merrick K, Lokan C (2014) An online evolutionary rule learning algorithm with incremental attribute discretization. In: IEEE congress on evolutionary computation (CEC), pp 1116–1123

  15. Du K, Swamy M (2016) Particle swarm optimization. Springer, Berlin

    Google Scholar 

  16. Elragal H (2010) Using swarm intelligence for improving accuracy of fuzzy classifiers. Int J Electr Comput Energ Electron Commun Eng 4(8):11–19

    Google Scholar 

  17. Esmin AAA (2007) Generating fuzzy rules from examples using the particle swarm optimization algorithm. In: International conference on hybrid intelligent systems, pp 340–343

  18. Fayyad U, Irani K (1993) Multi-interval discretization of continuous-valued attributes for classification learning. In: Proceedings of the 13th international joint conference on artificial intelligence, pp 1022–1027

  19. Fernandez A, Lopez V, del Jesus MJ, Herrera a (2015) Revisiting evolutionary fuzzy systems: taxonomy, applications, new trends and challenges. Knowl-Based Syst 80:109–121

    Article  Google Scholar 

  20. Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32(200):675–701

    MATH  Article  Google Scholar 

  21. Gacto MJ, Galende M, Alcalá R, Herrera F (2014) METSK-HD\(^{e}\): a multi-objective evolutionary algorithm to learn accurate tsk-fuzzy systems in high-dimensional and large-scale regression problems. Inf Sci 276:63–79

    Google Scholar 

  22. Garcia S, Luengo J, Sáez JA, Lopez V, Herrera F (2013) A survey of discretization techniques: taxonomy and empirical analysis in supervised learning. IEEE Trans Knowl Data Eng 25(4):734–750

    Article  Google Scholar 

  23. He Y, Ma WJ, Zhang JP (2016) The parameters selection of pso algorithm influencing on performance of fault diagnosis. MATEC Web Conf 63:02019

    Article  Google Scholar 

  24. Jang J-S (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cybern 23(3):665–685

    Article  Google Scholar 

  25. Kato ER, Morandin O, Sgavioli M, Muniz BD (2009) Genetic tuning for improving wang and mendel’s fuzzy database. In: IEEE international conference on systems, man and cybernetics, pp 1015–1020

  26. Liu H, Cocea M (2018) Granular computing-based approach of rule learning for binary classification. Granul Comput 4(2):1–9

    Google Scholar 

  27. Mamdani EH (1977) Application of fuzzy logic to approximate reasoning using linguistic systems. IEEE Trans Comput 26(12):1182–1191

    MATH  Article  Google Scholar 

  28. Oliveira MVd, Schirru R (2009) Applying particle swarm optimization algorithm for tuning a neuro-fuzzy inference system for sensor monitoring. Prog Nucl Energy 51(1):177–183

    Article  Google Scholar 

  29. Patel M (2013) Various rule pruning techniques and accuracy measures for fuzzy rules. Int J Appl Innov Eng Manag 2(12):175–178

    Google Scholar 

  30. Permana KE, Hashim SZM (2010) Fuzzy membership function generation using particle swarm optimization. Int J Open Probl Compt Math 3(1):27–41

    MATH  Google Scholar 

  31. Prasad M, Chou K-P, Saxena A, Kawrtiya OP, Li D-L, Lin C-T (2014) Collaborative fuzzy rule learning for mamdani type fuzzy inference system with mapping of cluster centers. In: IEEE symposium on computational intelligence in control and automation, pp 1–6

  32. Ratner B (2017) Statistical and machine-learning data mining: techniques for better predictive modeling and analysis of big data. Chapman and Hall/CRC, London

    Google Scholar 

  33. Riza LS, Bergmeir CN, Herrera F, Benítez Sánchez JM (2015) FRBS: Fuzzy rule-based systems for classification and regression in R. J Stat Softw 65(6):1–30

    Article  Google Scholar 

  34. Rodríguez-Fdez I, Canosa A, Mucientes M, Bugarín A (2015) Stac: a web platform for the comparison of algorithms using statistical tests. In: IEEE international conference on fuzzy systems (FUZZ-IEEE), pp 1–8

  35. Rodríguez-Fdez I, Mucientes M, Bugarín A (2016) FRULER: fuzzy rule learning through evolution for regression. Inf Sci 354:1–18

    Article  Google Scholar 

  36. Shehzad K (2013) Simple hybrid and incremental postpruning techniques for rule induction. IEEE Trans Knowl Data Eng 25(2):476–480

    Article  Google Scholar 

  37. Shill PC, Akhand M, Murase K (2011) Simultaneous design of membership functions and rule sets for type-2 fuzzy controllers using genetic algorithms. In: International conference on computer and information technology (ICCIT), pp 554–559

  38. Takagi T, Sugeno M (1985) Fuzzy identification of systems and its applications to modeling and control. IEEE Trans Syst Man Cybern 15(1):116–132

    MATH  Article  Google Scholar 

  39. Tay KM, Lim CP (2011) Optimization of gaussian fuzzy membership functions and evaluation of the monotonicity property of fuzzy inference systems. In: IEEE international conference on fuzzy systems (FUZZ-IEEE), pp 1219–1224

  40. Timothy J et al (2010) Fuzzy logic with engineering applications. Wiley, Chichester

    Google Scholar 

  41. Vaneshani S, Jazayeri-Rad H (2011) Optimized fuzzy control by particle swarm optimization technique for control of cstr. World Acad Sci Eng Technol 59:686–691

    Google Scholar 

  42. Visalakshi P, Sivanandam S (2009) Dynamic task scheduling with load balancing using hybrid particle swarm optimization. Int J Open Probl Compt Math 2(3):475–488

    Google Scholar 

  43. Wang LX, Mendel JM (1992) Generating fuzzy rules by learning from examples. IEEE Trans Syst Man Cybern 22(6):1414–1427

    MathSciNet  Article  Google Scholar 

  44. Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1(6):80–83

    Article  Google Scholar 

  45. Xue B, Zhang M, Browne WN, Yao X (2016) A survey on evolutionary computation approaches to feature selection. IEEE Trans Evol Comput 20(4):606–626

    Article  Google Scholar 

  46. Zadeh L (1965) Fuzzy sets. Inform Control 8(3):338–353

    MATH  Article  Google Scholar 

  47. Zadeh L (1975) The concept of a linguistic variable and its application to approximate reasoning-i. Inf Sci 8(3):199–249

    MathSciNet  MATH  Article  Google Scholar 

  48. Zanganeh M, Yeganeh-Bakhtiary A, Bakhtyar R (2011) Combined particle swarm optimization and fuzzy inference system model for estimation of current-induced scour beneath marine pipelines. J Hydroinform 13(3):558–573

    Article  Google Scholar 

  49. Zeinalkhani M, Eftekhari M (2014) Fuzzy partitioning of continuous attributes through discretization methods to construct fuzzy decision tree classifiers. Inf Sci 278:715–735

    MathSciNet  MATH  Article  Google Scholar 

Download references

Acknowledgements

This study was not funded at all.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mahdi Eftekhari.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by V. Loia.

Appendices

Appendix A: overview of Mamdani FRBSs

The standard architecture of a fuzzy rule-based system consists of four main modules as below (Riza et al. 2015):

Fuzzification—The fuzzification module converts the crisp input values into linguistic ones. Linguistic values refer to ordinary concepts like high, medium, low, etc., which are used in our conversation to declare measures. MFs are defined based on these linguistic variables to map input values into fuzzy concepts.

Knowledge Base (KB)—KB is one of the essential components of FRBS. It comprises two fundamental parts including Data Base (DB) and Rule Base (RB). The DB includes fuzzy set definitions and the MF’s parameters while the RB contains a set of linguistic fuzzy If-Then rules. Fuzzy rules are applied in the decision-making process. They are written in the following format: Ifx is A, Theny is B where the If part is called antecedent and the Then part is called consequent of the rule. This rule is described as: If the antecedent conditions are satisfied, Then the consequent can be inferred.

Inference Engine—The inference engine is where FRBS performs the process of reasoning; it uses fuzzy If-Then rules and input data to make the inference operation.

Defuzzification module—In defuzzification procedure, the acquired output is transformed from a fuzzy value into a crisp one.

Among the different fuzzy inference systems (Timothy 2010), two types of Mamdani FIS (Mamdani 1977) and Takagi–Sugeno–Kang (TSK) FIS (Takagi and Sugeno 1985) have been widely used in the regression applications. The most important differences between these FISs are related to the consequent part of the rules, and accordingly, their aggregation and defuzzification methods are different. In Mamdani fuzzy inference systems, both antecedent and consequent parts of a rule are determined by fuzzy sets, while in TSK systems only the antecedent part is represented by fuzzy sets and the consequent part is computed through a weighted linear function or a constant value. Mamdani fuzzy systems are usually applied to get more interpretable models, whereas TSK systems focus on the accuracy and precision (Blej and Azizi 2016).

The general form of a fuzzy If-Then rule in Mamdani system is like this:

$$\begin{aligned} \mathrm{Rule}^i :&\mathbf {If} ~ X^1 ~ is ~ A_i^1 ~ and ~ X^2 ~ is ~ A_i^2 ~ and ~ \ldots ~ and ~ X^n ~ is ~ A_i^n \nonumber \\&\mathbf {Then} ~ Y ~ is ~ B_i ; ~ i=1, ~\ldots ,~ p \end{aligned}$$
(18)

where \(X^k\) is the kth input variable, n is the number of input variables, Y is the output variable, p is the number of all rules, and \(A_i^j\) and \(B_i\) are the linguistic variables defined by MFs in the DB. The main advantage of this composition of inputs and output is its close relationship with the way of human thinking.

Fig. 8
figure8

MFs of different variables; left figures are related to the initial MFs, and right figures are the same left MFs which have been tuned

After generating or learning the constituents of DB and RB, the process of inferring in Mamdani system is done through the following five steps:

  1. (1)

    Fuzzification of the input values using the defined MFs.

  2. (2)

    Applying the fuzzy operator (AND or OR) to the antecedents in order to combine the fuzzified inputs and obtain the rule strength \((h_i)\). \(h_i\) is also called firing strength of the ith rule. It measures the degree of matching the input vector \((X^1, X^2, \ldots , X^n)\) to the ith rule.

    $$\begin{aligned} h_i=T(A_i^1 (X^1), A_i^2 (X^2), \ldots , A_i^n (X^n)) \end{aligned}$$
    (19)

    where \(A_i^k\) is the fuzzy value of the kth input variable \((X^k)\) and T is the T-norm conjunctive operator. Mamdani specifically recommended the use of the minimum T-norm (Mamdani 1977).

  3. (3)

    Indicating the final consequent for each rule; it is achieved by combining the computed rule strength and the linguistic fuzzy term related to the output of a certain rule as:

    $$\begin{aligned} B^{\prime }_i (Y)=T(h_i ,B_i (Y)) \end{aligned}$$
    (20)

    \(B^{\prime }_i\) is the conclusive consequent of the ith rule, \(B_i\) is the fuzzy value of output variable Y and T is the T-norm conjunctive operator. Mamdani has also recommended the use of the minimum T-norm in this case (Mamdani 1977).

  4. (4)

    Aggregation of all rule’s consequent; this is usually done by using the fuzzy OR operator as:

    $$\begin{aligned} O(Y)=\bigcup _{i=1}^p B^{\prime }_i (Y) \end{aligned}$$
    (21)

    where O(Y) is the result MF.

  5. (5)

    Defuzzification of the result; the output MF is not enough in many applications and the crisp value is needed. Some methods like centroid, weighted average, mean-max membership, etc., have been suggested computing the crisp value of the result. The most common method is centroid which uses the center of gravity of the fuzzy sets in which the crisp output y is calculated as:

    $$\begin{aligned} y= \int O(Y) Y \mathrm{d}y\diagup \int O(Y) \mathrm{d}y \end{aligned}$$
    (22)

    Due to the capability of Mamdani FIS in generating simple and interpretable models and regarding our objective in reducing the complexity of regression solutions, it is used in this study. Mamdani weakness in getting lower accuracy (in comparison with TSK models), however, has been overcome using optimization methods.

Appendix B: an illustrative example of EEFR-R

In this section, an illustrative example of EEFR-R is described. For this purpose, the first problem of Table 2, Ele-1, has been taken into account. Ele-1 is a real-world benchmark problem which estimates the length of low-voltage lines in rural towns using some available inputs. This dataset includes 3 attributes, namely Inhabitants, Distance, and Length; Inhabitants and Distance are the system inputs which are used to estimate the output Length (Cordón et al. 1999).

Table 14 Results of rule generation

EEFR-R is performed for Ele-1 in the three stages as the descriptions of Sect. 3. In the following, the results of one run of EEFR-R for this dataset are demonstrated. In the preprocessing stage, Sect. 3.1, a set of CPs is generated for each variable as:

$$\begin{aligned}&S_\mathrm{Inhabitants}= & {} \left\{ 1,~20,~320 \right\}&\\&S_\mathrm{Distance}= & {} \left\{ 60,~362.5,~ 500.8,~665.83,~1673.3 \right\}&\\&S_\mathrm{Length}= & {} \left\{ 80,~1568,~3492.5,~7675\right\}&\end{aligned}$$

These sets are utilized in the model generation stage to define initial MFs in Sect. 3.2.1; the initial MFs of the three variables are shown in the left column of Fig. 8. In the following of the second stage, all rules are extracted from the data samples of the Ele-1 dataset using the rule generation process of Sect. 3.2.2; these rules and their weights are represented in Table 14. After that, the rule pruning method, 3.2.3, is carried out, and among the 7 generated rules, just 3 of them (the highlighted ones in italic) remain in the RB. Table 15 shows the structure of final RB.

Table 15 Results of rule pruning and rules’ weights adjustment

Finally, as Sect. 3.3, the post-processing tasks including MFs tuning and rules’ weights adjustment are done. The right column of Fig. 8 shows the MFs of the three variables after the tuning. Also, the optimized weights of the final rules are indicated in the weight column of Table 15. The test error of this run of EEFR-R for the Ele-1 dataset was equal to 2.0527E+05.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Aghaeipoor, F., Eftekhari, M. EEFR-R: extracting effective fuzzy rules for regression problems, through the cooperation of association rule mining concepts and evolutionary algorithms. Soft Comput 23, 11737–11757 (2019). https://doi.org/10.1007/s00500-018-03726-1

Download citation

Keywords

  • Discretization
  • Fuzzy rule learning
  • Rule pruning
  • Support and confidence
  • Particle swarm optimization