Extending information processing in a Fuzzy Random Forest ensemble

Abstract

Imperfect information inevitably appears in real situations for a variety of reasons. Although efforts have been made to incorporate imperfect data into classification techniques, there are still many limitations as to the type of data, uncertainty, and imprecision that can be handled. In this paper, we will present a Fuzzy Random Forest ensemble for classification and show its ability to handle imperfect data into the learning and the classification phases. Then, we will describe the types of imperfect data it supports. We will devise an augmented ensemble that can operate with others type of imperfect data: crisp, missing, probabilistic uncertainty, and imprecise (fuzzy and crisp) values. Additionally, we will perform experiments with imperfect datasets created for this purpose and datasets used in other papers to show the advantage of being able to express the true nature of imperfect information.

Appendix: Combination methods

In this appendix, we use the notation above defined in Sect. 3.2. For each strategy for fuzzy classifier module in the FRF ensemble that is described in Sect. 3.2 we can define several functions Faggre1₁, Faggre1₂ and Faggre2. The overall set of functions is described in Bonissone et al. (2010). In this appendix, the utilized methods in this paper are described in detail:

1.1 Non-trainable methods

In these combination methods, a transformation is applied to the matrix L_FRF in Step 2 in Algorithms 3 and 4 so that each leaf reached assigns a simple vote to the majority class.

$$ \begin{aligned} {\rm For}\,t &= 1,\ldots,T \\ &{\rm For}\, n = 1,\ldots, N\\ & \quad {\rm For}\,i = 1,\ldots, I\\ &\qquad L\_FRF {-}mod_{t,n,i} = \left\{ \begin{array}{ll} 1 & {\rm if}\ i=\arg\displaystyle\max_{j, j=1,\ldots,I} \{L\_FRF_{t,n,j}\}\\ 0 &{\rm otherwise} \\ \end{array}\right. \end{aligned} $$

Within this group we define the following methods. We get two versions depending on the strategy used:

• Simple Majority vote:

• Strategy 1→ method SM1

  The function Faggre1₁ in Algorithm 3 is defined as

  $$ Faggre1_1(t,i,L\_FRF)=\left\{ \begin{array}{ll} 1 & {\rm if}\ i=\arg\displaystyle\max_{j, j=1,\ldots,I} \left\{ \sum_{n=1}^{N_t} L\_FRF-mod_{t,n,j} \right\}\\ 0 & {\rm otherwise} \\ \end{array}\right. $$

  In this method, each tree t assigns a simple vote to the most voted class among the N _t reached leaves by example e in the tree.

  The function Faggre1₂ in Algorithm 3 is defined as

  $$ Faggre1_2(i,T\_FRF)=\sum_{t=1}^{T} T\_FRF_{t,i} $$

• Strategy 2 → method SM2

  For Strategy 2, it is necessary to define the function Faggre2 combining information from all leaves reached in the ensemble by example e. Thus, the function Faggre2 in Algorithm 4 is defined as

  $$ Faggre2(i,L\_FRF)= \sum_{t=1}^{T} \sum_{n=1}^{N_t} L\_FRF_mod{t,n,i} $$

1.2 Trainable explicitly dependent methods

• Majority vote Weighted by Leaf:

  In this combination method, a transformation is applied to the matrix L_FRF in Step 2 of Algorithms 3 and 4 so that each leaf reached assigns a weighted vote to the majority class. The vote is weighted by the degree of satisfaction with which example e reaches the leaf.

  $$ \begin{aligned} {\rm For}\,t &= 1,\ldots,T \\ &{\rm For}\, n = 1,\ldots, N\\ & \quad {\rm For}\,i = 1,\ldots, I\\ &\qquad L\_FRF {-}mod_{t,n,i} = \left\{ \begin{array}{ll} \chi_{t,n(e)} & {\rm if}\ i=\arg\displaystyle\max_{j, j=1,\ldots,I} \{L\_FRF_{t,n,j}\}\\ 0 & {\rm otherwise} \\ \end{array}\right. \end{aligned} $$

  Again, we have two versions according to the strategy used.

• Strategy 1 → method MWL1

  The functions Faggre1₁ and Faggre1₂ are defined as

  $$ Faggre1_1(t,i,L\_FRF)=\left\{ \begin{array}{ll} 1 & {\rm if}\ i=\arg\displaystyle\max_{j, j=1,\ldots,I} \left\{ \sum_{n=1}^{N_t} L\_FRF-mod_{t,n,j} \right\}\\ 0 & {\rm otherwise} \\ \end{array}\right. $$
  $$ Faggre1_2(i,T\_FRF)=\sum_{t=1}^{T} T\_FRF_{t,i} $$

• Strategy 2 → method MWL2

  The function Faggre2 is defined as

  $$ Faggre2(i,L\_FRF)= \sum_{t=1}^{T} \sum_{n=1}^{N_t} L\_FRF_{t,n,i} $$

  • Majority vote Weighted by Leaf and by Tree:

  Again, in this combination method, a transformation is applied to the matrix L_FRF in Step 2 of Algorithms 3 and 4 so that each leaf reached assigns a weighted vote to the majority class. The vote is weighted by the degree of satisfaction with which example e reaches the leaf.

  $$ \begin{aligned} {\rm For}\,t &= 1,\ldots,T \\ &{\rm For}\, n = 1,\ldots, N\\ & \quad {\rm For}\,i = 1,\ldots, I\\ &\qquad L\_FRF {-}mod_{t,n,i} = \left\{ \begin{array}{ll} \chi_{t,n}(e) & {\rm if}\ i=\arg\displaystyle\max_{j, j=1,\ldots,I} \{L\_FRF_{t,n,j}\}\\ 0 & {\rm otherwise} \\ \end{array}\right. \end{aligned} $$

  In addition, in this method a weight for each tree obtained is introduced by testing each individual tree with the OOB dataset. Let $\overline{p}=(p_1,p_2,\ldots,p_T)$ be the vector with the weights assigned to each tree. Each p _t is obtained as $\frac{N\_success\_OOB_t}{size\_OOB_t}$ where N_success_OOB _t is the number of examples classified correctly from the OOB dataset used for testing the tth tree and size_OOB _t is the total number of examples in this dataset.

  Strategy 1 → method MWLT1

  The function Faggre1₁ is defined as:

  $$ Faggre1_1(t,i,L\_FRF)=\left\{ \begin{array}{ll} 1 & {\hbox{if}}\ i=\arg\displaystyle\max_{j, j=1,\ldots,I} \left\{ \sum_{n=1}^{N_t} L\_FRF-mod_{t,n,j} \right\}\\ 0 & {\hbox{otherwise}} \\ \end{array}\right. $$

  Vector $\overline{p}$ is used in the definition of function Faggre1₂:

  $$ Faggr1_2(i,T\_FRF)=\sum_{t=1}^{T} p_t \cdot T\_FRF_{t,i} $$

  Strategy 2 → method MWLT2

  The vector of weights $\overline{p}$ is applied to Strategy 2.

  $$ Faggre2(i,L\_FRF)=\sum_{t=1}^{T} p_t \sum_{n=1}^{N_t} L\_FRF_{t,n,i} $$

  • Minimum Weighted by Leaf and by membership Function:

    In this combination method, a transformation is applied to the matrix L_FRF in Step 2 of Algorithms 3.

    $$ \begin{aligned} {\rm For}\,t &= 1,\ldots,T \\ &{\rm For}\, n = 1,\ldots, N\\ & \quad {\rm For}\,i = 1,\ldots, I\\ &\qquad L\_FRF {-}mod_{t,n,i} = \chi_{t,n}(e) \times \frac{E_i}{E_n} \end{aligned} $$

    Strategy 1 → method MIWLF1

    The function Faggre1₁ is defined as:

    $$ Faggre1_1(t,i,L\_FRF)=\left\{ \begin{array}{ll} 1 & {\hbox{if}} \ i=\displaystyle\arg\max_{j, j=1,\ldots,I}\left\{ min(L\_FRF-mod_{t,1,j},L\_FRF-mod_{t,2,j},\right.\\ & \phantom{si\ i=\displaystyle\arg\max_{j, j=1,\ldots,I}}\left.\ldots ,L\_FRF-mod_{t,N_t,j})\right\} \\ 0 & {\hbox{otherwise}} \\ \end{array}\right. $$

    The function Faggre1₂ incorporates the weighting, defined by a fuzzy membership function, for each tree:

    $$ Faggre1_2(i,T\_FRF)=\sum_{t=1}^{T} \mu_{pond}\left(\frac{errors_{(OOB_t)}}{size_{(OOB_t)}}\right) \cdot T\_FRF_{t,i} $$

    The membership function is defined by μ _pond (x):

    $$ \mu_{pond}(x)=\left\{\begin{array}{ll} 1 & 0 \leq x \leq (pmin+marg)\\ \frac{(pmax+marg)-x}{(pmax-pmin)} & (pmin+marg)\leq x \leq (pmax+marg)\\ 0 & (pmax+marg) \leq x \\ \end{array}\right. $$

    where

    • pmax is the maximum rate of errors in the trees of the FRF ensemble ($pmax=\max_{t=1,\ldots,T}\left\{\frac{errors_{(OOB_t)}}{size_{(OOB_t)}}\right\}$). The rate of errors in a tree t is obtained as $\frac{errors_{(OOB_t)}}{size_{(OOB_t)}}$ where errors _{(OOB_t)} is the number of classification errors of the tree t (using the OOB _t dataset as test set), and size _{(OOB_t)} is the cardinal of the OOB _t dataset. As we indicated above, the OOB _t examples are not used to build the tree t and they constitute an independent sample to test tree t. So we can measure the goodness of a tree t as the number of errors when classifying the set of examples OOB _t ;

    • pmin is the minimum rate of errors in the trees of the FRF ensemble; and

    • $marg=\frac{pmax-pmin}{4}$

1.3 Trainable implicitly dependent methods

Within this group we define the following methods:

• Minimum Weighted by membership Function: In this combination method, no transformation is applied to matrix L_FRF in Step 2 of Algorithm 3.

  Strategy 1 → method MIWF1

  The function Faggre1₁ is defined as

  $$ Faggre1_1(t,i,L\_FRF)=\left\{ \begin{array}{ll} 1 & {\rm if}\ i=\displaystyle\arg\max_{j, j=1,\ldots,I}\left\{ min(L\_FRF_{t,1,j},L\_FRF_{t,2,j}, \right. \\ & \phantom{si\ i=\displaystyle\arg\max_{j, j=1,\ldots,I}}\left.\ldots ,L\_FRF_{t,N_t,j})\right\} \\ 0 & {\rm otherwise} \\ \end{array}\right. $$

  The function Faggre1₂ incorporates the weighting defined by the previous fuzzy membership function for each tree:

  $$ Faggre1_2(i,T\_FRF)=\sum_{t=1}^{T} \mu_{pond}\left(\frac{errors_{(OOB_t)}}{size_{(OOB_t)}}\right) \cdot T\_FRF_{t,i} $$