Explainable Machine Learning for Categorical and Mixed Data with Lossless Visualization

Abstract

Building accurate and explainable/interpretable Machine Learning (ML) models for heterogeneous/mixed data is a long-standing challenge for algorithms designed for numeric data. This work focuses on developing numeric coding schemes for non-numeric attributes for ML algorithms to support accurate and explainable ML models, methods for lossless visualization of n-D non-numeric categorical data with visual rule discovery in these visualizations, and accurate and explainable ML models for categorical data. This study proposes a classification of mixed data types and analyzes their important role in Machine Learning. It presents a toolkit for enforcing interpretability of all internal operations of ML algorithms on mixed data with a visual data exploration on mixed data. A new Sequential Rule Generation (SRG) algorithm for explainable rule generation with categorical data is proposed and successfully evaluated in multiple computational experiments. This work is one of the steps to the full scope ML algorithms for mixed data supported by lossless visualization of n-D data in General Line Coordinates beyond Parallel Coordinates.

Appendix

1.1 Experiment 1 with Algorithm SRG0 for Discovering Rules on Mushroom with Sequential Triples

In this experiment we tested SRG0 algorithm for discovering rules on Mushroom data [27] to solve a two-class classification problem (poisonous - eatable). We found rules with 100% precision for the target class C₁ – poisonous mushrooms. One of these rules is a rule reported in the literature [30]. Below we present discovered rules R1-R7 using notation of the case x = (x₁,x₂,…,x_n). If any of these rules is true, then x belongs to class C₁.

Rules R1–R7

R₁: [(x₅ = 3) $\vee$ (x₅ = 4) $\vee$ (x₅ = 5) $\vee$ (x₅ = 6) $\vee$ (x₅ = 8) $\vee$ (x₅ = 9)] ⇒ x ∈ C₁

R₂: [(x₉ = 6) $\vee$ (x₉ = 3)] ⇒ x ∈ C₁

R₃: [(x₁₉ = 2) & (x₂₀ = 8) & (x₂₁ ≠ 2) & (x₂₂ ≠ 2)] ⇒ x ∈ C₁

R₄: [(x₁₅ = 3) $\vee$ (x₁₅ = 2) $\vee$ (x₁₅ = 9)] ⇒ x ∈ C₁

R₅: [(x₁₉ ≠ 2) & (x₂₀ ≠ 6) & (x₂₁ = 5) & (x₂₂ = 1)] ⇒ x ∈ C₁

R₆: [(x₁₉ = 6) & (x₂₀ = 5) & (x₂₁ ≠ 1) & (x₂₂ = !6)] ⇒ x ∈ C₁

R₇: [(x₂₀ = 8) & (x₂₁ = 2) & (x₂₂ = !6)] ⇒ x ∈ C₁

Table 3 presents characteristics of discovered 7 rules, R1-R7, which cover 100% of the mushroom data for the “poisonous” class with 100% precision. The total number of cases in the poisonous N_total class is 3916 cases. The rules were with attributes broken up into six sequential triples and one group with 4 attributes: x₁₉, x₂₀, x₂₁, x₂₂. These 6 groups are (x₁, x₂, x₃), (x₄, x₅, x₆), (x₇, x₈, x₉), (x₁₀, x₁₁, x₁₂), (x₁₃, x₁₄, x₃₅), (x₁₆, x₁₇, x₁₈).

Table 3 Characteristics of discovered rules R1–R7

1.2 Experiment 2 with SRG1 Algorithm for Discovering Rules on Mushroom Data with Rule Overlap Minimization

The analysis of rules presented in the previous section shows that, Rule R₁ covers 96.94% of the cases with target value “poisonous”, which means that the other 6 rules add to the coverage only 3.04% of these cases (118 cases). For this reason, to get a better understanding of these rules we calculated the overlap between cases classified by different rules. See Tables 4 and 5. In these tables, the overlap, OL, is the total number of cases that are in the intersection of rules R_i and R_j, OL(R_i,R_j) = |Cases(R_i) ∩ Cases(R_j)|, the overlap percentage is OL(R_i,R_j) divided by the total number of cases in the union of cases covered by both rules, OL(R_i,R_j)/|cases(R_i)$\cup$  cases(R_j)|. Table 4 shows the relations of rule R₁ with other rules and Table 5 of remaining rules with each other.

Table 4 Overlap between dominant rule R₁ and other rules

Table 5 Overlap percentage and number of cases for rules R₂–R₇

For instance, the analysis of the relations between rules R₁ and R₂ show that together they cover 3820 cases of poisonous mushrooms and overlap in 1728 cases (45.24% of 3820 cases). Rule R₁ covers 3796 cases, thus rule R₂ adds only 3820–3796 = 24 new cases to R₁, which is 20.34% of 120 cases not covered by rule R₁. R₆ does not overlap with rule R₁ it adds 72 new cases to R₁, which is 60% of 120 cases not covered by rule R₁.

Thus, rules R₁&R₂, R₁&R₆ differ in an important characteristic. Heavy overlap of R₁ and R₂ (45.24%) increases the confidence in classification of those overlapped cases. The non-overlap of R₁ and R₆, increases coverage expanding the number of cases that the rule predicts two more than with adding R₂ to R₁.

For rules R₂-R₇, Table 5 shows overlap percentage for each pair of rules, the total number of distinct cases covered by two rules and the number of cases in their overlap. Only rules R₂ and R₃ heavily overlap. All other rules either do not overlap (7 pairs) or overlap in no more than 72 cases, which less than 10% of the total number of cases in each pair. As expected, many of the rules overlap with each other. The heaviest overlapping rules are R₂ and R₃, where the overlap is 64% together of the total cases predicted. This means that these two rules are closely related.

Table 6 shows the relations between rules CR₁-CR₃ from [30]. Rules CR1-CR3 use 5 attributes (x₅, x₈, ×₁₂, ×₂₀, ×₂₁), while our rules R1-R7 use 7 attributes (x₅, x₉, ×₁₅, ×₁₉, ×₂₀, ×₂₁, ×₂₂).

Table 6 Relations between rules CR₁–CR₃

Rule R₁ is the same as rule CR₁ from [30]. Tables 4 and 5 show that rules R₂-R₇ are more general than rules CR₂ and CR₃ discovered in [30]. Rule CR₂ covers 72 cases and rule CR₃ covers 912 cases, while our rule R₇ with the smallest coverage cover 52 cases. Rules R₂-R₇ cover in total 2188 cases, while rules CR₂ and CR₃ cover only 984 cases together.

1.3 Experiment 3 with SRG2 Algorithm on Mushroom Data with Complimentary Rules Generation

1.3.1 Rule Generation

Below we present rules generated at Step 3 first for the poisonous class and then complementary rules for the eatable class on Mushroom data with most frequent attributes generated from all 22 mushroom attributes. In this experiment the following attribute groups are used: Group A1: x₉, x₅, x₇, ×₁₁; Group A2: ×₁₃, ×₁₄, ×₁₅, x₆; Group A3: x₁, x₂, x₄, ×₂₁, ×₂₂.

According to the design of SRG algorithm it runs at the several levels of thresholds that can be set up by a user. We experimented with 3 level of precision: 75%. 85% and 95% with fixed level of coverage of 0.5%. This means that rules that have lower precision coverage are filtered out and not selected. For coverage with 3916 cases in the poisonous class it means that rules that cover less than 20 cases are filtered out considered as overfitting rules.

The thresholds 75%, 85%, and 95% limit only the low margin of the rule quality, but do not limit their upper level. Therefore, we also computed the actual precision and coverage reached for the poisonous class at each level.

Table 7 shows that rules at all levels missed only 4 cases from the poisonously class, giving 99.89% coverage and none of the poisonous cases was misclassified as eatable, giving 100% precision. All classification errors came from classifying some eatable cases as poisonous, that ranged from 800 cases for 75% threshold to 192 cases for 95% threshold.

Table 7 Results of sequential rule generation for poisonous class in Experiment 3: Rules R₁–R₁₃

We conducted the further analysis for 13 rules selected at level 3 with 95% threshold for rules. See Table 8. This table shows that all 192 misclassified cases belong to the rule R₁ that has 98.47% coverage and 95.2% precision. All other rules have 100% precision and coverage that smaller than for dominant rule R₁.

Table 8 Characteristics of discovered rules R₁–R₁₃ for poisonous class in Experiment 3

Rules R₁–R₁₃ (Poisonous)

R₁: [(x₅ ≠ 7) & (x₇ ≠ 2) & (x₉ ≠ 7) & (x₁₁ ≠ 2)] ⇒ x ∈ C₁

R₂: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) V (x₅ = 9)] ⇒ x ∈ C₁

R₃: [(x₉ = 3) ∨ (x₉ = 6)] ⇒ x ∈ C₁

R₄: [(x₆ ≠ 1) & (x₁₃ ≠ 4) & (x₁₄ ≠ 8) & (x₁₅ ≠ 1)] ⇒ x ∈ C₁

R₅: [(x₁ ≠ 6) & (x₂ ≠ 3) & (x₄ ≠ 1) & (x₂₁ ≠ 6) & (x₂₂ = 7)] ⇒ x ∈ C₁

R₆: [(x₉ = 5) & (x₁₁ = 1)] ⇒ x ∈ C₁

R₇: [(x₁₅ = 3) ∨ (x₁₅ = 3) ∨ (x₁₅ = 9)] ⇒ x ∈ C₁

R₈: [(x₁ = 5) & (x₄ = 2) & (x₂₁ = 5) & (x₂₂ ≠ 2)] ⇒ x ∈ C₁

R₉: [(x₂₁ = 5) & (x₂₂ = 1)] ⇒ x ∈ C₁

R₁₀: [(x₆ = 3) & (x₁₃ = 2) & (x₁₄ ≠ 1) & (x₁₅ ≠ 8)] ⇒ x ∈ C₁

R₁₁: [(x₂ = 3) & (x₂₁ = 2) & (x₂₂ ≠ 6)] ⇒ x ∈ C₁

R₁₂: [(x₁ = 1) & (x₂₁ = 5) & (x₂₂ ≠ 2)] ⇒ x ∈ C₁

R₁₃: [(x₁ ≠ 6) & (x₂ ≠ 1) & (x₄ ≠ 2) & (x₂₁ = 5) & (x₂₂ = 3)] ⇒ x ∈ C₁

The rules R14 and R15 generated for the eatable class C₂ are presented below:

R₁₄: [(x₅ = 1) & (x₉ ≠ 1)] = > x C₂

R₁₅: [(x₅ = 2) & (x₉ ≠ 1)] = > x C₂

Table 9 shows the analysis of both R14 and R15 for class eatable class C₂. Rules R₁₄ and R₁₅ together cover and correctly predict all 192 cases misclassed by Rule R₁. While both R₁₄ and R₁₅ cover 336 cases each, these cases are different. In fact, rules R₁₄ and R₁₅ have 0% overlap and combined cover 672 cases of eatable class C₂.

Table 9 Characteristics of discovered rules R₁₄, R₁₅ for eatable class for Experiment 3

1.3.2 Combining Rules for Two Classes

Now we have rules R₁-R₁₃ for the target class and rules R₁₄-R₁₅ for the non-target class and can accomplish step 3 of combining them. The only rule for the target class that misclassified some cases is R₁. So, we need to improve only this rule. It is done by creating a new rule R_N that combines rule R₁ with R₁₄ and R₁₅ as follows.

$${\text{R}}_{\text{N}} (\rm{x})\, = \,{\text{R}}_{1} \left( \rm{x} \right) \, \& \neg ({\text{R}}_{{14}} \left( \rm{x} \right) \vee {\text{R}}_{{15}} \left( \rm{x} \right))$$

resulted in

$${\text{R}}_{\text{N}}{ :} \, [(x_{5} \, \ne \,{7}) \, \& \, (x_{7} \, \ne \,{2}) \, \& \, (x_{9} \, \ne \,{7}) \, \& \, (x_{{11}} \, \ne \,{2})] \, \&$$

$$\neg ([\left( {x_{5} = {1}} \right) \, \& \, (x_{9} \ne {1})] \vee [\left( {x_{5} = {2}} \right) \, \& \, (x_{9} \ne {1})]) = > \rm{x} \in {\text{C}}_{1}$$

after putting actual rules R₁, R₁₄ and R₁₅ to the formula. Rule R_N is false, R_N(x) = 0, for all 192 cases x misclassified by rule R₁ as poisonous, for which R₁(x) = 1, because for those x rule R₁₄ or R₁₅ is true. If each rule R₁₄ and R₁₅ would independently cover all 192 cases misclassed by R₁ then R_N can be defined simpler in two ways by using any of these rules:

$${\text{R}}_{\text{N}} (\rm{x})\, = \,{\text{R}}_{1} \left( \rm{x} \right) \, \& \neg {\text{R}}_{{14}} \left( \rm{x} \right),{\text{ R}}_{\text{N}} (\rm{x})\, = \,{\text{R}}_{1} \left( \rm{x} \right) \, \& \neg {\text{R}}_{{15}} \left( \rm{x} \right)$$

1.4 Experiment 4 with SRG3 on Mushroom Data with All 30 Random Groups

To evaluate this algorithm, we performed the test with 30 groups of 3 attributes were my generated from the total 22 mushroom attributes, as shown in Table 10. The algorithm generated all possible rules for these groups on the mushroom data. Then, with all these rules the rule combination process and selection process were ran and the final rules were selected, as shown in both Table 11 and 12. Below are the randomly generated groups and the result of this test.

Table 10 30 Randomly generated groups using 22 attributes

Table 11 Sequential rule generation with 30 randomly generated attribute groupings of 3

Table 12 Characteristics of discovered rules R₁-R₇ for poisonous class in Experiment 4

Rules R1–R7 (Poisonous)

R₁: [(x₅ = 3) $\vee$ (x₅ = 4) $\vee$ (x₅ = 5) $\vee$ (x₅ = 6) $\vee$ (x₅ = 8) $\vee$ (x₅ = 9)] ⇒ x ∈ C₁

R₂: [(x₁₂ = 3) & (x₂₀ ≠ 6) & (x₇ ≠ 2)] ⇒ x ∈ C₁

R₃: [(x₉ = 3) $\vee$ (x₉ = 6)] ⇒ x ∈ C₁

R₄: [(x₁₉ ≠ 6) & (x₃ = 10)] ⇒ x ∈ C₁

R₅: [(x₃ = 2) & (x₁₁ = 1)] ⇒ x ∈ C₁

R₆: [(x₁₂ ≠ 1) & (x₂₀ = 5) & (x₇ = 1)] ⇒ x ∈ C₁

R₇: [(x₁₆ = 1) & (x₂₁ = 2) & (x₁₁ ≠ 7)] ⇒ x ∈ C₁

At all levels tests there is no unclassified cases of the poisonous class, and misclassified cases by all rules, with 100% coverage of the poisonous class and 100% accuracy.

The analysis of the tables above shows that with 30 random groups the SRG algorithm was able to achieve 100% coverage and 100% precision requiring only 7 rules in the level 3 test. In addition, this result is also positive because it was able to pick up a better rule with small coverage (52 cases vs. 42 cases in the alternative rule).

1.5 Experiment 5 with SRG3 on Mushroom Data with 13 Most Frequent Attributes from 30 Groups

Here we tested the sequential rule generation algorithm SRG3 with the 13 most frequent attributes used in the 30 random triples test as seen above, where 7 rules reached 100% precision and 100% coverage. The 13 most frequent attributes were broken up into three different groups to ensure that the test would finish in reasonable time (Tables 13 and 14).

Rules R₁–R₇ (Poisonous):

R₁: [(x₅ = 3) $\vee$ (x₅ = 4) $\vee$ (x₅ = 5) $\vee$ (x₅ = 6) $\vee$ (x₅ = 8) $\vee$ (x₅ = 9)] ⇒ x ∈ C₁

R₂: [(x₉ = 3) $\vee$ (x₉ = 6)] ⇒ x ∈ C₁

R₃: [(x₉ = 5) & (x₁₁ = 1)] ⇒ x ∈ C₁

R₄: [(x₁₆ = 1) & (x₁₉ ≠ 2) & (x₂₀ = 5) & (x₂₁ = 5)] ⇒ x ∈ C₁

R₅: [(x₁₁ = 2) & (x₁₂ ≠ 4)] ⇒ x ∈ C₁

R₆: [(x₁₆ = 1) & (x₁₉ ≠ 2) & (x₂₀ = 8) & (x₂₁ = 2)] ⇒ x ∈ C₁

R₇: [(x₃ = 10) & (x₅ = 7)] ⇒ x ∈ C₁

Table 13 13 used attributes in 30 random triple test results

Table 14 Characteristics of discovered rules R₁-R₇ for poisonous class in Experiment 5

Here the SRG3 algorithm did not reduce the number of selected rules below 7.

1.6 Experiment 6 with SRG3 Algorithm on Mushroom Data and Tenfold Cross Validation with Generated Rules

Here we tested the abilities of SRG3 algorithm to generate beneficial rules in the tenfold cross validation with the sequential triples attribute groups. These groups are G1: {x₁, x₂, x₃}; G2: {x₄, x₅, x₆}, …, G7: {x₁₉, x₂₀, x₂₁, x₂₂}. The tenfold cross validation test was run with these groups to ensure that performance on the training data can be confirmed in the validation data. Table 15 shows the result of this test with 95% precision threshold for rule generation. In all 10 tests all rules provided 100% precision, 100% coverage of the target class.

Table 15 Tenfold cross validation results for sequential triple attribute groups

The Table 15 shows that the tenfold cross validation achieved 100% accuracy in every test/fold. It generated and selected the four rules that were previously generated using all the data and the given attribute groups. This confirms the efficiency of the SRG algorithm to train and generate rules for newly added cases.

1.7 Experiment 7 with SRG3 Algorithm on Mushroom Data and Tenfold Cross Validation with 30 Random Randomly Generated Triples

Here we tested the abilities of SRG3 algorithm to generate beneficial rules using the tenfold cross validation algorithm with 30 random triples of attribute groups as defined below. We ran this test four times to validate the accuracy of results. The result validation is necessary due to the variability of the randomly generated triples.

1.7.1 Run 1

See Tables 16 and 17.

Table 16 30 Randomly generated triples of attributes for Run 1

Table 17 Tenfold cross validation results for sequential triple attribute groups (95% precision threshold) for run 1

In all 10 tests all rules provided 100% precision, 100% coverage of the target class and 100% accuracy.

Rules generated using 10-Fold Cross Validation:

CR₁= R₁: [(x₅ = 3) $\vee$ (x₅ = 4) $\vee$ (x₅ = 5) $\vee$ (x₅ = 6) $\vee$ (x₅ = 8) $\vee$ (x₅ = 9)] ⇒ x ∈ C₁.

R₂: [(x₄ ≠ 2) & (x₂₀ = 5)] ⇒ x ∈ C₁

R₃: [(x₁₂ = 3) & (x₁₆ = 1) & (x₁₉ ≠ 6)] ⇒ x ∈ C₁

R₄: [(x₄ ≠ 2) & (x₇ = 2) & (x₁₀ = 1)] ⇒ x ∈ C₁

R₅: [(x₁₉ ≠ 6) & (x₃ = 10)] ⇒ x ∈ C₁

Complexity 16/3520 = 0.004545.

1.7.2 Run 2

See Tables 18 and 19.

Table 18 30 randomly generated triples for run 2

Table 19 Tenfold cross validation results for sequential triple attribute groups (95% precision threshold) for run 2

At all 10 tests precision of all rules is 100% and target class coverage by rules is 100%.

Rules generated using 10-Fold Cross Validation:

CR₁= R₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁.

R₂: [(x₄ ≠ 2) & (x₂₀ = 5)] ⇒ x ∈ C₁

R₃: [(x₁₂ = 3) & (x₁₈ ≠ 3)] ⇒ x ∈ C₁

R₄: [(x₈ ≠ 1) & (x₁₄ ≠ 8)] ⇒ x ∈ C₁

R₅: [(x₂₂ = 2) & (x₄ ≠ 2)] ⇒ x ∈ C₁

Complexity = 14/3533 = 0.00396.

1.7.3 Run 3

See Tables 20 and 21.

Table 20 30 randomly generated triples for run 3

Table 21 Tenfold cross validation results for sequential triple attribute groups (95% precision threshold) for run 3

In all 10 tests all rules provided 100% precision, 100% coverage of the target class.

Rules generated using 10-Fold Cross Validation:

CR₁= R₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁.

R₂: [(x₅ ≠ 2) & (x₂₀ = 5)] ⇒ x ∈ C₁

R₃: [(x₁₃ = 2) & (x₁₆ = 1) & (x₁₉ ≠ 6)] ⇒ x ∈ C₁

R₄: [(x₁₅ = 8) & (x₂₂ = 2)] ⇒ x ∈ C₁

Complexity = 13/3525 = 0.003688.

1.7.4 Run 4

See Tables 22 and 23.

Table 22 30 randomly generated triples for run 4

Table 23 Tenfold cross validation results for sequential triple attribute groups (95% precision threshold) for run 4

In all 10 tests all rules provided 100% precision and 100% coverage of the target class.

Rules generated using 10-Fold Cross Validation:

CR₁= R₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁.

R₂: [(x₁ ≠ 6) & (x₁₆ = 1) & (x₂₀ = 5)] ⇒ x ∈ C₁

R₃: [(x₁₃ = 2) & (x₁₆ = 1) & (x₁₉ ≠ 6)] ⇒ x ∈ C₁

R₄: [(x₁₅ = 8) & (x₂₂ = 2)] ⇒ x ∈ C₁

Complexity = 14/3531 = 0.00396 (Table 24).

Table 24 Summary of runs

In all runs all rules provided 100% precision, 100% coverage of the target class and 100% accuracy. While the tests are accurate and precise, the generated rules have a moderate amount of variation in complexity and attributes used. The variation is due to the random group generation and the tenfold data partition.

1.7.5 Rule Generation

Below we present rules generated at Step 3 first for the poisonous class and then complementary rules for the eatable class on Mushroom data with most frequent attributes generated from all 22 mushroom attributes. In this experiment the following attribute groups are used: Group A1: x₉, x₅, x₇, ×₁₁; Group A2: ×₁₃, ×₁₄, ×₁₅, x₆; Group A3: x₁, x₂, x₄, ×₂₁, ×₂₂.

According to the design of SRG algorithm it runs at the several levels of thresholds that can be set up by a user. We experimented with 3 level of precision: 75%. 85% and 95% with fixed level of coverage of 0.5%. This means that rules that have lower precision coverage are filtered out and not selected. For coverage with 3916 cases in the poisonous class it means that rules that cover less than 20 cases are filtered out considered as overfitting rules.

The thresholds 75%, 85%, and 95% limit the low margin of the rule quality, but not their upper level. Therefore, we computed the actual precision and coverage reached for the poisonous class at each level. Table 25 shows that rules at all levels missed only 4 cases from the poisonously class, giving 99.89% coverage and none of the poisonous cases was misclassified, giving 100% precision. All misclassified are eatable cases that ranged from 800 cases for 75% threshold to 192 cases for 95% threshold.

Table 25 Results of sequential rule generation for poisonous class in Experiment 7: Rules R₁–R₁₃

We conducted the further analysis for 13 rules selected at level 3 with 95% threshold for rules. See Table 26. This table shows that all 192 misclassified cases belong to the rule R₁ that has 98.47% coverage and 95.2% precision. All other rules have 100% precision and coverage that smaller than for dominant rule R₁.

Table 26 Characteristics of discovered rules R₁–R₁₃ for poisonous class in Experiment 7

Rules R₁–R₁₃ (Poisonous)

R₁: [(x₅ ≠ 7) & (x₇ ≠ 2) & (x₉ ≠ 7) & (x₁₁ ≠ 2)] ⇒ x ∈ C₁

R₂: [(x₅ = 3) $\vee$ (x₅ = 4) $\vee$ (x₅ = 5) $\vee$ (x₅ = 6) $\vee$ (x₅ = 8) $\vee$ (x₅ = 9)] ⇒ x ∈ C₁

R₃: [(x₉ = 3) $\vee$ (x₉ = 6)] ⇒ x ∈ C₁

R₄: [(x₆ ≠ 1) & (x₁₃ ≠ 4) & (x₁₄ ≠ 8) & (x₁₅ ≠ 1)] ⇒ x ∈ C₁

R₅: [(x₁ ≠ 6) & (x₂ ≠ 3) & (x₄ ≠ 1) & (x₂₁ ≠ 6) & (x₂₂ = 7)] ⇒ x ∈ C₁

R₆: [(x₉ = 5) & (x₁₁ = 1)] ⇒ x ∈ C₁

R₇: [(x₁₅ = 3) $\vee$ (x₁₅ = 3) $\vee$ (x₁₅ = 9)] ⇒ x ∈ C₁

R₈: [(x₁ = 5) & (x₄ = 2) & (x₂₁ = 5) & (x₂₂ ≠ 2)] ⇒ x ∈ C₁

R₉: [(x₂₁ = 5) & (x₂₂ = 1)] ⇒ x ∈ C₁

R₁₀: [(x₆ = 3) & (x₁₃ = 2) & (x₁₄ ≠ 1) & (x₁₅ ≠ 8)] ⇒ x ∈ C₁

R₁₁: [(x₂ = 3) & (x₂₁ = 2) & (x₂₂ ≠ 6)] ⇒ x ∈ C₁

R₁₂: [(x₁ = 1) & (x₂₁ = 5) & (x₂₂ ≠ 2)] ⇒ x ∈ C₁

R₁₃: [(x₁ ≠ 6) & (x₂ ≠ 1) & (x₄ ≠ 2) & (x₂₁ = 5) & (x₂₂ = 3)] ⇒ x ∈ C₁

The rules R14 and R15 generated for the eatable class C₂ are presented below:

R₁₄: [(x₅ = 1) & (x₉ ≠ 1)] = > x C₂

R₁₅: [(x₅ = 2) & (x₉ ≠ 1)] = > x C₂

Table 27 shows the analysis of both R14 and R15 for class eatable class C₂. Rules R₁₄ and R₁₅ together cover and correctly predict all 192 cases misclassed by Rule R₁. While both R₁₄ and R₁₅ cover 336 cases each, these cases are different. In fact, rules R₁₄ and R₁₅ have 0% overlap and combined cover 672 cases of eatable class C₂.

Table 27 Characteristics of discovered rules R₁₄, R₁₅ for eatable class for Experiment 7

1.7.6 Combining Rules for Two Classes

Now we have rules R₁–R₁₃ for the target class and rules R₁₄–R₁₅ for the non-target class and can accomplish step 3 of combining them. The only rule for the target class that misclassified some cases is R₁. So, we need to improve only this rule. It is done by creating a new rule R_N that combines rule R₁ with R₁₄ and R₁₅ as follows.

$${\text{R}}_{\mathrm{N}} ({\mathbf{x}})\, = \,{\mathrm{R}}_{\mathrm{1}} \left( {\mathbf{x}} \right){\mathrm{ }}\& \neg ({\mathrm{R}}_{{\mathrm{14}}} \left( {\mathbf{x}} \right) \vee {\mathrm{R}}_{{\mathrm{15}}} \left( {\mathbf{x}} \right))$$

resulted in

$${\text{R}}_{\mathrm{N}} {\mathrm{: }}[(x_{\mathrm{5}} \, \ne \,{\mathrm{7}}){\mathrm{ }}\& {\mathrm{ }}\left( {x_{\mathrm{7}} \, \ne \,{\mathrm{2}}} \right){\mathrm{ }}\& {\mathrm{ }}\left( {x_{\mathrm{9}} \, \ne \,{\mathrm{7}}} \right){\mathrm{ }}\& {\mathrm{ }}\left( {x_{{\mathrm{11}}} \, \ne \,{\mathrm{2}}} \right)]{\mathrm{ }}\&$$

$$\neg (\left[ {\left( {x_{\text{5}} = {\mathrm{1}}} \right){\mathrm{ }}\& {\mathrm{ }}\left( {x_{\mathrm{9}} \ne {\mathrm{1}}} \right)} \right] \vee \left[ {\left( {x_{\mathrm{5}} = {\mathrm{2}}} \right){\mathrm{ }}\& {\mathrm{ }}\left( {x_{\mathrm{9}} \ne {\mathrm{1}}} \right)} \right]) = > {\mathbf{x}} \in {\mathrm{C}}_{\mathrm{1}}$$

after putting actual rules R₁, R₁₄ and R₁₅ to the formula.

Rule R_N is false, R_N(x) = 0, for all 192 cases x misclassified by rule R₁ as poisonous, for which R₁(x) = 1, because for those x rule R₁₄ or R₁₅ is true. If each rule R₁₄ and R₁₅ would independently cover all 192 cases misclassed by R₁ then R_N can be defined simpler in two ways by using any of these rules:

$${\text{R}}_{\mathrm{N}} ({\mathbf{x}})\, = \,{\mathrm{R}}_{\mathrm{1}} \left( {\mathbf{x}} \right){\mathrm{ }}\& \neg {\mathrm{R}}_{{\mathrm{14}}} \left( {\mathbf{x}} \right),{\mathrm{ R}}_{\mathrm{N}} ({\mathbf{x}})\, = \,{\mathrm{R}}_{\mathrm{1}} \left( {\mathbf{x}} \right){\mathrm{ }}\& \neg {\mathrm{R}}_{{\mathrm{15}}} \left( {\mathbf{x}} \right)$$

1.8 Experiment 8 with Algorithm SRG4 Based on Expert Selected Groups

The results of this test are shown in Tables 28 and 29. The groups used are: Group 1 (Cap): {x₁, x₂, x₃}; Group 2 (Odor): {x₅}; Group 3 (Gill): {x₆, x₇, x₈, x₉}; Group 4 (Stalk): {x₁₀, x₁₁, x₁₃, x₁₅}; Group 5 (Veil): {x₁₇, x₁₈, x₁₉}; Group 6 (Spore): {x₂₀}; Group 7 (Distribution): {x₂₁, x₂₂}. The analysis of Tables 28 and 29 shows that using the groups generated by the biologist the SRG algorithm was able to get a good result of 99.81% coverage of the target class with 100% precision using just 3 rules. Although this is considered to be a good result, it did not cover the whole of class C₁. One way this can be achieved is by altering the coverage threshold to a value less than 0.5%. Making this adjustment would allow more small coverage rules to be generated an in turn would allow the combination phase to combine these smaller coverage rules with larger coverage rules to produce high coverage and general rules that cover more class C₁ cases.

Table 28 Sequential rule generation test using expert groups from Biologist (95% precision threshold) for Experiment 8

Table 29 Characteristics of discovered rules R₁–R₃ for poisonous class (95% precision threshold) for Experiment 8

R₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁

R₂: [(x₂₀ = 5)] ⇒ x ∈ C₁

R₃: [(x₁₁ ≠ 1) & (x₁₃ ≠ 4) & (x₁₅ ≠ 8)] ⇒ x ∈ C₁

1.9 Experiment 9 with SRG5 and 7 Successful Attributes

Here we tested SRG algorithm with the 7 used attributes in our previous 100% precision test. This test was run with these attributes in hopes that only using the 7 used attributes will allow the rule generation process to generate and select less rules while keeping the 100% precision and total coverage of the target class.

The 7 attributes were broken up into two different groups to be able to finish the test in reasonable time, where group 1: x₅, x₉, x₁₅; and Group 2: x₁₉, x₂₀, x₂₁, x₂₂. The results are shown in Tables 30 and 31.

Table 30 Sequential rule generation test using successful attributes for Experiment 9

Table 31 Characteristics of discovered rules R₁-R₇ for poisonous class for Experiment 9

Rules R₁–R₇ (Poisonous):

R₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁

R₂: [(x₉ = 3) ∨ (x₉ = 6)] ⇒ x ∈ C₁

R₃: [(x₁₉ = 2) & (x₂₀ = 8) & (x₂₁ ≠ !2) & (x₂₂ ≠ !2)] ⇒ x ∈ C₁

R₄: [(x₁₅ = 3) ∨ (x₁₅ = 2) ∨ (x₁₅ = 9)] ⇒ x ∈ C₁

R₅: [(x₁₉ ≠ 2) & (x₂₀ ≠ 6) & (x₂₁ = 5) & (x₂₂ = 1)] ⇒ x ∈ C₁

R₆: [(x₁₉ = 6) & (x₂₀ = 5) & (x₂₁ ≠ 1) & (x₂₂ ≠ 6)] ⇒ x ∈ C₁

R₇: [(x₂₀ = 8) & (x₂₁ = 2) & (x₂₂ ≠ 6)] ⇒ x ∈ C₁

At all levels all rules provided 100% precision, 100% coverage of the target class.

These tables allow to conclude that the sequential rule generation algorithm using the attribute groups created by using the 7 attributes was unable to reduce the number of rules selected to cover all of class C₁. This result is likely due to the SRG2 process used. Other existing or new versions SRG algorithm can be more successful in future.

1.10 Experiment 10 with SPG 5 and Comparison of Rule Complexity

In this experiment we used SRG5 algorithm based on SRG1 algorithm, with attributes that have been successful for mushroom data in [1, 30]. We compared the complexity of rules generated in this process with rules from [1, 30], which we denote as CR rules.

The formulas for computing complexity of rules and sets of rules are given in Sect. 4.2. Several sets of rules were generated in [1, 30]. We use only the final rules from [1, 30] listed below in our notation.

CR Rules [1, 30]

CR₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁,

Complexity 6/3796 = 0.0016.

CR₂: [(x₂₀ = 5)] ⇒ x ∈ C₁, Complexity 1/72 = 0.014.

CR₃: [(x₈ = 2) & (x₁₂ = 3)] v [(x₈ = 2) & (x₁₂ = 2)] ∨ [(x₈ = 2) & (x₂₁ = 2)] ⇒ x ∈ C₁.

Complexity 6/912 = 0.0066.

Complexity of the set of rules: (6 + 1 + 6)/(3796 + 72 + 912) = 13/ 4780 = 0.0027.

Attribute Groups that we used in SRG algorithm are G1 = {x₅}; G2 = {x₂₀}; G3 = {x₈, x₁₂, x₂₁}, which directly correspond to CR rules above.

Our Rules

R₁: [(x₅ = 3) ∨ (x₅ = 4) ∨ (x₅ = 5) ∨ (x₅ = 6) ∨ (x₅ = 8) ∨ (x₅ = 9)] ⇒ x ∈ C₁

Complexity 6/3796 = 0.0016.

R₂: [(x₂₀ = 5)] ⇒ x ∈ C₁, Complexity 1/72 = 0.014

R₃: [(x₁₂ = 3) & (x₂₁ = 5)] ⇒ x ∈ C₁, Complexity 2/1544 = 0.0013

R₄: [(x₈ ≠ 1) & (x₂₁ = 2)] ⇒ x ∈ C₁, Complexity 2/16 = 0.125

Complexity of a set of rules = (6 + 1 + 2 + 2)/(3796 + 72 + 1544 + 16) = 11/5428 = 0.002.

The algorithm SRG1 generated simpler rules (11 clauses vs. 13 clauses) with the same precision as in in [1, 30] using the attribute groups derived from CR rules. This result shows that the algorithm SRG1 can generate rules that are less complex than CR rules and suggests that better rules are possible with more testing and preprocessing of attribute groups.

1.11 Toolkit

The toolkit includes the Data Type Editor integrated with visualization system VisCanvas 2.0 [29, 32] for multidimensional data visualization based on the adjustable parallel coordinates. The data type editor supports saving data in the explainable measurement coding format for pattern discovery and data visualization. Figure 16 illustrates setting up and applying a coding scheme for data that converts letter grades for 4 classes X₁-X₄ as follows: A to 4, B to 3, C to 2, and so on.

Fig. 16

Example of applying coding schema for class grades

The toolkit supports Nominal, Ordinal, Interval, and Ratio data measurement types. The attributes of the absolute measurement data type are encoded as the ratio data type because it differs from it only by presence of the fixed measurement unit.

The data type editor provides helpful descriptions and examples in case the user is not familiar with these data types. The user interface allows a user: to assign data type for each attribute, and to group values of each attribute. The scheme loader allows a user to assign data measurement type: nominal, ordinal, interval, and ratio.

A typical example of mixed data are the mushroom data [27], which contain 8124 instances and 22 attributes. These attributes include nominal data such as habitat (grass, leaves, meadows, paths, urban, waste, woods), ordinal data, such as gill size (broad, narrow) and gill spacing (crowded, close, distant), absolute data such as the number of rings (0,1,2), which the scheme loader treats as ratio data.

The colors of different parts of the mushroom such as cap color represent an interesting data type. It can be treated as: (1) nominal (red, blue, green and so on), (2) three numeric attributes like R,G,B, (3) some scalar function from R,G,B like R + G + B, and (4) a single numeric ratio data type that uses wavelength. The first one does not capture the similarity between colors, the second one is expanding the number of attributes, the third one corrupts similarity relations between some colors, and the last one is the most physically meaningful.

Since, each color covers a wavelength interval, grouping wavelength values according to colors is a natural way to encode the colors. These groups are ordered and can be encoded by integers starting from 0. In general, grouping attribute values decreases the space size and run time of the algorithms.

Manual coding is time consuming and tedious work for the tasks with many attributes and multiple values of each attribute. The toolkit allows to speed up this process. The editor has the “All Ordinal” and “All Nominal” options that allow to assign initially Nominal or Ordinal type and to assign integer code values from 1 to n to all attributes with abilities to edit this assignment later.

Grouping. Figure 17 illustrates grouping and binary coding keys for a nominal attribute. Figure 18 illustrates setting up groups for the numeric interval and ratio attributes by creating intervals where user sets on the starting value for the group and the length of its interval. Original values of the attributes may not correctly represent its data type for the task. The user can select keeping the existing values or generate new ones.

Hierarchy of attribute groups. When we have hundreds of attributes, a hierarchy of attributes allows to deal with them efficiently. The system supports a user in constructing a hierarchy and picking up a level at which attributes will be visualized.

Fig. 17

Grouping keys and assigning binary codes for a nominal attribute

Fig. 18

Setting up groups for the numeric interval and ratio attributes