Ensemble-based noise detection: noise ranking and visual performance evaluation

Abstract

Noise filtering is most frequently used in data preprocessing to improve the accuracy of induced classifiers. The focus of this work is different: we aim at detecting noisy instances for improved data understanding, data cleaning and outlier identification. The paper is composed of three parts. The first part presents an ensemble-based noise ranking methodology for explicit noise and outlier identification, named Noise- Rank, which was successfully applied to a real-life medical problem as proven in domain expert evaluation. The second part is concerned with quantitative performance evaluation of noise detection algorithms on data with randomly injected noise. A methodology for visual performance evaluation of noise detection algorithms in the precision-recall space, named Viper, is presented and compared to standard evaluation practice. The third part presents the implementation of the NoiseRank and Viper methodologies in a web-based platform for composition and execution of data mining workflows. This implementation allows public accessibility of the developed approaches, repeatability and sharing of the presented experiments as well as the inclusion of web services enabling to incorporate new noise detection algorithms into the proposed noise detection and performance evaluation workflows.

Appendices

Appendix 1: Evaluation of HARF with different agreement levels

We here present in more detail our High Agreement Random Forest noise detection algorithm (HARF), introduced in our paper (Sluban et al. 2010). The HARF noise detection algorithm is used as an example of a new algorithm whose noise detection performance is evaluated and compared to the performance of other existing approaches by our Viper visual performance evaluation methodology. This appendix briefly presents HARF and experimentally shows that HARF-70 and HARF-80 are the best performing variants of the HARF noise detection algorithm.

Classification filters using Random Forest (RF) classifiers (actually) act like an ensemble noise filter, since randomly generated decision trees vote for the class to be assigned to an instance. However, if the class which got the maximal number of votes won only by a single vote (over the 50–50 % voting outcome), this result is not reliable, since the decision trees are generated randomly. Therefore, we decided to demand high agreement of classifiers (decision trees), i.e., a significantly increased majority of votes, namely, for two class domains used in our experiments this means that over 50 % (e.g., 60, 70, 80 or 90 %) of the randomly generated decision trees should classify an instance into the opposite class in order to accept it as noisy.

We have tested and implemented this high agreement based approach in the classification filtering manner with the RF classifier with 500 decision trees and with four different agreement levels: 60, 70, 80 and 90 %, referred to as HARF-60, HARF-70, HARF-80 and HARF-90, respectively. The obtained noise detection results are presented in part C of Table 3.

The results in part C of Table 3 prove that our request for high agreement of decision trees in the Random Forest leads to excellent performance results of the HARF noise detection algorithm. Figure 15 clearly indicates the increase in precision of noise detection based on the increased agreement level, for all experimental domains with 5 % injected noise.¹³ However, the best results of the proposed HARF noise detection algorithm in terms of $F_{0.5}$-scores are achieved at agreement levels 70 and 80 % as shown in Figure 16 and presented in bold in part C of Table 3. Therefore, only the HARF noise detection algorithms using RF500 with 70 and 80 % agreement level—referred to as HARF-70 and HARF-80, respectively—are used in further evaluation and comparisons of noise detection algorithms in this paper.

Fig. 15

Precision of HARF noise detection based on the agreement level for RF on all domains with 5 % injected noise

Fig. 16

$F_{0.5}$-scores of HARF noise detection based on the agreement level for RF on all domains with 5 % injected noise

Appendix 2: Discussion of tabular noise detection results

This appendix presents and discusses the performance results of different noise detection algorithms and their ensembles evaluated on four domains with three different levels of randomly injected class noise. The precision and $F_{0.5}$-score results in Tables 3 and 4 present averages and standard deviations over ten repetitions of noise detection on a given domain with a given level of randomly injected class noise. The best precision and the best $F_{0.5}$-score results in each table are printed in bold.

Observing the results in part (A) of Table 3 we can see that among the classification filters the RF and SVM filters generally perform best, RF on the KRKP and CHD datasets, SVM on the TTT dataset and both RF and SVM on the NAKE dataset. The NN filter is quite good on NAKE at lower noise levels, but otherwise achieves only average performance. Noise detection with the Bayes classification filter shows overall the worst performance results, except for the CHD domain where it outperforms NN and is comparable to SVM.

The performance of the saturation filters presented in part (B) of Table 3 is between the worst and the best classification filters. Except for lower noise levels on the KRKP dataset, where SF even outperforms the best classification filter (RF), the saturation filters are not really comparable to the best classification filtering results. On the other hand, among the saturation filters, SF achieves higher or comparable (on the NAKE dataset) precision results like PruneSF, whereas the $F_{0.5}$-scores of PruneSF are not much lower, expect for the KRKP dataset, which indicates that PruneSF has higher recall of injected class noise than SF.

Ensemble filtering using a majority voting scheme (a simple majority of votes of elementary noise filters in the given ensemble) did not outperform the precision of injected noise detection achieved by the best elementary filtering algorithms alone.¹⁴ On the other hand, with consensus voting we significantly improved the precision of noise detection as well as the $F_{0.5}$-scores. Therefore, from now on, when we talk about ensemble noise filters in this paper we actually refer to ensemble noise filters using the consensus voting scheme.

Performance results of the ten ensemble noise filters constructed from different elementary noise detection algorithms can be found in Table 4. The results show that in terms of precision using many diverse elementary noise detection algorithms in the ensemble proves to be better. Even including a different type of worst performing noise filter like Bayes, this does not reduce the ensemble’s precision but it actually improves it, as can be seen by comparing precision results of Ens2 and Ens3, Ens5 and Ens6, as well as Ens8 and Ens9. The highest precision among noise detection ensembles was hence not surprisingly achieved by Ens10 which uses all elementary noise filters examined in this work. However, considering the $F_{0.5}$-scores it is interesting to notice that ensembles without SF perform best, namely Ens1, Ens2 and Ens3 perform better on all datasets at higher noise levels (5–10 %), whereas Ens7, Ens8 and Ens9 are better at lower noise levels (2–5 %).

Last we discuss the results presented in part (C) of Table 3 for our HARF noise detection algorithm (described in more detail in Appendix 1) with four different agreement levels: 60, 70, 80 and 90 %, also referred to as HARF-60, HARF-70, HARF-80 and HARF-90, respectively. The results in part (C) of Table 3 as well as Figs. 15 and 16 in Appendix 1 show that HARF-80 and HARF-90 are the most precise noise detection algorithms presented in Table 3 and achieve comparable or higher precision than the precisest ensemble filters in Table 4. Furthermore, in terms of the $F_{0.5}$-scores the best results are achieved by HARF-70 and HARF-80, which shows to be also significantly better than the results achieved by elementary noise detection algorithms, expect in one case where SVM outperforms the HARF algorithms on the TTT dataset with 10 % noise level. Compared to ensemble noise filters the HARF-70 and HARF-80 achieve lower $F_{0.5}$-scores only on the TTT dataset, but are slightly better on the KRKP and CHD datasets and better on the NAKE dataset.

At this point we can observe another interesting phenomenon, indicating that classification noise filters are robust noise detection algorithms. By analyzing the results in Tables 3 and 4 we see that the proportion of detected noise (i.e., the precision of noise detection) increases with the increased abundance of noise (increased levels of artificially inserted noise 2, 5, 10 %). Clearly, with increased noise levels the classifiers are less accurate (their accuracy drops), therefore leading to the increased number of misclassified instances, i.e., the number of instances which are considered potentially noisy increases. As an observed side effect, the proportion of detected artificially corrupted instances increases as well.