UniDis: a universal discretization technique

Abstract

Discretization techniques have played an important role in machine learning and data mining as most methods in such areas require that the training data set contains only discrete attributes. Data discretization unification (DDU), one of the state-of-the-art discretization techniques, trades off classification errors and the number of discretized intervals, and unifies existing discretization criteria. However, it suffers from two deficiencies. First, the efficiency of DDU is very low as it conducts a large number of parameters to search good results, which does not still guarantee to obtain an optimal solution. Second, DDU does not take into account the number of inconsistent records produced by discretization, which leads to unnecessary information loss. To overcome the above deficiencies, this paper presents a Uni versal Dis cretization technique, namely UniDis. We first develop a non-parametric normalized discretization criteria which avoids the effect of relatively large difference between classification errors and the number of discretized intervals on discretization results. In addition, we define a new entropy-based measure of inconsistency for multi-dimensional variables to effectively control information loss while producing a concise summarization of continuous variables. Finally, we propose a heuristic algorithm to guarantee better discretization based on the non-parametric normalized discretization criteria and the entropy-based inconsistency. Besides theoretical analysis, experimental results demonstrate that our approach is statistically comparable to DDU evaluated by a popular statistical test and it yields a better discretization scheme which significantly improves the accuracy of classification than previously other known discretization methods except for DDU by running J4.8 decision tree and Naive Bayes classifier.

Appendix

In this section, we show the monotonicity of f(β) and H _β (R _i ) with regard to β.

Theorem 1

f(β) and H _β (R _i ) are the monotonous decreasing functions of β in the interval (0,1].

Proof

Let β ₁ and β ₂ are two values in the interval (0,1], and β ₁ < β ₂. For f(β) in (1), we have

$$\begin{array}{rll} f(\beta_1)-f(\beta_2)&=& \frac{1-(\frac{1}{N})^{\beta_1}}{\beta_1}-\frac{1-(\frac{1}{N})^{\beta_2}}{\beta_2}\\ &=& \frac{\beta_2-\beta_1+\beta_1\big(\frac{1}{N}\big)^{\beta_2}-\beta_2\big(\frac{1}{N}\big)^{\beta_1}}{\beta_1\beta_2} \end{array}$$

• $\because ~ 0<\beta_1< \beta_2\leq 1$

• $\therefore \beta_2-\beta_1>0,~~~~ \beta_1\beta_2>0, ~~~~\beta_1\big(\frac{1}{N}\big)^{\beta_2}-\beta_2\big(\frac{1}{N}\big)^{\beta_1}<0$

• $\because N \gg 1$

• $\therefore~ \beta_2-\beta_1 > \mid \beta_1\big(\frac{1}{N}\big)^{\beta_2}-\beta_2\big(\frac{1}{N}\big)^{\beta_1} \mid$

• $\therefore f(\beta_1)>f(\beta_2)$

Therefore, f(β) increases with the reduction of β in the interval (0,1]. Similarly, $H_{\beta_1}(R_{i})>H_{\beta_2}(R_{i})$. Therefore, H _β (R _i ) is also monotone decreasing with regard to β in the interval (0,1].□

Theorem 2

$f(\beta)\in \big[1-\frac{1}{N}, \ln N \big]$ , and H ₁(R _i ) ≤ H _β (R _i ) ≤ logS.

Proof

According to Theorem 1, f(β) achieves a minimum value when β = 1 and achieves a maximum value when β→0. Then, we have

$$ 1-\frac{1}{N} \leq f(\beta)<\displaystyle \lim\limits_{\beta\rightarrow 0}~\frac{1-(\frac{1}{N})^{\beta}}{\beta}=\ln N $$

Similarly,

$$\begin{array}{rll} H_{1}(R_{i}) \leq H_{\beta}(R_{i})&<&\lim\limits_{\beta\rightarrow 0} H_{\beta}(R_{i})\\ &=&\lim\limits_{\beta\rightarrow 0}\sum\limits_{j=1}^{S}\frac{N_{ij}}{N_{i\cdot}}\Big[1-\left(\frac{N_{ij}}{N_{i\cdot}}\right)^{\beta}\Big]\Big/\beta \\&=& \sum\limits_{j=1}^{S} \frac{N_{ij}}{N_{i\cdot}} \log \frac{N_{i\cdot}}{N_{ij}}\\&=&H(R_{i}) \end{array}$$

where H(R _i ) is Shannon’s entropy (Cover and Thomas 2006) of interval R _i . According to the extremum property of entropy, H(R _i ) ≤ logS. Therefore, the theorem is proven.□