A supervised term selection technique for effective text categorization

Abstract

Term selection methods in text categorization effectively reduce the size of the vocabulary to improve the quality of classifier. Each corpus generally contains many irrelevant and noisy terms, which eventually reduces the effectiveness of text categorization. Term selection, thus, focuses on identifying the relevant terms for each category without affecting the quality of text categorization. A new supervised term selection technique have been proposed for dimensionality reduction. The method assigns a score to each term of a corpus based on its similarity with all the categories, and then all the terms of the corpus are ranked accordingly. Subsequently the significant terms of each category are selected to create the final subset of terms irrespective of the size of the category. The performance of the proposed term selection technique is compared with the performance of nine other term selection methods for categorization of several well known text corpora using kNN and SVM classifiers. The empirical results show that the proposed method performs significantly better than the other methods in most of the cases of all the corpora.

Appendix

In this section we shall justify the claims that have been made in Sect 4.2 for all possible TRL values between a term and a category. Let $t_{1}, t_{2a}, t_{2b}, t_{2c}, t_{2d}$ and $t_{2e}$ be the terms obtained respectively by case 1, case 2(a), case 2(b), case 2(c), case 2(d) and case 2(e) and all these terms belong to the same category i.e., $C_{i}$. Let $t_{3a}, t_{3b}, t_{3c}, t_{3d}$ and $t_{3e}$ be the terms obtained respectively by case 3(a), case 3(b), case 3(c), case 3(d) and case 3(e) and all these terms belong to the same $C_{i}$.

Claim 1

$t_{1}$ gets highest preference among all the terms.

Justification: The minimum TRL value of a term t and a category c is 0 and it can be obtained only when $P(t, C_{i})=P(t)=P(C_{i})$ and for all the other cases the TRL values are greater than 0. $t_{1}$ is such a term where $TRL(t_{1},C_{i})=0$. Note that the minimum TRL value indicates the optimum term. The terms of all the sub-cases of case 2 and case 3 can be obtained by using either TF or TRF or TCR. It has been explained in Sect. 4 that the values of TF, TRF and TCR lie between (0, 1) whenever they are applied to find TRL between a term and a category. Thus the TRL values of all the sub-cases of case 2 and case 3 lie between (0, 1). Hence $t_{1}$ gets highest preference among all the terms.

Claim 2(a) $t_{2a}$ gets higher preference than $t_{2b}$ by TRL when $P(t_{2a},C_{i})=P(t_{2b},C_{i})$.

Justification: As $1+P(t_{2b}) > 1+P(C_{i})$ we have

$$\begin{aligned} \displaystyle \frac{1+P(t_{2a},C_{i})}{1+P(C_{i})} = \displaystyle \frac{1+P(t_{2b},C_{i})}{1+P(C_{i})} > \displaystyle \frac{1+P(t_{2b},C_{i})}{1+P(t_{2b})} \end{aligned}$$

$$\begin{aligned} \therefore\, TRL(t_{2a},C_{i}) &= {} 1- \displaystyle \frac{1+P(t_{2a},C_{i})}{1+P(C_{i})} \times E(C_{i}) \\&< {} 1- \displaystyle \frac{1+P(t_{2b},C_{i})}{1+P(t_{2b})} \times E(C_{i}) \\&= {} TRL(t_{2b},C_{i}) \end{aligned}$$

Hence the terms of case 2(a) gets higher preference than the terms of case 2(b) for the same $P(C_{i})$ and $P(t,C_{i})$ values.

Claim 2(b)

If $P(t_{2b})=P(t_{2c})$ then $t_{2b}$ gets higher preference than $t_{2c}$ by TRL.

Justification: It can be seen that $P(t_{2b},C_{i})>P(t_{2c},C_{i})$, since they both exist in the same category $C_{i}$ and $P(t_{2b},C_{i})=P(C_{i})$ and $P(t_{2c},C_{i})<P(C_{i})$.

$$\begin{aligned} \therefore\, TRL(t_{2b},C_{i})& = {} 1- \frac{1+P(t_{2b},C_{i})}{1+P(t_{2b})} \times E(C_{i}) \\&< {} 1- \frac{P(t_{2b},C_{i})}{P(t_{2b})} \times E(C_{i}) \\&= {} 1- \frac{P(t_{2b},C_{i})}{P(t_{2c})} \times E(C_{i})\\&< {} 1- \frac{P(t_{2c},C_{i})}{P(t_{2c})} \times E(C_{i}) \\&= {} 1- \frac{P(t_{2c})-P(t_{2c},\overline{C_{i}})}{P(t_{2c})} \times E(C_{i}) \\&= {} TRL(t_{2c},C_{i}) \end{aligned}$$

Hence for the same $P(C_{i})$ and P(t) values the terms of case 2(b) get higher preference than the terms of case 2(c).

Claim 2(c)

If $P(t_{2b})=P(t_{2d})$ then $t_{2b}$ gets higher preference than $t_{2d}$ by TRL.

Justification: It can be seen that $P(t_{2b},C_{i})>P(t_{2d},C_{i})$, since they both exist in the same category $C_{i}$ and $P(t_{2b},C_{i})=P(C_{i})$ and $P(t_{2d},C_{i})<P(C_{i})$.

$$\begin{aligned} \therefore\, TRL(t_{2b},C_{i})& = {} 1- \frac{1+P(t_{2b},C_{i})}{1+P(t_{2b})} \times E(C_{i}) \\& < {} 1- \frac{P(t_{2b},C_{i})}{P(t_{2b})} \times E(C_{i}) \\ & = {} 1- \frac{P(C_{i})}{P(t_{2b})} \times E(C_{i})\\& < {} 1- \frac{P(C_{i})-P(t_{2d},C_{i})}{P(t_{2d})-P(t_{2d},C_{i})} \times E(C_{i}) \\& < {} 1- \frac{P(C_{i})-P(t_{2d},C_{i})}{P(t_{2d})-P(t_{2d},C_{i})} \\&\times \frac{P(t_{2d})-P(t_{2d},\overline{C_{i}})}{P(t_{2d})} \times E(C_{i}) \\& = {} TRL(t_{2d},C_{i}) \end{aligned}$$

as the multiplication of two fractional values is less than their individual values (e.g., $0.4 \times 0.3 =0.12$ is less than both 0.4 and 0.3). Hence for the same $P(C_{i})$ and P(t) values the terms of case 2(b) get higher preference than the terms of case 2(d).

Claim 2(d)

$t_{2a}$ gets higher preference than $t_{2e}$ by TRL, if $P(t_{2a})=P(t_{2e})$.

Justification: Note that $P(t_{2a})= P(t_{2a},C_{i})$ and $P(t_{2a},C_{i})>P(t_{2e},C_{i})$ as both of $t_{2a}$ and $t_{2e}$ belong to $C_{i}$.

$$\begin{aligned} \therefore\, TRL(t_{2a},C_{i})& = {} 1- \frac{1+P(t_{2a},C_{i})}{1+P(C_{i})} \times E(C_{i}) \\&< {} 1- \frac{P(t_{2a},C_{i})}{P(C_{i})} \times E(C_{i}) \\&= {} 1- \frac{P(t_{2e})}{P(C_{i})} \times E(C_{i}) \\& < {} 1- \frac{P(t_{2e})-P(t_{2e},C_{i})}{P(C_{i})-P(t_{2e},C_{i})} \times E(C_{i}) \\&< {} 1-\frac{P(t_{2e})-P(t_{2e},C_{i})}{P(C_{i})-P(t_{2e},C_{i})} \\&\times \frac{P(t_{2e})-P(t_{2e}, \overline{C_{i}})}{P(t_{2e})} \times E(C_{i}) \\&= {} TRL(t_{2e},C_{i}) \end{aligned}$$

as the multiplication of two fractional values is less than their individual values. Thus the terms of case 2(a) get higher preference than the terms of case 2(d) for the same $P(C_{i})$ and $P(t,C_{i})$ values.

Claim 2(e)

$t_{2c}$ gets higher preference than $t_{2d}$ by TRL when $P(t_{2c},C_{i})=P(t_{2d},C_{i})$.

Justification:

$$\begin{aligned} TRL(t_{2c},C_{i})&= {} 1- \frac{P(t_{2c})-P(t_{2c},\overline{C_{i}})}{P(t_{2c})} \times E(C_{i}) \\&< {} 1- \frac{P(t_{2d})-P(t_{2d},\overline{C_{i}})}{P(t_{2d})} \times E(C_{i}) \\&[\because\, P(t_{2c},C_{i})=P(t_{2d},C_{i}) \text{ and } \\&\qquad P(t_{2d})>P(t_{2c})] \\&< {} 1- \frac{P(C_{i})-P(t_{2e},C_{i})}{P(t_{2e})-P(t_{2e},C_{i})} \\&\times \frac{P(t_{2d})-P(t_{2d},\overline{C_{i}})}{P(t_{2d})} \times E(C_{i}) \\&= {} TRL(t_{2d},C_{i}) \end{aligned}$$

since the multiplication of two fractional values is less than their individual values. Therefore the terms of case 2(c) get higher preference than the terms of case 2(d) for the same $P(C_{i})$ and $P(t,C_{i})$ values.

Claim 3(a)

$t_{3a}, t_{3b}, t_{3c}, t_{3d}$ and $t_{3e}$ get lower preference than $t_{2a}, t_{2b}, t_{2c}, t_{2d}$ and $t_{2e}$ in $C_{i}$.

Justification: It has been justified in Claim 2(d) that $t_{2a}$ gets higher preference than $t_{2e}$, if $P(t_{2a}) =P(t_{2e})$. Claim 2(b) states that $t_{2b}$ gets higher preference than $t_{2c}$, if $P(t_{2b})=P(t_{2c})$. Hence we have to 3(a).

• (i) $t_{3a}$ gets lower preference than $t_{2c}$ by TRL.

  $$\begin{aligned} TRL(t_{2c},C_{i})= & {} 1- \frac{P(t_{2c})-P(t_{2c},\overline{C_{i}})}{P(t_{2c})} \times E(C_{i}) \\ TRL(t_{3a},C_{i})= & {} 1- \frac{1+P(t_{3a},C_{i})}{1+P(C_{i})} \times E(C_{i}) \end{aligned}$$

  As $P(C_{i})>> P(t_{3a})$ and $P(t_{3a})=P(t_{3a},C_{i})$ then $1+P(t_{3a},C_{i}) << 1+P(C_{i})$ and $\displaystyle \frac{1+P(t_{3a},C_{i})}{1+P(C_{i})} << 1$, i.e., close to 0. As a result $TRL(t_{3a},C_{i})$ is close to 1. $P(t_{2c},C_{i}) < P(C_{i}) = P(t_{2c})$ and $P(t_{2c})$ is close to $P(t_{2c},C_{i})$. Therefore $\displaystyle \frac{P(t_{2c})-P(t_{2c},\overline{C_{i}})}{P(t_{2c})}$ is close to 1. Hence $TRL(t_{2c},C_{i})$ is less than $TRL(t_{3a},C_{i})$ and thus $t_{3a}$ gets lower preference than $t_{2c}$ by TRL.

• (ii) $t_{3a}$ gets lower preference than $t_{2d}$ by TRL.

  $$\begin{aligned} TRL(t_{2d},C_{i})&= {} 1- \frac{P(C_{i})-P(t_{2d},C_{i})}{P(t_{2d})-P(t_{2d},C_{i})} \\&\times \frac{P(t_{2d})-P(t_{2d},\overline{C_{i}})}{P(t_{2d})} \times E(C_{i}) \end{aligned}$$
  $$\begin{aligned} TRL(t_{3a},C_{i}) = 1- \frac{1+P(t_{3a},C_{i})}{1+P(C_{i})} \times E(C_{i}) \end{aligned}$$

  It has been shown that $TRL(t_{3a},C_{i})$ is close to 1. Note that $P(t_{2d},C_{i}) < P(C_{i}) < P(t_{2d})$ and $P(t_{2d},C_{i})$ and $P(t_{2d})$ both are close to $P(C_{i})$. Thus $P(t_{2d})$ is close to $P(t_{2d},C_{i})$. Therefore $\displaystyle \frac{P(t_{2d})-P(t_{2d},\overline{C_{i}})}{P(t_{2d})}$ and $\displaystyle \frac{P(C_{i})-P(t_{2d},C_{i})}{P(t_{2d})-P(t_{2d},C_{i})}$ both are close to 1 and $TRL(t_{2d},C_{i})$ becomes close to 0. Hence $TRL(t_{2d},C_{i}) < TRL(t_{3a},C_{i})$ and $t_{3a}$ gets lower preference than $t_{2c}$.

• (iii) $t_{3a}$ gets lower preference than $t_{2e}$ by TRL.

  $$\begin{aligned} TRL(t_{2e},C_{i})= & {} 1- \frac{P(t_{2e})-P(t_{2e},C_{i})}{P(C_{i})-P(t_{2e},C_{i})} \\&\times \frac{P(t_{2e})-P(t_{2e},\overline{C_{i}})}{P(t_{2e})} \times E(C_{i}) \end{aligned}$$
  $$\begin{aligned} TRL(t_{3a},C_{i}) = 1- \frac{1+P(t_{3a},C_{i})}{1+P(C_{i})} \times E(C_{i}) \end{aligned}$$

  $P(t_{2e},C_{i}) < P(t_{2e}) < P(C_{i})$ and $P(t_{2e},C_{i})$ and $P(t_{2e})$ both are close to $P(C_{i})$. Thus $P(t_{2e})$ is close to $P(t_{2e},C_{i})$. Therefore $\displaystyle \frac{P(t_{2e})-P(t_{2e},\overline{C_{i}})}{P(t_{2e})}$ and $\displaystyle \frac{P(t_{2e})-P(t_{2e},C_{i})}{P(C_{i})-P(t_{2e},C_{i})}$ both are close to 1 and as a result $TRL(t_{2e},C_{i})$ is close to 0. Note that $TRL(t_{3a},C_{i})$ is close to 1 and thus $TRL(t_{2e},C_{i}) < TRL(t_{3a},C_{i})$.

Fig. 1

All possible TRL values of a term t and a category c

Claim 3(b)

$t_{3a}$ gets higher preference than $t_{3b}$ by TRL when $P(t_{3a},C_{i})=P(t_{3b},C_{i})$.

Claim 3(c)

If $P(t_{3b})=P(t_{3c})$ then $t_{3b}$ gets higher preference than $t_{3c}$ by TRL.

Claim 3(d)

If $P(t_{3b})=P(t_{3d})$ then $t_{3b}$ gets higher preference than $t_{3d}$ by TRL.

Claim 3(e)

$t_{3a}$ gets higher preference than $t_{3e}$ by TRL, if $P(t_{3a})=P(t_{3e})$.

Claim 3(f)

$t_{3c}$ gets higher preference than $t_{3d}$ by TRL, if $P(t_{3c},C_{i}) = P(t_{3d},C_{i})$.

Justification: Claim 3(b) can be justified in the same way as claim 2(a) has been justified. Claim 3(c), claim 3(d), claim 3(e) and claim 3(f) can be justified respectively in the same way as claim 2(b), claim 2(c), claim 2(d) and claim 2(e) have been justified.