Multi-class E-mail Classification with a Semi-Supervised Approach Based on Automatic Feature Selection and Information Retrieval

Abstract

Nowadays, millions of data are generated every day, and their use and interpretation have become fundamental in all fields. This is particularly true in the area of e-mails classification where, beyond its key role in organizing huge amounts of incoming information, it presents several challenging aspects to be solved. To the well-known problems presented by textual data (as its ambiguity) e-mails are usually characterized by their short lenght and informal language. These difficulties are increased when a relatively large number of highly imbalanced classes need to be considered and manual labeling is expensive and must be carried out by specialized personnel. Those are the main issues addressed in the present work, where Spanish-language e-mails sent by students of an Argentinian university needs to be categorized in 16 different classes.

Our proposal to address this problem consists of a semi-supervised approach based on an automatic feature selection process complemented with an information retrieval strategy. From an initial data set of manually labeled e-mails, the main features are selected for each class, using three techniques: logistic regression, TF-IDF, and SS3. Then, the remaining (non labeled) instances are indexed with a general-purpose search engine (Elasticsearch) and documents of each class are retrieved based on the selected features identified by each technique.

Our very simple approach shows that classifiers trained with labeled documents plus those retrieved in an automatic way obtain an improvement in performance (up to 6%) regarding classifiers trained only with manually labeled instances. Those improvements are observed in both, traditional learning algorithms like SVM, but also in more recent, state of the arts, transformer-based models (BERT).