Multi-class feature selection via Sparse Softmax with a discriminative regularization

Abstract

Feature selection plays a critical role in many machine learning applications as it effectively addresses the challenges posed by “the curse of dimensionality” and enhances the generalization capability of trained models. However, existing approaches for multi-class feature selection (MFS) often combine sparse regularization with a simple classification model, such as least squares regression, which can result in suboptimal performance. To address this limitation, this paper introduces a novel MFS method called Sparse Softmax Feature Selection ($S^2FS$). $S^2FS$ combines a $l_{2,0}$-norm regularization with the Softmax model to perform feature selection. By utilizing the $l_{2,0}$-norm, $S^2FS$ produces a more precise sparsity solution for the feature selection matrix. Additionally, the Softmax model improves the interpretability of the model’s outputs, thereby enhancing the classification performance. To further enhance discriminative feature selection, a discriminative regularization, derived based on linear discriminate analysis (LDA), is incorporated into the learning model. Furthermore, an efficient optimization algorithm, based on the alternating direction method of multipliers (ADMM), is designed to solve the objective function of $S^2FS$. Extensive experiments conducted on various datasets demonstrate that $S^2FS$ achieves higher accuracy in classification tasks compared to several contemporary MFS methods.