Applying Softmax Classifiers to Open Set

Abstract

Many artificial neural network classifiers assume a closed-world, where the composition of classes is fixed and known, however problems where this assumption does not hold are found in nearly every case where multi-class classification is applied. For example, in network traffic classification the actual number of classes often dramatically exceeds the number of classes known or labelled at training time. Various treatments have been proposed to adapt closed-set classifiers for application in open-set scenarios, both formal and informal. To demonstrate the effectiveness of these treatments, we conducted an empirical study using a simple example based on the MNIST digit classification problem. Our results show the various treatments make trade-offs between classification recall and precision. We demonstrate that softmax output equalisation during training can significantly improve performance in open set classification.