Deep neural network-based automatic unknown protocol classification system using histogram feature

Abstract

The protocol reverse engineering technique can be used to extract the specification of an unknown protocol. However, there is no standardized method, and in most cases, the extracting process is executed manually or semiautomatically. Since only frequently seen values are extracted as fields from the messages of a protocol, it is difficult to understand the complete specification of the protocol. Therefore, if the information about the structure of an unknown protocol could be acquired in advance, it would be easy to conduct reverse engineering. As such, one of the most important techniques for classifying unknown protocols is a feature extraction algorithm. In this paper, we propose a new feature extraction algorithm based on average histogram for classification of an unknown protocol and design unknown protocol classifier using deep belief networks, one of deep learning algorithms. In order to verify the performance of the proposed system, we performed the training using eight open protocols to evaluate the performance using unknown data. Experimental results show that the proposed technique gives significantly more reliable results of about 99% classification performance, regardless of the strength of the modification of the protocol.