Automating Incident Classification Using Sentiment Analysis and Machine Learning

Abstract

The first step in an incident response plan of an organization is to establish whether the reported event is in fact an incident. This is not an easy task especially if it is a novel event, which has not previously been documented. A typical classification of a novel event includes consulting a database of events with similar keywords and making a subjective decision by human. Efforts have been made to categorize events but there is no universal list of all possible incidents because each incident can be described in multiple different ways. In this paper we propose automating the process of receiving and classifying an event based on the assumption that the main difference between an event and an incident in the field of security is that an event is a positive or a neutral occurrence whereas an incident has strictly negative connotations. We applied sentiment analysis on event reports from the RISI dataset, and the results supported our assumption. We further observed that the sentiment analysis score and magnitude parameters of similar incidents were also very similar and we used them as features in a machine learning model along with other features obtained from each report such as impact and duration in order to predict the likelihood that an event is an incident. We found that using sentiment analysis as a feature of the model increases its accuracy, precision, and recall by at least 10%. The difference between our approach and the typical incident classification approach is that in our approach we train the system to recognize the incidents before any incident actually takes place and our system can recognize incidents even if their descriptions do not include keywords previously encountered by the system.