A Comprehensive Analysis on Question Classification Using Machine Learning and Deep Learning Techniques

Abstract

The competence of any online Web site depends on the type of experience it gives to its users, which depends largely on the content they put up on their Web site. Hence, the content which is being put online should be really taken care. There are many Web sites that provide content to their user in terms of questions and answers, for example, the online Web site Quora, which has large scale of data in terms of questions and answers of users. Users are the one who put up questions and also provide answers to those questions. In this paper, a system is proposed by considering significant amount of data from Kaggle, where it is utilized for different approaches to predict the insincere questions. The goal of the proposed work is to develop a model that considers question as an input in English language and produces output as either 0 or 1. In that, 0 represents sincere type of questions and 1 represents insincere type of questions. Hence, the data are transformed into result including F1 score using preprocessing, TF-IDF and pre-trained word embedding. For each method, preprocessing tasks like tokenization, case normalization and punctuation removal are used. Each individual word after preprocessing is used as word vector in this work. These word vectors are then used for the insincere and sincere question classification which is given as input to the machine learning models like SVM, Naïve Bayes, logistic regression and deep learning model called RNN with word embedding. It is used to detect the toxic content in the reviews as 1 for insincere and 0 for sincere questions. The accuracy of these different methods is critically examined with the help of LSTM, word embedding and model ensembling.