A Method for Sharing Cell State for LSTM-Based Language Model

Abstract

With the rapid growth in the integration of AI technology in various Natural Language Processing, the demands on the development within the fields of Natural Language Understanding and Natural Language Generation have been rapidly increasing as well. Both of these techniques analyze language, as it is naturally spoken or written by users, and must contend with a degree of ambiguity not present in formal language. For that reason, Language Modeling has been used as a key role in this area. Recently, the emerging field of deep learning, which applies complex Deep Neural Networks for machine learning tasks, has been applied to language modeling. Long-Short Term Memory (LSTM), a type of a recurrent neural network, has been adopted and has achieved reasonable results than the traditional language models. However, although LSTM-based language models have shown reasonable results by memorizing preceding cells’ values, it is difficult to memorize all the information of preceding cells because they only use a 2-dimensional matrix to memorize all the information. To compensate for this limitation of memorizing problems, we propose a method for sharing cell state for a neural network-based language model, which considers all preceding cell states as a cell-stack. Our model achieved better performance compared to a traditional LSTM-based language model improving average perplexity scores from 133.88 to 124.32 for various time steps and from 141.29 to 133.62 for various hidden sizes, respectively.