Gated Memory Unit: A Novel Recurrent Neural Network Architecture for Sequential Analysis

Abstract

The predominant models used to analyze sequential data today are recurrent neural networks, specifically Long-Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, which utilize a temporal value known as the hidden state. These recurrent neural networks process sequential data by storing and modifying a hidden state through the use of mathematical functions known as gates. However, these networks hold many flaws such as limited temporal vision, insufficient memory capacity, and ineffective training times. In response, we propose a simple architecture, the Gated Memory Unit, which utilizes a new element, the hidden stack, a data stack implementation of the hidden state, as well as novel gates. This, along with a parameterized bounded activation function (PBA), allows the Gated Memory Unit (GMU) to outperform existing recurrent models effectively and efficiently. Trials on three datasets were used to display the new architecture’s superior performance and reduced training time as well as the utility of the novel hidden stack compared to existing recurrent networks. On data which measures the daily death rate of SARS-Cov-2, the GMU was able to reduce losses to half that of comparable models and did so in nearly half the training time. Additionally, through the use of a generated spiking dataset, the GMU depicted its ability to use its hidden stack to store information past directly observable time steps. We prove that the Gated Memory Unit performs well on a variety of tasks and can outperform existing recurrent architectures.