CAST: A convolutional attention spatiotemporal network for predictive learning

Abstract

Predictive learning is receiving growing interests with wide applications. Combining RNN and CNN, recent works attempt to capture temporal dependencies and spatial correlations simultaneously. However, these methods tend to be deeper networks without feature compression in scale and easily suffer from high computational costs, especially for high-resolution frames. To reduce resource burden, we introduce a novel and efficient network named CASTnet for spatiotemporal predictive learning. In this paper, we present a concise architecture cascading multiple stages, which can directly accept the raw frames without information loss in the first stage and compress features in the following stages leading to low computation relatively. Then, we adopt a special prediction head to aggregate multi-level features into predictions, which helps our model capture multi-scale targets. As for the spatiotemporal block, we adopt bidirectional convolutional attention (BCA) operations to capture the local and long-range information simultaneously without quadratic calculation complexity. In order to further improve the model performance while not causing much additional computation burden, we propose frame-wise knowledge distillation, which enables each frame at low-level to learn from each one at high-level. To evaluate our model, we conduct quantitative qualitative and ablation experiments on the MovingMNIST and Radar Echo datasets. The results show that our CASTnet achieves competitive results with lower computational costs and fewer network parameters than the state-of-the-art models.