Learning Contextual Dependencies for Optical Flow with Recurrent Neural Networks

Conference paper

DOI: 10.1007/978-3-319-54190-7_5

Part of the Lecture Notes in Computer Science book series (LNCS, volume 10114)
Cite this paper as:
Lu M., Deng Z., Li ZN. (2017) Learning Contextual Dependencies for Optical Flow with Recurrent Neural Networks. In: Lai SH., Lepetit V., Nishino K., Sato Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science, vol 10114. Springer, Cham


Pixel-level prediction tasks, such as optical flow estimation, play an important role in computer vision. Recent approaches have attempted to use the feature learning capability of Convolutional Neural Networks (CNNs) to tackle dense per-pixel predictions. However, CNNs have not been as successful in optical flow estimation as they are in many other vision tasks, such as image classification and object detection. It is challenging to adapt CNNs designated for high-level vision tasks to handle pixel-level predictions. First, CNNs do not have a mechanism to explicitly model contextual dependencies among image units. Second, the convolutional filters and pooling operations result in reduced feature maps and hence produce coarse outputs when upsampled to the original resolution. These two aspects render CNNs limited ability to delineate object details, which often result in inconsistent predictions. In this paper, we propose a recurrent neural network to alleviate this issue. Specifically, a row convolutional long short-term memory (RC-LSTM) network is introduced to model contextual dependencies of local image features. This recurrent network can be integrated with CNNs, giving rise to an end-to-end trainable network. The experimental results demonstrate that our model can learn context-aware features for optical flow estimation and achieve competitive accuracy with the state-of-the-art algorithms at a frame rate of 5 to 10 fps.

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.College of Computer ScienceZhejiang UniversityHangzhouChina
  2. 2.School of Computing ScienceSimon Fraser UniversityBurnabyCanada

Personalised recommendations