Learning Depth from Monocular Sequence with Convolutional LSTM Network

  • Chia-Hung YehEmail author
  • Yao-Pao Huang
  • Chih-Yang Lin
  • Min-Hui Lin
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1036)


Resolving depth from monocular RGB image has been a long-standing task in computer vision and robotics. Recently, deep learning based methods has become a popular algorithm on depth estimation. Most existing learning based methods take image-pair as input and utilize feature matching across frames to resolve depth. However, two-frame methods require sufficient and static camera motion to reach optimal performance, while camera motion is usually uncontrollable in most application scenarios. In this paper we propose a recurrent neural network based depth estimation network. With the ability of taking multiple images as input, recurrent neural network will decide by itself which image to reference during estimation. We train a u-net like network architecture which utilizes convolutional LSTM in the encoder. We demonstrate our proposed method with the TUM RGB-D dataset, where our proposed method shows the ability of estimating depth with various sequence lengths as input.


Multi-view depth estimation Deep learning Convolutional LSTM Recurrent neural network 



The authors would like to thank the Ministry of Science and Technology, Taiwan, R.O.C. for financially supporting this research under grants MOST 107-2218-E-003-003-, MOST 107-2218-E-110-004-, MOST 105-2221-E-110-094-MY3 and MOST 106-2221-E-110-083-MY2.


  1. 1.
    Pizzoli, M., Forster, C., Scaramuzza, D.: REMODE: probabilistic, monocular dense reconstruction in real time. In: Proceedings of IEEE International Conference on Robotics and Automation, pp. 2609–2616, May 2014Google Scholar
  2. 2.
    Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851–1858, June 2017Google Scholar
  3. 3.
    Gwn Lore, K., Reddy, K., Giering, M., Bernal, E.A.: Generative adversarial networks for depth map estimation from RGB video. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1177–1185 (2018)Google Scholar
  4. 4.
    Ummenhofer, B., Zhou, H., Uhrig, J., Mayer, N., Ilg, E., Dosovitskiy, A., Brox, T.: Demon: depth and motion network for learning monocular stereo. In: Proceedings of IEEE Conference on computer vision and pattern recognition, vol. 5, p. 6, Jul 2017Google Scholar
  5. 5.
    Wang, K., Shen, S.: MVDepthNet: real-time multiview depth estimation neural network. In: Proceedings of International Conference on 3D Vision (2018)Google Scholar
  6. 6.
    Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems, pp. 802–810 (2015)Google Scholar
  7. 7.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241, October 2015Google Scholar
  8. 8.
    Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: Proceedings of IEEE International Conference on Intelligent Robots and Systems, pp. 573–580, Oct 2012Google Scholar
  9. 9.
    Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: Advances in Neural Information Processing Systems, pp. 2366–2374 (2014)Google Scholar
  10. 10.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361 (2012)Google Scholar
  11. 11.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Proceedings of European Conference on Computer Vision (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Chia-Hung Yeh
    • 1
    • 2
    Email author
  • Yao-Pao Huang
    • 2
  • Chih-Yang Lin
    • 3
  • Min-Hui Lin
    • 2
  1. 1.Department of Electrical EngineeringNational Taiwan Normal UniversityTaipeiTaiwan
  2. 2.Department of Electrical EngineeringNational Sun Yat-sen UniversityKaohsiungTaiwan
  3. 3.Department of Electrical EngineeringYuan Ze UniversityTaoyuanTaiwan

Personalised recommendations