Memory Selection Network for Video Propagation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12360)


Video propagation is a fundamental problem in video processing where guidance frame predictions are propagated to guide predictions of the target frame. Previous research mainly treats the previous adjacent frame as guidance, which, however, could make the propagation vulnerable to occlusion, large motion and inaccurate information in the previous adjacent frame. To tackle this challenge, we propose a memory selection network, which learns to select suitable guidance from all previous frames for effective and robust propagation. Experimental results on video object segmentation and video colorization tasks show that our method consistently improves performance and can robustly handle challenging scenarios in video propagation.

Supplementary material

504470_1_En_11_MOESM1_ESM.pdf (655 kb)
Supplementary material 1 (pdf 654 KB)


  1. 1.
    Bao, L., Wu, B., Liu, W.: CNN in MRF: video object segmentation via inference in a CNN-based higher-order spatio-temporal MRF. In: CVPR (2018)Google Scholar
  2. 2.
    Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: CVPR (2017)Google Scholar
  3. 3.
    Chai, Y.: Patchwork: a patch-wise attention network for efficient object detection and segmentation in video streams. In: ICCV (2019)Google Scholar
  4. 4.
    Cheng, J., Tsai, Y.H., Hung, W.C., Wang, S., Yang, M.H.: Fast and accurate online video object segmentation via tracking parts. In: CVPR (2018)Google Scholar
  5. 5.
    Cheng, J., Tsai, Y.H., Wang, S., Yang, M.H.: Segflow: joint learning for video object segmentation and optical flow. In: ICCV (2017)Google Scholar
  6. 6.
    Ci, H., Wang, C., Wang, Y.: Video object segmentation by learning location-sensitive embeddings. In: ECCV (2018)Google Scholar
  7. 7.
    Duarte, K., Rawat, Y.S., Shah, M.: Capsulevos: semi-supervised video object segmentation using capsule routing. In: ICCV (2019)Google Scholar
  8. 8.
    Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian conference on Image analysis (2003)Google Scholar
  9. 9.
    Griffin, B.A., Corso, J.J.: Bubblenets: learning to select the guidance frame in video object segmentation by deep sorting frames. In: CVPR (2019)Google Scholar
  10. 10.
    He, K., Sun, J., Tang, X.: Guided image filtering. TPAMI (2013)Google Scholar
  11. 11.
    Hu, Y.T., Huang, J.B., Schwing, A.: Maskrnn: instance level video object segmentation. In: NeurIPS (2017)Google Scholar
  12. 12.
    Hu, Y.T., Huang, J.B., Schwing, A.G.: Videomatch: matching based video object segmentation. In: ECCV (2018)Google Scholar
  13. 13.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)Google Scholar
  14. 14.
    Jampani, V., Gadde, R., Gehler, P.V.: Video propagation networks. In: CVPR (2017)Google Scholar
  15. 15.
    Johnander, J., Danelljan, M., Brissman, E., Khan, F.S., Felsberg, M.: A generative appearance model for end-to-end video object segmentation. In: CVPR (2019)Google Scholar
  16. 16.
    Khoreva, A., Benenson, R., Ilg, E., Brox, T., Schiele, B.: Lucid data dreaming for multiple object tracking. arXiv:1703.09554 (2017)
  17. 17.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  18. 18.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: TOG (2004)Google Scholar
  19. 19.
    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. TPAMI (2008)Google Scholar
  20. 20.
    Li, X., et al.: Video object segmentation with re-identification. In: The 2017 DAVIS Challenge on Video Object Segmentation-CVPR Workshops (2017)Google Scholar
  21. 21.
    Li, X., Change Loy, C.: Video object segmentation with joint re-identification and attention-aware mask propagation. In: ECCV (2018)Google Scholar
  22. 22.
    Li, Z., Chen, Q., Koltun, V.: Interactive image segmentation with latent diversity. In: CVPR (2018)Google Scholar
  23. 23.
    Liu, S., De Mello, S., Gu, J., Zhong, G., Yang, M.H., Kautz, J.: Learning affinity via spatial propagation networks. In: NeurIPS (2017)Google Scholar
  24. 24.
    Liu, S., Zhong, G., De Mello, S., Gu, J., Yang, M.H., Kautz, J.: Switchable temporal propagation network. arXiv:1804.08758 (2018)
  25. 25.
    Luiten, J., Voigtlaender, P., Leibe, B.: Premvos: proposal-generation, refinement and merging for video object segmentation. In: ACCV (2018)Google Scholar
  26. 26.
    Maninis, K.K., et al.: Video object segmentation without temporal information. TPAMI 41, 1515–1530 (2018)CrossRefGoogle Scholar
  27. 27.
    Miksik, O., Munoz, D., Bagnell, J.A., Hebert, M.: Efficient temporal consistency for streaming video scene analysis. In: ICRA (2013)Google Scholar
  28. 28.
    Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: ICCV (2019)Google Scholar
  29. 29.
    Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., Sorkine-Hornung, A.: Learning video object segmentation from static images. In: CVPR (2017)Google Scholar
  30. 30.
    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: CVPR (2016)Google Scholar
  31. 31.
    Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv:1704.00675 (2017)
  32. 32.
    Rick Chang, J.H., Frank Wang, Y.C.: Propagated image filtering. In: CVPR (2015)Google Scholar
  33. 33.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
  34. 34.
    Tokmakov, P., Alahari, K., Schmid, C.: Learning video object segmentation with visual memory. In: ICCV (2017)Google Scholar
  35. 35.
    Ventura, C., Bellver, M., Girbau, A., Salvador, A., Marques, F., Giro-i Nieto, X.: Rvos: end-to-end recurrent network for video object segmentation. In: CVPR (2019)Google Scholar
  36. 36.
    Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., Chen, L.C.: Feelvos: fast end-to-end embedding learning for video object segmentation. In: CVPR (2019)Google Scholar
  37. 37.
    Voigtlaender, P., Leibe, B.: Online adaptation of convolutional neural networks for video object segmentation. arXiv:1706.09364 (2017)
  38. 38.
    Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., Murphy, K.: Tracking emerges by colorizing videos. In: ECCV (2018)Google Scholar
  39. 39.
    Wug Oh, S., Lee, J.Y., Sunkavalli, K., Joo Kim, S.: Fast video object segmentation by reference-guided mask propagation. In: CVPR (2018)Google Scholar
  40. 40.
    Xu, K., Wen, L., Li, G., Bo, L., Huang, Q.: Spatiotemporal CNN for video object segmentation. In: CVPR (2019)Google Scholar
  41. 41.
    Xu, N., et al.: Youtube-VOS: Sequence-to-sequence video object segmentation. In: ECCV (2018)Google Scholar
  42. 42.
    Xu, N., et al.: Youtube-VOS: a large-scale video object segmentation benchmark. arXiv:1809.03327 (2018)
  43. 43.
    Yang, L., Wang, Y., Xiong, X., Yang, J., Katsaggelos, A.K.: Efficient video object segmentation via network modulation. In: CVPR (2018)Google Scholar
  44. 44.
    Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. TIP 15, 1120–1129 (2006)Google Scholar
  45. 45.
    Zeng, X., Liao, R., Gu, L., Xiong, Y., Fidler, S., Urtasun, R.: DMM-net: Differentiable mask-matching network for video object segmentation. In: ICCV (2019)Google Scholar
  46. 46.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016)Google Scholar
  47. 47.
    Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. TOG (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.The Chinese University of Hong KongHong KongChina
  2. 2.University of Hong KongHong KongChina
  3. 3.SmartMoreShenzhenChina

Personalised recommendations