Advertisement

Switchable Temporal Propagation Network

  • Sifei LiuEmail author
  • Guangyu Zhong
  • Shalini De Mello
  • Jinwei Gu
  • Varun Jampani
  • Ming-Hsuan Yang
  • Jan Kautz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11211)

Abstract

Videos contain highly redundant information between frames. Such redundancy has been studied extensively in video compression and encoding, but is less explored for more advanced video processing. In this paper, we propose a learnable unified framework for propagating a variety of visual properties of video images, including but not limited to color, high dynamic range (HDR), and segmentation mask, where the properties are available for only a few key-frames. Our approach is based on a temporal propagation network (TPN), which models the transition-related affinity between a pair of frames in a purely data-driven manner. We theoretically prove two essential properties of TPN: (a) by regularizing the global transformation matrix as orthogonal, the “style energy” of the property can be well preserved during propagation; and (b) such regularization can be achieved by the proposed switchable TPN with bi-directional training on pairs of frames. We apply the switchable TPN to three tasks: colorizing a gray-scale video based on a few colored key-frames, generating an HDR video from a low dynamic range (LDR) video and a few HDR frames, and propagating a segmentation mask from the first frame in videos. Experimental results show that our approach is significantly more accurate and efficient than the state-of-the-art methods.

Supplementary material

474212_1_En_6_MOESM1_ESM.pdf (16.6 mb)
Supplementary material 1 (pdf 17024 KB)
474212_1_En_6_MOESM2_ESM.pptx (62.8 mb)
Supplementary material 2 (pptx 64305 KB)

References

  1. 1.
    Gadde, R., Jampani, V., Gehler, P.V.: Semantic video CNNs through representation warping. In: Proceedings of IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  2. 2.
    He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 35(6), 1397–1409 (2013)CrossRefGoogle Scholar
  3. 3.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM Transactions on Graphics (TOG) (2004)CrossRefGoogle Scholar
  4. 4.
    Jampani, V., Gadde, R., Gehler, P.: Video propagation networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  5. 5.
    Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R., Unger, J.: HDR image reconstruction from a single exposure using deep CNNs. In: ACM Transactions on Graphics (SIGGRAPH Asia) (2017)CrossRefGoogle Scholar
  6. 6.
    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 30(2), 228–242 (2008)CrossRefGoogle Scholar
  7. 7.
    Liu, S., Mello, S.D., Gu, J., Zhong, G., Yang, M., Kautz, J.: Learning affinity via spatial propagation networks. In: Neural Information Processing Systems (NIPS) (2017)Google Scholar
  8. 8.
    Jampani, V., Kiefel, M., Gehler, P.V.: Learning sparse high dimensional filters: image filtering, dense CRFs and bilateral neural networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  9. 9.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_40CrossRefGoogle Scholar
  10. 10.
    Zhang, R., Zhu, J.Y., Isola, P., Geng, X., Lin, A.S., Yu, T., Efros, A.A.: Real-time user-guided image colorization with learned deep priors. In: ACM Transactions on Graphics (SIGGRAPH) (2017)Google Scholar
  11. 11.
    Debevec, P., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM Transactions on Graphics (SIGGRAPH) (1997)Google Scholar
  12. 12.
    Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting. Morgan Kaufmann, San Francisco (2010)Google Scholar
  13. 13.
    Hu, J., Gallo, O., Pulli, K., Sun, X.: HDR deghosting: how to deal with saturation? In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  14. 14.
    Oh, T., Lee, J., Tai, Y., Kweon, I.: Robust high dynamic range imaging by rank minimization. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 37(6), 1219–1232 (2015)CrossRefGoogle Scholar
  15. 15.
    Gallo, O., Troccoli, A., Hu, J., Pulli, K., Kautz, J.: Locally non-right registration for mobile HDR photography. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  16. 16.
    Kang, S., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. In: ACM Transactions on Graphics (SIGGRAPH) (2003)CrossRefGoogle Scholar
  17. 17.
    Kalantari, N., Shechtman, E., Barnes, C., Darabi, S., Goldman, D., Sen, P.: Patch-based high dynamic range video. In: ACM Transactions on Graphics (SIGGRAPH) (2013)Google Scholar
  18. 18.
    Tocci, M., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. In: ACM Transactions on Graphics (SIGGRAPH) (2011)Google Scholar
  19. 19.
    Nayar, S., Mitsunaga, T.: High dynamic range imaging: Spatially varying pixel exposure. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2000)Google Scholar
  20. 20.
    Gu, J., Hitomi, Y., Mitsunaga, T., Nayar, S.: Coded rolling shutter photography: flexible space-time sampling. In: Proceedings of IEEE International Conference on Computational Photography (ICCP) (2010)Google Scholar
  21. 21.
    Kalantari, N., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. In: ACM Transactions on Graphics (SIGGRAPH) (2017)CrossRefGoogle Scholar
  22. 22.
    Zhang, J., Lalonde, J.: Learning high dynamic range from outdoor panoramas. In: Proceedings of IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  23. 23.
    Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. CoRR abs/1508.06576 (2015)Google Scholar
  24. 24.
    Liu, S., Pan, J., Yang, M.-H.: Learning recursive filters for low-level vision via a hybrid neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 560–576. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_34CrossRefGoogle Scholar
  25. 25.
    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  26. 26.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_48CrossRefGoogle Scholar
  27. 27.
    Wang, X., Farhadi, A., Gupta, A.: Actions \(\sim \) transformations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  28. 28.
    Grossberg, M.D., Nayar, S.K.: Modeling the space of camera response functions. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 26(10), 1272–1282 (2004)CrossRefGoogle Scholar
  29. 29.
    Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. Comput. Graph. Forum 22(3), 419–426 (2003)CrossRefGoogle Scholar
  30. 30.
    Reinhard, E., Devlin, K.: Dynamic range reduction inspired by photoreceptor physiology. IEEE Trans. Vis. Comput. Graph. 11(1), 13–24 (2005)CrossRefGoogle Scholar
  31. 31.
    Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Sifei Liu
    • 1
    Email author
  • Guangyu Zhong
    • 1
    • 3
  • Shalini De Mello
    • 1
  • Jinwei Gu
    • 1
  • Varun Jampani
    • 1
  • Ming-Hsuan Yang
    • 1
    • 2
  • Jan Kautz
    • 1
  1. 1.NVIDIASanta ClaraUSA
  2. 2.UC MercedMercedUSA
  3. 3.Dalian University of TechnologyDalianChina

Personalised recommendations