Flow and Color Inpainting for Video Completion

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8753)

Abstract

We propose a framework for temporally consistent video completion. To this end we generalize the exemplar-based inpainting method of Criminisi et al. [7] to video inpainting. Specifically we address two important issues: Firstly, we propose a color and optical flow inpainting to ensure temporal consistency of inpainting even for complex motion of foreground and background. Secondly, rather than requiring the user to hand-label the inpainting region in every single image, we propose a flow-based propagation of user scribbles from the first to subsequent video frames which drastically reduces the user input. Experimental comparisons to state-of-the-art video completion methods demonstrate the benefits of the proposed approach.

Keywords

Video completion Video inpainting Disocclusion Temporal consistency Segmentation Optical flow 

References

  1. 1.
    Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics, pp. 217–226. ACM (2001)Google Scholar
  2. 2.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), Article 24, 1–11 (2009)Google Scholar
  3. 3.
    Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 355–362 (2001)Google Scholar
  4. 4.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  5. 5.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. 6.
    Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 721–728, June 2003Google Scholar
  7. 7.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  8. 8.
    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346. ACM (2001)Google Scholar
  9. 9.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1033–1038 (1999)Google Scholar
  10. 10.
    Granados, M., Kim, K.I., Tompkin, J., Kautz, J., Theobalt, C.: Background inpainting for videos with dynamic objects and a free-moving camera. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part I. LNCS, vol. 7572, pp. 682–695. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  11. 11.
    Granados, M., Tompkin, J., Kim, K.I., Grau, O., Kautz, J., Theobalt, C.: How not to be seen - object removal from videos of crowded scenes. Comput. Graph. Forum 31(2), 219–228 (2012)CrossRefGoogle Scholar
  12. 12.
    Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. Image Process. 11(2), 68–76 (2002)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Masnou, S., Morel, J.M.: Level lines based disocclusion. In: International Conference on Image Processing, vol. 3, pp. 259–263 (1998)Google Scholar
  14. 14.
    Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P.: Towards fast, generic video inpainting. In: Proceedings of the 10th European Conference on Visual Media Production, CVMP ’13, pp. 1–8. ACM, New York (2013)Google Scholar
  15. 15.
    Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P.: Video inpainting of complex scenes, January 2014. http://hal.archives-ouvertes.fr/hal-00937795
  16. 16.
    Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive multi-label segmentation. IEEE Trans. Patt. Anal. Mach. Intell. 35(5), 1234–1247 (2013)CrossRefGoogle Scholar
  17. 17.
    Patwardhan, K., Sapiro, G., Bertalmio, M.: Video inpainting of occluding and occluded objects. In: IEEE International Conference on Image Processing, vol. 2, pp. 69–72 (2005)Google Scholar
  18. 18.
    Patwardhan, K.A., Sapiro, G., Bertalmo, M.: Video inpainting under constrained camera motion. IEEE Trans. Image Process. 16(2), 545–553 (2007)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Sethian, J.A.: A fast marching level set method for monotonically advancing fronts. Proc. Natl. Acad. Sci. 93(4), 1591–1595 (1996)MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Shih, T., Tang, N., Hwang, J.N.: Exemplar-based video inpainting without ghost shadow artifacts by maintaining temporal continuity. IEEE Trans. Circuits Syst. Video Technol. 19(3), 347–360 (2009)CrossRefGoogle Scholar
  21. 21.
    Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. Tools 9(1), 23–34 (2004)CrossRefGoogle Scholar
  22. 22.
    Tsitsiklis, J.N.: Efficient algorithms for globally optimal trajectories. IEEE Trans. Autom. Control 40(9), 1528–1538 (1995)MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Wang, J., Lu, K., Pan, D., He, N., kun Bao, B.: Robust object removal with an exemplar-based image inpainting approach. Neurocomputing 123, 150–155 (2014), contains Special issue articles: Advances in Pattern Recognition Applications and MethodsGoogle Scholar
  24. 24.
    Werlberger, M.: Convex approaches for high performance video processing. Ph.D. thesis, Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria, June 2012Google Scholar
  25. 25.
    Wexler, Y., Shechtman, E., Irani, M.: Space-time video completion. In: International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 120–127, June 2004Google Scholar
  26. 26.
    Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Patt. Anal. Mach. Intell. 29(3), 463–476 (2007)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Michael Strobel
    • 1
  • Julia Diebold
    • 1
  • Daniel Cremers
    • 1
  1. 1.Technical University of MunichMunichGermany

Personalised recommendations