Multimedia Tools and Applications

, Volume 78, Issue 2, pp 1805–1829 | Cite as

Tensor based approach for inpainting of video containing sparse text

  • Baburaj MEmail author
  • Sudhish N. George


Videos received from certain sources may contain irrelevant contents which might reduce the amount of information conveyed by it. This paper proposes an effective tensor based video inpainting approach to improve the quality of these videos by removing and replacing unwanted contents with relevant information. The proposed method employs reweighted tensor decomposition technique to identify and discard the inappropriate sparse components of the video data. These sparse components are substituted with proper contents by utilising spatio-temporal consistency through reweighted tensor completion. The replacement is carried out in such a way that the resulting video possesses superior temporal consistency and visual credibility. The proposed method is applied to sparse text removal of videos having dynamic content in various extents and is found that our method outperforms its counterparts.


Video inpainting Low rank recovery Tensor decomposition Tensor completion 



  1. 1.
    Arizona State University, Yuv video sequences.
  2. 2.
    Baburaj M, George SN (2017) Twist tensor total variation regularized-reweighted nuclear norm based tensor completion for video missing area recovery, Information SciencesGoogle Scholar
  3. 3.
    Bornard R, Lecan E, Laborelli L, Chenot JH (2002) Missing data correction in still images and image sequences. In: Proceedings of the tenth ACM international conference on multimedia. ACM, pp 355–361Google Scholar
  4. 4.
    Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122CrossRefGoogle Scholar
  5. 5.
    Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell 23(11):1222–1239CrossRefGoogle Scholar
  6. 6.
    Candes EJ, Wakin MB, Boyd SP (2008) Enhancing sparsity by reweighted 1 minimization. J Fourier Anal Appl 14(5):877–905MathSciNetCrossRefGoogle Scholar
  7. 7.
    Carroll JD, Chang JJ (1970) Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika 35 (3):283–319CrossRefGoogle Scholar
  8. 8.
    Choi JK, Dong B, Zhang X (2017) An edge driven wavelet frame model for image restoration. arXiv:170107158
  9. 9.
    Cichocki A, Mandic D, De Lathauwer L, Zhou G, Zhao Q, Caiafa C, Phan HA (2015) Tensor decompositions for signal processing applications: from two-way to multiway component analysis. IEEE Signal Proc Mag 32(2):145–163CrossRefGoogle Scholar
  10. 10.
    Comon P (2014) Tensors: a brief introduction. IEEE Signal Proc Mag 31(3):44–53CrossRefGoogle Scholar
  11. 11.
    Criminisi A, Perez P, Toyama K (2003) Object removal by exemplar-based inpainting. In: 2003. Proceedings. 2003 IEEE computer society conference on computer vision and pattern recognition. IEEE, vol 2, pp II–IIGoogle Scholar
  12. 12.
    De Lathauwer L, De Moor B, Vandewalle J (2000) A multilinear singular value decomposition. SIAM J Matrix Anal Appl 21(4):1253–1278MathSciNetCrossRefGoogle Scholar
  13. 13.
    De Lathauwer L (2009) A survey of tensor methods. In: 2009. ISCAS 2009. IEEE international symposium on circuits and systems. IEEE, pp 2773–2776Google Scholar
  14. 14.
    Demanet L, Song B, Chan T (2003) Image inpainting by correspondence maps: a deterministic approach. Appl Comput Math 1100(217-50):99Google Scholar
  15. 15.
    Ding T, Sznaier M, Camps OI (2007) A rank minimization approach to video inpainting. In: 2007. ICCV 2007. IEEE 11th international conference on computer vision. IEEE, pp 1–8Google Scholar
  16. 16.
    Drori I, Cohen-Or D, Yeshurun H (2003) Fragment-based image completion. In: ACM transactions on graphics (TOG). ACM, vol 22, pp 303–312Google Scholar
  17. 17.
    Ebdelli M, Le Meur O, Guillemot C (2015) Video inpainting with short-term windows: application to object removal and error concealment. IEEE Trans Image Process 24(10):3034–3047MathSciNetCrossRefGoogle Scholar
  18. 18.
    Efros AA, Leung TK (1999) Texture synthesis by non-parametric sampling. In: 1999. The proceedings of the seventh IEEE international conference on computer vision. IEEE, vol 2, pp 1033–1038Google Scholar
  19. 19.
    Goldfarb D, Qin Z (2014) Robust low-rank tensor recovery: models and algorithms. SIAM J Matrix Anal Appl 35(1):225–253MathSciNetCrossRefGoogle Scholar
  20. 20.
    Granados M, Tompkin J, Kim K, Grau O, Kautz J, Theobalt C (2012) How not to be seen—object removal from videos of crowded scenes. In: Computer graphics forum, Wiley online library, vol 31, pp 219–228Google Scholar
  21. 21.
    Grasedyck L, Kressner D, Tobler C (2013) A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36(1):53–78MathSciNetCrossRefGoogle Scholar
  22. 22.
    Guo Q, Zhang C, Zhang Y, Liu H (2016) An efficient svd-based method for image denoising. IEEE Trans Circ Syst Video Technol 26(5):868–880CrossRefGoogle Scholar
  23. 23.
    He K, Sun J (2012) Statistics of patch offsets for image completion. In: Computer vision–ECCV 2012. Springer, pp 16–29Google Scholar
  24. 24.
    He W, Zhang H, Zhang L, Shen H (2016) Total-variation-regularized low-rank matrix factorization for hyperspectral image restoration. IEEE Trans Geosci Remote Sens 54(1):178–188CrossRefGoogle Scholar
  25. 25.
    Hu W, Tao D, Zhang W, Xie y, Yang Y (2016) The twist tensor nuclear norm for video completion.. In: IEEE transactions on neural networks and learning systemsGoogle Scholar
  26. 26.
    Ji H, Huang S, Shen Z, Xu Y (2011) Robust video restoration by joint sparse and low rank matrix approximation. SIAM J Imaging Sci 4(4):1122–1142MathSciNetCrossRefGoogle Scholar
  27. 27.
    Jia J, Tang CK (2003) Image repairing: Robust image synthesis by adaptive nd tensor voting. In: 2003. Proceedings. 2003 IEEE computer society conference on computer vision and pattern recognition. IEEE, vol 1, pp I–IGoogle Scholar
  28. 28.
    Jia J, Tai YW, Wu TP, Tang CK (2006) Video repairing under variable illumination using cyclic motions. IEEE Trans Pattern Anal Mach Intell 28(5):832–839CrossRefGoogle Scholar
  29. 29.
    Kilmer ME, Braman K, Hao N, Hoover RC (2013) Third-order tensors as operators on matrices: a theoretical and computational framework with applications in imaging. SIAM J Matrix Anal Appl 34(1):148–172MathSciNetCrossRefGoogle Scholar
  30. 30.
    Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500MathSciNetCrossRefGoogle Scholar
  31. 31.
    Komodakis N, Tziritas G (2007) Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans Image Process 16(11):2649–2661MathSciNetCrossRefGoogle Scholar
  32. 32.
    Kressner D, Steinlechner M, Vandereycken B (2014) Low-rank tensor completion by riemannian optimization. BIT Numer Math 54(2):447–468MathSciNetCrossRefGoogle Scholar
  33. 33.
    Krishnamurthy A, Singh A (2013) Low-rank matrix and tensor completion via adaptive sampling. In: Advances in neural information processing systems, pp 836–844Google Scholar
  34. 34.
    Ling CH, Lin CW, Su CW, Chen YS, Liao HYM (2011) Virtual contour guided video object inpainting using posture mapping and retrieval. IEEE Trans Multimed 13(2):292–302CrossRefGoogle Scholar
  35. 35.
    Liu Y, Caselles V (2013) Exemplar-based image inpainting using multiscale graph cuts. IEEE Trans Image Process 22(5):1699–1711MathSciNetCrossRefGoogle Scholar
  36. 36.
    Lu C, Feng J, Chen Y, Liu W, Lin Z, Yan S (2016) Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5249–5257Google Scholar
  37. 37.
    Martin CD, Shafer R, LaRue B (2013) An order-p tensor factorization with applications in imaging. SIAM J Sci Comput 35(1):A474–A490MathSciNetCrossRefGoogle Scholar
  38. 38.
    Newson A, Almansa A, Fradet M, Gousseau Y, Pérez P (2014) Video inpainting of complex scenes. SIAM J Imaging Sci 7(4):1993–2019MathSciNetCrossRefGoogle Scholar
  39. 39.
    Patwardhan KA, Sapiro G, Bertalmio M (2005) Video inpainting of occluding and occluded objects. In: 2005. ICIP 2005. IEEE international conference on image processing, vol 2. IEEE, pp II–69Google Scholar
  40. 40.
    Patwardhan KA, Sapiro G, Bertalmío M (2007) Video inpainting under constrained camera motion. IEEE Trans Image Process 16(2):545–553MathSciNetCrossRefGoogle Scholar
  41. 41.
    Peng Y, Suo J, Dai Q, Xu W (2014) Reweighted low-rank matrix recovery and its application in image restoration. IEEE Trans Cybern 44(12):2418–2430CrossRefGoogle Scholar
  42. 42.
    Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 151–158Google Scholar
  43. 43.
    Semerci O, Hao N, Kilmer ME, Miller EL (2014) Tensor-based formulation and nuclear norm regularization for multienergy computed tomography. IEEE Trans Image Process 23(4):1678–1693MathSciNetCrossRefGoogle Scholar
  44. 44.
    Umeda Y, Arakawa K (2012) Removal of film scratches using exemplar-based inpainting with directional median filter. In: International symposium on communications and information technologies (ISCIT), 2012. IEEE, pp 6–11Google Scholar
  45. 45.
    Venkatesh MV, Cheung ScS, Zhao J (2009) Efficient object-based video inpainting. Pattern Recogn Lett 30(2):168–179CrossRefGoogle Scholar
  46. 46.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRefGoogle Scholar
  47. 47.
    Wexler Y, Shechtman E, Irani M (2004) Space-time video completion. In: 2004. CVPR 2004. Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition. IEEE, vol 1, pp I–IGoogle Scholar
  48. 48.
    Xu Y, Hao R, Yin W, Su Z (2013) Parallel matrix factorization for low-rank tensor completion. arXiv:13121254
  49. 49.
    Yoo S, Park RH (2009) Red-eye detection and correction using inpainting in digital photographs. IEEE Trans Consum Electron 55(3):1006–1014Google Scholar
  50. 50.
    Zhang Z, Aeron S (2017) Exact tensor completion using t-svd. IEEE Trans Signal Process 65(6):1511–1526MathSciNetCrossRefGoogle Scholar
  51. 51.
    Zhang Z, Ely G, Aeron S, Hao N, Kilmer M (2014) Novel methods for multilinear data completion and de-noising based on tensor-svd. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3842–3849Google Scholar
  52. 52.
    Zhou X, Yang C, Zhao H, Yu W (2015) Low-rank modeling and its applications in image analysis. ACM Comput Surv (CSUR) 47(2):36Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electronics, Communication EngineeringNational Institute of Technology Calicut, NIT Campus Post OfficeCalicutIndia

Personalised recommendations