Abstract
For a damaged image, recovering an image with missing entire rows or columns is a challenging problem arising in many real applications, such as digital image inpainting. For this kind of information missing situation, the diffusion-based inpainting methods are tend to produce blur, the exemplar-based methods are prone to error filling and the neural network-based methods are highly dependent on data. Many existing approaches formulate this problem as a general low-rank matrix approximate one which cannot handle this special structural missing very well. In this paper, we propose a novel image inpainting algorithm named nonlocal low-rank tensor completion (NLLRTC) based on the nonlocal self-similarity prior and the low-rank prior. By using the nonlocal self-similarity of image patches, we directly stack these patches into a three-dimensional similar tensor instead of pulling them into column vectors, then the similar tensor can be completed by tensor ring (TR) decomposition. By leveraging the alternating direction method under the augmented Lagrangian multiplier framework, the optimization results can be obtained. Moreover, a weighted nuclear norm is added to the tensor completion model to achieve better inpainting performance, which we call weighted nonlocal low-rank tensor completion (WNLLRTC) algorithm. Our empirical studies show encouraging results on both quantitative assessment and visual interpretation of our proposed methods in comparison to some state-of-the-art algorithms.
Similar content being viewed by others
References
Barnes C, Zhang FL (2017) A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3(01):5–22
Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp 417–424
Cai J, Candés EJ, Shen Z (2008) A singular value thresholding algorithm for matrix completion. Siam Journal on Optimization 20(4):1956–1982
Candès EJ, Recht B (2009) Exact matrix completion via convex optimization. Found Comput Math 9(6):717
Cao X, Ren W, Zuo W, Guo X, Foroosh H (2015) Scene text deblurring using text-specific multiscale dictionaries. IEEE Trans Image Process 24 (4):1302–1314
Criminisi A, Perez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13(9):1200–1212
Daisy M, Buyssens P, Tschumperlé D, Lézoray O (2014) A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence. In: 2014 IEEE international conference on image processing (ICIP), IEEE, pp 4622–4626
Dong W, Shi G, Li X (2012) Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans Image Process 22(2):700–711
Efros AA, Leung TK (2002) Texture synthesis by non-parametric sampling. In: Proceedings of the Seventh IEEE international conference on computer vision
Elad M, Starck JL, Querre P, Donoho DL (2005) Simultaneous cartoon and texture image inpainting using morphological component analysis (mca). Appl Comput Harmon Anal 19(3):340–358
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680
Gu S, Xie Q, Meng D, Zuo W, Feng X, Zhang L (2017) Weighted nuclear norm minimization and its applications to low level vision. Int J Comput Vis 121(2):183–208
Guo Q, Gao S, Zhang X, Yin Y, Zhang C (2017) Patch-based image inpainting via two-stage low rank approximation. IEEE Trans Vis Comput Graph 24(6):2023–2036
He L, Wang Y (2014) Iterative support detection-based split bregman method for wavelet frame-based image inpainting. IEEE Trans Image Process 23 (12):5470–5485
Hillar CJ, Lim L (2013) Most tensor problems are np-hard. Journal of the ACM (JACM) 60(6):1–39
Hu SM, Zhang FL, Wang M, Martin RR, Wang J (2013) Patchnet: A patch-based image representation for interactive library-driven image editing. ACM Transactions on Graphics (TOG) 32(6):1–12
Hu Y, Zhang D, Ye J, Li X, He X (2012) Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Trans Pattern Anal Mach Intell 35(9):2117–2130
Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500
Komodakis N (2006) Image completion using global optimization. In: 2006 IEEE computer society conference on computer vision and pattern recognition
Le Meur O, Gautier J, Guillemot C (2011) Examplar-based inpainting based on local geometry. In: 2011 18Th IEEE international conference on image processing, IEEE, pp 3401–3404
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
Li F, Zeng T (2014) A universal variational framework for sparsity-based image inpainting. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 23(10):4242
Li W, Zhao L, Lin Z, Xu D, Lu D (2015) Non-local image inpainting using low-rank matrix completion. In: Computer graphics forum, vol 34, Wiley Online Library, pp 111–122
Lin Z, Chen M, Ma Y (2010) The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint arXiv:1009.5055
Liu J, Musialski P, Wonka P, Ye J (2012) Tensor completion for estimating missing values in visual data. IEEE Trans Pattern Anal Mach Intell 35 (1):208–220
Luo SJ, Sun YT, Shen IC, Chen BY, Chuang YY (2015) Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Transactions on Visualization & Computer Graphics 21(1):56–67
Masci J, Meier U, Cireṡan D., Schmidhuber J (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. In: International conference on artificial neural networks, Springer, pp 52–59
Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759
Oseledets IV (2011) Tensor-train decomposition. SIAM J Sci Comput 33(5):2295–2317
Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544
Perona P, Malik J (2002) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639
Ren W, Cao X, Pan J, Guo X, Zuo W, Yang MH (2016) Image deblurring via enhanced low-rank prior. IEEE Trans Image Process 25 (7):3426–3437
Steck H (2010) Training and testing of recommender systems on data missing not at random. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 713–722
Toh KC, Yun S (2010) An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of Optimization 6(615-640):15
Tschumperlé D (2006) Fast anisotropic smoothing of multi-valued images using curvature-preserving pde’s. Int J Comput Vis 68(1):65–82
Wen Z, Yin W, Zhang Y (2012) Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm. Mathematical Programming Computation 4(4):333–361
Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 19(5):1153
Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 19(5):1153
Ye X, Yang J, Sun X, Li K, Hou C, Wang Y (2015) Foreground–background separation from video clips via motion-assisted matrix restoration. IEEE Trans Circuits Syst Video Technol 25(11):1721–1734
Yeh R, Chen C, Lim TY, Hasegawa-Johnson M, Do MN (2016) Semantic image inpainting with perceptual and contextual losses. arXiv preprint arXiv:1607.07539 2(3)
Yuan L, Li C, Mandic D, Cao J, Zhao Q (2019) Tensor ring decomposition with rank minimization on latent space: an efficient approach for tensor completion. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9151–9158
Zhang Z, Ganesh A, Liang X, Ma Y (2012) Tilt: Transform invariant low-rank textures. Int J Comput Vis 99(1):1–24
Zhang FL, Wang J, Shechtman E, Zhou ZY, Shi JX, Hu SM (2017) Plenopatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization & Computer Graphics 23(5):1561–1573
Zhang J, Zhao D, Gao W (2014) Group-based sparse representation for image restoration. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 23(8):3336
Zhao Q, Zhou G, Xie S, Zhang L, Cichocki A (2016) Tensor ring decomposition. arXiv preprint arXiv:1606.05535
Zhou M, Chen H, Paisley J, Ren L, Li L, Xing Z, Dunson D, Sapiro G, Carin L (2012) Nonparametric bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 21(1):130–44
Acknowledgements
The research has been supported in part by the National Natural Science Foundation of China (12071263, 61671276, 11971269), the Natural Science Foundation of Shandong Province of China (ZR2019MF045), and the National Science Fund for Distinguished Young Scholars (61625102).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liu, X., Hao, C., Su, Z. et al. Image inpainting algorithm based on tensor decomposition and weighted nuclear norm. Multimed Tools Appl 82, 3433–3458 (2023). https://doi.org/10.1007/s11042-022-12635-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-12635-3