MARLow: A Joint Multiplanar Autoregressive and Low-Rank Approach for Image Completion

  • Mading Li
  • Jiaying LiuEmail author
  • Zhiwei Xiong
  • Xiaoyan Sun
  • Zongming Guo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9911)


In this paper, we propose a novel multiplanar autoregressive (AR) model to exploit the correlation in cross-dimensional planes of a similar patch group collected in an image, which has long been neglected by previous AR models. On that basis, we then present a joint multiplanar AR and low-rank based approach (MARLow) for image completion from random sampling, which exploits the nonlocal self-similarity within natural images more effectively. Specifically, the multiplanar AR model constraints the local stationarity in different cross-sections of the patch group, while the low-rank minimization captures the intrinsic coherence of nonlocal patches. The proposed approach can be readily extended to multichannel images (e.g. color images), by simultaneously considering the correlation in different channels. Experimental results demonstrate that the proposed approach significantly outperforms state-of-the-art methods, even if the pixel missing rate is as high as 90 %.


Image completion Multiplanar autoregressive model Low-rank minimization 



This work was supported by National High-tech Technology R&D Program (863 Program) of China under Grant 2014AA015205, National Natural Science Foundation of China under contract No. 61472011 and Beijing Natural Science Foundation under contract No. 4142021.


  1. 1.
    Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Cai, J.F., Cands, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20(4), 1956–1982 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Candès, E., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2009). MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Chen, Y.L., Hsu, C.T., Liao, H.Y.: Simultaneous tensor decomposition and completion using factor priors. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 577–591 (2014)CrossRefGoogle Scholar
  5. 5.
    Chierchia, G., Pustelnik, N., Pesquet-Popescu, B., Pesquet, J.C.: A nonlocal structure tensor-based approach for multicomponent image recovery problems. IEEE Trans. Image Process. 23(12), 5531–5544 (2014)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Dong, W., Shi, G., Li, X.: Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process. 22(2), 700–711 (2013)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Dong, W., Zhang, L., Lukac, R., Shi, G.: Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Trans. Image Process. 22(4), 1382–1394 (2013)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Goh, W., Chong, M., Kalra, S., Krishnan, D.: Bi-directional 3D auto-regressive model approach to motion picture restoration. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. 2275–2278, May 1996Google Scholar
  10. 10.
    He, L., Wang, Y.: Iterative support detection-based split bregman method for wavelet frame-based image inpainting. IEEE Trans. Image Process. 23(12), 5470–5485 (2014)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Heide, F., Heidrich, W., Wetzstein, G.: Fast and flexible convolutional sparse coding. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5135–5143, June 2015Google Scholar
  12. 12.
    Ji, H., Liu, C., Shen, Z., Xu, Y.: Robust video denoising using low rank matrix completion. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1791–1798, June 2010Google Scholar
  13. 13.
    Kokaram, A., Rayner, P.: Detection and interpolation of replacement noise in motion picture sequences using 3D autoregressive modelling. In: IEEE International Symposium on Circuits and Systems, vol. 3, pp. 21–24, May 1994Google Scholar
  14. 14.
    Li, X., Orchard, M.: New edge-directed interpolation. IEEE Trans. Image Process. 10(10), 1521–1527 (2001)CrossRefGoogle Scholar
  15. 15.
    Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. In: IEEE International Conference on Computer Vision, pp. 2114–2121, September 2009Google Scholar
  16. 16.
    Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Trans. Image Process. 21(9), 3952–3966 (2012)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: IEEE International Conference on Computer Vision, pp. 2272–2279, September 2009Google Scholar
  18. 18.
    Ono, S., Miyata, T., Yamada, I.: Cartoon-texture image decomposition using blockwise low-rank texture characterization. IEEE Trans. Image Process. 23(3), 1128–1142 (2014)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Roth, S., Black, M.: Fields of experts: a framework for learning image priors. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 860–867, June 2005Google Scholar
  20. 20.
    Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vis. 82(2), 205–229 (2009)CrossRefGoogle Scholar
  21. 21.
    Takeda, H., Farsiu, S., Milanfar, P.: Robust kernel regression for restoration and reconstruction of images from sparse noisy data. In: IEEE International Conference on Image Processing, pp. 1257–1260 (2006)Google Scholar
  22. 22.
    Zhang, D., Hu, Y., Ye, J., Li, X., He, X.: Matrix completion by truncated nuclear norm regularization. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2192–2199, June 2012Google Scholar
  23. 23.
    Zhang, J., Zhao, D., Gao, W.: Group-based sparse representation for image restoration. IEEE Trans. Image Process. 23(8), 3336–3351 (2014)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Zhang, J., Zhao, D., Xiong, R., Ma, S., Gao, W.: Image restoration using joint statistical modeling in a space-transform domain. IEEE Trans. Circ. Syst. Video Technol. 24(6), 915–928 (2014)CrossRefGoogle Scholar
  25. 25.
    Zhang, X., Wu, X.: Image interpolation by adaptive 2-d autoregressive modeling and soft-decision estimation. IEEE Trans. Image Process. 17(6), 887–896 (2008)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Zhang, Z., Ely, G., Aeron, S., Hao, N., Kilmer, M.: Novel methods for multilinear data completion and de-noising based on tensor-svd. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3842–3849, June 2014Google Scholar
  27. 27.
    Zhou, D., Shen, X., Dong, W.: Image zooming using directional cubic convolution interpolation. IET Image Process. 6(6), 627–634 (2012)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Zhou, M., Chen, H., Paisley, J., Ren, L., Li, L., Xing, Z., Dunson, D., Sapiro, G., Carin, L.: Nonparametric bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Trans. Image Process. 21(1), 130–144 (2012)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Mading Li
    • 1
  • Jiaying Liu
    • 1
    Email author
  • Zhiwei Xiong
    • 2
  • Xiaoyan Sun
    • 3
  • Zongming Guo
    • 1
  1. 1.Institute of Computer Science and TechnologyPeking UniversityBeijingChina
  2. 2.University of Science and Technology of ChinaHefeiChina
  3. 3.Microsoft Research AsiaBeijingChina

Personalised recommendations