Skip to main content
SpringerLink
Log in

Adapting total generalized variation for blind image restoration

  • Published:
Multidimensional Systems and Signal Processing Aims and scope Submit manuscript

Abstract

In this paper, a fast blind deconvolution approach is proposed for image deblurring by modifying a recent well-known natural image model, i.e., the total generalized variation (TGV). As a generalization of total variation, TGV aims at reconstructing a higher-quality image with high-order smoothness as well as sharp edge structures. However, when it turns to the blind issue, as demonstrated either empirically or theoretically by several previous blind deblurring works, natural image models including TGV actually prefer the blurred images rather than their counterpart sharp ones. Inspired by the discovery, a simple, yet effective modification strategy is applied to the second-order TGV, resulting in a novel L0–L1-norm-based image regularization adaptable to the blind deblurring problem. Then, a fast numerical scheme is deduced with O(NlogN) complexity for alternatingly estimating the intermediate sharp images and blur kernels via coupling operator splitting, augmented Lagrangian and also fast Fourier transform. Experiment results on a benchmark dataset and real-world blurred images demonstrate the superiority or comparable performance of the proposed approach to state-of-the-art ones, in terms of both deblurring quality and speed. Another contribution in this paper is the application of the newly proposed image prior to single image nonparametric blind super-resolution, which is a fairly more challenging inverse imaging task than blind deblurring. In spite of that, we have shown that both blind deblurring and blind super-resolution (SR) can be formulated into a common regularization framework. Experimental results demonstrate well the feasibility and effectiveness of the proposed blind SR approach, and also its advantage over the recent method by Michaeli and Irani in terms of estimation accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. Since 20 March, 2013, the authors of (Xu et al. 2013) have successively released two executable software (implemented in C ++) for blind motion deblurring, i.e., Robust Motion Deblurring System. The first version is v3.0.1 which implements the algorithm as detailed in Xu et al. (2013), and the second version is v3.1 which incorporates the algorithms in both Xu et al. (2013) and Dong et al. (2014), i.e. Xu et al. (2013), Dong et al. (2014), for more accurate blur-kernel estimation.

References

  • Aharon, M., Elad, M., & Bruckstein, A. M. (2006). The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation. IEEE Transaction On Signal Processing, 54(11), 4311–4322.

    Article  MATH  Google Scholar 

  • Almeida, M., & Almeida, L. (2010). Blind and semi-blind deblurring of natural images. IEEE Transactions on Image Processing, 19(1), 36–52.

    Article  MathSciNet  MATH  Google Scholar 

  • Amizic, B., Molina, R., & Katsaggelos, A. K. (2012). Sparse Bayesian blind image deconvolution with parameter estimation. EURASIP Journal on Image and Video Processing, 20(2012), 1–15.

    Google Scholar 

  • Babacan, S. D., Molina, R., Do, M. N., & Katsaggelos, A. K. (2012). Bayesian blind deconvolution with general sparse image priors. In A. Fitzgibbon et al. (Ed.), ECCV 2012, Part VI, LNCS (Vol. 75771, pp. 341–355).

  • Begin, I., & Ferrie, F. R. (2007). PSF recovery from examples for blind super-resolution. In Proceedings of IEEE conference on image processing (pp. 421–424).

  • Benichoux, A., Vincent, E., & Gribonval, R. (2013). A fundamental pitfall in blind deconvolution with sparse and shift-invariant priors. In Proceedings of international conference on acoustics, speech and signal processing (pp. 6108–6112).

  • Bredies, K., Kunisch, K., & Pock, T. (2010). Total generalized variation. SIAM Journal on Imaging Sciences, 3(3), 492–526.

    Article  MathSciNet  MATH  Google Scholar 

  • Chan, T. F., & Wong, C. K. (1998). Total variation blind deconvolution. IEEE Transactions on Image Processing, 7(3), 370–375.

    Article  Google Scholar 

  • Chang, H., Yeung, D.-Y., & Xiong, Y. (2004). Super-resolution through neighbor embedding. In Proceedings of IEEE international conference on computer vision (pp. 275–282).

  • Cho, S., & Lee, S. (2009). Fast motion deblurring. ACM Transactions on Graphics, 28(5), 145.

    Article  Google Scholar 

  • Cho, S. & Lee, S. (2017). Convergence analysis of MAP based blur kernel estimation. Unpublished. arXiv:1611.07752v1.

  • Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.

    Article  MathSciNet  Google Scholar 

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In Proceedings of european conference on computer vision (pp. 194–199).

  • Efrat, N., Glasner, D., Apartsin, A., Nadler, B., & Levin, A. (2013). Accurate blur models versus image priors in single image super-resolution. In Proceedings of IEEE conference on computer vision (pp. 2832–2839).

  • Fattal, R. (2007). Image upsampling via imposed edge statistics. ACM Transactions on Graphics, 26, 95.

    Article  Google Scholar 

  • Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., & Freeman, W. T. (2006). Removing camera shake from a single photograph. ACM Transactions on Graphics, 25(3), 787–794.

    Article  MATH  Google Scholar 

  • Glasner, D., Bagon, S., & Irani, M. (2009). Super-resolution from a single image. In Proceedings of IEEE international conference on computer vision (pp. 349–356).

  • Gu, S., Zhang, L., Zuo, W., & Feng, X. (2014). Weighted nuclear norm minimization with application to image denoising. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2862–2869).

  • He, C., Hu, C., Zhang, W., & Shi, B. (2014). A fast adaptive parameter estimation for total variation image restoration. IEEE Transactions on Image Processing, 23(12), 4954–4967.

    Article  MathSciNet  MATH  Google Scholar 

  • He, K., Sun, J., & Tang, X. (2011). Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2341–2353.

    Article  Google Scholar 

  • He, Y., Yap, K. H., Chen, L., & Chau, L. P. (2009). A soft MAP framework for blind super-resolution image reconstruction. Image and Vision Computing, 27, 364–373.

    Article  Google Scholar 

  • Joshi, N., Szeliski, R., & Kriegman, D. J. (2008). PSF estimation using sharp edge prediction. In Proceedings of IEEE conference on CVPR (pp. 1–8).

  • Kim, J., Lee, J. K., & Lee, K. M. (2015). Deeply-recursive convolutional network for image super-resolution. arXiv:1511.04491.

  • Kotera, J., Sroubek, F., & Milanfar, P. (2013). Blind deconvolution using alternating maximum a posteriori estimation with heavy-tailed priors. In R. Wilson et al. (Eds.), CAIP, Part II, LNCS (Vol. 8048, pp. 59–66).

  • Krishnan, D. & Fergus, R. (2009). Fast image deconvolution using hyper-laplacian priors. In Proceedings of international conference on neural information and processing systems (pp. 1033–1041).

  • Krishnan, D., Tay, T., & Fergus, R. (2011). Blind deconvolution using a normalized sparsity measure. In Proceedings of international conference on computer vision and pattern recognition (pp. 233–240).

  • Laghrib, A., Ezzaki, M., Rhabi, M., Hakim, A., Monasse, P., & Raghay, S. (2018). Simultaneous deconvolution and denoising using a second order variational approach applied to image super resolution. Computer Vision and Image Understanding, 168, 50–63.

    Article  Google Scholar 

  • Lai, W.-S., Huang, J.-B., Hu, Z., Ahuja, N., & Yang, M.-H. (2016). A comparative study for single image blind deblurring. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 1701–1709).

  • Levin, A., Weiss, Y., Durand, F., & Freeman, W. T. (2011a). Efficient marginal likelihood optimization in blind deconvolution. In Proceedings of international conference on computer vision and pattern recognition (pp. 2657–2664).

  • Levin, A., Weiss, Y., Durand, F., & Freeman, W. T. (2011b). Understanding blind deconvolution algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2354–2367.

    Article  Google Scholar 

  • Marquina, A., & Osher, S. J. (2008). Image super-resolution by TV-regularization and Bregman iteration. Journal of Scientific Computing, 37, 367–382.

    Article  MathSciNet  MATH  Google Scholar 

  • Michaeli, T. & Irani, M. (2013). Nonparametric blind super-resolution. In Proceedings of IEEE conference on computer vision (pp. 945–952).

  • Money, J. H., & Kang, S. H. (2008). Total variation minimizing blind deconvolution with shock filter reference. Image and Vision Computing, 26(2), 302–314.

    Article  Google Scholar 

  • Ohkoshi, K., Sawada, M., Goto, T., Hirano, S., & Sakurai, M. (2014). Blind image restoration based on total variation regularization and shock filter for blurred images. In 2014 IEEE international conference on consumer electronics (ICCE), Las Vegas, NV (pp. 217–218).

  • Osher, S., & Rudin, L. I. (1990). Feature-oriented image enhancement using shock filters. SIAM Journal on Numerical Analysis, 27(4), 919–940.

    Article  MATH  Google Scholar 

  • Pan, J., Hu, Z., Su, Z., & Yang, M.-H. (2014). Deblurring text images via L0-regularized intensity and gradient prior. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 1628–1636).

  • Pan, J., & Su, Z. (2013). Fast L 0-regularized kernel estimation for robust motion deblurring. IEEE Signal Processing Letters, 20(9), 1107–1114.

    Google Scholar 

  • Pan, J., Sun, D., Pfister, H., & Yang, M.-H. (2016). Blind image deblurring using dark channel prior. In IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 2901–2908).

  • Peleg, T., & Elad, M. (2014). A statistical prediction model based on sparse representations for single image super-resolution. IEEE Transactions on Image Processing, 23, 2569–2582.

    Article  MathSciNet  MATH  Google Scholar 

  • Perrone, D., & Favaro, P. (2016a). A clearer picture of total variation blind deconvolution. IEEE Transactions on Pattern Analysis Machine Intelligence, 38(6), 1041–1055.

    Article  Google Scholar 

  • Perrone, D., & Favaro, P. (2016b). A logarithmic image prior for blind deconvolution. International Journal of Computer Vision, 117(2), 159–172.

    Article  MathSciNet  MATH  Google Scholar 

  • Roth, S., & Black, M. J. (2009). Fields of experts. International Journal of Computer Vision, 82(2), 205–229.

    Article  Google Scholar 

  • Rubinstein, R., Peleg, T., & Elad, M. (2013). Analysis K-SVD: A dictionary-learning algorithm for the analysis sparse model. IEEE Transaction on Signal Processing, 61(3), 661–677.

    Article  MathSciNet  MATH  Google Scholar 

  • Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60, 259–268.

    Article  MathSciNet  MATH  Google Scholar 

  • Ruiz, P., Zhou, X., Mateos, J., Molina, R., & Katsaggelos, A. K. (2015). Variational Bayesian blind image deconvolution: A review. Digital Signal Processing, 52, 116–127.

    Article  MathSciNet  Google Scholar 

  • Sampat, M., Wang, Z., Gupta, S., Bovik, A., & Markey, M. (2009). Complex Wavelet structural similarity: A new image similarity index. IEEE Transactions on Image Processing, 18(11), 2385–2401.

    Article  MathSciNet  MATH  Google Scholar 

  • Shao, W.-Z., Deng, H.-S., Ge, Q., Li, H.-B., & Wei, Z.-H. (2016). Regularized motion blur-kernel estimation with adaptive sparse image prior learning. Pattern Recognition, 51, 402–424.

    Article  Google Scholar 

  • Shao, W.-Z., Li, H.-B., & Elad, M. (2015). Bi-l 0-l 2-norm regularization for blind motion deblurring. Journal of Visual Communication and Image Representation, 33, 42–59.

    Article  Google Scholar 

  • Shearer, P., Gilbert, A. C., & Hero A. O., III. (2013). Correcting camera shake by incremental sparse approximation. In Proceedings of IEEE international conference on image processing (pp. 572–576).

  • Takeda, H., Farsiu, S., & Milanfar, P. (2007). Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 16(2), 349–366.

    Article  MathSciNet  Google Scholar 

  • Timofte, R., Smet, V. D., & Gool, L. V. (2013). Anchored neighborhood regression for fast exampled-based super-resolution. In Proceedings of IEEE international conference on computer vision (pp. 1920–1927).

  • Timofte, R., Smet, V. D., & Gool, L. V. (2014). A + : Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of Asian conference on computer vision (pp. 111–126).

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wang, Q., Tang, X., & Shum, H. (2005). Patch based blind image super resolution. In Proceedings of IEEE conference on computer vision (pp. 709–716).

  • Wang, R. & Tao, D. (2014). Recent progress in image deblurring. Unpublished. arXiv:1409.6838

  • Wipf, D. & Zhang, H. (2013). Analysis of Bayesian blind deconvolution. In Proceedings of international conference on energy minimization methods in computer vision and pattern recognition (pp. 40–53).

  • Wipf, D., & Zhang, H. (2014). Revisiting Bayesian blind deconvolution. Journal of Machine Learning Research, 15, 3595–3634.

    MathSciNet  MATH  Google Scholar 

  • Xu, L. & Jia, J. (2010). Two-phase kernel estimation for robust motion deblurring. In ECCV, Part I, LNCS (Vol. 6311, pp. 157–170).

  • Xu, L., Zheng, S., & Jia, J. (2013) Unnatural L 0 sparse representation for natural image deblurring. In Proceedings of international conference on computer vision and pattern recognition (pp. 1107–1114).

  • Yan, J., & Lu, W.-S. (2015). Image denoising by generalized total variation regularization and least squares fidelity. Multidimensional Systems and Signal Processing, 26(1), 243–266.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J., Lin, Z., & Cohen, S. (2013). Fast image super-resolution based on in-place example regression. In Proceedings of IEEE conference on CVPR (pp. 1059–1066).

  • Yang, C.-Y., Ma, C., & Yang, M.-H. (2014). Single image super resolution: A benchmark. In Proceedings of european conference on computer vision (pp. 372–386).

  • Yang, J., Wang, Z., Lin, Z., Cohen, S., & Huang, T. (2012). Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing, 21, 3467–3478.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J., Wright, J., Huang, T., & Ma, Y. (2010). Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19, 2861–2873.

    Article  MathSciNet  MATH  Google Scholar 

  • Zeyde, R., Elad, M., & Protter, M. (2012). On single image scale-up using sparse-representations. In Proceedings of curves and surfaces (pp. 711–730).

  • Zhang, H., Tang, L., Fang, Z., Xiang, C., & Li, C. (2017). Nonconvex and nonsmooth total generalized variation model for image restoration. Signal Processing, 143, 69–85.

    Article  Google Scholar 

  • Zhu, X., & Milanfar, P. (2010). Automatic parameter selection for denoising algorithms using a no-reference measure of image content. IEEE Transactions on Image Processing, 19(12), 3116–3132.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

Many thanks are given to the anonymous reviewers for their pertinent and helpful comments which have improved the paper a lot. The first author is very grateful to Prof. Zhi-Hui Wei, Prof. Yi-Zhong Ma, Dr. Min Wu, and Mr. Ya-Tao Zhang for their kind supports in the past years. The study was supported in part by the Natural Science Foundation (NSF) of China (61771250, 61402239, 61602257, 11671004), NSF of Jiangsu Province (BK20130868, BK20160904) and Guangxi Provinces (2014GXNSFAA118360), NSF for the Jiangsu Institutions (16KJB520035), and also the Open Project Fund of both Jiangsu Key Laboratory of Image and Video Understanding for Social Safety (Nanjing University of Science and Technology, 30920140122007) and National Engineering Research Center of Communications and Networking (Nanjing University of Posts and Telecommunications, TXKY17008).

Author information

Authors and Affiliations

  1. College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, China

    Wen-Ze Shao & Feng Wang

  2. National Engineering Research Center of Communications and Networking, Nanjing University of Posts and Telecommunications, Nanjing, China

    Wen-Ze Shao

  3. Jiangsu Key Laboratory of Image and Video Understanding for Social Safety, Nanjing University of Science and Technology, Nanjing, China

    Wen-Ze Shao

  4. Institutes of Advanced Technology, Zhejiang Dahua Technology Co., Ltd., Hangzhou, China

    Feng Wang

  5. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

    Li-Li Huang

Authors
  1. Wen-Ze Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li-Li Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen-Ze Shao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, WZ., Wang, F. & Huang, LL. Adapting total generalized variation for blind image restoration. Multidim Syst Sign Process 30, 857–883 (2019). https://doi.org/10.1007/s11045-018-0586-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11045-018-0586-0

Keywords

Search

Navigation