Deblurring Shaken and Partially Saturated Images

Abstract

We address the problem of deblurring images degraded by camera shake blur and saturated (over-exposed) pixels. Saturated pixels violate the common assumption that the image-formation process is linear, and often cause ringing in deblurred outputs. We provide an analysis of ringing in general, and show that in order to prevent ringing, it is insufficient to simply discard saturated pixels. We show that even when saturated pixels are removed, ringing is caused by attempting to estimate the values of latent pixels that are brighter than the sensor’s maximum output. Estimating these latent pixels is likely to cause large errors, and these errors propagate across the rest of the image in the form of ringing. We propose a new deblurring algorithm that locates these error-prone bright pixels in the latent sharp image, and by decoupling them from the remainder of the latent image, greatly reduces ringing. In addition, we propose an approximate forward model for saturated images, which allows us to estimate these error-prone pixels separately without causing artefacts. Results are shown for non-blind deblurring of real photographs containing saturated regions, demonstrating improved deblurred image quality compared to previous work.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Notes

  1. 1.

    We note that the regulariser \(\varvec{\phi }\) in Eq. (26) is not continuously differentiable, however this can be avoided by using the standard approximation \( |x|\simeq \sqrt{\epsilon +x^2}\), for some small number \(\epsilon \).

  2. 2.

    http://cg.postech.ac.kr/research/deconv_outliers/ (accessed November 12, 2011).

References

  1. Afonso, M., Bioucas-Dias, J., & Figueiredo, M. (2010). Fast image recovery using variable splitting and constrained optimization. IEEE Transactions on Image Processing, 19(9), 2345–2356.

    MathSciNet  Article  Google Scholar 

  2. Almeida, M. S. C., & Figueiredo, M. A. T. (2013). Deconvolving images with unknown boundaries using the alternating direction method of multipliers. IEEE Transactions on Image Processing, 22(8), 3074–3086.

    MathSciNet  Article  Google Scholar 

  3. Bar, L., Sochen, N., & Kiryati, N. (2006). Image deblurring in the presence of impulsive noise. International Journal of Computer Vision, 70(3), 279–298.

    Article  Google Scholar 

  4. Blake, A., & Zisserman, A. (1987). Visual Reconstruction. London: MIT Press.

    Google Scholar 

  5. Bouman, C., & Sauer, K. (1993). A generalized Gaussian image model for edge-preserving MAP estimation. IEEE Transactions on Image Processing, 2(3), 296–310.

    Article  Google Scholar 

  6. Cai, J.-F., Ji, H., Liu, C., Shen, Z (2009). Blind motion deblurring from a single image using sparse approximation. In: Proc. CVPR

  7. Chen, C., & Mangasarian, O. L. (1996). A class of smoothing functions for nonlinear and mixed complementarity problems. Computational Optimization and Applications, 5(2), 97–138.

    MathSciNet  Article  MATH  Google Scholar 

  8. Cho, S., & Lee, S. (2009). Fast motion deblurring. ACM Transactions on Graphics, 28(5), 145–148.

    Article  Google Scholar 

  9. Cho, S., Wang, J., & Lee, S. (2011). Handling outliers in non-blind image deconvolution. In:Proc ICCV

  10. Chou, P. B., & Brown, C. M. (1990). The theory and practice of Bayesian image labeling. International Journal of Computer Vision, 4(3), 185–210.

    Article  Google Scholar 

  11. Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., & Freeman, W. T. (2006). Removing camera shake from a single photograph. ACM Transactions on Graphics, 25(3), 787–794.

    Article  Google Scholar 

  12. Gamelin, T. W. (2001). Complex Analysis. New York: Springer-Verlag.

    Google Scholar 

  13. Gupta, A., Joshi, N., Zitnick, C. L., Cohen, M., Curless, B. (2010). Single image deblurring using motion density functions. In: Proc. ECCV

  14. Harmeling, S., Hirsch, M., Schölkopf, B (2010a) Space-variant single-image blind deconvolution for removing camera shake. In:NIPS

  15. Harmeling, S., Sra, S., Hirsch, M., Schölkopf, B (2010b) Multiframe blind deconvolution, super-resolution, and saturation correction via incremental EM. In: Proc. ICIP.

  16. Ji, H., & Wang, K. (2012). Robust image deblurring with an inaccurate blur kernel. IEEE Transactions on Image Processing, 21(4), 1624–1634.

    MathSciNet  Article  Google Scholar 

  17. Joshi, N., Kang, S. B., Zitnick, C. L., & Szeliski, R. (2010). Image deblurring using inertial measurement sensors. ACM Transactions on Graphics, 29(4), 30–39.

    Article  Google Scholar 

  18. Krishnan, D., Fergus, R. (2009). Fast image deconvolution using hyper-Laplacian priors. In NIPS.

  19. Krishnan, D., Tay, T., Fergus, R (2011). Blind deconvolution using a normalized sparsity measure. In: Proc. CVPR.

  20. Levin, A., Fergus, R., Durand, F., & Freeman, W. T. (2007). Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics, 26(3), 70–79.

    Article  Google Scholar 

  21. Levin, A., Weiss, Y., Durand, F., & Freeman, W. T. (2009). Understanding and evaluating blind deconvolution algorithms. MIT: Technical report.

  22. Levin, A., Weiss, Y., Durand, F., & Freeman, W. T. (2011). Efficient marginal likelihood optimization in blind deconvolution. CVPR: In Proc.

  23. Lucy, L. B. (1974). An iterative technique for the rectification of observed distributions. Astronomical Journal, 79(6), 745–754.

    Article  Google Scholar 

  24. Richardson, W. H. (1972). Bayesian-based iterative method of image restoration. The Journal of the Optical Society of America, 62(1), 55–59.

    Article  Google Scholar 

  25. Schultz, R. R., & Stevenson, R. L. (1994). A bayesian approach to image expansion for improved definition. IEEE Transactions on Image Processing, 3(3), 233–242.

    Article  Google Scholar 

  26. Shan, Q., Jia, J., & Agarwala, A. (2008). High-quality motion deblurring from a single image. ACM Transactions on Graphics (Proc. SIGGRAPH 2008), 27(3), 73-1–73-10.

  27. Tai, Y.-W., Du, H., Brown, M. S., & Lin, S. (2008). Image/video deblurring using a hybrid camera. CVPR: In Proc.

  28. Tai, Y.-W., Tan, P., & Brown, M. S. (2011). Richardson-Lucy deblurring for scenes under a projective motion path. IEEE PAMI, 33(8), 1603–1618.

    Article  Google Scholar 

  29. Wang, Y., Yang, J., Yin, W., & Zhang, Y. (2008). A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences, 1(3), 248–272.

    MathSciNet  Article  MATH  Google Scholar 

  30. Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  31. Welk, M. (2010). Robust variational approaches to positivity-constrained image deconvolution. Technical Report 261, Saarland University, Saarbrücken, Germany.

  32. Whyte, O. (2012). Removing camera shake blur and unwanted occluders from photographs. PhD thesis, ENS Cachan.

  33. Whyte, O., Sivic, J., Zisserman, A. (2011). Deblurring shaken and partially saturated images. In: Proc. CPCV, with ICCV.

  34. Whyte, O., Sivic, J., Zisserman, A., Ponce, J. (2010). Non-uniform deblurring for shaken images. In: Proc. CVPR.

  35. Whyte, O., Sivic, J., Zisserman, A., & Ponce, J. (2012). Non-uniform deblurring for shaken images. International Journal of Computer Vision, 98(2), 168–186.

    MathSciNet  Article  MATH  Google Scholar 

  36. Wiener, N. (1949). Extrapolation, interpolation, and smoothing of stationary time series. London: MIT Press.

    Google Scholar 

  37. Xu, L., & Jia, J. (2010). Two-phase kernel estimation for robust motion deblurring. In Proc. ECCV

  38. Yang, J., Zhang, Y., & Yin, W. (2009). An efficient TVL1 algorithm for deblurring multichannel images corrupted by impulsive noise. The SIAM Journal on Scientific Computing, 31(4), 2842–2865. doi:10.1137/080732894.

    MathSciNet  Article  MATH  Google Scholar 

  39. Yuan, L., Sun, J., Quan, L., & Shum, H.-Y. (2007). Image deblurring with blurred/noisy image pairs. ACM Transsactions on Graphics (Proc. SIGGRAPH 2007), 26(3), 1-1–1-10.

  40. Yuan, L., Sun, J., Quan, L., & Shum, H.-Y. (2008). Progressive inter-scale and intra-scale non-blind image deconvolution. ACM Transactions on Graphics (Proc. SIGGRAPH 2008), 27(3), 74-1–74-10.

  41. Zoran, D., & Weiss, Y. (2011). From learning models of natural image patches to whole image restoration. In Proc. ICCV

Download references

Acknowledgments

This work was partly supported by the MSR-INRIA laboratory, the EIT ICT labs and ERC grant VisRec no. 228180.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Oliver Whyte.

Additional information

Parts of this work were previously published in the IEEE Workshop on Color and Photometry in Computer Vision, with ICCV 2011 Whyte et al. (2011).

Communicated by Dr. Srinivas Narasimhan, Dr. Frédo Durand and Dr. Wolfgang Heidrich.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Whyte, O., Sivic, J. & Zisserman, A. Deblurring Shaken and Partially Saturated Images. Int J Comput Vis 110, 185–201 (2014). https://doi.org/10.1007/s11263-014-0727-3

Download citation

Keywords

  • Non-blind deblurring
  • Saturation
  • Ringing
  • Outliers