Skip to main content
SpringerLink
Log in

Video Denoising via Empirical Bayesian Estimation of Space-Time Patches

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

In this paper we present a new patch-based empirical Bayesian video denoising algorithm. The method builds a Bayesian model for each group of similar space-time patches. These patches are not motion-compensated, and therefore avoid the risk of inaccuracies caused by motion estimation errors. The high dimensionality of spatiotemporal patches together with a limited number of available samples poses challenges when estimating the statistics needed for an empirical Bayesian method. We therefore assume that groups of similar patches have a low intrinsic dimensionality, leading to a spiked covariance model. Based on theoretical results about the estimation of spiked covariance matrices, we propose estimators of the eigenvalues of the a priori covariance in high-dimensional spaces as simple corrections of the eigenvalues of the sample covariance matrix. We demonstrate empirically that these estimators lead to better empirical Wiener filters. A comparison on classic benchmark videos demonstrates improved visual quality and an increased PSNR with respect to state-of-the-art video denoising methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. We refer to our method as an empirical Bayesian approach because although it is based on a Bayesian model for patches, the parameters of the prior are determined from the data using frequentist estimators. The empirical (or data-driven) Bayesian approach is a usual choice when the parameters of the prior are unknown, but it is a departure from a strict Bayesian methodology which would require a hyperprior to estimate the parameters (see [1] for a closely related method considering a Bayesian estimation of the parameters of the prior).

  2. In [32] the unbiased estimator for the covariance matrix is used, which differs from the MLE in that the sum is divided by \(n-1\) instead of n. In practice, except for small values of n, the choice between these two estimators has little effect on the results. Here we prefer to use the MLE because it is the one studied by Paul in [18]. Later, in Sect. 3.2 we will exploit Paul’s results to derive estimators for the eigenvalues of the covariance matrix.

  3. The distance threshold \(\rho \) is applied in the second step, where the distance is computed using patches from the basic estimate. For this reason (as in [30]) we do not consider a noise-dependent threshold.

  4. It might seem counterintuitive that the filter computed from the estimated eigenvalues \(\widehat{\lambda }^{\text {S}}\) outperforms the oracular one. It is a consequence of the error in the eigenvectors. In particular, eigenvectors corresponding to eigenvalues below the recoverability threshold are better discarded.

  5. In order to reduce the parameter space, we did not optimize the choice of the distance threshold \(\rho \).

  6. We use the default parameters in [12], with the exception of the number of bins, which we set to one since we are not interested in a noise curve, but in just an average noise level.

  7. http://dev.ipol.im/~pariasm/video_nlbayes/.

  8. In [8] the error is measured as the root mean square error (RMSE) of the central frame. For color sequences the average channel RMSE is used. To allow a direct comparison, we adopt in Table 3 these error measurements, expressing them as a PSNR in decibels. For color sequences, this amounts to \(\text {PSNR} = 20\log _{10}\big (255 / \frac{1}{3}(\text {RMSE}_{R} +\text {RMSE}_{G} + \text {RMSE}_{B})\big ).\)

  9. We are thankful to the authors of [8] for kindly sharing with us their results.

References

  1. Aguerrebere, C., Almansa, A., Delon, J., Gousseau, Y., Musé, P.: Inverse Problems in Imaging: A Hyperprior Bayesian Approach (2016). https://hal.archives-ouvertes.fr/hal-01107519. Working paper or preprint

  2. Arias, P., Morel, J.M.: Towards a Bayesian video denoising method. In: Battiato, S., Blanc-Talon, J., Gallo, G., Philips, W., Popescu, D., Scheunders, P. (eds.) Advanced Concepts for Intelligent Vision Systems. Lecture Notes in Computer Science, vol. 9386, pp. 107–117. Springer International Publishing (2015). doi:10.1007/978-3-319-25903-1_10

  3. Boulanger, J., Kervrann, C., Bouthemy, P.: Space-time adaptation for patch-based image sequence restoration. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1096–1102 (2007). doi:10.1109/TPAMI.2007.1064

    Article  Google Scholar 

  4. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, New York (2004)

    Book  MATH  Google Scholar 

  5. Brailean, J.C., Kleihorst, R.P., Efstratiadis, S., Katsaggelos, A.K., Lagendijk, R.: Noise reduction filters for dynamic image sequences: a review. Proc. IEEE 83(9), 1272–1292 (1995)

    Article  Google Scholar 

  6. Buades, A., Coll, B., Morel, J.M.: Denoising image sequences does not require motion estimation. In: Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 70–74 (2005)

  7. Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  8. Buades, A., Lisani, J.L., Miladinović, M.: Patch-based video denoising with optical flow estimation. IEEE Trans. Image Process. 25(6), 2573–2586 (2016). doi:10.1109/TIP.2016.2551639

    Article  MathSciNet  Google Scholar 

  9. Cai, T., Ma, Z., Wu, Y.: Optimal estimation and rank detection for sparse spiked covariance matrices. Probab. Theory Relat. Fields 161(3), 781–815 (2015). doi:10.1007/s00440-014-0562-z

    Article  MathSciNet  MATH  Google Scholar 

  10. Cai, T.T., Ren, Z., Zhou, H.H.: Estimating structured high-dimensional covariance and precision matrices: optimal rates and adaptive estimation. Electron. J. Stat. 10(1), 1–59 (2016). doi:10.1214/15-EJS1081

    Article  MathSciNet  MATH  Google Scholar 

  11. Chatterjee, P., Milanfar, P.: Patch-based near-optimal image denoising. IEEE Trans. Image Process. 21(4), 1635–1649 (2012). doi:10.1109/TIP.2011.2172799

    Article  MathSciNet  MATH  Google Scholar 

  12. Colom, M., Buades, A.: Analysis and extension of the percentile method, estimating a noise curve from a single image. Image Process. On Line 3, 332–359 (2013). doi:10.5201/ipol.2013.90

    Article  Google Scholar 

  13. Dabov, K., Foi, A.: Image denoising with block-matching and 3D filtering. Electronic... 6064, 1–12 (2006). doi:10.1117/12.643267. http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=728740

  14. Dabov, K., Foi, A., Egiazarian, K.: Video denoising by sparse 3D transform-domain collaborative filtering. In: EUSIPCO, pp. 145–149 (2007). doi:10.1109/TIP.2007.901238

  15. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3d transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  16. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: BM3D image denoising with shape-adaptive principal component analysis. In: Gribonval, R. (ed.) SPARS’09-Signal Processing with Adaptive Sparse Structured Representations. Inria Rennes-Bretagne Atlantique, Saint Malo, France (2009). https://hal.inria.fr/inria-00369582

  17. Dai, J., Au, O., Pang, C., Zou, F.: Color video denoising based on combined interframe and intercolor prediction. IEEE Trans. Circuits Syst. Video Technol. 23(1), 128–141 (2013). doi:10.1109/TCSVT.2012.2203203

    Article  Google Scholar 

  18. Debashis, P.: Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Stat. Sin. 17, 1617–1642 (2007)

    MathSciNet  MATH  Google Scholar 

  19. Deledalle, C.A., Salmon, J., Dalalyan, A.: Image denoising with patch based PCA: local versus global. In: Proceedings of the British Machine Vision Conference 2011, pp. 25.1–25.10 (2011). doi:10.5244/C.25.25. http://www.bmva.org/bmvc/2011/proceedings/paper25/index.html

  20. Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)

    Article  MathSciNet  Google Scholar 

  21. Ercole, C., Foi, A., Katkovnik, V., Egiazarian, K.: Spatio-temporal pointwise adaptive denoising of video: 3D non-parametric regression approach. In: Proceedings of the First International Workshop of Video Processing and Quality Metrics for Consumer Electronics (2005)

  22. Ghael, S., Sayeed, A., Baraniuk, R.G.: Improved wavelet denoising via empirical wiener filtering. In: SPIE Technical Conference on Wavelet Applications in Signal Processing (1997)

  23. Ghoniem, M., Chahir, Y., Elmoataz, A.: Nonlocal video denoising, simplification and inpainting using discrete regularization on graphs. Signal Process. 90(8), 2445–2455 (2010). doi:10.1016/j.sigpro.2009.09.004. http://linkinghub.elsevier.com/retrieve/pii/S016516840900379X

  24. Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: The Twenty-Seventh IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

  25. Guillemot, T., Almansa, A., Boubekeur, T.: Covariance trees for 2d and 3d processing. In: The Twenty-Seventh IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

  26. Halko, N., Martinsson, P.G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011). doi:10.1137/090771806

    Article  MathSciNet  MATH  Google Scholar 

  27. Jin, F., Fieguth, P., Winger, L.: Wavelet video denoising with regularized multiresolution motion estimation. EURASIP J. Adv. Signal Process. 2006, 1–11 (2006). doi:10.1155/ASP/2006/72705

    MATH  Google Scholar 

  28. Johnstone, I.M., Lu, A.Y., Nadler, B., Witten, D.M., Hastie, T., Tibshirani, R., Ramsay, J.O., Johnstone, I.M., Lu, A.Y.: On consistency and sparsity for principal components analysis in high dimensions [with comments]. J. Am. Stat. Assoc. 104(486), 682–703 (2009). http://www.jstor.org/stable/40592215

  29. Kambhatla, N., Leen, T.K.: Dimension reduction by local principal component analysis. Neural Comput. 9, 1493–1516 (1997)

    Article  Google Scholar 

  30. Kervrann, C., Boulanger, J.: Optimal spatial adaptation for patch-based image denoising. IEEE Trans. Image Process. 15(10), 2866–2878 (2006). doi:10.1109/TIP.2006.877529

    Article  Google Scholar 

  31. Laus, F., Nikolova, M., Persch, J., Steidl, G.: A nonlocal denoising algorithm for manifold-valued images using second order statistics (2016). http://arxiv.org/abs/1607.08481. Preprint

  32. Lebrun, M., Buades, A., Morel, J.M.: A nonlocal Bayesian image denoising algorithm. SIAM J. Imaging Sci. 6(3), 1665–1688 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  33. Lebrun, M., Buades, A., Morel, J.M.: Implementation of the “Non-Local Bayes” (NL-Bayes) image denoising algorithm. Image Process. On Line 3, 1–42 (2013). doi:10.5201/ipol.2013.16

    Article  Google Scholar 

  34. Lebrun, M., Colom, M., Buades, A., Morel, J.M.: Secrets of image denoising cuisine. Acta Numer. 21(1), 475–576 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  35. Lebrun, M., Colom, M., Morel, J.M.: The noise clinic: a blind image denoising algorithm. Image Process. On Line 5, 1–54 (2015). doi:10.5201/ipol.2015.125

    Article  Google Scholar 

  36. Li, W., Zhang, J., Dai, Q.H.: Video denoising using shape-adaptive sparse representation over similar spatio-temporal patches. Sig. Process. Image Commun. 26, 250–265 (2011)

    Article  Google Scholar 

  37. Liu, C., Freeman, W.T.: A high-quality video denoising algorithm based on reliable motion estimation. In: ECCV, pp. 706–719 (2010)

  38. Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising using separable 4D nonlocal spatiotemporal transforms. In: Proceedings of SPIE, 7870 (2011)

  39. Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Trans. Image Process. 21(9), 3952–3966 (2012). doi:10.1109/TIP.2012.2199324

    Article  MathSciNet  MATH  Google Scholar 

  40. Maggioni, M., Foi, A.: Implementation of V-BM4D. http://www.cs.tut.fi/~foi/GCF-BM3D/

  41. Maggioni, M., Katkovnok, V., Egiazarian, K., Foi, A.: A nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Trans. Image Process. 22(1), 119–133 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  42. Mairal, J., Sapiro, G., Elad, M.: Learning multiscale sparse representations for image and video restoration. Multiscale Model. Simul. 7(1), 214–241 (2008). doi:10.1137/070697653

    Article  MathSciNet  MATH  Google Scholar 

  43. Monzón, N., Salgado, A., Sánchez, J.: Robust discontinuity preserving optical flow methods. Image Process. On Line 6, 165–182 (2016). doi:10.5201/ipol.2016.172

    Article  Google Scholar 

  44. Muresan, D.D., Parks, T.W.: Adaptive principal components and image denoising. In: Proceedings 2003 International Conference on Image Processing, vol. 1, pp. I–101–4 vol. 1 (2003). doi:10.1109/ICIP.2003.1246908

  45. Nadler, B.: Finite sample approximation results for principal component analysis: a matrix perturbation approach. Ann. Stat. 36(6), 2791–2817 (2008). doi:10.1214/08-AOS618

    Article  MathSciNet  MATH  Google Scholar 

  46. Niknejad, M., Rabbani, H., Babaie-Zadeh, M.: Image restoration using Gaussian mixture models with spatially constrained patch clustering. IEEE Trans. Image Process. 24(11), 3624–3636 (2015)

    Article  MathSciNet  Google Scholar 

  47. Oktem, R., Ponomarenko, N.: Image filtering based on discrete cosine transform. Telecommun. Radio Eng. 66(18), 1685–1701 (2007)

  48. Portilla, J., Strela, V., Wainwright, M., Simoncelli, E.: Image denoising using scale mixtures of gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003). doi:10.1109/TIP.2003.818640

    Article  MathSciNet  MATH  Google Scholar 

  49. Protter, M., Elad, M.: Sparse and redundant representations and motion-estimation-free algorithm for video denoising. In: Proceedings of SPIE, 67011, pp. 67,011D–67,011D–12 (2007). doi:10.1117/12.731851

  50. Protter, M., Elad, M.: Image sequence denoising via sparse and redundant representations. IEEE Trans. Image Process. 18(1), 27–35 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  51. Rajpoot, N., Yao, Z., Wilson, R.: Adaptive wavelet restoration of noisy video sequences. In: Image Processing, 2004. ICIP ’04. 2004 International Conference on, vol. 2, pp. 957–960 (2004). doi:10.1109/ICIP.2004.1419459

  52. Salmon, J., Harmany, Z., Deledalle, C.A., Willett, R.: Poisson noise reduction with non-local PCA. J. Math. Imaging Vis. 48(2), 279–294 (2014). doi:10.1007/s10851-013-0435-6

    Article  MathSciNet  MATH  Google Scholar 

  53. Sánchez, J., Meinhardt-Llopis, E., Facciolo, G.: TV-L1 optical flow estimation. Image Process. On Line 3, 137–150 (2013). doi:10.5201/ipol.2013.26

    Article  Google Scholar 

  54. Selesnick, I.W., Li, K.Y.: Video denoising using 2D and 3D dual-tree complex wavelet transforms. In: Proceedings of SPIE—The International Society for Optical Engineering 5207(2), 607–618 (2003). doi:10.1117/12.504896

  55. Seshadrinathan, K., Bovik, A.C.: Motion tuned spatio-temporal quality assessment of natural videos. Trans. Image Process. 19(2), 335–350 (2010). doi:10.1109/TIP.2009.2034992

    Article  MathSciNet  MATH  Google Scholar 

  56. Sutour, C., Deledalle, C.A., Aujol, J.F.: Adaptive regularization of the nl-means: application to image and video denoising. IEEE Trans. Image Process. 23(8), 3506–3521 (2014). doi:10.1109/TIP.2014.2329448

    Article  MathSciNet  MATH  Google Scholar 

  57. Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analysis. Neural Comput. 11, 443–482 (1999)

    Article  MATH  Google Scholar 

  58. Yu, G., Sapiro, G.: DCT image denoising: a simple and effective image denoising algorithm. Image Process. On Line (2011). doi:10.5201/ipol.2011.ys-dct

  59. Yu, G., Sapiro, G., Mallat, S.: Solving inverse problems with piecewise linear estimators: from gaussian mixture models to structured sparsity. IEEE Trans. Image Process. 21(5), 2481–2499 (2012). doi:10.1109/TIP.2011.2176743

    Article  MathSciNet  MATH  Google Scholar 

  60. Zach, C., Pock, T., Bischof, H.: A Duality Based Approach for Realtime TV-L1 Optical Flow, pp. 214–223. Springer, Berlin (2007). doi:10.1007/978-3-540-74936-3_22

  61. Zhang, L., Dong, W., Zhang, D., Shi, G.: Two-stage image denoising by principal component analysis with local pixel grouping. Pattern Recognit. 43(4), 1531–1549 (2010)

    Article  MATH  Google Scholar 

  62. Zlokolica, V., Ptžurica, A., Philips, W.: Wavelet-domain video denoising based on reliability measures. IEEE Trans. Circuits Syst. Video Technol. 16(8), 993–1007 (2006). doi:10.1109/TCSVT.2006.879994

    Article  Google Scholar 

  63. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 479–486 (2011). doi:10.1109/ICCV.2011.6126278

Download references

Author information

Authors and Affiliations

  1. CMLA, ENS Cachan, CNRS, Université Paris-Saclay, 94235, Cachan, France

    Pablo Arias & Jean-Michel Morel

Authors
  1. Pablo Arias
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Michel Morel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pablo Arias.

Additional information

This work is partly founded by BPIFrance and Région Ile de France, in the framework of the FUI 18 Plein Phare project; by the Office of Naval research by grant N00014-17-1-2552; by ANR-DGA project ANR-12-ASTR-0035; and by ANR-DGA project ANR-14-CE27-001 (MIRIAM).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arias, P., Morel, JM. Video Denoising via Empirical Bayesian Estimation of Space-Time Patches. J Math Imaging Vis 60, 70–93 (2018). https://doi.org/10.1007/s10851-017-0742-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-017-0742-4

Keywords

Search

Navigation