Real-time video denoising on multicores and GPUs with Kalman-based and Bilateral filters fusion

  • Sergio G. Pfleger
  • Patricia D. M. Plentz
  • Rodrigo C. O. Rocha
  • Alyson D. Pereira
  • Márcio Castro
Original Research Paper
  • 146 Downloads

Abstract

In the context of video processing, image noise caused by acquisition, transfer and image compression can be attenuated by video denoising algorithms. However, their computational cost must be as low as possible to allow them to be applied to real-time applications. In this paper, we propose stmkf, a real-time video denoising algorithm based on Kalman and Bilateral filters. We evaluate the effectiveness of stmkf using several common videos used in the literature and we compare it to other denoising algorithms using both the PSNR and SSIM metrics. Our experimental results show that stmkf is competitive with other filters, especially for videos that feature stationary backgrounds such as in videoconferencing, video lectures and video surveillance. We also evaluate the performance of our parallel implementations of stmkf for CPUs and GPUs. stmkf achieved a performance improvement of up to \(2.9\times \) on a Intel i7 multicore processor with 4 cores compared to the sequential solution. The results obtained with the GPU version of stmkf on a NVIDIA Tesla K40 showed a performance improvement of up to \(7.6\times \) compared to the Intel i7 multicore processor.

Keywords

Spatiotemporal video denoising Kalman filter Bilateral filter Multicore GPU 

Supplementary material

Supplementary material 1 (mp4 129997 KB)

References

  1. 1.
    Bardu, T.: Variational image denoising approach with diffusion porous media flow. Abstr. Appl. Anal. 2013, 8 (2013)MathSciNetGoogle Scholar
  2. 2.
    Buades, A., Coll, B., Morel, J.-M.: Nonlocal image and movie denoising. Int. J. Comput. Vision 76(2), 123–139 (2008)CrossRefGoogle Scholar
  3. 3.
    Buades, A., Coll, B., Morel, J.-M.: A non-local algorithm for image denoising. In: Conference on Computer Vision and Pattern Recognition, CVPR ’05, pp. 60–65. Washington, DC, USA, IEEE Computer Society (2005)Google Scholar
  4. 4.
    Chan, T.-W., Au, O.C., Chong, T.-S., Chau, W.-S.: A novel content-adaptive video denoising filter. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 649–652, Philadelphia, USA (2005)Google Scholar
  5. 5.
    Chaudhury, K.N.: Acceleration of the shiftable algorithm for bilateral filtering and nonlocal means. IEEE Trans. Image Process. 22(4), 1291–1300 (2013)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chen, T.-Y., Chen, T.-H., Su, C.-P., Chen, Y.-J.: The study on video enhancement in the low-light environment by spatio-temporal filtering. In: International Conference on Intelligent Systems Design and Applications (ISDA), vol. 3, pp. 561–564, Kaohsiung, Taiwan (2008)Google Scholar
  7. 7.
    Chenglin Z., Yu, L., Xin, T., Wei, W., Maojun, Z. (2013) Video denoising based on a spatiotemporal Kalman-bilateral mixture model. Sci. World J. 2013 (2013)Google Scholar
  8. 8.
    Dabov, K., Foi, A., Egiazarian, K.: Video denoising by sparse 3D transform-domain collaborative filtering. In: European Signal Processing Conference, pp. 145–149, Poznan, Poland. IEEE (2007)Google Scholar
  9. 9.
    Davis, L., Rosenfeld, A.: Noise cleaning by iterated cleaning. IEEE Trans. Syst. Man Cybern. SMC 8(9), 705–710 (1978)CrossRefGoogle Scholar
  10. 10.
    Dufaux, F., Callet, P.L., Mantiuk, R., Mrak, M.:  High Dynamic Range Video: From Acquisition, to Display and Applications. Elsevier (2016). ISBN 9780128030394Google Scholar
  11. 11.
    Farooque, M.A., Sohankar, J.S.: Survey on various noises and techniques for denoising the color image. Int. J. Appl. Innov. Eng. Manage. (IJAIEM) 2, 217 (2013)Google Scholar
  12. 12.
    Garg, R., Kumar, A.: Comparision of various noise removals using bayesian framework. Int. J. Mod. Eng. Res. (IJMER) 2, 265 (2012)Google Scholar
  13. 13.
    Han, Y., Chen, R.: Efficient video denoising based on dynamic nonlocal means. Image Vision Comput. 30, 78–85 (2012)CrossRefGoogle Scholar
  14. 14.
    Hong-Zhi, W., Ling, C., Shu-Liang, X.: Improved video denoising algorithm based on spatial-temporal combination. In: International Conference on Image and Graphics (ICIG), pp. 64–67, Qingdao, China. IEEE (2013)Google Scholar
  15. 15.
    Jojy, C., Nair, M.S., Subrahmanyam, G.R.K.S., Raji, R.: Discontinuity adaptive non-local means with importance sampling unscented Kalman filter for de-speckling SAR images. IEEE J. Sel. Top. Appl. Earth Obser. Remote Sens. 6(4), 1964–1970 (2013)CrossRefGoogle Scholar
  16. 16.
    Jung, B., Sukhatme, G.S.: Detecting moving objects using a single camera on a mobile robot in an outdoor environment. In: International Conference on Intelligent Autonomous Systems, pp. 980–987 (2004)Google Scholar
  17. 17.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82(D), 35–45 (1960)CrossRefGoogle Scholar
  18. 18.
    Karnati, V., Uliyar, M., Dey, S.: Fast non-local algorithm for image denoising. In: International Conference on Image Processing (ICIP), pp. 3873–3876. IEEE (Nov 2009)Google Scholar
  19. 19.
    Kirk, D.B., Wen-mei W.H.: Programming Massively Parallel Processors: A Hands-on Approach. Morgan Kaufmann Publishers Inc., San Francisco, 1st edn. (2010). ISBN 0123814723Google Scholar
  20. 20.
    Kokkonis, G., Psannis, K.E., Roumeliotis, M., Ishibashi, Y.: Efficient Algorithm for transferring a real-time HEVC stream with haptic data through the internet. J. Real-Time Image Process. pp. 1–13, (2015). ISSN 1861-8219. doi:10.1007/s11554-015-0505-7
  21. 21.
    Kokkonis, G., Psannis, K.E., Roumeliotis, M., Schonfeld, D.: Real-time wireless multisensory smart surveillance with 3D-HEVC streams for internet-of-things (iot). J. Supercomput. pp 1–19, (2016). ISSN 1573-0484. doi:10.1007/s11227-016-1769-9
  22. 22.
    Kostadin D., Alessandro F., Vladimir K., Karen E.: Image denoising with block-matching and 3D filtering. In: SPIE-IS&T Electronic Imaging, p. 6064 (2006)Google Scholar
  23. 23.
    Li, W., Zhang, J., Dai, Q.: Video denoising using shape-adaptive sparse representation over similar spatio-temporal patches. Signal Proc.: Image. Communication 26(4–5), 250–265 (2011)Google Scholar
  24. 24.
    Li, X., Zheng, Y.: Patch-based video proc.: a variational bayesian approach. IEEE Trans. Circuits Syst Video Technol 19(1), 27–40 (2009)CrossRefGoogle Scholar
  25. 25.
    Mahmoud, R.O., Faheem, M.T., Sarhan, A.: Intelligent denoising technique for spatial video denoising for real-time applications. In: International Conference on Computer Engineering Systems (ICCES), pp. 407–412, Cairo, Egypt. IEEE (2008)Google Scholar
  26. 26.
    Mahmoudi, M., Sapiro, G.: Fast image and video denoising via nonlocal means of similar neighborhoods. IEEE Signal Process. Lett. 12(12), 839–842 (2005)CrossRefGoogle Scholar
  27. 27.
    Memos, V.A., Psannis, K.E.: Encryption algorithm for efficient transmission of hevc media. J. Real-Time Image Process. pp. 1–10, (2015). ISSN 1861-8219. doi:10.1007/s11554-015-0509-3
  28. 28.
    Mitchell, H.B., Mashkit, N.: Noise smoothing by a fast k-nearest neighbour algorithm. Signal Process. Image Commun. 4(3), 227–232 (1992)CrossRefGoogle Scholar
  29. 29.
    OpenMP Architecture Review Board. OpenMP application program interface version 4.0, July 2013. URL http://www.openmp.org/mp-documents/OpenMP4.0.0.pdf
  30. 30.
    Pauwels, K., Tomasi, M., Alonso, J. Diaz., Ros, E., Van Hulle, M. M.: A comparison of fpga and GPU for real-time phase-based optical flow, stereo, and local image features. IEEE Trans. Comput. 61(7): 999–1012, (2012). ISSN 0018-9340Google Scholar
  31. 31.
    Pizurica, A., Zlokolica, V., Philips, W.: Noise reduction in video sequences using wavelet-domain and temporal filtering. In: Photonics Technologies for Robotics, Automation, and Manufacturing, Int. Soc. for Optics and Photonics, pp. 48–59 (2004)Google Scholar
  32. 32.
    Psannis, K.E.: Hevc in wireless environments. J. Real-Time Image Process. pp. 1–8, (2015). ISSN 1861-8219. doi:10.1007/s11554-015-0514-6
  33. 33.
    Pulli, K., Baksheev, A., Kornyakov, K., Eruhimov, V.: Real-time computer vision with OpenCV. Commun. ACM 55(6): 61–69, (2012). ISSN 0001-0782Google Scholar
  34. 34.
    Rahman, S.M.M., Ahmad, M.O., Swamy, M.N.S.: Video denoising based on inter-frame statistical modeling of wavelet coefficients. IEEE Trans. Circuits Syst. Video Technol. 17(2), 187–198 (2007)CrossRefGoogle Scholar
  35. 35.
    Ryu, J., Nishimura, T. H.: Fast image blurring using lookup table for real time feature extraction. In: 2009 IEEE International Symposium on Industrial Electronics, pp. 1864–1869 (2009)Google Scholar
  36. 36.
    Seiller, N., Singhal, N., Park, I.K.: Object oriented framework for real-time image processing on GPU. In: International Conference on Image Processing (ICIP), pp. 4477–4480, Hong Kong, China. IEEE (2010)Google Scholar
  37. 37.
    Selesnick, I.W, Li, K.Y.: Video denoising using 2D and 3D dual-tree complex wavelet transforms. In: Annual Meeting on Optical Science and Technology (SPIE), Int. Soc. for Optics and Photonics, pp. 607–618. (2003)Google Scholar
  38. 38.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: International Conference on Computer Vision, Bombay, India, pp. 839–846, IEEE (1998)Google Scholar
  39. 39.
    Van De Ville, D., Kocher, M.: SURE-based non-local means. IEEE Signal Process. Lett. 16(11), 973–976 (2009)CrossRefGoogle Scholar
  40. 40.
    Wang, Z., Bovik, A.C., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Proc. 13(4), 600 (2004)CrossRefGoogle Scholar
  41. 41.
    Wolf, W., Ozer, B., Lv, T.: Smart cameras as embedded systems. Computer 35(9), 48–53 (2002)CrossRefGoogle Scholar
  42. 42.
    Zlokolica, V., Pizurica, A., Philips, W.: Wavelet-domain video denoising based on reliability measures. IEEE Trans. Circuits Syst. Video Technol. 16(8), 993–1007 (2006)CrossRefGoogle Scholar
  43. 43.
    Zlokolica, V., Philips, W., Van De Ville, D.: A new non-linear filter for video processing. In: IEEE Benelux Signal Processing Symposium, pp. 221–224 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Sergio G. Pfleger
    • 1
  • Patricia D. M. Plentz
    • 1
  • Rodrigo C. O. Rocha
    • 2
  • Alyson D. Pereira
    • 1
  • Márcio Castro
    • 1
  1. 1.Department of Informatics and Statistics (INE)Federal University of Santa Catarina (UFSC)FlorianópolisBrazil
  2. 2.Computer Science DepartmentPontifical Catholic University of Minas Gerais (PUC Minas)Belo HorizonteBrazil

Personalised recommendations