Skip to main content
SpringerLink
Log in

A novel 5-D depth–velocity filter for enhancing noisy light field videos

  • Published:
Multidimensional Systems and Signal Processing Aims and scope Submit manuscript

Abstract

A 5-D depth–velocity filter is proposed for enhancing moving objects in noisy light field videos (LFVs) (also known as plenoptic videos). The proposed filter consists of an ultra-low complexity 5-D IIR depth filter and a 5-D FIR velocity filter. The 5-D IIR depth filter is employed to denoise a noisy LFV. The denoised LFV is then utilized to estimate the 3-D apparent velocity of the moving object of interest. The 5-D FIR velocity filter is designed based on the estimated 3-D apparent velocity and is used to enhance the moving object of interest while attenuating other interfering moving objects. Experimental results confirm the effectiveness of the proposed 5-D depth–velocity filter compared to previously reported 5-D depth–velocity filters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  • Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In M. Landy & J. A. Movshon (Eds.), Computation models of visual processing (pp. 3–20). Cambridge, MA: MIT Press.

    Google Scholar 

  • Agathoklis, P., & Bruton, L. T. (1983). Practical-BIBO stability of n-dimensional discrete systems. IEE Proceedings G-Electronic Circuits and Systems, 130(6), 236–242.

    Article  Google Scholar 

  • Bruton, L. T., & Bartley, N. (1986). The enhancement and tracking of moving objects in digital images using adaptive three-dimensional recursive filters. IEEE Transactions on Circuits and Systems, CAS–33(6), 604–612.

    Article  Google Scholar 

  • Bruton, L. T., & Bartley, N. R. (1985). Three-dimensional image processing using the concept of network resonance. IEEE Transactions on Circuits and Systems (CAS), 32(7), 664–672.

    Article  Google Scholar 

  • Dansereau, D., & Bruton, L. (2003). A 4D frequency-planar IIR filter and its application to light field processing. In Proceedings of the IEEE international symposium on circuits and systems (Vol. 4, pp. IV-476–IV-479).

  • Dansereau, D., & Bruton, L. T. (2007). A 4-D dual-fan filter bank for depth filtering in light fields. IEEE Transactions on Signal Processing, 55(2), 542–549.

    Article  MathSciNet  Google Scholar 

  • Dansereau, D. G., Bongiorno, D. L., Pizarro, O., & Williams, S. B. (2013a). Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter. In Proceedings of the SPIE conference on computational imaging XI (Vol. 8657, pp. 86570P-1–86570P-14).

  • Dansereau, D. G., Pizarro, O., & Williams, S. B. (2013b). Decoding, calibration and rectification for lenselet-based plenoptic cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1027–1034).

  • Dansereau, D. G., Pizarro, O., & Williams, S. B. (2015). Linear volumetric focus for light field cameras. ACM Transactions on Graphics, 34(2), 15:1–15:20.

    Article  Google Scholar 

  • Dudgeon, D. E., & Mersereau, R. M. (1984). Multidimensional digital signal processing. Englewood Cliffs, NJ: Prentice-Hall.

    MATH  Google Scholar 

  • Edussooriya, C. U. S. (2015). Low-complexity multi-dimensional filters for plenoptic signal processing. PhD thesis, Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC, Canada

  • Edussooriya, C. U. S., Bruton, L. T., & Agathoklis, P. (2014). A low-complexity 3D spatio-temporal FIR filter for enhancing linear trajectory signals. In Proceedings of the IEEE international conference on acoustics, speech and signal processing (pp. 1165–1169).

  • Edussooriya, C. U. S., Bruton, L. T., & Agathoklis, P. (2015a). A 5-D IIR depth-velocity filter for enhancing objects moving on linear-trajectories in light field videos. In Proceedings of the IEEE international symposium on circuits and systems (pp. 2381–2384).

  • Edussooriya, C. U. S., Bruton, L. T., & Agathoklis, P. (2015b). Enhancing moving objects in light field videos using 5-D IIR adaptive depth-velocity filters. In Proceedings of the IEEE Pacific Rim conference on communications, computers and signal processing (pp. 1–5).

  • Edussooriya, C. U. S., Bruton, L. T., & Agathoklis, P. (2015c). Velocity filtering for attenuating moving artifacts in videos using an ultra-low complexity 3-D linear-phase IIR filter. Multidimensional Systems and Signal Processing, 1–20. doi:10.1007/s11045-015-0362-3.

  • Edussooriya, C. U. S., Dansereau, D. G., Bruton, L. T., & Agathoklis, P. (2015d). Five-dimensional depth-velocity filtering for enhancing moving objects in light field videos. IEEE Transactions on Signal Processing, 63(8), 2151–2163.

    Article  MathSciNet  Google Scholar 

  • Hahne, C., & Aggoun, A. (2014). Embedded FIR filter design for real-time refocusing using a standard plenoptic video camera. In Proceedings of the SPIE electronic imaging (Vol. 9023, pp. 902305-1–902305-12).

  • Handagala, S., Madanayake, A., & Udayanga, N. (2015). Design of a millimeter-wave dish-antenna based 3-D IIR radar digital velocity filter. In Proceedings of the international workshop multidimensional (nD) systems (pp. 1–6).

  • Isaksen, A., McMillan, L., & Gortler, S. J. (2000). Dynamically reparameterized light fields. In Proceedings of the annual conference on computer graphics (SIGGRAPH) (pp. 297–306).

  • Kennedy, H. L. (2014). Multidimensional digital filters for point-target detection in cluttered infrared scenes. Journal of Electronic Imaging, 23(6), 063019-1–063019-11.

    Article  Google Scholar 

  • Kondo, K., & Hamada, N. (2000). Design of optimal filter for detecting linear trajectory signals utilizing object shape and velocity vector. Electronics and Communications in Japan (Part I), 83(2), 42–51.

    Article  Google Scholar 

  • Kuenzle, B., & Bruton, L. T. (2006). 3-D IIR filtering using decimated DFT-polyphase filter bank structures. IEEE Transactions on Circuits and Systems I, 53(2), 394–408.

    Article  Google Scholar 

  • Levoy, M., & Hanrahan, P. (1996). Light field rendering. In Proceedings of the annual conference on computer graphics (SIGGRAPH) (pp. 31–42).

  • Pei, S. C., & Jaw, S. B. (1994). Two-dimensional general fan-type FIR digital filter design. Signal Processing, 37(2), 265–274.

    Article  MATH  Google Scholar 

  • Schauland, S., Velten, J., & Kummert, A. (2008). Detection of moving objects in image sequences using 3D velocity filters. International Journal of Applied Mathematics and Computer Science, 18(1), 21–31.

    Article  MATH  MathSciNet  Google Scholar 

  • Schwerdtfeger, T., Velten, J., & Kummert, A. (2014). A multidimensional wave digital filter bank for video-based motion analysis. Multidimensional Systems and Signal Processing, 25(2), 295–311.

    Article  MATH  Google Scholar 

  • Vaish, V., Wilburn, B., Joshi, N., & Levoy, M. (2004). Using plane+ parallax for calibrating dense camera arrays. In Procedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, pp. I-2–I-9).

  • Vaish, V., Garg, G., Talvala, E. V., Antunez, E., Wilburn, B., Horowitz, M., et al. (2005). Synthetic aperture focusing using a shear-warp factorization of the viewing transform. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 129–129).

  • Wang, Y., Ostermann, J., & Zhang, Y. Q. (2002). Video processing and communications. Upper Saddle River, NJ: Prentice-Hall.

    Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

Download references

Acknowledgments

The authors thank to Ms. Ioana Sevcenco and Mr. Hussam Shubayli for helping in generating the Lytro-LF-camera-based LFV used in the experiments. Furthermore, a special thank goes to Dr. Donald Dansereau for providing the MATLAB LFToolbox to decode the static LFs of the Lytro-LF-camera-based LFV.

Author information

Authors and Affiliations

  1. Department of Electronic and Telecommunication Engineering, University of Moratuwa, Moratuwa, Sri Lanka

    Chamira U. S. Edussooriya

  2. Department of Electrical and Computer Engineering, University of Calgary, Calgary, AB, Canada

    Len T. Bruton

  3. Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC, Canada

    Panajotis Agathoklis

Authors
  1. Chamira U. S. Edussooriya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Len T. Bruton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Panajotis Agathoklis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chamira U. S. Edussooriya.

Additional information

The authors gratefully acknowledge the financial support of the Natural Sciences and Engineering Research Council (NSERC) of Canada.

Appendix: Derivation of the ideal infinite-extent impulse response \(g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)\)

Appendix: Derivation of the ideal infinite-extent impulse response \(g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)\)

Following Pei and Jaw (1994), the ideal infinite-extent impulse response \(g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)\) can be derived as

$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{B_uB_v(aB_u+2B_t)}{4\pi ^3}, \qquad \bar{n}_u=0,\bar{n}_v=0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_t=0 \end{aligned}$$
(15a)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{B_u\sin (B_v\bar{n}_v)(aB_u+2B_t)}{4\pi ^3\bar{n}_v}, \qquad \bar{n}_u=0,\bar{n}_v\ne 0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_t=0 \end{aligned}$$
(15b)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{B_v}{2\pi ^3\bar{n}_u}\left[ (aB_u+B_t)\sin (B_u\bar{n}_u)+\frac{a(\cos (B_u\bar{n}_u)-1)}{\bar{n}_u}\right] , \nonumber \\&\qquad \bar{n}_u\ne 0,\bar{n}_v=0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_t=0 \end{aligned}$$
(15c)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{\sin (B_v\bar{n}_v)}{2\pi ^3\bar{n}_u\bar{n}_v}\left[ (aB_u+B_t)\sin (B_u\bar{n}_u) +\frac{a(\cos (B_u\bar{n}_u)-1)}{\bar{n}_u}\right] , \nonumber \\&\qquad \bar{n}_u\ne 0,\bar{n}_v\ne 0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_t=0 \end{aligned}$$
(15d)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{B_v}{4\pi ^3\bar{n}_t}\left[ B_u\sin (B_t\bar{n}_t)-\frac{\cos (B_t\bar{n}_t)-\cos (B_u\bar{n}_u -aB_u\bar{n}_t-B_t\bar{n}_t)}{\bar{n}_u-a\bar{n}_t}\right] , \nonumber \\&\qquad \bar{n}_v=0,\bar{n}_t\ne 0, \bar{n}_u+a\bar{n}_t=0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_u-a\bar{n}_t\ne 0 \end{aligned}$$
(15e)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{\sin (B_v\bar{n}_v)}{4\pi ^3\bar{n}_v\bar{n}_t}\left[ B_u\sin (B_t\bar{n}_t)-\frac{\cos (B_t\bar{n}_t) -\cos (B_u\bar{n}_u-aB_u\bar{n}_t-B_t\bar{n}_t)}{\bar{n}_u-a\bar{n}_t}\right] , \nonumber \\&\qquad \bar{n}_v\ne 0,\bar{n}_t\ne 0, \bar{n}_u+a\bar{n}_t=0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_u-a\bar{n}_t\ne 0 \end{aligned}$$
(15f)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{B_v}{4\pi ^3\bar{n}_t}\left[ B_u\sin (B_t\bar{n}_t)+\frac{\cos (B_t\bar{n}_t)-\cos (B_u\bar{n}_u +aB_u\bar{n}_t+B_t\bar{n}_t)}{\bar{n}_u+a\bar{n}_t}\right] , \nonumber \\&\qquad \bar{n}_v=0,\bar{n}_t\ne 0, \bar{n}_u+a\bar{n}_t\ne 0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_u-a\bar{n}_t=0 \end{aligned}$$
(15g)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{\sin (B_v\bar{n}_v)}{4\pi ^3\bar{n}_v\bar{n}_t}\left[ B_u\sin (B_t\bar{n}_t)+\frac{\cos (B_t\bar{n}_t) -\cos (B_u\bar{n}_u+aB_u\bar{n}_t+B_t\bar{n}_t)}{\bar{n}_u+a\bar{n}_t}\right] , \nonumber \\&\qquad \bar{n}_v\ne 0,\bar{n}_t\ne 0, \bar{n}_u+a\bar{n}_t\ne 0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_u-a\bar{n}_t=0 \end{aligned}$$
(15h)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{B_v}{4\pi ^3\bar{n}_t}\left[ \frac{\cos (B_t\bar{n}_t)-\cos (B_u\bar{n}_u+aB_u\bar{n}_t+B_t\bar{n}_t)}{\bar{n}_u+a\bar{n}_t}\right. \nonumber \\&\qquad \quad \left. -\frac{\cos (B_t\bar{n}_t)-\cos (B_u\bar{n}_u-aB_u\bar{n}_t-B_t\bar{n}_t)}{\bar{n}_u-a\bar{n}_t}\right] , \nonumber \\&\qquad \quad \bar{n}_v=0,\bar{n}_t\ne 0, \bar{n}_u+a\bar{n}_t\ne 0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_u-a\bar{n}_t\ne 0 \end{aligned}$$
(15i)
$$\begin{aligned} g_{uvt}^{I}(\bar{n}_u,\bar{n}_v,\bar{n}_t)&= \frac{\sin (B_v\bar{n}_v)}{4\pi ^3\bar{n}_v\bar{n}_t}\left[ \frac{\cos (B_t\bar{n}_t)-\cos (B_u\bar{n}_u +aB_u\bar{n}_t+B_t\bar{n}_t)}{\bar{n}_u+a\bar{n}_t}\right. \nonumber \\&\qquad \quad \left. -\frac{\cos (B_t\bar{n}_t)-\cos (B_u\bar{n}_u-aB_u\bar{n}_t-B_t\bar{n}_t)}{\bar{n}_u-a\bar{n}_t}\right] , \nonumber \\&\qquad \quad \bar{n}_v\ne 0,\bar{n}_t\ne 0, \bar{n}_u+a\bar{n}_t\ne 0{{\mathrm{\,}}}\mathrm {and}{{\mathrm{\,}}}\bar{n}_u-a\bar{n}_t\ne 0, \end{aligned}$$
(15j)

where \(a=\tan (\theta )\).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Edussooriya, C.U.S., Bruton, L.T. & Agathoklis, P. A novel 5-D depth–velocity filter for enhancing noisy light field videos. Multidim Syst Sign Process 28, 353–369 (2017). https://doi.org/10.1007/s11045-016-0460-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11045-016-0460-x

Keywords

Search

Navigation