Skip to main content
SpringerLink
Log in

Massively parallel Lucas Kanade optical flow for real-time video processing applications

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

This paper deals with dense optical flow estimation from the perspective of the trade-off between quality of the estimated flow and computational cost which is required by real-world applications. We propose a fast and robust local method, denoted by eFOLKI, and describe its implementation on GPU. It leads to very high performance even on large image formats such as 4 K (3,840 × 2,160) resolution. In order to assess the interest of eFOLKI, we first present a comparative study with currently available GPU codes, including local and global methods, on a large set of data with ground truth. eFOLKI appears significantly faster while providing quite accurate and highly robust estimated flows. We then show, on four real-time video processing applications based on optical flow, that eFOLKI reaches the requirements both in terms of estimated flows quality and of processing rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Ayuso, F., Botella, G., Garcia, C., Prieto, M., Tirado, F.: Gpu-based acceleration of bio-inspired motion estimation model. Concurr. Comput.: Pract. Exp. 25(8), 1037–1056 (2013)

    Article  Google Scholar 

  2. Baker, S., Matthews, I.: Lucas-Kanade 20 years on: A unifying framework. Int. J. Comput. Vis. 56(3), 221–255 (2004)

    Article  Google Scholar 

  3. Baker, S., Bennett, E., Kang, S.B., Szeliski, R.: Removing rolling shutter wobble. In: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, pp. 2392–2399 (2010)

  4. Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2011)

    Article  Google Scholar 

  5. Barranco, F., Tomasi, M., Diaz, J., Vanegas, M., Ros, E.: Parallel architecture for hierarchical optical flow estimation based on fpga. Very Large Scale Integr. (VLSI) Syst. IEEE Trans. 20(6), 1058–1067, (2012)

    Google Scholar 

  6. Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)

    Article  Google Scholar 

  7. Bergen, J.R., Anandan, P., Hanna, K.J., Hingorani, R.: Hierarchical model-based motion estimation. In: European Conference on Computer Vision ECCV’92, pp. 237–252. Springer (1992)

  8. Botella, G., García, A., Rodríguez-Álvarez, M., Ros, E., Meyer-Baese, U., Molina, M.C.: Robust bioinspired architecture for optical-flow computation. Very Large Scale Integr. (VLSI) Syst. IEEE Trans. 18(4), 616–629, (2010)

    Google Scholar 

  9. Bouguet, J.Y.: Pyramidal implementation of the affine Lucas Kanade feature tracker description of the algorithm. Intel Corporation 5 (2001)

  10. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: European Conference on Computer Vision, ECCV04, pp. 25–36. Springer (2004)

  11. Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets horn/schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)

    Article  Google Scholar 

  12. Champagnat, F., Le Sant, Y.: Efficient cubic b-spline image interpolation on a GPU. J. Graph. Tools 16(4), 218–232 (2012)

    Article  Google Scholar 

  13. Champagnat, F., Plyer, A., Le Besnerais, G., Leclaire, B., Davoust, S., Le Sant, Y.: Fast and accurate PIV computation using highly parallel iterative correlation maximization. Exp. Fluids 50(4), 1169–1182 (2011)

    Article  Google Scholar 

  14. Díaz, J., Ros, E., Pelayo, F., Ortigosa, E.M., Mota, S.: Fpga-based real-time optical-flow system. Circuits Syst. Video Technol. IEEE Trans. 16(2), 274–279 (2006)

    Article  Google Scholar 

  15. Elad, M., Hel-Or, Y.: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Trans. Image Process. 10(8), 1187–1193 (2001)

    Article  MATH  Google Scholar 

  16. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Lecture Notes in Computer Science, pp. 363–370. Springer (2003)

  17. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  18. Fishbain, B., Yaroslavsky, L.P., Ideses, I.A.: Real-time stabilization of long range observation system turbulent video. J. Real-Time Image Process. 2(1), 11–22 (2007)

    Article  Google Scholar 

  19. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: Computer Vision and Pattern Recognition (CVPR), Providence, USA (2012)

  20. Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A., Weickert, J.: A highly efficient GPU implementation for variational optic flow based on the euler-lagrange framework. In: Trends and Topics in Computer Vision, pp. 372–383. Springer (2012)

  21. Harris, M.: Optimizing parallel reduction in cuda. CUDA SDK Whitepaper (2007)

  22. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1), 185–203 (1981)

    Article  Google Scholar 

  23. Le Besnerais, G., Champagnat, F.: Dense optical flow by iterative local window registration. In: IEEE International Conference on Image Processing, ICIP05, IEEE, pp. I–137 (2005)

  24. Leclaire, B., Le Sant, Y., Le Besnerais, G., Champagnat, F.: On the stability and spatial resolution of image deformation PIV methods. In: Proceedings of 8th International Symposium on Particle Image Velocimetry-PIV09, Kobe (2011)

  25. Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision. IJCAI 81, 674–679 (1981)

    Google Scholar 

  26. Marzat, J., Dumortier, Y., Ducrot, A., et al.: Real-time dense and accurate parallel optical flow using cuda. In: Proceedings of the 17th International Conference in Central European Computer Graphics, Visualization and Computer Vision (WSCG09) (2009)

  27. Milanfar, P.: Super-resolution imaging. CRC Press (2010)

  28. Mitzel, D., Pock, T., Schoenemann, T., Cremers, D.: Video super resolution using duality based tv-l 1 optical flow. In: Pattern Recognition, pp. 432–441. Springer (2009)

  29. NVIDIA: NVIDIA CUDA Compute Unified Device Architecture—Programming Guide (2007)

  30. Ottonelli, C., Hervé, A., Leclaire, B., Sipp, D., Schmid, P.: Non-linear reduced-order model of a cavity flow from TR-PIV measurements. In: 9th European Fluid Mechanics Conference (2012)

  31. Pauwels, K., Tomasi, M., Diaz Alonso, J., Ros, E., Van Hulle, M.M.: A comparison of fpga and GPU for real-time phase-based optical flow, stereo, and local image features. Comput IEEE Trans 61(7), 999–1012 (2012)

    Article  MathSciNet  Google Scholar 

  32. Pereiro, S., Goussard, Y.: Unsupervised 3-d restoration of tomographic images by constrained wiener filtering. Engineering in Medicine and Biology Society, 1997. In: Proceedings of the 19th Annual International Conference of the IEEE, IEEE, vol. 2, pp. 557–560 (1993)

  33. Plyer, A., Le Besnerais, G., Champagnat, F.: Folki-GPU : a powerful and versatile cuda code for real-time optical flow computation. In: GPU Technology Conference (2009)

  34. Raffel, M., Willert, C.E., Kompenhans, J.: Particle Image Velocimetry: A Practical Guide. Springer (2007)

  35. Sanfourche, M., Vittori, V., Le Besnerais, G.: eVO: a realtime embedded stereo odometry for mav applications. In: Proceedings of IROS’13, Tokyo, Japan (2013)

  36. Schrijer, F., Scarano, F.: Effect of predictor-corrector filtering on the stability and spatial resolution of iterative PIV interrogation. Exp. Fluids 45(5), 927–941 (2008)

    Article  Google Scholar 

  37. Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, pp. 2432–2439 (2010)

  38. Tao, M., Bai, J., Kohli, P., Paris, S.: Simpleflow: a non-iterative, sublinear optical flow algorithm. Comput. Graph. Forum, Wiley Online Libr. 31, 345–353 (2012)

    Article  Google Scholar 

  39. Unger, M., Pock, T., Werlberger, M., Bischof, H.: A convex approach for variational super-resolution. In: Pattern Recognition, pp. 313–322. Springer (2010)

  40. Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. In: Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, IEEE, vol. 2, pp. 722–729 (1999)

  41. Weiß, A.R., Adomeit, U., Chevalier, P., Landeau, S., Bijl, P., Champagnat, F., Dijk, J., Göhler, B., Landini, S., Reynolds, J.P., et al.: A standard data set for performance analysis of advanced IR image processing techniques. SPIE Def. Secur. Sens. Int. Soc. Optics Photonics, pp. 512–835 (2012)

  42. Werlberger, M.: Convex approaches for high performance video processing. PhD thesis, Institute for Computer Graphics and Vision, Graz University of Technology, Graz (2012)

  43. Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: European Conference on Computer Vision, ECCV’94, pp. 151–158 (1994)

  44. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: Pattern Recognition, pp. 214–223. Springer (2007)

  45. Zhang, Z.: Parameter estimation techniques: a tutorial with application to conic fitting. Tech. Rep. 2767, INRIA (1995)

  46. Zhao, W., Sawhney, H.S.: Is super-resolution with optical flow feasible? In: European Conference on Computer Vision, ECCV02, pp. 599–613. Springer (2002)

Download references

Acknowledgments

The authors are most grateful to Benjamin Leclaire and Yves Le Sant at ONERA/DAFE for years of fruitful collaboration.

Author information

Authors and Affiliations

  1. ONERA, Chemin de la Hunière et des Joncherettes, 91123, Palaiseau, France

    Aurélien Plyer, Guy Le Besnerais & Frédéric Champagnat

Authors
  1. Aurélien Plyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guy Le Besnerais
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frédéric Champagnat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aurélien Plyer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Plyer, A., Le Besnerais, G. & Champagnat, F. Massively parallel Lucas Kanade optical flow for real-time video processing applications. J Real-Time Image Proc 11, 713–730 (2016). https://doi.org/10.1007/s11554-014-0423-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-014-0423-0

Keywords

Search

Navigation