Advertisement

Journal of Real-Time Image Processing

, Volume 11, Issue 4, pp 713–730 | Cite as

Massively parallel Lucas Kanade optical flow for real-time video processing applications

  • Aurélien PlyerEmail author
  • Guy Le Besnerais
  • Frédéric Champagnat
Special Issue Paper

Abstract

This paper deals with dense optical flow estimation from the perspective of the trade-off between quality of the estimated flow and computational cost which is required by real-world applications. We propose a fast and robust local method, denoted by eFOLKI, and describe its implementation on GPU. It leads to very high performance even on large image formats such as 4 K (3,840 × 2,160) resolution. In order to assess the interest of eFOLKI, we first present a comparative study with currently available GPU codes, including local and global methods, on a large set of data with ground truth. eFOLKI appears significantly faster while providing quite accurate and highly robust estimated flows. We then show, on four real-time video processing applications based on optical flow, that eFOLKI reaches the requirements both in terms of estimated flows quality and of processing rate.

Keywords

Optical flow Motion GPU Super-resolution Dense 

Notes

Acknowledgments

The authors are most grateful to Benjamin Leclaire and Yves Le Sant at ONERA/DAFE for years of fruitful collaboration.

References

  1. 1.
    Ayuso, F., Botella, G., Garcia, C., Prieto, M., Tirado, F.: Gpu-based acceleration of bio-inspired motion estimation model. Concurr. Comput.: Pract. Exp. 25(8), 1037–1056 (2013)CrossRefGoogle Scholar
  2. 2.
    Baker, S., Matthews, I.: Lucas-Kanade 20 years on: A unifying framework. Int. J. Comput. Vis. 56(3), 221–255 (2004)CrossRefGoogle Scholar
  3. 3.
    Baker, S., Bennett, E., Kang, S.B., Szeliski, R.: Removing rolling shutter wobble. In: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, pp. 2392–2399 (2010)Google Scholar
  4. 4.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2011)CrossRefGoogle Scholar
  5. 5.
    Barranco, F., Tomasi, M., Diaz, J., Vanegas, M., Ros, E.: Parallel architecture for hierarchical optical flow estimation based on fpga. Very Large Scale Integr. (VLSI) Syst. IEEE Trans. 20(6), 1058–1067, (2012)Google Scholar
  6. 6.
    Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)CrossRefGoogle Scholar
  7. 7.
    Bergen, J.R., Anandan, P., Hanna, K.J., Hingorani, R.: Hierarchical model-based motion estimation. In: European Conference on Computer Vision ECCV’92, pp. 237–252. Springer (1992)Google Scholar
  8. 8.
    Botella, G., García, A., Rodríguez-Álvarez, M., Ros, E., Meyer-Baese, U., Molina, M.C.: Robust bioinspired architecture for optical-flow computation. Very Large Scale Integr. (VLSI) Syst. IEEE Trans. 18(4), 616–629, (2010)Google Scholar
  9. 9.
    Bouguet, J.Y.: Pyramidal implementation of the affine Lucas Kanade feature tracker description of the algorithm. Intel Corporation 5 (2001) Google Scholar
  10. 10.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: European Conference on Computer Vision, ECCV04, pp. 25–36. Springer (2004) Google Scholar
  11. 11.
    Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets horn/schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)CrossRefGoogle Scholar
  12. 12.
    Champagnat, F., Le Sant, Y.: Efficient cubic b-spline image interpolation on a GPU. J. Graph. Tools 16(4), 218–232 (2012)CrossRefGoogle Scholar
  13. 13.
    Champagnat, F., Plyer, A., Le Besnerais, G., Leclaire, B., Davoust, S., Le Sant, Y.: Fast and accurate PIV computation using highly parallel iterative correlation maximization. Exp. Fluids 50(4), 1169–1182 (2011)CrossRefGoogle Scholar
  14. 14.
    Díaz, J., Ros, E., Pelayo, F., Ortigosa, E.M., Mota, S.: Fpga-based real-time optical-flow system. Circuits Syst. Video Technol. IEEE Trans. 16(2), 274–279 (2006)CrossRefGoogle Scholar
  15. 15.
    Elad, M., Hel-Or, Y.: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Trans. Image Process. 10(8), 1187–1193 (2001)CrossRefzbMATHGoogle Scholar
  16. 16.
    Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Lecture Notes in Computer Science, pp. 363–370. Springer (2003)Google Scholar
  17. 17.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Fishbain, B., Yaroslavsky, L.P., Ideses, I.A.: Real-time stabilization of long range observation system turbulent video. J. Real-Time Image Process. 2(1), 11–22 (2007)CrossRefGoogle Scholar
  19. 19.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: Computer Vision and Pattern Recognition (CVPR), Providence, USA (2012)Google Scholar
  20. 20.
    Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A., Weickert, J.: A highly efficient GPU implementation for variational optic flow based on the euler-lagrange framework. In: Trends and Topics in Computer Vision, pp. 372–383. Springer (2012)Google Scholar
  21. 21.
    Harris, M.: Optimizing parallel reduction in cuda. CUDA SDK Whitepaper (2007)Google Scholar
  22. 22.
    Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1), 185–203 (1981)CrossRefGoogle Scholar
  23. 23.
    Le Besnerais, G., Champagnat, F.: Dense optical flow by iterative local window registration. In: IEEE International Conference on Image Processing, ICIP05, IEEE, pp. I–137 (2005) Google Scholar
  24. 24.
    Leclaire, B., Le Sant, Y., Le Besnerais, G., Champagnat, F.: On the stability and spatial resolution of image deformation PIV methods. In: Proceedings of 8th International Symposium on Particle Image Velocimetry-PIV09, Kobe (2011)Google Scholar
  25. 25.
    Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision. IJCAI 81, 674–679 (1981)Google Scholar
  26. 26.
    Marzat, J., Dumortier, Y., Ducrot, A., et al.: Real-time dense and accurate parallel optical flow using cuda. In: Proceedings of the 17th International Conference in Central European Computer Graphics, Visualization and Computer Vision (WSCG09) (2009)Google Scholar
  27. 27.
    Milanfar, P.: Super-resolution imaging. CRC Press (2010)Google Scholar
  28. 28.
    Mitzel, D., Pock, T., Schoenemann, T., Cremers, D.: Video super resolution using duality based tv-l 1 optical flow. In: Pattern Recognition, pp. 432–441. Springer (2009)Google Scholar
  29. 29.
    NVIDIA: NVIDIA CUDA Compute Unified Device Architecture—Programming Guide (2007)Google Scholar
  30. 30.
    Ottonelli, C., Hervé, A., Leclaire, B., Sipp, D., Schmid, P.: Non-linear reduced-order model of a cavity flow from TR-PIV measurements. In: 9th European Fluid Mechanics Conference (2012)Google Scholar
  31. 31.
    Pauwels, K., Tomasi, M., Diaz Alonso, J., Ros, E., Van Hulle, M.M.: A comparison of fpga and GPU for real-time phase-based optical flow, stereo, and local image features. Comput IEEE Trans 61(7), 999–1012 (2012)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Pereiro, S., Goussard, Y.: Unsupervised 3-d restoration of tomographic images by constrained wiener filtering. Engineering in Medicine and Biology Society, 1997. In: Proceedings of the 19th Annual International Conference of the IEEE, IEEE, vol. 2, pp. 557–560 (1993)Google Scholar
  33. 33.
    Plyer, A., Le Besnerais, G., Champagnat, F.: Folki-GPU : a powerful and versatile cuda code for real-time optical flow computation. In: GPU Technology Conference (2009)Google Scholar
  34. 34.
    Raffel, M., Willert, C.E., Kompenhans, J.: Particle Image Velocimetry: A Practical Guide. Springer (2007)Google Scholar
  35. 35.
    Sanfourche, M., Vittori, V., Le Besnerais, G.: eVO: a realtime embedded stereo odometry for mav applications. In: Proceedings of IROS’13, Tokyo, Japan (2013)Google Scholar
  36. 36.
    Schrijer, F., Scarano, F.: Effect of predictor-corrector filtering on the stability and spatial resolution of iterative PIV interrogation. Exp. Fluids 45(5), 927–941 (2008)CrossRefGoogle Scholar
  37. 37.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, pp. 2432–2439 (2010)Google Scholar
  38. 38.
    Tao, M., Bai, J., Kohli, P., Paris, S.: Simpleflow: a non-iterative, sublinear optical flow algorithm. Comput. Graph. Forum, Wiley Online Libr. 31, 345–353 (2012)CrossRefGoogle Scholar
  39. 39.
    Unger, M., Pock, T., Werlberger, M., Bischof, H.: A convex approach for variational super-resolution. In: Pattern Recognition, pp. 313–322. Springer (2010)Google Scholar
  40. 40.
    Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. In: Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, IEEE, vol. 2, pp. 722–729 (1999)Google Scholar
  41. 41.
    Weiß, A.R., Adomeit, U., Chevalier, P., Landeau, S., Bijl, P., Champagnat, F., Dijk, J., Göhler, B., Landini, S., Reynolds, J.P., et al.: A standard data set for performance analysis of advanced IR image processing techniques. SPIE Def. Secur. Sens. Int. Soc. Optics Photonics, pp. 512–835 (2012)Google Scholar
  42. 42.
    Werlberger, M.: Convex approaches for high performance video processing. PhD thesis, Institute for Computer Graphics and Vision, Graz University of Technology, Graz (2012)Google Scholar
  43. 43.
    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: European Conference on Computer Vision, ECCV’94, pp. 151–158 (1994)Google Scholar
  44. 44.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: Pattern Recognition, pp. 214–223. Springer (2007) Google Scholar
  45. 45.
    Zhang, Z.: Parameter estimation techniques: a tutorial with application to conic fitting. Tech. Rep. 2767, INRIA (1995)Google Scholar
  46. 46.
    Zhao, W., Sawhney, H.S.: Is super-resolution with optical flow feasible? In: European Conference on Computer Vision, ECCV02, pp. 599–613. Springer (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Aurélien Plyer
    • 1
    Email author
  • Guy Le Besnerais
    • 1
  • Frédéric Champagnat
    • 1
  1. 1.ONERAPalaiseauFrance

Personalised recommendations