Skip to main content
SpringerLink
Log in

Sparsity in optical flow and trajectories

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In this article, we apply sparse constraints to improve optical flow and trajectories. We apply sparsity in two ways. First, with two-frame optical flow, we enforce a sparse representation of flow patches using a learned overcomplete dictionary. Second, we apply a low-rank constraint to trajectories via robust coupling. Optical flow is an ill-posed underconstrained inverse problem. Many recent approaches use total variation to constrain the flow solution to satisfy color constancy. In our first results presented, we find that learning a 2D overcomplete dictionary from the total variation result and then enforcing a sparse constraint on the flow improves the result. A new technique using partially overlapping patches accelerates the calculation. This approach is implemented in a coarse-to-fine strategy. Our results show that combining total variation and a sparse constraint from a learned dictionary is more effective than total variation alone. In the second part, we compute optical flow and trajectories from an image sequence. Sparsity in trajectories is measured by matrix rank. We introduce a low-rank constraint of linear complexity using random subsampling of the data. We demonstrate that, by using a robust coupling with the low-rank constraint, our approach outperforms baseline methods on general image sequences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. MPI Sintel Flow Dataset. http://sintel.is.tue.mpg.de/ (2014)

  2. The Middlebury Computer Vision Pages. http://vision.middlebury.edu (2014)

  3. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. IJCV 92(1), 1–31 (2011)

    Article  Google Scholar 

  4. Black, M.: Recursive non-linear estimation of discontinuous flow fields. In: Computer Vision—ECCV’94, pp. 138–145 (1994)

  5. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2010)

    Article  MATH  Google Scholar 

  6. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Computer Vision—ECCV 2004, 4 May, pp. 25–36 (2004)

  7. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE TPAMI 33(3), 500–513 (2011)

    Article  Google Scholar 

  8. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: European Conference on Computer Vision (ECCV), pp. 611–625 (2012)

  9. Cai, J., Candès, E., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20(4), 1–26 (2010)

    Article  Google Scholar 

  10. Condat, L.: A generic proximal algorithm for convex optimization—application to total variation minimization. IEEE Signal Process. Lett. 21(8), 985–989 (2014)

    Article  Google Scholar 

  11. Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)

    Article  MathSciNet  Google Scholar 

  12. Engan, K., Aase, S.O., Hakon Husoy, J.: Method of optimal directions for frame design. In: Proceedings of the Acoustics, Speech, and Signal Processing, 1999, pp. 2443–2446 (1999)

  13. Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. IJCV 104, 286–314 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  14. Gibson, J., Marques, O.: Sparse regularization of TV-L1 optical flow. In: Elmoataz, A., Lezoray, O., Nouboud, F., Mammass, D. (eds.) ICISP 2014, Image and Signal Processing, vol. LNCS 8509, pp. 460–467. Springer, Cherbourg (2014)

  15. Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  16. Irani, M.: Multi-frame correspondence estimation using subspace constraints. IJCV 48(153), 173–194 (2002)

    Article  MATH  Google Scholar 

  17. Jia, K., Wang, X., Tang, X.: Optical flow estimation using learned sparse model. In: ICCV, 60903115 (2011)

  18. Liu, R., Lin, Z., Su, Z., Gao, J.: Linear time principal component pursuit and its extensions using 1 filtering. Neurocomputing 142, 529–541 (2014). doi:10.1016/j.neucom.2014.03.046

    Article  Google Scholar 

  19. Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11, 19–60 (2010)

    MathSciNet  MATH  Google Scholar 

  20. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)

    Article  MathSciNet  Google Scholar 

  21. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103, 127–152 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  22. Nir, T., Bruckstein, A.M., Kimmel, R.: Over-parameterized variational optical flow. IJCV 76(2), 205–216 (2007)

    Article  Google Scholar 

  23. Pizarro, D., Bartoli, A.: Feature-based deformable surface detection with self-occlusion reasoning. IJCV 97(1), 54–70 (2011)

    Article  Google Scholar 

  24. Ricco, S., Tomasi, C.: Dense lagrangian motion estimation with occlusions. CVPR pp. 1800–1807 (2012)

  25. Ricco, S., Tomasi, C.: Video motion for every visible point. In: ICCV, i (2013)

  26. Sand, P., Teller, S.: Particle video: long-range motion estimation using point trajectories. IJCV 80(1), 72–91 (2008)

    Article  Google Scholar 

  27. Shen, X., Wu, Y.: Sparsity model for robust optical flow estimation at motion discontinuities. IEEE CVPR 2010 1, 2456–2463 (2010)

    Google Scholar 

  28. Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: 2011 International Conference on Computer Vision (ICCV), pp. 1116–1123. IEEE (2011)

  29. Wedel, A., Cremers, D.: Structure-and motion-adaptive regularization for high accuracy optic flow. In: Computer Vision, 2009, pp. 1663–1668 (2009)

  30. Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for TV-L 1 optical flow. In: Statistical and Geometrical Approaches to Visual Motion Analysis, pp. 23–45. Springer, Berlin (2009)

  31. Wright, J., Yigang, P., Ma, Y., Ganesh, A., Rao, S.: Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. In: NIPS, pp. 1–9 (2009)

  32. Yuan, X., Yang, J.: Sparse and low-rank matrix decomposition via alternating direction methods. Optimization Online pp. 1–11 (2009)

  33. Zhang, S., Yao, H., Sun, X., Lu, X.: Sparse coding based visual tracking: review and experimental comparison. Pattern Recognit. 46(7), 1772–1788 (2013). doi:10.1016/j.patcog.2012.10.006

    Article  Google Scholar 

  34. Zhang, S., Zhou, H., Jiang, F., Li, X.: Robust visual tracking using structurally random projection and weighted least squares. IEEE Transactions on Circuits and Systems for Video Technology pp. 1–12 (2015). doi:10.1109/TCSVT.2015.2406194

  35. Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. IJCV 93(3), 368–388 (2011)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Florida Atlantic University, 777 Glades Road, Boca Raton, FL, 33431, USA

    Joel Gibson & Oge Marques

Authors
  1. Joel Gibson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oge Marques
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joel Gibson.

Appendix: RLR implementation details

Appendix: RLR implementation details

For all tests, \(\tau = 0.00004\) and \(\lambda = 0.001\). For convenience of the GPU implementation, the Minimization 1 was fixed at 1200 iterations. The actual convergence was seen to often be much less. For the MPI-Sintel tests \(\gamma =0.1\), \(R=3\), the number of warps was two, and the alternating iterations between optimization steps was eight. For the flag sequence \(\gamma =0.05\), \(R=9\), the number of warps was four, and the alternating iterations was 15. Images were preprocessed with ROF structure/texture decomposition.

For Garg et al.’s MFSF code, the grayscale version was implemented. \(R=12\) was found to work best on a set of the MPI-Sintel tests and fixed for all sequences.

The two-frame results presented in Table 2 were obtained by setting \(\tau =0\) in our algorithm, effectively disabling the low-rank calculation. Our algorithm then degenerates to a TV-L\(^1\) two-frame optical flow calculation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gibson, J., Marques, O. Sparsity in optical flow and trajectories. SIViP 10, 487–494 (2016). https://doi.org/10.1007/s11760-015-0767-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-015-0767-3

Keywords

Search

Navigation