Skip to main content
SpringerLink
Log in

Detecting Occlusions as an Inverse Problem

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Occlusions generally become apparent when integrated over time because violations of the brightness-constancy constraint of optical flow accumulate in occluded areas. Based on this observation, we propose a variational model for occlusion detection that is formulated as an inverse problem. Our forward model adapts the brightness constraint of optical flow to emphasize occlusions by exploiting their temporal behavior, while spatio-temporal regularizers on the occlusion set make our model robust to noise and modeling errors. In terms of minimization, we approximate the resulting variational problem by a sequence of convex optimizations and develop efficient algorithms to solve them. Our experiments show the benefits of the proposed formulation, both forward model and regularizers, in comparison to the state-of-the-art techniques that detect occlusion as the residual of optical-flow estimation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. As the dependence of \(\varvec{u}_{c-j} = \varvec{\bar{u}}_{c-j}(x - \sum _{l=1}^{j-1} \varvec{u}_{c-l}(x) )\) on \(\varvec{u}_{k}\) for \(c-j<k\) disappears when we fix \(\varvec{u}_{c-j}\) to \( \varvec{u}^t_{c-j}~=~\varvec{\bar{u}}_{c-j}(x - \sum _{l=0}^{j-1} \varvec{u}^t_{c-l}(x) )\), we use the standard linearization of optical flow.

References

  1. Adato, Y., Vasilyev, Y., Ben-Shahar, O., Zickler, T.: Toward a theory of shape from specular flow. In: International Conference on Computer Vision (ICCV), pp. 1–8 (2007). doi:10.1109/ICCV.2007.4408883

  2. Alvarez, L., Deriche, R., Papadopoulo, T., Sánchez, J.: Symmetrical dense optical flow estimation with occlusions detection. Int. J. Comput. Vis. 75(3), 371–385 (2007). doi:10.1007/s11263-007-0041-4

    Article  Google Scholar 

  3. Ayvaci, A., Raptis, M., Soatto, S.: Sparse occlusion detection with optical flow. Int. J. Comput. Vis. 97(3), 322–338 (2011). doi:10.1007/s11263-011-0490-7

    Article  MathSciNet  Google Scholar 

  4. Ayvaci, A., Soatto, S.: Motion segmentation with occlusions on the superpixel graph. In: IEEE International Conference on Computer Vision, pp. 727–734 (2009). doi:10.1109/ICCVW.2009.5457630

  5. Ayvaci, A., Soatto, S.: Detachable object detection with efficient model selection. In: Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 191–204 (2011)

  6. Ballester, C., Garrido, L., Lazcano, V., Caselles, V.: A TV-L1 Optical Flow Method with Occlusion Detection. Pattern Recognit. Lecture Notes in Computer Science 7476, 31–40 (2012)

  7. Black, M.J., Fleet, D.J.: Probabilistic detection and tracking of motion boundaries. Int. J. Comput. Vis. 38(3), 231–245 (2000)

    Article  MATH  Google Scholar 

  8. Boyd, S., Vandenberghe, L.: Convex Optimization Theory, vol. 25, 7th edn. Cambridge University Press, New York (2004). doi:10.1080/10556781003625177

    Book  Google Scholar 

  9. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. Eur. Conf. Comput. Vis. 4(May), 25–36 (2004)

    Google Scholar 

  10. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011). doi:10.1109/TPAMI.2010.143

    Article  Google Scholar 

  11. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, C. Schmid (eds.) Computer Vision ECCV 2012 SE - 44. Lecture Notes in Computer Science, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33783-3_44

  12. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2010). doi:10.1007/s10851-010-0251-1

    Article  MathSciNet  Google Scholar 

  13. Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006). doi:10.1137/040615286

    Article  MathSciNet  MATH  Google Scholar 

  14. Chin, T., Karl, W., Willsky, A.S.: Probabilistic and sequential computation of optical flow using temporal coherence. IEEE Trans. Image Process. 3(6), 773–788 (1994)

    Article  Google Scholar 

  15. Deriche, R., Kornprobst, P., Aubert, G.: Optical-flow estimation while preserving its discontinuities: a variational approach. Recent Dev. Comput. Vis. 9, 69–80 (1996). doi:10.1007/BF01246637

    Article  Google Scholar 

  16. Garg, R., Pizarro, L., Rueckert, D., Agapito, L.: Dense multi-frame optic flow for non-rigid objects using subspace constraints. In: Asian Conference on Computer Vision, pp. 460–473 (2011)

  17. He, X., Yuille, A.: Occlusion boundary detection using pseudo-depth. Comput. Vis. 2010, 539–552 (2010)

    Google Scholar 

  18. Horn, B.K.P.: Robot Vis. MIT Press, Cambridge (1986)

    Google Scholar 

  19. Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981). doi:10.1016/0004-3702(81)90024-2

    Article  Google Scholar 

  20. Humayun, A., Mac Aodha, O., Brostow, G.J.: Learning to find occlusion regions. In: International Conference on Computer Vision and Pattern Recognition, pp. 2161–2168 (2011). doi:10.1109/CVPR.2011.5995517

  21. Ince, S., Konrad, J.: Occlusion-aware optical flow estimation. IEEE Trans. Image Process. 17(8), 1443–1451 (2008). doi:10.1109/TIP.2008.925381

    Article  MathSciNet  Google Scholar 

  22. Jackson, J.D., Yezzi, A.J., Soatto, S.: Dynamic shape and appearance modeling via moving and deforming layers. Int. J. Comput. Vis. 79(1), 71–84 (2007). doi:10.1007/s11263-007-0097-1

    Article  Google Scholar 

  23. Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions using graph cuts. IEEE Int. Conf. Comput. Vis. 2, 508–515 (2001). doi:10.1109/ICCV.2001.937668

    Google Scholar 

  24. Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: International Conference on Computer Vision (2013)

  25. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. Imaging Underst. Workshop 130, 121–130 (1981)

    Google Scholar 

  26. Maver, J., Bajcsy, R.: Occlusions as a guide for planning the next view (1993). doi:10.1109/34.211463

  27. Nagel, H.H.: Extending the ’oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In: European Conference on Computer Vision (ECCV), pp. 139–148 (1990)

  28. Pock, T., Unger, M., Cremers, D., Bischof, H.: Fast and exact solution of total variation models on the GPU. Conf. Comput. Vis. Pattern Recognit. Workshop 1, 1–8 (2008). doi:10.1109/CVPRW.2008.4563099

    Google Scholar 

  29. Rockafellar, R.T.: Convex Analysis, vol. 28. Princeton University Press, Princeton (1997)

    MATH  Google Scholar 

  30. Roth, S., Domini, F., Black, M.J.: Specular flow and the perception of surface reflectance. Int. Conf. Comput. Vis. Pattern Recognit. 1, 1869–1876 (2006). doi:10.1167/3.9.413

    Google Scholar 

  31. Salgado, A., Sánchez, J.: Optical Flow Estimation with Large Displacements : A Temporal Regularizer. Tech. Rep. April, Cuadernos del Instituto Universitario de Ciencias y Tecnologías Cibernéticas, Las Palmas, España (2006)

  32. Sargin, M.E., Bertelli, L., Manjunath, B.S., Rose, K.: Probabilistic occlusion boundary detection on spatio-temporal lattices. In: International Conference on Computer Vision, pp. 560–567 (2009). doi:10.1109/ICCV.2009.5459190

  33. Smith, P., Drummond, T., Cipolla, R.: Layered motion segmentation and depth ordering by tracking edges. IEEE Trans. Pattern Anal. Mach. Intell. 26(4), 479–494 (2004). doi:10.1109/TPAMI.2004.1265863

    Article  Google Scholar 

  34. Stein, A.N., Hebert, M.: Occlusion boundaries from motion: low-level detection and mid-level reasoning. Int. J. Comput. Vis. 82(3), 325–357 (2009). doi:10.1007/s11263-008-0203-z

    Article  Google Scholar 

  35. Strecha, C., Fransens, R., Gool, L.V.: A probabilistic approach to large displacement optical flow and occlusion detection. Stat. Methods Video Process. 3247, 71–82 (2004)

    Article  Google Scholar 

  36. Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2432–2439 (2010). doi:10.1109/CVPR.2010.5539939

  37. Sun, D., Sudderth, E.B., Black, M.J.: Layered segmentation and optical flow estimation over time. In: International Conference on Computer Vision and Pattern Recognition, pp. 1768–1775 (2012). doi:10.1109/CVPR.2012.6247873

  38. Sun, J., Li, Y.: Symmetric stereo matching for occlusion handling. IEEE Conf. Comput. Vis. Pattern Recognit. 2, 399–406 (2005)

    Google Scholar 

  39. Sundberg, P., Brox, T.: Occlusion boundary detection and figure/ground assignment from optical flow. In: International Conference on Computer Vision and Pattern Recognition (2011)

  40. Vasilyev, Y., Zickler, T., Gortler, S., Ben-Shahar, O.: Shape from specular flow: is one flow enough? In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2561–2568 (2011). doi:10.1109/CVPR.2011.5995662

  41. Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: 2011 International Conference on Computer Vision, pp. 1116–1123 (2011). doi:10.1109/ICCV.2011.6126359

  42. Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for TV-L 1 optical flow. Stat. Geom. Approach. Vis. Motion Anal. Springer B, 23–45 (2009)

    Article  Google Scholar 

  43. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow : large displacement optical flow with deep matching. In: International Conference on Computer Vision (2013)

  44. Xiao, J., Cheng, H., Sawhney, H.: Bilateral filtering-based optical flow estimation with occlusion detection. Eur. Conf. Comput. Vis. 3951, 211–224 (2006)

    Google Scholar 

  45. Xiao, J., Shah, M.: Motion layer extraction in the presence of occlusion using graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1644–1659 (2005). doi:10.1109/TPAMI.2005.202

    Article  Google Scholar 

  46. Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012)

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the authors of [2] for sharing their code and D. Davis from UCLA for discussions on optimization. V. Estellers is supported by the Swiss National Science Foundation under Grant P2ELP2_148890.

Author information

Authors and Affiliations

  1. UCLA Vision Lab, University of California, Los Angeles, USA

    V. Estellers & S. Soatto

Authors
  1. V. Estellers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Soatto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. Estellers.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Estellers, V., Soatto, S. Detecting Occlusions as an Inverse Problem. J Math Imaging Vis 54, 181–198 (2016). https://doi.org/10.1007/s10851-015-0596-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-015-0596-6

Keywords

Search

Navigation