We tackle the problem of detecting occluded regions in a video stream. Under assumptions of Lambertian reflection and static illumination, the task can be posed as a variational optimization problem, and its solution approximated using convex minimization. We describe efficient numerical schemes that reach the global optimum of the relaxed cost functional, for any number of independently moving objects, and any number of occlusion layers. We test the proposed algorithm on benchmark datasets, expanded to enable evaluation of occlusion detection performance, in addition to optical flow.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Alvarez, L., Deriche, R., Papadopoulo, T., & Sánchez, J. (2007). Symmetrical dense optical flow estimation with occlusions detection. International Journal of Computer Vision, 75(3), 371–385.
Ayvaci, A., & Soatto, S. (2011). Efficient model selection for detachable object detection. In Proc. of energy minimization methods in computer vision and pattern recognition, July 2011.
Ayvaci, A., Raptis, M., & Soatto, S. (2010). Occlusion detection and motion estimation with convex optimization. In Advances in neural information processing systems.
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In Proc. of the international conference on computer vision (pp. 1–8).
Becker, S., Bobin, J., & Candes, E. (2009). Nesta: a fast and accurate first-order method for sparse recovery. Arxiv preprint arXiv, 904.
Ben-Ari, R., & Sochen, N. (2007). Variational stereo vision with sharp discontinuities and occlusion handling. In Proc. of international conference on computer vision (pp. 1–7).
Black, M., & Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104.
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Proc. of European conference on computer vision (pp. 25–36).
Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 61(3), 211–231.
Candes, E., Wakin, M., & Boyd, S. (2008). Enhancing sparsity by reweighted L1 minimization. The Journal of Fourier Analysis and Applications, 14(5), 877–905.
Caselles, V., Coll, B., & Morel, J.-M. (1999). Topographic maps and local contrast changes in natural images. International Journal of Computer Vision, 33(1), 5–27.
Chan, T., Esedoglu, S., & Nikolova, M. (2006). Algorithms for finding global minimizers of denoising and segmentation models. SIAM Journal on Applied Mathematics, 66(1), 1632–1648
Dahl, J., Hansen, P., Jensen, S., & Jensen, T. (2009). Algorithms and software for total variation image reconstruction via first-order methods. Numerical Algorithms, 67–92.
Gibson, J. J. (1984). The ecological approach to visual perception. LEA.
Goldstein, T., & Osher, S. (2009). The split Bregman method for L1 regularized problems. SIAM Journal on Imaging Sciences, 2(2), 323–343.
He, X., & Yuille, A. (2010). Occlusion boundary detection using pseudo-depth. In Proc. of the European conference on computer vision.
Horn, B., & Schunck, B. (1981). Determining optical flow. Computer Vision, 17, 185–203.
Humayun, A., Mac Aodha, O., & Brostow, G. J. (2011). Learning to find occlusion regions. In Proc. of conference on computer vision and pattern recognition.
Ince, S., & Konrad, J. (2008). Occlusion-aware optical flow estimation. IEEE Transactions on Image Processing, 17(8), 1443–1451.
Jackson, J. D., Yezzi, A. J., & Soatto, S. (2005). Dynamic shape and appearance modeling via moving and deforming layers. In Proc. of workshop on energy minimization in computer vision and pattern recognition (EMMCVPR) (pp. 427–438).
Jackson, J., Yezzi, A. J., & Soatto, S. (2008). Dynamic shape and appearance modeling via moving and deforming layers. International Journal of Computer Vision.
Kim, Y., Martínez, A., & Kak, A. (2005). Robust motion estimation under varying illumination. Image and Vision Computing, 23(4), 365–375.
Kolmogorov, V., & Zabih, R. (2001). Computing visual correspondence with occlusions via graph cuts. In Proc. of international conference on computer vision (pp. 508–515).
Lim, K., Das, A., & Chong, M. (2002). Estimation of occlusion and dense motion fields in a bidirectional Bayesian framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 712–718.
Negahdaripour, S. (1998). Revised definition of optical flow: integration of radiometric and geometric cues for dynamic scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 961–979.
Nesterov, Y. (1983). A method for unconstrained convex minimization problem with the rate of convergence O (1/k 2). Doklady Akademii Nauk SSSR, 269, 543–547.
Nesterov, Y. (2005). Smooth minimization of non-smooth functions. Mathematical Programming, 103(1), 127–152.
Proesmans, M., Van Gool, L., & Oosterlinck, A. (1994). Determination of optical flow and its discontinuities using a non-linear diffusion. In Proc. of European conference of computer vision.
Robert, C. P. (2001). The Bayesian choice. New York: Springer.
Rudin, L., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica. D, 60, 259–268.
Shulman, D., & Herve, J. (1989). Regularization of discontinuous flow fields. In Proc. of workshop on visual motion (pp. 81–86).
Soatto, S. (2011). Actionable information in vision. In Machine learning for computer vision. Berlin: Springer.
Soatto, S., & Yezzi, A. (2002). Deformotion: deforming motion, shape average and the joint segmentation and registration of images. In Proc. of the European conference on computer vision (Vol. 3, pp. 32–47).
Soatto, S., Yezzi, A. J., & Jin, H. (2003). Tales of shape and radiance in multiview stereo. In Proc. of international conference on computer vision (pp. 974–981). October 2003.
Stein, A., & Hebert, M. (2009). Occlusion boundaries from motion: low-level detection and mid-level reasoning. International Journal of Computer Vision, 82(3), 325–357.
Strecha, C., Fransens, R., & Van Gool, L. (2004). A probabilistic approach to large displacement optical flow and occlusion detection. In ECCV workshop SMVP (pp. 71–82). Berlin: Springer.
Sun, J., Li, Y., Kang, S., & Shum, H. (2005). Symmetric stereo matching for occlusion handling. In Proc. of conference on computer vision and pattern recognition (Vol. 2, p. 399).
Sun, D., Roth, S., & Black, M. (2010). Secrets of optical flow estimation and their principles. In Proc. of conference on computer vision and pattern recognition (pp. 2432–2439).
Sundaramoorthi, G., Petersen, P., Varadarajan, V. S., & Soatto, S. (2009). On the set of images modulo viewpoint and contrast changes. In Proc. of conference on computer vision and pattern recognition.
Teng, C., Lai, S., Chen, Y., & Hsu, W. (2005). Accurate optical flow computation under non-uniform brightness variations. Computer Vision and Image Understanding, 97(3), 315–346.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B. Methodological, 58(1), 267–288.
Verri, A., & Poggio, T. (1989). Motion field and optical flow: Qualitative properties. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(5), 490–498.
Wang, J., & Adelson, E. (1994). Representing moving images with layers. IEEE Transactions on Image Processing, 3(5), 625–638.
Wang, Y., Yin, W., & Zhang, Y. (2007). A fast algorithm for image deblurring with total variation regularization (CAAM Technical Reports). Rice University.
Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2008). An improved algorithm for TV-L1 optical flow. In Proc. of statistical and geometrical approaches to visual motion analysis: International Dagstuhl seminar.
Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In Proc. of international conference on computer vision.
Weiss, P., Blanc-Féraud, L., & Aubert, G. (2009). Efficient schemes for total variation minimization under constraints in image processing. SIAM Journal on Scientific Computing, 31, 2047.
Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009). Anisotropic Huber-L1 optical flow. In Proc. of British machine vision conference.
Xiao, J., Cheng, H., Sawhney, H., Rao, C., Isnardi, M., et al. (2006). Bilateral filtering-based optical flow estimation with occlusion detection. In Proc. of European conference of computer vision (Vol. 3951, p. 211).
A. Ayvaci and M. Raptis contributed equally to this work.
About this article
Cite this article
Ayvaci, A., Raptis, M. & Soatto, S. Sparse Occlusion Detection with Optical Flow. Int J Comput Vis 97, 322–338 (2012). https://doi.org/10.1007/s11263-011-0490-7
- Occlusion detection
- Optical flow
- Convex optimization
- Sparse optimization
- Nesterov’s algorithm
- Split-Bregman method