In this work we present a new way of simultaneously solving the problems of motion detection and background image reconstruction. An accurate estimation of the background is only possible if we locate the moving objects. Meanwhile, a correct motion detection is achieved if we have a good available background model. The key of our joint approach is to define a single random process that can take two types of values, instead of defining two different processes, one symbolic (motion detection) and one numeric (background intensity estimation). It thus allows to exploit the (spatio-temporal) interaction between a decision (motion detection) and an estimation (intensity reconstruction) problem. Consequently, the meaning of solving both tasks jointly, is to obtain a single optimal estimate of such a process. The intrinsic interaction and simultaneity between both problems is shown to be better modeled within the so-called mixed-state statistical framework, which is extended here to account for symbolic states and conditional random fields. Experiments on real sequences and comparisons with existing motion detection methods support our proposal. Further implications for video sequence inpainting will be also discussed.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Benboudjema, D., & Pieczynski, W. (2007). Unsupervised statistical segmentation of nonstationary images using triplet Markov fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1367–1378.
Benedek, C., Sziranyi, T., Kato, Z., & Zerubia, J. (2007). A multi-layer Mrf model for object-motion detection in unregistered airborne image-pairs. In IEEE international conference on image processing, 2007 (ICIP07), 16 2007–Oct. 19 2007 (Vol. 6, pp. 141–144).
Besag, J. (1974). Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society. Series B, 36, 192–236.
Besag, J. (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society. Series B, 48(3), 259–302.
Black, M. J., & Rangarajan, A. (1996). On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. International Journal of Computer Vision, 19(1), 57–91.
Blanchet, J., & Forbes, F. (2008). Triplet Markov fields for the classification of complex structure data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(6), 1055–1067.
Bouthemy, P., & Lalande, P. (1993). Recovery of moving object masks in an image sequence using local spatiotemporal contextual information. Optical Engineering, 32(6), 1205–1212.
Bouthemy, P., Hardouin, Ch., Piriou, G., & Yao, J.-F. (2006). Mixed-state auto-models and motion texture modeling. Journal of Mathematical Imaging and Vision, 25(3), 387–402.
Bugeau, A., & Pérez, P. (2007). Detection and segmentation of moving objects in highly dynamic scenes. In CVPR ’07: proc. of the 2007 IEEE conf. on computer vision and pattern recognition, Minneapolis, MI.
Caillol, H., Hillion, A., & Pieczynski, W. (1993). Fuzzy random fields and unsupervised image segmentation. IEEE Transactions on Geoscience and Remote Sensing, 31, 801–810.
Carincotte, C., Derrode, S., Sicot, G., & Boucher, J. M. (2004). Unsupervised image segmentation based on a new fuzzy hmc model. In ICASSP’04 (pp. 17–21).
Carincotte, C., Derrode, S., & Bourennane, S. (2006). Unsupervised change detection on sar images using fuzzy hidden Markov chains. IEEE Transactions on Geoscience and Remote Sensing, 44(2), 432–441.
Cernuschi-Frias, B. (2007). Mixed states Markov random fields with symbolic labels and multidimensional real values. Technical Report 6255, INRIA, July 2007.
Chellappa, R. (1985). Two-dimensional discrete Gaussian Markov random field models for image processing. In Progress in pattern recognition 2 (Vol. 85, pp. 79–112).
Chen, J., & Tang, C. (2007). Spatio-temporal Markov random field for video denoising. In Proc. IEEE conf. on comp. vision and pattern recognition (CVPR’07), June (pp. 1–8).
Collet, C., & Murtagh, F. (2004). Segmentation based on a hierarchical Markov model. Pattern Recognition, 37(12), 2337–2347.
Criminisi, A., Cross, G., Blake, A., & Kolmogorov, V. (2006). Bilayer segmentation of live video. In CVPR ’06: proceedings of the IEEE computer society conference on computer vision and pattern recognition (pp. 53–60). Washington: IEEE Computer Society.
Crivelli, T., Cernuschi-Frias, B., Bouthemy, P., & Yao, J.-F. (2006). Mixed-state Markov random fields for motion texture modeling and segmentation. In Proc. IEEE int. conf. on image processing (ICIP’06), Atlanta, USA (pp. 1857–1860).
Crivelli, T., Piriou, G., Bouthemy, P., Cernuschi-Frías, B., & Yao, J.-F. (2008). Simultaneous motion detection and background reconstruction with a mixed-state conditional Markov random field. In ECCV ’08: proceedings of the 10th European conference on computer vision, Marseille, France (pp. 113–126).
Crivelli, T., Bouthemy, P., Cernuschi-Frías, B., & Yao, J.-F. (2009). Learning mixed-state Markov models for statistical motion texture tracking. In MLVMA’09: 2nd IEEE int. workshop on machine learning for vision-based motion analysis, Kyoto, Japan.
Elfadel, I., & Picard, R. (1994). Gibbs random fields, cooccurrences, and texture modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(1), 24–37.
Elgammal, A. M., Harwood, D., & Davis, L. S. (2000). Non-parametric model for background subtraction. In ECCV ’00: proc. of the 6th European conf. on comp. vision-part II, London, UK (pp. 751–767).
Fablet, R., & Bouthemy, P. (2003). Motion recognition using non-parametric image motion models estimated from temporal and multiscale co-occurrence statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12), 1619–1624.
Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721–741.
Guyon, X. (1995). Random fields on a network: modeling, statistics and applications. New York: Springer.
Hardouin, C., & Yao, J. (2008). Multi-parameter auto-models and their applications. Biometrika, 95(2), 335–349.
Heitz, F., & Bouthemy, P. (1993). Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(12), 1217–1232.
Jojic, N., & Frey, B. J. (2001). Learning flexible sprites in video layers. In IEEE int. conference on computer vision and pattern recognition 2001 (pp. 199–206).
Kasetkasem, T., & Varshney, P. K. (2002). An image change detection algorithm based on Markov random field models. IEEE Transactions on Geoscience and Remote Sensing, 40(8), 1815–1823.
Ko, T., Soatto, S., & Estrin, D. (2008). Background subtraction on distributions. In ECCV08 (pp. 276–289).
Koller, D., Lerner, U., & Angelov, D. (1999). A general algorithm for approximate inference and its application to hybrid Bayes nets. In Proc. of the fifteenth conference on uncertainty in artificial intelligence, Stockholm, Sweden (pp. 324–333).
Kumar, S., & Hebert, M. (2006). Discriminative random fields. International Journal of Computer Vision, 68(2), 179–201.
Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. 18th international conf. on machine learning, Williamstown, MA, USA (pp. 282–289).
Li, Y., & Huttenlocher, D. P. (2008). Learning for optical flow using stochastic optimization. In ECCV ’08: proceedings of the 10th European conference on computer vision, Marseille, France (pp. 379–391).
Lorette, A., Descombes, X., & Zerubia, J. (2000). Texture analysis through a Markovian modelling and fuzzy classification: Application to urban area extraction from satellite images. International Journal of Computer Vision, 36(3), 221–236.
Lu, W. L., Murphy, K. P., Little, J. J., Sheffer, A., & Fu, H. B. (2009). A hybrid conditional random field for estimating the underlying ground surface from airborne lidar data. IEEE Transactions on Geoscience and Remote Sensing, 47(8), 2913–2922.
Migdal, J., & Grimson, W. E. (2005). Background subtraction using Markov thresholds. In WACV-MOTION ’05: proceedings of the IEEE workshop on motion and video computing (pp. 58–65). Washington: IEEE Computer Society.
Mittal, A., & Paragios, N. (2004). Motion-based background subtraction using adaptive kernel density estimation. In CVPR ’04: proc. of the 2004 IEEE conf. on computer vision and pattern recognition (Vol. 2, pp. 302–309).
Monnet, A., Mittal, A., Paragios, N., & Visvanathan, R. (2003). Background modeling and subtraction of dynamic scenes. In Proc. of the ninth IEEE int. conf. on computer vision (Vol. 2, pp. 1305–1312).
Murphy, K. P. (1999). A variational approximation for Bayesian networks with discrete and continuous latent variables. In Uncertainty in artificial intelligence (Vol. 15, pp. 457–466). San Mateo: Morgan Kaufmann.
Parag, T., Elgammal, A., & Mittal, A. (2006). A framework for feature selection for background subtraction. In CVPR ’06: proc. of the IEEE conf. on computer vision and pattern recognition, Washington, DC, USA (pp. 1916–1923).
Pieczynski, W., & Tebbache, A. (2000). Pairwise Markov random fields and segmentation of textured images. In Machine graphics and vision (pp. 705–718).
Salzenstein, F., & Collet, C. (2006). Fuzzy Markov random fields versus chains for multispectral image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1753–1767.
Salzenstein, F., & Pieczynski, W. (1997). Parameter estimation in hidden fuzzy Markov random fields and image segmentation. Graphical Models and Image Processing, 59(4), 205–220.
Sheikh, Y., & Shah, M. (2005). Bayesian modeling of dynamic scenes for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(11), 1778–1792.
Stauffer, C., & Grimson, W. E. L. (2000). Learning patterns of activity using real-time tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 747–757.
Sun, J., Zhang, W., Tang, X., & Shum, H. (2006). Background cut. In Proc. European conf. comp. vision, ECCV 2006 (pp. 628–641).
Veit, Th., Cao, F., & Bouthemy, P. (2006). An a contrario decision framework for region-based motion detection. International Journal of Computer Vision, 68(2), 163–178.
Wang, T., Li, J., Diao, Q., Hu, W., Zhang, Y., & Dulong, C. (2006). Semantic event detection using conditional random fields. In CVPRW ’06: proceedings of the conference on computer vision and pattern recognition workshop, Washington, DC, USA.
Wren, C. R., Azarbayejani, A., Darrell, T., & Pentland, A. P. (1997). Pfinder: real-time tracking of the human body. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 780–785.
Wright, J., Ganesh, A., Rao, S., & Ma, Y. (2009). Robust principal component analysis: exact recovery of corrupted low-rank matrices. In NIPS 2009. 0905.0233.
Wu, J., & Chung, A. C. S. (2007). A segmentation model using compound Markov random fields based on a boundary model. IEEE Transactions on Image Processing, 16(1), 241–252.
Zivkovic, Z., & van der Heijden, F. (2004). Recursive unsupervised learning of finite mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(5), 651–656.
Zivkovic, Z., & van der Heijden, F. (2006). Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters, 27(7), 773–780.
About this article
Cite this article
Crivelli, T., Bouthemy, P., Cernuschi-Frías, B. et al. Simultaneous Motion Detection and Background Reconstruction with a Conditional Mixed-State Markov Random Field. Int J Comput Vis 94, 295–316 (2011). https://doi.org/10.1007/s11263-011-0429-z
- Motion detection
- Background reconstruction
- Mixed-state Markov models
- Conditional random fields