Spatiotemporal Background Subtraction Using Minimum Spanning Tree and Optical Flow

  • Mingliang Chen
  • Qingxiong Yang
  • Qing Li
  • Gang Wang
  • Ming-Hsuan Yang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8695)


Background modeling and subtraction is a fundamental research topic in computer vision. Pixel-level background model uses a Gaussian mixture model (GMM) or kernel density estimation to represent the distribution of each pixel value. Each pixel will be process independently and thus is very efficient. However, it is not robust to noise due to sudden illumination changes. Region-based background model uses local texture information around a pixel to suppress the noise but is vulnerable to periodic changes of pixel values and is relatively slow. A straightforward combination of the two cannot maintain the advantages of the two. This paper proposes a real-time integration based on robust estimator. Recent efficient minimum spanning tree based aggregation technique is used to enable robust estimators like M-smoother to run in real time and effectively suppress the noisy background estimates obtained from Gaussian mixture models. The refined background estimates are then used to update the Gaussian mixture models at each pixel location. Additionally, optical flow estimation can be used to track the foreground pixels and integrated with a temporal M-smoother to ensure temporally-consistent background subtraction. The experimental results are evaluated on both synthetic and real-world benchmarks, showing that our algorithm is the top performer.


Background Modeling Video Segmentation Tracking Optical Flow 


  1. 1.
    Bao, L., Song, Y., Yang, Q., Yuan, H., Wang, G.: Tree filtering: Efficient structure-preserving smoothing with a minimum spanning tree. IEEE Transactions on Image Processing (2014)Google Scholar
  2. 2.
    Bao, L., Yang, Q., Jin, H.: Fast edge-preserving patchmatch for large displacement optical flow. In: CVPR (2014)Google Scholar
  3. 3.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patchmatch: a randomized correspondence algorithm for structural image editing. TOG (2009)Google Scholar
  4. 4.
    Barnich, O., Droogenbroeck, M.V.: Vibe: A powerful random technique to estimate the background in video sequences. In: IEEE Int. Conf. on Acoustics, Speech and Signal Processing (2009)Google Scholar
  5. 5.
    Brutzer, S., Hoferlin, B., Heidemann, G.: Evaluation of background subtraction techniques for video surveillance. In: CVPR (2011)Google Scholar
  6. 6.
    Chu, C., Glad, I., Godtliebsen, F., Marron, J.: Edgepreserving smoothers for image processing. Journal of the American Statistical Association (1998)Google Scholar
  7. 7.
    Comaniciu, D., Zhu, Y., Davis, L.: Sequential kernel density approximation and its application to real-time visual tracking. PAMI 30(7), 1186–1197 (2008)CrossRefGoogle Scholar
  8. 8.
    Elgammal, A.M., Duraiswami, R., Harwood, D., Davis, L.S.: Background and foreground modeling using non-parametric kernel density estimation for visual surveillance. Proceedings of the IEEE (2002)Google Scholar
  9. 9.
    Elgammal, A., Harwood, D., Davis, L.: Non-parametric model for background subtraction. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 751–767. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  10. 10.
    Elqursh, A., Elgammal, A.: Online moving camera background subtraction. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 228–241. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  11. 11.
    Goyette, N., Jodoin, P.M., Porikli, F., Konrad, J., Ishwar, P.: A new change detection benchmark dataset. In: IEEE Workshop on Change Detection at CVPR (2012)Google Scholar
  12. 12.
    Han, B., Comaniciu, D., Davis, L.: Sequential kernel density approximation through mode propagation: Applications to background modeling. In: ACCV (2004)Google Scholar
  13. 13.
    Han, B., Davis, L.S.: Density-based multifeature background subtraction with support vector machine. PAMI 34(5), 1017–1023 (2012)CrossRefGoogle Scholar
  14. 14.
    Han, B., Davis, L.: Adaptive Background Modeling and Subtraction: A Density-Based Approach with Multiple Features. CRC Press (2010)Google Scholar
  15. 15.
    Heikkila, M., Pietikainen, M.: A texture-based method for modeling the background and detecting moving objects. PAMI 28(4), 657–662 (2006)CrossRefGoogle Scholar
  16. 16.
    Hu, T.: The maximum capacity route problem. Operations Research (1961)Google Scholar
  17. 17.
    Kim, K., Chalidabhongse, T., Harwood, D., Davis, L.: Real-time foreground-background segmentation using codebook model. Real-Time Imaging 11(3) (2005)Google Scholar
  18. 18.
    Li, L., Huang, W., Gu, I., Tian, Q.: Foreground object detection from videos containing complex background. In: Int. Conf. on Multimedia, pp. 2–10 (2003)Google Scholar
  19. 19.
    Maddalena, L., Petrosino, A.: A self-organizing approach to background subtraction for visual surveillance applications. IEEE Transactions on Image Processing 17(7) (2008)Google Scholar
  20. 20.
    McFarlane, N., Schofield, C.: Segmentation and tracking of piglets in images. Machine Vision and Applications 8(3), 187–193 (1995)CrossRefGoogle Scholar
  21. 21.
    McKenna, S.J., Jabri, S., Duric, Z., Rosenfeld, A., Wechsler, H.: Tracking groups of people. Computer Vision and Image Understanding 80(1) (2000)Google Scholar
  22. 22.
    Oliver, N., Rosario, B., Pentland, A.: A bayesian computer vision system for modeling human interactions. PAMI 22(8), 831 (2000)CrossRefGoogle Scholar
  23. 23.
    Parag, T., Elgammal, A.M., Mittal, A.: A framework for feature selection for background subtraction. In: CVPR (2006)Google Scholar
  24. 24.
    Pollack, M.: The maximum capacity through a network. Operations Research (1960)Google Scholar
  25. 25.
    Prati, A., Mikic, I., Trivedi, M., Cucchiara, R.: Detecting moving shadows: Formulation, algorithms and evaluation. PAMI 25(7), 918–924 (2003)CrossRefGoogle Scholar
  26. 26.
    Shimada, A., Nagahara, H., Taniguchi, R.: Background modeling based on bidirectional analysis. In: CVPR, pp. 1979–1986 (2013)Google Scholar
  27. 27.
    Stauffer, C., Grimson, E.: Adaptive background mixture models for real-time tracking. In: CVPR, pp. 2246–2252 (1999)Google Scholar
  28. 28.
    Tanaka, T., Shimada, A., Taniguchi, R.-I., Yamashita, T., Arita, D.: Towards robust object detection: Integrated background modeling based on spatio-temporal features. In: Zha, H., Taniguchi, R.-I., Maybank, S. (eds.) ACCV 2009, Part I. LNCS, vol. 5994, pp. 201–212. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  29. 29.
    Toyama, K., Krumm, J., Brumitt, B., Meyers, B.: Wallflower: Principle and practice of background maintenance. In: ICCV (1999)Google Scholar
  30. 30.
    Yang, Q.: A non-local cost aggregation method for stereo matching. In: CVPR, pp. 1402–1409 (2012)Google Scholar
  31. 31.
    Yoshinaga, S., Shimada, A., Nagahara, H., Taniguchi, R.-I.: Object detection using local difference patterns. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part IV. LNCS, vol. 6495, pp. 216–227. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  32. 32.
    Zhang, S., Yao, H., Liu, S.: Dynamic background modeling and subtraction using spatio-temporal local binary patterns. In: ICIP (2008)Google Scholar
  33. 33.
    Zivkovic, Z., Heijden, F.: Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters 27, 773–780 (2006)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Mingliang Chen
    • 1
    • 2
  • Qingxiong Yang
    • 1
    • 2
  • Qing Li
    • 1
    • 2
  • Gang Wang
    • 3
  • Ming-Hsuan Yang
    • 4
  1. 1.Department of Computer Science, Multimedia software Engineering Research Centre (MERC)City University of Hong KongHong KongChina
  2. 2.MERC-ShenzhenGuangdongChina
  3. 3.Nanyang Technological UniversitySingapore
  4. 4.University of CaliforniaMercedUSA

Personalised recommendations