G-LBM: Generative Low-Dimensional Background Model Estimation from Video Sequences

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12357)


In this paper, we propose a computationally tractable and theoretically supported non-linear low-dimensional generative model to represent real-world data in the presence of noise and sparse outliers. The non-linear low-dimensional manifold discovery of data is done through describing a joint distribution over observations, and their low-dimensional representations (i.e. manifold coordinates). Our model, called generative low-dimensional background model (G-LBM) admits variational operations on the distribution of the manifold coordinates and simultaneously generates a low-rank structure of the latent manifold given the data. Therefore, our probabilistic model contains the intuition of the non-probabilistic low-dimensional manifold learning. G-LBM selects the intrinsic dimensionality of the underling manifold of the observations, and its probabilistic nature models the noise in the observation data. G-LBM has direct application in the background scenes model estimation from video sequences and we have evaluated its performance on SBMnet-2016 and BMC2012 datasets, where it achieved a performance higher or comparable to other state-of-the-art methods while being agnostic to different scenes. Besides, in challenges such as camera jitter and background motion, G-LBM is able to robustly estimate the background by effectively modeling the uncertainties in video observations in these scenarios. (The code and models are available at:


Background estimation Foreground segmentation Non-linear manifold learning Deep neural network Variational auto-encoding 

Supplementary material (6.9 mb)
Supplementary material 1 (zip 7017 KB)


  1. 1.
    Bianco, S., Ciocca, G., Schettini, R.: Combination of video change detection algorithms by genetic programming. IEEE Trans. Evol. Comput. 21(6), 914–928 (2017)CrossRefGoogle Scholar
  2. 2.
    Blei, D.M., Kucukelbir, A., McAuliffe, J.D.: Variational Inference: A Review for Statisticians. ArXiv e-prints, January 2016Google Scholar
  3. 3.
    Bouwmans, T., Garcia-Garcia, B.: Background subtraction in real applications: challenges, current models and future directions. arXiv preprint arXiv:1901.03577 (2019)
  4. 4.
    Bouwmans, T., Javed, S., Sultana, M., Jung, S.K.: Deep neural network concepts for background subtraction: a systematic review and comparative evaluation. Neural Netw. 117, 8–66 (2019)CrossRefGoogle Scholar
  5. 5.
    Bouwmans, T., Zahzah, E.H.: Robust PCA via principal component pursuit: a review for a comparative evaluation in video surveillance. Comput. Vis. Image Underst. 122, 22–34 (2014)CrossRefGoogle Scholar
  6. 6.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM (JACM) 58(3), 1–37 (2011)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Cao, X., Yang, L., Guo, X.: Total variation regularized RPCA for irregularly moving object detection under dynamic background. IEEE Trans. Cybern. 46(4), 1014–1027 (2015)CrossRefGoogle Scholar
  8. 8.
    De Gregorio, M., Giordano, M.: Background estimation by weightless neural networks. Pattern Recogn. Lett. 96, 55–65 (2017)CrossRefGoogle Scholar
  9. 9.
    Djerida, A., Zhao, Z., Zhao, J.: Robust background generation based on an effective frames selection method and an efficient background estimation procedure (FSBE). Sig. Process.: Image Commun. 78, 21–31 (2019)Google Scholar
  10. 10.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)Google Scholar
  11. 11.
    Farnoosh, A., Rezaei, B., Ostadabbas, S.: DeepPBM: deep probabilistic background model estimation from video sequences. arXiv preprint arXiv:1902.00820 (2019)
  12. 12.
    Haines, T.S., Xiang, T.: Background subtraction with Dirichlet process mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 36(4), 670–683 (2013)CrossRefGoogle Scholar
  13. 13.
    Halfaoui, I., Bouzaraa, F., Urfalioglu, O.: CNN-based initial background estimation. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 101–106. IEEE (2016)Google Scholar
  14. 14.
    He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.S.: Neural collaborative filtering. In: Proceedings of the 26th International Conference on World Wide Web, pp. 173–182 (2017)Google Scholar
  15. 15.
    Javed, S., Mahmood, A., Bouwmans, T., Jung, S.K.: Spatiotemporal low-rank modeling for complex scene background initialization. IEEE Trans. Circ. Syst. Video Technol. 28(6), 1315–1329 (2016)CrossRefGoogle Scholar
  16. 16.
    Javed, S., Mahmood, A., Bouwmans, T., Jung, S.K.: Background-foreground modeling based on spatiotemporal sparse subspace clustering. IEEE Trans. Image Process. 26(12), 5840–5854 (2017). Scholar
  17. 17.
    Javed, S., Narayanamurthy, P., Bouwmans, T., Vaswani, N.: Robust PCA and robust subspace tracking: a comparative evaluation. In: 2018 IEEE Statistical Signal Processing Workshop (SSP), pp. 836–840. IEEE (2018)Google Scholar
  18. 18.
    Jiang, S., Lu, X.: WeSamBE: a weight-sample-based method for background subtraction. IEEE Trans. Circ. Syst. Video Technol. 28(9), 2105–2115 (2017)CrossRefGoogle Scholar
  19. 19.
    Jodoin, P.M., Maddalena, L., Petrosino, A., Wang, Y.: Extensive benchmark and survey of modeling methods for scene background initialization. IEEE Trans. Image Process. 26(11), 5244–5256 (2017)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes. ArXiv e-prints, December 2013Google Scholar
  21. 21.
    Laugraud, B., Van Droogenbroeck, M.: Is a memoryless motion detection truly relevant for background generation with LaBGen? In: Blanc-Talon, J., Penne, R., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2017. Lecture Notes in Computer Science, vol. 10617, pp. 443–454. Springer, Cham (2017). Scholar
  22. 22.
    Lim, L.A., Keles, H.Y.: Foreground segmentation using convolutional neural networks for multiscale feature encoding. Pattern Recogn. Lett. 112, 256–262 (2018)CrossRefGoogle Scholar
  23. 23.
    Liu, X., Zhao, G., Yao, J., Qi, C.: Background subtraction based on low-rank and structured sparse decomposition. IEEE Trans. Image Process. 24(8), 2502–2514 (2015)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Maddalena, L., Petrosino, A.: Extracting a background image by a multi-modal scene background model. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 143–148. IEEE (2016)Google Scholar
  25. 25.
    Mondéjar-Guerra, V., Rouco, J., Novo, J., Ortega, M.: An end-to-end deep learning approach for simultaneous background modeling and subtraction. In: British Machine Vision Conference (BMVC), Cardiff (2019)Google Scholar
  26. 26.
    Mseddi, W.S., Jmal, M., Attia, R.: Real-time scene background initialization based on spatio-temporal neighborhood exploration. Multimedia Tools Appl. 78(6), 7289–7319 (2018). Scholar
  27. 27.
    Oreifej, O., Li, X., Shah, M.: Simultaneous video stabilization and moving object detection in turbulence. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 450–462 (2012)CrossRefGoogle Scholar
  28. 28.
    Ortego, D., SanMiguel, J.C., Martínez, J.M.: Rejection based multipath reconstruction for background estimation in SBMnet 2016 dataset. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 114–119. IEEE (2016)Google Scholar
  29. 29.
    Papadimitriou, C.H., Raghavan, P., Tamaki, H., Vempala, S.: Latent semantic indexing: a probabilistic analysis. J. Comput. Syst. Sci. 61(2), 217–235 (2000)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Pathak, D., Girshick, R., Dollár, P., Darrell, T., Hariharan, B.: Learning features by watching objects move. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2701–2710 (2017)Google Scholar
  31. 31.
    Qu, Z., Yu, S., Fu, M.: Motion background modeling based on context-encoder. In: 2016 Third International Conference on Artificial Intelligence and Pattern Recognition (AIPR), pp. 1–5. IEEE (2016)Google Scholar
  32. 32.
    Ramirez-Alonso, G., Ramirez-Quintana, J.A., Chacon-Murguia, M.I.: Temporal weighted learning model for background estimation with an automatic re-initialization stage and adaptive parameters update. Pattern Recogn. Lett. 96, 34–44 (2017)CrossRefGoogle Scholar
  33. 33.
    Rezaei, B., Huang, X., Yee, J.R., Ostadabbas, S.: Long-term non-contact tracking of caged rodents. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1952–1956. IEEE (2017)Google Scholar
  34. 34.
    Rezaei, B., Ostadabbas, S.: Background subtraction via fast robust matrix completion. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1871–1879 (2017)Google Scholar
  35. 35.
    Rezaei, B., Ostadabbas, S.: Moving object detection through robust matrix completion augmented with objectness. IEEE J. Sel. Top. Sig. Process. 12(6), 1313–1323 (2018)CrossRefGoogle Scholar
  36. 36.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds.) Medical Image Computing and Computer-Assisted Intervention – CCAI 2015, MICCAI 2015. Lecture Notes in Computer Science, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  37. 37.
    St-Charles, P.L., Bilodeau, G.A., Bergevin, R.: A self-adjusting approach to change detection based on background word consensus. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 990–997. IEEE (2015)Google Scholar
  38. 38.
    Sultana, M., Mahmood, A., Javed, S., Jung, S.K.: Unsupervised deep context prediction for background estimation and foreground segmentation. Machine Vis. Appl. 30(3), 375–395 (2018). Scholar
  39. 39.
    Tao, Y., Palasek, P., Ling, Z., Patras, I.: Background modelling based on generative UNet. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6. IEEE (2017)Google Scholar
  40. 40.
    Vacavant, A., Chateau, T., Wilhelm, A., Lequièvre, L.: A benchmark dataset for outdoor foreground/background extraction. In: Park, J.I., Kim, J. (eds.) ACCV 2012. Lecture Notes in Computer Science, vol. 7728, pp. 291–300. Springer, Heidelberg (2012). Scholar
  41. 41.
    Vaswani, N., Bouwmans, T., Javed, S., Narayanamurthy, P.: Robust subspace learning: robust PCA, robust subspace tracking, and robust subspace recovery. IEEE Sig. Process. Mag. 35(4), 32–55 (2018)CrossRefGoogle Scholar
  42. 42.
    Wang, Y., Jodoin, P.M., Porikli, F., Konrad, J., Benezeth, Y., Ishwar, P.: CDnet 2014: an expanded change detection benchmark dataset. In: Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition Workshops, pp. 387–394 (2014)Google Scholar
  43. 43.
    Wang, Y., Luo, Z., Jodoin, P.M.: Interactive deep learning method for segmenting moving objects. Pattern Recogn. Lett. 96, 66–75 (2017)CrossRefGoogle Scholar
  44. 44.
    Xin, B., Tian, Y., Wang, Y., Gao, W.: Background subtraction via generalized fused lasso foreground modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4676–4684 (2015)Google Scholar
  45. 45.
    Xu, P., Ye, M., Li, X., Liu, Q., Yang, Y., Ding, J.: Dynamic background learning through deep auto-encoder networks. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 107–116 (2014)Google Scholar
  46. 46.
    Xu, Z., Min, B., Cheung, R.C.: A robust background initialization algorithm with superpixel motion detection. Sig. Process.: Image Commun. 71, 1–12 (2019)Google Scholar
  47. 47.
    Xue, G., Song, L., Sun, J.: Foreground estimation based on linear regression model with fused sparsity on outliers. IEEE Trans. Circ. Syst. Video Technol. 23(8), 1346–1357 (2013)CrossRefGoogle Scholar
  48. 48.
    Yang, B., Lei, Y., Liu, J., Li, W.: Social collaborative filtering by trust. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1633–1647 (2016)CrossRefGoogle Scholar
  49. 49.
    Zheng, W., Wang, K., Wang, F.Y.: A novel background subtraction algorithm based on parallel vision and Bayesian GANs. Neurocomputing 394, 178–200 (2019)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Augmented Cognition Lab, Electrical and Computer Engineering DepartmentNortheastern UniversityBostonUSA

Personalised recommendations