Applied Intelligence

, Volume 49, Issue 5, pp 1771–1784 | Cite as

Challenging situations for background subtraction algorithms

  • Silvio R. R. SanchesEmail author
  • Claiton Oliveira
  • Antonio C. Sementille
  • Valdinei Freire


Background subtraction is the prerequisite for a wide range of applications including video surveillance, smart environments and content retrieval. Real environments present some challenging situations even for the most recent algorithms, such as shadows, illumination changes, dynamic background, among others. If a real environment is previously known and the challenging situations of this environment can be predicted, the choice of an appropriate algorithm to deal with such situations may be essential for obtaining better segmentation results. In our work, we identify the main situations that affect the performance of background subtraction algorithms and present a classification of these challenging situations. In addition, we present a solution that uses videos and ground-truths from existing datasets to evaluate the performance of segmentation algorithms when they need to deal with a specific challenging situation.


Background subtraction Foreground extraction Algorithm evaluation Challenging situation 



  1. 1.
    Babaee M, Dinh DT, Rigoll G (2018) A deep convolutional neural network for video sequence background subtraction. Pattern Recog 76:635–649. CrossRefGoogle Scholar
  2. 2.
    Badura S, Lieskovsky A, Mokrys M (2011) With shadow elimination towards effective foreground extraction. In: 2011 IEEE international symposium on signal processing and information technology (ISSPIT). IEEE, pp 404–408.
  3. 3.
    Barnich O, Droogenbroeck MV (2011) Vibe: a universal background subtraction algorithm for video sequences. IEEE Trans Image Process 20 (6):1709–1724. MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Berjón D, Cuevas C, Morán F, García N (2018) Real-time nonparametric background subtraction with tracking-based foreground update. Pattern Recog 74:156–170. CrossRefGoogle Scholar
  5. 5.
    Bianco S, Ciocca G, Schettini R (2017) Combination of video change detection algorithms by genetic programming. IEEE Trans Evol Comput 21(6):914–928. CrossRefGoogle Scholar
  6. 6.
    Bloisi D, Iocchi L (2012) Independent multimodal background subtraction. In: Proceedings of the third international conference on computational modeling of objects presented in images: fundamentals, methods and applications. Rome, Italy., pp 39–44
  7. 7.
    Bloisi DD, Pennisi A, Iocchi L (2017) Parallel multi-modal background modeling. Pattern Recogn Lett 96:45–54. CrossRefGoogle Scholar
  8. 8.
    Bouwmans T, Maddalena L, Petrosino A (2017) Scene background modeling and initialization: a taxonomy. Pattern Recog Lett 96:3–11. CrossRefGoogle Scholar
  9. 9.
    Brutzer S, Höferlin B, Heidemann G (2011) Evaluation of background subtraction techniques for video surveillance. In: CVPR 2011, pp 1937–1944.
  10. 10.
    Calderara S, Prati A, Cucchiara R (2008) HECOL: Homography And epipolar-based consistent labeling for outdoor park surveillance. Comput Vis Image Underst 111(1):21–42. CrossRefGoogle Scholar
  11. 11.
    Dey B, Kundu MK (2015) Efficient foreground extraction from HEVC compressed video for application to real-time analysis of surveillance ‘Big’ data. IEEE Trans Image Process 24(11):3574–3585. MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Dou J, Li J (2014) Robust human action recognition based on spatio-temporal descriptors and motion temporal templates. Optik - Int J Light Electron Opt 125(7):1891–1896. CrossRefGoogle Scholar
  13. 13.
    Fisher R (2016) CAVIAR test case scenariosGoogle Scholar
  14. 14.
    Gallego J, Pardàs M, Haro G (2012) Enhanced foreground segmentation and tracking combining bayesian background, shadow and foreground modeling. Pattern Recog Lett 33(12):1558–1568. CrossRefGoogle Scholar
  15. 15.
    Ge W, Dong Y, Guo Z, Chen Y (2014) Background subtraction with dynamic noise sampling and complementary learning. In: 2014 22nd international conference on pattern recognition. IEEE., pp 2341–2346
  16. 16.
    Ge W, Guo Z, Dong Y, Chen Y (2016) Dynamic background estimation and complementary learning for pixel-wise foreground/background segmentation. Pattern Recogn 59:112–125. CrossRefGoogle Scholar
  17. 17.
    Goyette N, Jodoin P, Porikli F, Konrad J, Ishwar P (2012) A new change detection benchmark dataset. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW)., pp 1–8
  18. 18.
    Gregorio M, Giordano M (2017) Wisardrp for change detection in video sequences. In: 25th European symposium on artificial neural networks, computational intelligence and machine learning, pp 453–458Google Scholar
  19. 19.
    Hofmann M, Tiefenbacher P, Rigoll G (2012) Background segmentation with feedback: the pixel-based adaptive segmenter. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops., pp 38–43
  20. 20.
    Hsiao HH, Leou JJ (2011) An effective foreground/background segmentation approach for bootstrapping video sequences. In: 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, Foreground Extraction., pp 1177–1180
  21. 21.
    i-Lids (2007) i-Lids dataset for AVSS 2007Google Scholar
  22. 22.
    Isik S, Ozkan K, Gunal S, Gerek ON (2018) Swcd: a sliding window and self-regulated learning-based background updating method for change detection in videos. J Electronic Imaging 27(2):27–27–11. CrossRefGoogle Scholar
  23. 23.
    Li J, Li X, Yang B, Sun X (2015) Segmentation-based image copy-move forgery detection scheme. IEEE Trans Inf Forensics Secur 10(3):507–518. CrossRefGoogle Scholar
  24. 24.
    Liu Y, Pados DA (2016) Compressed-Sensed-Domain L 1 -PCA video surveillance. IEEE Trans Multimedia 18(3):351–363. CrossRefGoogle Scholar
  25. 25.
    Mandellos NA, Keramitsoglou I, Kiranoudis CT (2011) A background subtraction algorithm for detecting and tracking vehicles. Expert Syst Appl 38(3):1619–1631. CrossRefGoogle Scholar
  26. 26.
    Liu N, Wu H, Lin L (2015) Hierarchical ensemble of background models for PTZ-based video surveillance. IEEE Trans Cybern 45(1):89–102. CrossRefGoogle Scholar
  27. 27.
    Noh S, Jeon M (2013) A new framework for background subtraction using multiple cues. In: Lee K M, Matsushita Y, Rehg J M, Hu Z (eds) Computer vision – ACCV, 2012. Springer, Berlin, pp 493–506Google Scholar
  28. 28.
    Qin L, Sheng B, Lin W, Wu W, Shen R (2015) GPU-accelerated Video Background subtraction using gabor detector. J Vis Commun Image Represent 32:1–9. CrossRefGoogle Scholar
  29. 29.
    Singh M, Nain N, Panwar S (2015) Foreground object extraction using thresholding with automatic shadow removal. In: 2015 11th international conference on signal-image technology & internet-based systems (SITIS), pp 655–662.
  30. 30.
    Sobral A, Bouwmans T (2014) Bgs library: a library framework for algorithm’s evaluation in foreground/background segmentation. In: Background modeling and foreground detection for video surveillance. CRC Press, Taylor and Francis GroupGoogle Scholar
  31. 31.
    Sobral A, Vacavant A (2014) A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos. Comput Vis Image Underst 122:4–21. CrossRefGoogle Scholar
  32. 32.
    St-Charles PL, Bilodeau GA (2014) Improving background subtraction using local binary similarity patterns. In: IEEE winter conference on applications of computer vision (WACV), 2014., pp 509–515
  33. 33.
    St-Charles PL, Bilodeau GA, Bergevin R (2015) A self-adjusting approach to change detection based on background word consensus. In: 2015 IEEE winter conference on applications of computer vision., pp 990–997
  34. 34.
    St-Charles PL, Bilodeau GA, Bergevin R (2015) Subsense: a universal change detection method with local adaptive sensitivity. IEEE Trans Image Process 24(1):359–373. MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Stauffer C, Grimson W (1999) Adaptive background mixture models for real-time tracking. In: Proceedings. 1999 IEEE computer society conference on computer vision and pattern recognition (cat. no PR00149), IEEE comput. Soc, vol 2, pp 246–252.
  36. 36.
    Thangarajah A, Wu Q J, Singh A, Mandon B, Chowdhury A (2015) Video foreground detection in non-static background using multi-dimensional color space. Procedia Computer Science 70:55–61. CrossRefGoogle Scholar
  37. 37.
    Toyama K, Krumm J, Brumitt B, Meyers B (1999) Wallflower: principles and practice of background maintenance. In: Proceedings of the seventh IEEE international conference on computer vision, vol 1, pp 255–261.
  38. 38.
    UK EPSRC REASON (2007) PETS 2007 benchmark dataGoogle Scholar
  39. 39.
    University of Reading (2001) PETS2001 DATASETSGoogle Scholar
  40. 40.
    Vacavant A, Chateau T, Wilhelm A, Lequièvre L (2013) A benchmark dataset for outdoor Foreground/Background extraction. Springer, Berlin, pp 291–300. Google Scholar
  41. 41.
    Wang J, Wang Y, Jiang M, Yan X, Song M (2014) Moving cast shadow detection using online sub-scene shadow modeling and object inner-edges analysis. J Vis Commun Image Represent 25(5):978–993. CrossRefGoogle Scholar
  42. 42.
    Wang K, Gou C, Wang FY (2018) m 4 c d : a robust change detection method for intelligent visual surveillance. IEEE Access 6:15505–15520. CrossRefGoogle Scholar
  43. 43.
    Wang L, Yin B, Guo A, Ma H, Cao J (2018) Skip-connection convolutional neural network for still image crowd counting. Appl Intell 48:3360–3371. CrossRefGoogle Scholar
  44. 44.
    Wang Y, Qi Y (2013) Memory-based cognitive modeling for robust object extraction and tracking. Appl Intell 39(3):614–629. CrossRefGoogle Scholar
  45. 45.
    Wang Y, Jodoin PM, Porikli F, Konrad J, Benezeth Y, Ishwar P (2014) CDnet 2014: an expanded change detection benchmark dataset. In: 2014 IEEE conference on computer vision and pattern recognition workshops. IEEE., pp 393–400
  46. 46.
    Wei W, Yang XL, Zhou B, Feng J, Shen PY (2012) Combined energy minimization for image reconstruction from few views. Math Probl Eng 2012:1–15. MathSciNetzbMATHGoogle Scholar
  47. 47.
    Wei W, Qiang Y, Zhang J (2013) A bijection between lattice-valued filters and lattice-valued congruences in residuated lattices. Mathematical Problems in Engineering 2013.
  48. 48.
    Wei W, Fan X, Song H, Wang H (2017) Video tamper detection based on multi-scale mutual information. Multimed Tools Appl 9:1–18. Google Scholar
  49. 49.
    Yang D, Zhao C, Zhang X, Huang S (2018) Background modeling by stability of adaptive features in complex scenes. IEEE Trans Image Process 27(3):1112–1125. MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Zhao Z, Bouwmans T, Zhang X, Fang Y (2012) Proceedings of second international conference multimedia and signal processing, Springer, Berlin, Heidelberg, chap a fuzzy background modeling approach for motion detection in dynamic backgrounds, pp 177–185.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Universidade Tecnológica Federal do ParanáCornélio ProcópioBrazil
  2. 2.Universidade Estadual Paulista “Júlio de Mesquita Filho”BauruBrazil
  3. 3.Universidade de São PauloSão PauloBrazil

Personalised recommendations