Skip to main content
SpringerLink
Log in

Robust watch-list screening using dynamic ensembles of SVMs based on multiple face representations

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Still-to-video face recognition (FR) is an important function in video surveillance (VS), where faces captured over a network of video cameras are matched against reference stills of target individuals. Screening faces against a watch-list is a challenging VS application because the appearance of faces varies due to changing capture conditions and operational domains. The facial models used for matching may not be representative of faces captured with video cameras because they are typically designed a priori with only one reference still. In this paper, a multi-classifier framework is proposed for robust still-to-video FR based on multiple and diverse face representations of a single reference face still. During enrollment of a target individual, the single reference face still is modeled using an ensemble of SVM classifiers based on different patches and face descriptors. Multiple feature extraction techniques are applied to patches isolated in the reference still to generate a diverse SVM pool that provides robustness to common nuisance factors (e.g., variations in illumination and pose). The estimation of discriminant feature subsets, classifier parameters, decision thresholds, and ensemble fusion functions is achieved using the high-quality reference still and a large number of faces captured in lower-quality video of non-target individuals in the scene. During operations, the most competent subset of SVMs is dynamically selected according to capture conditions. Finally, a head-face tracker gradually regroups faces captured from different people appearing in a scene, while each individual-specific ensemble performs face matching. The accumulation of matching scores per face track leads to a robust spatiotemporal FR when accumulated ensemble scores surpass a detection threshold. Experimental results obtained with the Chokepoint and COX-S2V datasets show a significant improvement in performance w.r.t. reference systems, especially when individual-specific ensembles (1) are designed using exemplar-SVMs rather than one-class SVMs and (2) exploit score-level fusion of local SVMs (trained using features extracted from each patch), rather than using either decision-level or feature-level fusion with a global SVM (trained by concatenating features extracted from patches).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. A facial model is defined as either a set of samples extracted from one or more reference face images (stored in a gallery for a template matcher), or a set of classifier parameters estimated from reference samples (for a pattern classifier).

  2. http://itee.uq.edu.au/~uqywong6/chokepoint.html.

  3. http://vipl.ict.ac.cn/resources/datasets/cox-face-dataset/COX-S2V.

References

  1. Abate, A.F., Nappi, M., Riccio, D., Sabatino, G.: 2d and 3d face recognition: a survey. Pattern Recognit. Lett. 28(14), 1885–1906 (2007)

    Article  Google Scholar 

  2. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)

    Article  MATH  Google Scholar 

  3. Ahonen, T., Rahtu, E., Ojansivu, V., Heikkila, J.: Recognition of blurred faces using local phase quantization. In: 19th International Conference on Pattern Recognition, 2008. ICPR 2008, pp 1–4. IEEE (2008)

  4. Amira, A., Farrell, P.: An automatic face recognition system based on wavelet transforms. In: IEEE International Symposium on Circuits and Systems, 2005. ISCAS 2005, pp. 6252–6255. IEEE (2005)

  5. Barr, J.R., Bowyer, K.W., Flynn, P.J., Biswas, S.: Face recognition from video: a review. Int. J. Pattern Recognit. Artif. Intell. 26(05), 1266002 (2012)

    Article  MathSciNet  Google Scholar 

  6. Bashbaghi, S., Granger, E., Sabourin, R., Bilodeau, G.A.: Watch-list screening using ensembles based on multiple face representations. In: 2014 22nd International Conference on Pattern Recognition (ICPR), pp. 4489–4494 (2014)

  7. Batuwita, R., Palade, V.: Fsvm-cil: fuzzy support vector machines for class imbalance learning. IEEE Trans. Fuzzy Syst. 18(3), 558–571 (2010)

    Article  Google Scholar 

  8. Bengio, S., Mariéthoz, J.: Biometric person authentication is a multiple classifier problem. In: Haindl, M., Kittler, J., Roli, F. (eds.) Multiple Classifier Systems: Proceedings of the 7th International Workshop, MCS 2007, Prague, Czech Republic, May 23–25, 2007, pp. 513–522. Springer, Berlin, Heidelberg (2007). doi:10.1007/978-3-540-72523-7_51

  9. Bereta, M., Pedrycz, W., Reformat, M.: Local descriptors and similarity measures for frontal face recognition: a comparative analysis. J. Vis. Commun. Image Represent. 24(8), 1213–1231 (2013)

    Article  Google Scholar 

  10. Best-Rowden, L., Klare, B., Klontz, J., Jain, A.K.: Video-to-video face matching: establishing a baseline for unconstrained face recognition. In: 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 1–8. IEEE (2013)

  11. Beveridge, J.R., Phillips, P.J., Bolme, D.S., Draper, B.A., Givens, G.H., Lui, Y.M., Teli, M.N., Zhang, H., Scruggs, W.T., Bowyer, K.W., Flynn, P.J., Cheng, S.: The challenge of face recognition from digital point-and-shoot cameras. In: 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 1–8 (2013)

  12. Chang, C.C., Lin, C.J.: Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2(3), 27 (2011)

    Google Scholar 

  13. Chellappa, R., Sinha, P., Phillips, P.J.: Face recognition by computers and humans. Computer 43(2), 46–55 (2010)

    Article  Google Scholar 

  14. Connaughton, R., Bowyer, K.W., Flynn, P.J.: Fusion of face and iris biometrics. In: Burge, M.J., Bowyer, K.W. (eds.) Handbook of Iris Recognition, pp. 219–237. Springer, London (2013). doi:10.1007/978-1-4471-4402-1_12

    Chapter  Google Scholar 

  15. Deng, W., Hu, J., Guo, J.: Extended src: undersampled face recognition via intraclass variant dictionary. IEEE Trans. PAMI 34(9), 1864–1870 (2012)

    Article  Google Scholar 

  16. Déniz, O., Bueno, G., Salido, J., De la Torre, F.: Face recognition using histograms of oriented gradients. Pattern Recognit. Lett. 32(12), 1598–1603 (2011)

    Article  Google Scholar 

  17. Dreuw, P., Steingrube, P., Hanselmann, H., Ney, H.: SURF-Face Face recognition under viewpoint consistency constraints. In: Cavallaro, A., Prince, S., Alexander, D. (eds) Proceedings of the British Machine Vision Conference, pp. 7.1–7.11. BMVA Press, London (2009). doi:10.5244/C.23.7

  18. Ekenel, H.K., Stallkamp, J., Stiefelhagen, R.: A video-based door monitoring system using local appearance-based face models. Comput. Vis. Image Underst. 114(5), 596–608 (2010)

    Article  Google Scholar 

  19. Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., Herrera, F.: A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 42(4), 463–484 (2012)

    Article  Google Scholar 

  20. Granger, E., Khreich, W., Sabourin, R., Gorodnichy, D.O.: Fusion of biometric systems using boolean combination: an application to iris-based authentication. Int. J. Biom. 4(3), 291–315 (2012)

    Article  Google Scholar 

  21. He, X., Yan, S., Hu, Y., Zhang, H.J.: Learning a locality preserving subspace for visual recognition. In: Proceedings Ninth IEEE International Conference on Computer Vision, 2003, pp. 385–392. IEEE (2003)

  22. Huang, Z., Shan, S., Zhang, H., Lao, S., Kuerban, A., Chen, X.: Benchmarking still-to-video face recognition via partial and local linear discriminant analysis on cox-s2v dataset. In: Computer Vision—ACCV 2012, pp. 589–600. Springer (2013)

  23. Huang, Z., Zhao, X., Shan, S., Wang, R., Chen, X.: Coupling alignments with recognition for still-to-video face recognition. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 3296–3303. IEEE (2013)

  24. Imam, T., Ting, K.M., Kamruzzaman, J.: z-SVM: An SVM for improved classification of imbalanced data. In: Sattar, A., Kang, B. (eds.) AI 2006: Advances in Artificial Intelligence: Proceedings of the19th Australian Joint Conference on Artificial Intelligence, Hobart, Australia, December 4–8, 2006, pp. 264–273. Springer, Berlin, Heidelberg (2006). doi:10.1007/11941439_30

  25. Jain, A.K., Ross, A., Prabhakar, S.: An introduction to biometric recognition. IEEE Trans. Circuits Syst. Video Technol. 14(1), 4–20 (2004)

    Article  Google Scholar 

  26. Juneja, M., Vedaldi, A., Jawahar, C., Zisserman, A.: Blocks that shout: Distinctive parts for scene classification. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 923–930 (2013)

  27. Kamgar-Parsi, B., Lawson, W.: Toward development of a face recognition system for watchlist surveillance. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 1925–1937 (2011)

    Article  Google Scholar 

  28. Kan, M., Shan, S., Su, Y., Xu, D., Chen, X.: Adaptive discriminant learning for face recognition. Pattern Recognit. 46(9), 2497–2509 (2013)

    Article  Google Scholar 

  29. Kemmler, M., Rodner, E., Wacker, E.S., Denzler, J.: One-class classification with gaussian processes. Pattern Recognit. 46(12), 3507–3518 (2013)

    Article  Google Scholar 

  30. Klare, B.F., Klein, B., Taborsky, E., Blanton, A., Cheney, J., Allen, K., Grother, P., Mah, A., Burge, M., Jain, A.K.: Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1931–1939 (2015)

  31. Krawczyk, B., Wozniak, M.: Diversity measures for one-class classifier ensembles. Neurocomputing 126, 36–44 (2014)

    Article  Google Scholar 

  32. Li, Q., Yang, B., Li, Y., Deng, N., Jing, L.: Constructing support vector machine ensemble with segmentation for imbalanced datasets. Neural Comput. Appl. 22(1), 249–256 (2013)

    Article  Google Scholar 

  33. Li, Y., Shen, W., Shi, X., Zhang, Z.: Ensemble of randomized linear discriminant analysis for face recognition with single sample per person. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2013)

  34. Liao, S., Jain, A.K., Li, S.Z.: Partial face recognition: alignment-free approach. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1193–1205 (2013)

    Article  Google Scholar 

  35. Liu, C., Wechsler, H.: Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans. Image process. 11(4), 467–476 (2002)

    Article  Google Scholar 

  36. Lu, J., Tan, Y.P., Wang, G.: Discriminative multimanifold analysis for face recognition from a single training sample per person. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 39–51 (2013)

    Article  Google Scholar 

  37. Malisiewicz, T., Gupta, A., Efros, A.A.: Ensemble of exemplar-svms for object detection and beyond. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 89–96. IEEE (2011)

  38. Misra, I., Shrivastava, A., Hebert, M.: Data-driven exemplar model selection. In: 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 339–346 (2014)

  39. Mokhayeri, F., Granger, E., Bilodeau, G.A.: Synthetic face generation under various operational conditions in video surveillance. In: ICIP (2015)

  40. Nourbakhsh, F., Granger, E., Fumera, G.: An extended sparse classification framework for domain adaptation in video surveillance. In: ACCV, Workshop on Human Identification for Surveillance (2016)

  41. Pagano, C., Granger, E., Sabourin, R., Marcialis, G., Roli, F.: Adaptive ensembles for face recognition in changing video surveillance environments. Inf. Sci. 286, 75–101 (2014)

    Article  Google Scholar 

  42. Pagano, C., Granger, E., Sabourin, R., Gorodnichy, D.O.: Detector ensembles for face recognition in video surveillance. In: The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2012)

  43. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)

    Article  Google Scholar 

  44. Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT press, Cambridge (2002)

    Google Scholar 

  45. Shaokang, C., Sandra, M., Mehrtash, T.H., Conrad, S., Abbas, B., Brian, C.L., et al.: Face recognition from still images to video sequences: a local-feature-based framework. EURASIP J. Image Video Process. 2011, 1 (2011)

    Google Scholar 

  46. Tan, X., Chen, S., Zhou, Z.H., Zhang, F.: Face recognition from a single image per person: a survey. Pattern Recognit. 39(9), 1725–1745 (2006)

    Article  MATH  Google Scholar 

  47. De la Torre Gomerra, M., Granger, E., Radtke, P.V., Sabourin, R., Gorodnichy, D.O.: Partially-supervised learning from facial trajectories for face recognition in video surveillance. Inf. Fusion 24, 31–53 (2015)

    Article  Google Scholar 

  48. Veropoulos, K., Campbell, C., Cristianini, N., et al.: Controlling the sensitivity of support vector machines. Proc. Int. Joint Conf. Artif. Intell. 1999, 55–60 (1999)

    Google Scholar 

  49. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  50. Wong, Y., Chen, S., Mau, S., Sanderson, C., Lovell, B.C.: Patch-based probabilistic image quality assessment for face selection and improved video-based face recognition. In: 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 74–81. IEEE (2011)

  51. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 210–227 (2009)

    Article  Google Scholar 

  52. Xie, C., Kumar, B.V.K.V., Palanivel, S., Yegnanarayana, B.: A still-to-video face verification system using advanced correlation filters. In: Zhang, D., Jain, A.K. (eds.) Biometric Authentication: Proceedings of the First International Conference, ICBA 2004, Hong Kong, China, July 15–17, 2004, pp. 102–108. Springer, Berlin, Heidelberg (2004). doi:10.1007/978-3-540-25948-0_15

  53. Yang, M., Van Gool, L., Zhang, L.: Sparse variation dictionary learning for face recognition with a single training sample per person. In: ICCV, pp. 689–696 (2013)

  54. Zeng, Z.-Q., Gao, J.: Improving SVM classification with imbalance data set. In: Leung, C.S., Lee, M. (eds.) Neural Information Processing: Proceedings of the16th International Conference, ICONIP 2009, Bangkok, Thailand, December 1–5, 2009, Part I, pp. 389–398. Springer, Berlin, Heidelberg (2009). doi:10.1007/978-3-642-10677-4_44

  55. Zhang, J., Yan, Y., Lades, M.: Face recognition: eigenface, elastic matching, and neural nets. Proc. IEEE 85(9), 1423–1435 (1997)

    Article  Google Scholar 

  56. Zhang, Y., Wang, D.: A cost-sensitive ensemble method for class-imbalanced datasets. Abstr. Appl. Anal. 2013 (2013)

  57. Zhang, Y., Martínez, A.M.: From stills to video: face recognition using a probabilistic approach. In: Conference on Computer Vision and Pattern Recognition Workshop, 2004. CVPRW’04, pp. 78–78. IEEE (2004)

  58. Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: a literature survey. Acm Comput. Surv. (CSUR) 35(4), 399–458 (2003)

    Article  Google Scholar 

  59. Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of human faces from video. Comput. Vis. Image Underst. 91(1), 214–245 (2003)

    Article  Google Scholar 

  60. Zong, W., Huang, G.B.: Face recognition based on extreme learning machine. Neurocomputing 74(16), 2541–2551 (2011)

    Article  Google Scholar 

  61. Zou, J., Ji, Q., Nagy, G.: A comparative study of local matching approach for face recognition. IEEE Trans. Image Process. 16(10), 2617–2628 (2007)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by the Fonds de recherche du Québec - Nature et technologies.

Author information

Authors and Affiliations

  1. Laboratoire d’imagerie de vision et d’intelligence artificielle, École de technologie supérieure, Université du Québec, Montreal, Canada

    Saman Bashbaghi, Eric Granger & Robert Sabourin

  2. Laboratoire d’interprétation et de traitement d’images et vidéo, Polytechnique Montréal, Montreal, Canada

    Guillaume-Alexandre Bilodeau

Authors
  1. Saman Bashbaghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Granger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Robert Sabourin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guillaume-Alexandre Bilodeau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saman Bashbaghi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bashbaghi, S., Granger, E., Sabourin, R. et al. Robust watch-list screening using dynamic ensembles of SVMs based on multiple face representations. Machine Vision and Applications 28, 219–241 (2017). https://doi.org/10.1007/s00138-016-0820-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-016-0820-4

Keywords

Search

Navigation