Multimedia Tools and Applications

, Volume 78, Issue 2, pp 2045–2072 | Cite as

People search based on attributes description provided by an eyewitness for video surveillance applications

  • Mayssa FrikhaEmail author
  • Emna Fendri
  • Mohamed Hammami


People search based on attributes description presents a paramount task for several forensics and surveillance applications. The aim is to locate a suspect or to find a missing person in public areas. However, semantic attributes provide a natural interface for this system as they present human understandable properties. These features can cover the whole body characteristics by describing the worn bags, carried objects, clothes, accessories, etc. Detecting semantic attributes under uncontrolled acquisition conditions still remains a challenging task. Most of state-of-the-art approaches assume independence among attributes where each attribute classifier is trained independently based on low-level features extracted from training samples. In this paper, we propose a novel people search system based on attributes description that relies on several components. An interactive query verification algorithm is introduced to prevent search failure. In addition, an attribute classification method that relies on two steps is introduced. We start by selecting the most relevant features in attribute adaptive way. Then, we explored the interactions among attributes to predict a semantic trait by involving the independent attribute classifier and the other correlated attribute classifiers. Several experiments were conducted to validate the effectiveness of the proposed people search system on the challenging VIPeR, CUHK, and HDA+ datasets benchmark.


People search Attribute description Semantic attributes Interaction model Appearance style rule database Weighted appearance interaction graph 



  1. 1.
    Adjeroh D, Cao D, Piccirilli M, Ross A (2010) Predictability and correlation in human metrology. In: 2010 IEEE international workshop on information forensics and security, pp 1–6.
  2. 2.
    Agrawal R, Srikant R (1994) Fast algorithms for mining association rules in large databases. In: Proceedings of the 20th international conference on very large data bases, VLDB ’94. Morgan Kaufmann Publishers Inc., San Francisco, pp 487–499.
  3. 3.
    Almudhahka NY, Nixon MS, Hare JS (2016) Unconstrained human identification using comparative facial soft biometrics. In: 2016 IEEE 8th international conference on biometrics theory, applications and systems (BTAS). IEEE, pp 1–6Google Scholar
  4. 4.
    An L, Chen X, Kafai M, Yang S, Bhanu B (2013) Improving person re-identification by soft biometrics based reranking. In: 2013 seventh international conference on distributed smart cameras (ICDSC), pp 1–6.
  5. 5.
    Antol S, Zitnick CL, Parikh D (2014) Zero-shot learning via visual abstraction. Springer International Publishing, Cham, pp 401–416. Google Scholar
  6. 6.
    Antonie ML, Zaïane OR (2004) Mining positive and negative association rules: an approach for confined rules. In: European conference on principles of data mining and knowledge discovery. Springer, pp 27–38Google Scholar
  7. 7.
    Breiman L (2001) Random forests. Mach Learn 45 (1):5–32. CrossRefzbMATHGoogle Scholar
  8. 8.
    Caruana R, Niculescu-Mizil A (2006) An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd international conference on machine learning. ACM, pp 161–168Google Scholar
  9. 9.
    Chen BC, Chen YY, Kuo YH, Hsu WH (2013) Scalable face image retrieval using attribute-enhanced sparse codewords. IEEE Trans Multimedia 15(5):1163–1173. CrossRefGoogle Scholar
  10. 10.
    Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math Control Signals Syst 2(4):303–314. MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30Google Scholar
  12. 12.
    Denman S, Halstead M, Fookes C, Sridharan S (2017) Locating people in surveillance video using soft biometric traits. In: Handbook of biometrics for forensic science. Springer, pp 267–288Google Scholar
  13. 13.
    Dollar P, Wojek C, Schiele B, Perona P (2012) Pedestrian detection: an evaluation of the state of the art. IEEE Trans Pattern Anal Mach Intell 34(4):743–761CrossRefGoogle Scholar
  14. 14.
    Farhadi A, Endres I, Hoiem D, Forsyth D (2009) Describing objects by their attributes. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009, pp 1778–1785.
  15. 15.
    Feris R, Bobbitt R, Brown L, Pankanti S (2014) Attribute-based people search: lessons learnt from a practical surveillance system. In: Proceedings of international conference on multimedia retrieval, ICMR ’14. ACM, New York, pp 153:153–153:160.
  16. 16.
    Figueira D, Taiana M, Nambiar A, Nascimento J, Bernardino A (2014) The hda + data set for research on fully automated re-identification systems. In: European conference on computer vision. Springer, pp 241–255Google Scholar
  17. 17.
    Frikha M, Fendri E, Hammami M (2014) A new appearance signature for real time person re-identification. Springer International Publishing, Cham, pp 175–182. Google Scholar
  18. 18.
    Gray D, Brennan S, Tao H (2007) Evaluating appearance models for recognition, reacquisition, and tracking. In: Proceedings of IEEE international workshop on performance evaluation for tracking and surveillance (PETS), vol 3. CiteseerGoogle Scholar
  19. 19.
    Han J, Pauwels EJ, de Zeeuw PM, de With PH (2012) Employing a rgb-d sensor for real-time tracking of humans across multiple re-entries in a smart environment. IEEE Trans Consum Electron 58(2):255–263CrossRefGoogle Scholar
  20. 20. Online. Accessed 20 April 2016
  21. 21. Online. Accessed 19 Sept 2015
  22. 22.
    Jain AK, Farrokhnia F (1991) Unsupervised texture segmentation using gabor filters. Pattern Recogn 24(12):1167–1186. CrossRefGoogle Scholar
  23. 23.
    Kalantidis Y, Kennedy L, Li LJ (2013) Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos. In: Proceedings of the 3rd ACM conference on international conference on multimedia retrieval, ICMR ’13. ACM, New York, pp 105–112.
  24. 24.
    Kittler J, Hatef M, Duin RP, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239CrossRefGoogle Scholar
  25. 25.
    Kovashka A, Parikh D, Grauman K (2015) Whittlesearch: interactive image search with relative attribute feedback. Int J Comput Vis 115(2):185–210. MathSciNetCrossRefGoogle Scholar
  26. 26.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  27. 27.
    Kumar N, Berg A, Belhumeur PN, Nayar S (2011) Describable visual attributes for face verification and image search. IEEE Trans Pattern Anal Mach Intell 33(10):1962–1977. CrossRefGoogle Scholar
  28. 28.
    Lampert CH, Nickisch H, Harmeling S (2009) Learning to detect unseen object classes by between-class attribute transfer. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009, pp 951–958.
  29. 29.
    Lampert CH, Nickisch H, Harmeling S (2014) Attribute-based classification for zero-shot visual object categorization. IEEE Trans Pattern Anal Mach Intell 36 (3):453–465. CrossRefGoogle Scholar
  30. 30.
    Layne R, Hospedales TM, Gong S, Mary Q (2012) Person re-identification by attributes. In: BMVC, vol 2, p 8Google Scholar
  31. 31.
    Layne R, Hospedales TM, Gong S (2014) Attributes-based re-identification. Springer London, London, pp 93–117. Google Scholar
  32. 32.
    Li W, Wang X (2013) Locally aligned feature transforms across views. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3594–3601Google Scholar
  33. 33.
    Liao S, Hu Y, Zhu X, Li SZ (2015) Person re-identification by local maximal occurrence representation and metric learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2197–2206Google Scholar
  34. 34.
    Liu J, Liang C, Ye M, Wang Z, Yang Y, Han Z, Sun K (2015) Person re-identification via attribute confidence and saliency. Springer International Publishing, Cham, pp 591–600. Google Scholar
  35. 35.
    Lumini A, Nanni L (2017) Overview of the combination of biometric matchers. Information Fusion 33:71–85CrossRefGoogle Scholar
  36. 36.
    Martinel N, Micheloni C, Foresti GL (2016) A pool of multiple person re-identification experts. Pattern Recogn Lett 71:23–30CrossRefGoogle Scholar
  37. 37.
    Martínez AM, Kak AC (2001) Pca versus lda. IEEE Trans Pattern Anal Mach Intell 23(2):228–233CrossRefGoogle Scholar
  38. 38.
    Nambiar A, Bernardino A, Nascimento J (2015) Shape context for soft biometrics in person re-identification and database retrieval. Pattern Recogn Lett 68:297–305CrossRefGoogle Scholar
  39. 39.
    Nguyen NB, Nguyen VH, Duc TN, Le DD, Duong DA (2015) AttRel: an approach to person re-identification by exploiting attribute relationships. Springer International Publishing, Cham, pp 50–60. Google Scholar
  40. 40.
    Odone F, Barla A, Verri A (2005) Building kernels from binary strings for image matching. IEEE Trans Image Process 14(2):169–180MathSciNetCrossRefGoogle Scholar
  41. 41.
    Provost F, Fawcett T (2001) Robust classification for imprecise environments. Mach Learn 42(3):203–231CrossRefGoogle Scholar
  42. 42.
    Saghafi MA (2014) Review of person re-identification techniques. IET Comput Vis 8:455–474(19). CrossRefGoogle Scholar
  43. 43.
    Schmid C (2001) Constructing models for content-based image retrieval. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, 2001. CVPR 2001, vol 2, pp II–39–II–45.
  44. 44.
    Shih FY (2010) Image processing and pattern recognition: fundamentals and techniques. Wiley, New YorkCrossRefGoogle Scholar
  45. 45.
    Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1):11–32. CrossRefGoogle Scholar
  46. 46.
    Tan S, Zheng F, Liu L, Han J, Shao L (2016) Dense invariant feature based support vector ranking for cross-camera person re-identification. IEEE Trans Circuits Syst Video Technol 28:356–363CrossRefGoogle Scholar
  47. 47.
    Umeda T, Sun Y, Irie G, Sudo K, Kinebuchi T (2016) Attribute discovery for person re-identification. Springer International Publishing, Cham, pp 268–276. Google Scholar
  48. 48.
    Vaquero DA, Feris RS, Tran D, Brown L, Hampapur A, Turk M (2009) Attribute-based people search in surveillance environments. In: 2009 workshop on applications of computer vision (WACV), pp 1–8.
  49. 49.
    Vezzani R, Baltieri D, Cucchiara R (2013) People reidentification in surveillance and forensics: a survey. ACM Comput Surv 46(2):29:1–29:37. CrossRefGoogle Scholar
  50. 50.
    Wang X (2013) Intelligent multi-camera video surveillance: a review. Pattern Recogn Lett 34(1):3–19. Extracting Semantics from Multi-Spectrum VideoCrossRefGoogle Scholar
  51. 51.
    Ye M, Liang C, Wang Z, Leng Q, Chen J, Liu J (2015) Specific person retrieval via incomplete text description. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, ICMR ’15. ACM, New York, pp 547–550.
  52. 52.
    Zhu J, Liao S, Lei Z, Li SZ (2015) Improve pedestrian attribute classification by weighted interactions from other attributes. Springer International Publishing, Cham, pp 545–557. Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.MIRACL-Faculty of Economics and ManagementSfax UniversitySfaxTunisia
  2. 2.MIRACL-Faculty of SciencesSfax UniversitySfaxTunisia

Personalised recommendations