Short-Term Person Re-identification Using RGB, Depth and Skeleton Information of RGB-D Sensors

Abstract

Person re-identification is to search for a correct match for a person of interest across different camera views among a large number of impostors. While the approaches based on RGB modality have been widely studied to re-identify people, other modalities could be exploited as additional information sources, like depth and skeleton modalities. In this paper, we perform multi- and single-modal person re-identification using RGB, depth and skeleton modalities obtained by RGB-D sensors. First of all, the depth and RGB images are divided into three regions of head, torso and legs. Then, each region is explained by the histograms of local vector patterns (LVP). In depth and RGB modalities, the depth values of pixels and the gray levels of pixels are used for extracting LVPs, respectively. The skeleton features are extracted by computing the various Euclidean distances for the joint points of skeleton images. Then, features extracted by different modalities are combined as double and triple combinations using score-level fusion. The experiments are evaluated on two RGBD-ID and KinectREID databases, and results illustrate the acceptable performance of the proposed method.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Barbosa BI, Cristani M, Bue AD, Bazzani L, Murino V (2012) Re-identification with RGB-D sensors. In: 1st international workshop on re-identification, pp 433–442

  2. Bazzani L, Cristani M, Perina A, Murino V (2012) Multiple-shot person re-identification by chromatic and epitomic analyses. Pattern Recognit Lett 33:898–903

    Article  Google Scholar 

  3. Bazzani L, Cristani M, Murino V (2013) Symmetry-driven accumulation of local features for human characterization and re-identification. Comput Vis Image Underst 117:130–144

    Article  Google Scholar 

  4. Chahla C, Snoussi H, Abdallah F, Dornaika F (2017) Discriminant quaternion local binary pattern embedding for person reidentification through prototype formation and color categorization. Eng Appl Artif Intell 58:27–33

    Article  Google Scholar 

  5. Doretto G, Sebastian Th, Tu P, Rittscher J (2011) Appearance-based person reidentification in camera networks: problem overview and current approaches. J Ambient Intell Hum Comput 2:127–151

    Article  Google Scholar 

  6. Fan KCh, Hung TY (2014) A novel local pattern descriptor—local vector pattern in high-order derivative space for face recognition. IEEE Trans Image Process 23:2877–2891

    MathSciNet  Article  Google Scholar 

  7. Fang W, Hu HM, Hu Z, Liao Sh, Li B (2017) Perceptual hash-based feature description for person re-identification. Neurocomput. https://doi.org/10.1016/j.neucom.2017.07.019

    Article  Google Scholar 

  8. Farenzena M, Bazzani L, Perina A, Murino V, Cristani M (2013) Symmetry-driven accumulation of local features for human characterization and re-identification. Comput Vis Image Underst 117:130–144

    Article  Google Scholar 

  9. Fathi A, Nilchi AN (2012) Noise tolerant local binary pattern operator for efficient texture analysis. Pattern Recognit Lett 33:1093–1100

    Article  Google Scholar 

  10. Forssen PE (2007) Maximally stable colour regions for recognition and matching. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1–8

  11. Gala AB, Shah ShK (2014) A survey of approaches and trends in person re-identification. Image Vis Comput 32:270–286

    Article  Google Scholar 

  12. Gala AB, Shah ShK (2011) Multiple person re-identification using part based spatio-temporal color appearance model. In: IEEE International conference on computer vision workshops (ICCV Workshops), pp 1721–1728

  13. Guo Z, Zhang L, Zhang D (2010a) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19:1657–1663

    MathSciNet  Article  Google Scholar 

  14. Guo Z, Zhang L, Zhang D (2010b) Rotation invariant texture classification using LBP variance (LBPV) with global matching. Pattern Recognit 43:706–719

    Article  Google Scholar 

  15. Hadid A (2008) The local binary pattern approach and its applications to face analysis. In: 1st Workshop on image processing theory, tools and applications (IPTA), pp 1–9

  16. Huang X, Li SZ, Wang Y (2004) Shape localization based on statistical method using extended local binary pattern. In: Proceedings of the 3rd international conference on image and graphics (ICIG), pp 184–187

  17. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C 41:765–781

    Article  Google Scholar 

  18. Imani Z, Soltanizadeh H (2016) Person reidentification using local pattern descriptors and anthropometric measures from videos of kinect sensor. IEEE Sensors J 16:6227–6238

    Article  Google Scholar 

  19. Imani Z, Soltanizadeh H (2018) Histogram of the node strength and histogram of the edge weight: two new features for RGB-D person re-identification. Sci China Inf Sci 61:092108. https://doi.org/10.1007/s11432-016-9086-8

    Article  Google Scholar 

  20. Imani Z, Soltanizadeh H (2019) Local binary pattern, local derivative pattern and skeleton features for RGB-D person re-identification. Natl Acad Sci Lett 42:233–238. https://doi.org/10.1007/s40009-018-0736-9

    Article  Google Scholar 

  21. Jin H, Liu Q, Lu H, Tong X (2004) Face detection using improved LBP under Bayesian framework. In: Proceedings of the 3rd international conference on image and graphics, pp 306–309

  22. Krucki K, Asari V, Borel-Donohue C, Bunker D (2014) Human re-identification in multi-camera systems. In: IEEE applied imagery pattern recognition workshop (AIPR), pp 1–7

  23. Layne R, Hospedales T, Gong S (2012) Person re-identification by attributes. In: Proceedings of the British machine vision conference (BMVC)

  24. Liu C, Gong S, Loy CC, Lin X (2012) Person re-identification: What features are important? In: Proceedings of the 1st international workshop on re-identification, pp 391–401

  25. Liu X, Song M, Tao D, Zhou X, Chen C, Bu J (2014) Semi-supervised coupled dictionary learning for person re-identification. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 3550–3557

  26. Lowe DG (2004) Distinctive image features from scale-invariant key-points. Int J Comput Vis 60:91–110

    Article  Google Scholar 

  27. Ma B, Su Y, Jurie F (2014) Covariance descriptor based on bio-inspired features for person re-identification and face verification. Image Vis Comput 32:379–390

    Article  Google Scholar 

  28. Martinel N, Micheloni C (2012) Re-identify people in wide area camera network. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), RI, USA, pp 31–36

  29. Munaro M, Fossati A, Basso A, Menegatti E, Gool LV (2014) One-shot person re-identification with a consumer depth camera. In: Advances in computer vision and pattern recognition. Springer, Berlin, pp 161–181

  30. Murala S, Wu QMJ (2015) Spherical symmetric 3D local ternary patterns for natural, texture and biomedical image indexing and retrieval. Neurocomput 149:1502–1514

    Article  Google Scholar 

  31. Murala S, Maheshwari RP, Balasubramanian R (2012) Local tetra patterns: a new feature descriptor for content-based image retrieval. IEEE Trans Image Process 21:2874–2886

    MathSciNet  Article  Google Scholar 

  32. Nanni L, Lumini A, Brahnam S (2010) Local binary patterns variants as texture descriptors for medical image analysis. Artif Intell Med 49:117–125

    Article  Google Scholar 

  33. Nanni L, Brahnam S, Lumini A (2012) A simple method for improving local binary patterns by considering non-uniform patterns. Pattern Recognit 45:3844–3852

    Article  Google Scholar 

  34. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recognit 29:51–59

    Article  Google Scholar 

  35. Oliveira IO, Pio JL (2009) People re-identification in a camera network. In: IEEE international conference on dependable, autonomic and secure computing (DASC), Chengdu, China, pp 461–466

  36. Pala F, Satta R, Fumera G, Roli F (2015) Multi-modal person re-identification using RGB-D cameras. IEEE Trans Circuit Syst Video Tech 26:788–799

    Article  Google Scholar 

  37. Prosser B, Zheng WS, Gong S, Xiang T (2010) Person re-identification by support vector ranking. In: Proceedings of the British machine vision conference (BMVA), pp 21.1–21.11

  38. Ross A, Jain A (2003) Information fusion in biometrics. Pattern Recognit Let 24:2115–2125

    Article  Google Scholar 

  39. Schwartz WR, Davis LS (2009) Learning discriminative appearance-based models using partial least squares. In: 22nd Brazilian symposium on computer graphics and image processing (SIBGRAPI), pp 322–329

  40. Sun N, Zheng W, Sun C, Zou C, Zhao L (2006) Gender classification based on boosting local binary pattern. In: Proceedings of 3rd international symposium on neural networks, pp 194–201

  41. Tao D, Guo Y, Song M, Li Y, Yu Zh, Tang YY (2016) Person re-identification by dual-regularized KISS metric learning. IEEE Trans Image Process 25:2726–2738

    MathSciNet  Article  Google Scholar 

  42. Vezzani R, Baltieri D, Cucchiara R (2014) People reidentification in surveillance and forensics: a survey. ACM Comput Surveys 46:29

    Google Scholar 

  43. Wu L, Shen C, Hengel A (2017) Deep linear discriminant analysis on Fisher networks: a hybrid architecture for person re-identification. Pattern Recognit 65:238–250

    Article  Google Scholar 

  44. Xiang ZJ, Chen Q, Liu Y (2014) Person re-identification by fuzzy space color histogram. Multimed Tools Appl 73:91–107

    Article  Google Scholar 

  45. Xiong F, Gou M, Camps O, Sznaier M (2014) Person re-identification using kernel-based metric learning methods. In: 13th European conference on computer vision (ECCV), pp 1–16

  46. Yang B, Chen S (2013) A comparative study on local binary pattern (LBP) based face recognition: LBP histogram versus LBP image. Neurocomput 120:365–379

    Article  Google Scholar 

  47. Zhang Y, Li S (2011) Gabor-lbp based region covariance descriptor for person re-identification. In: Sixth International Conference on Image and Graphics (ICIG), pp 368–371

  48. Zhang B, Gao Y, Zhao S, Liu J (2010) Local derivative pattern versus local binary pattern: face recognition with high order local pattern descriptor. IEEE Trans Image Process 19:533–544

    MathSciNet  Article  Google Scholar 

  49. Zhao R, Ouyang W, Wang X (2013) Unsupervised salience learning for person re-identification. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 3586–3593

  50. Zhao Y, Zhao X, Luo R, Liu Y (2015) Person Re-identification by encoding free energy feature maps. Multimed Tools Appl 75:4795–4813

    Article  Google Scholar 

  51. Zheng W, Gong S, Xiang T (2011) Person re-identification by probabilistic relative distance comparison. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 649–656

  52. Zheng W, Gong S, Xiang T (2013) Reidentification by relative distance comparison. IEEE Trans Pattern Anal Mach Intell 35:653–668

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Hadi Soltanizadeh.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Imani, Z., Soltanizadeh, H. & Orouji, A.A. Short-Term Person Re-identification Using RGB, Depth and Skeleton Information of RGB-D Sensors. Iran J Sci Technol Trans Electr Eng 44, 669–681 (2020). https://doi.org/10.1007/s40998-019-00249-9

Download citation

Keywords

  • Person re-identification
  • LVP
  • Skeleton
  • Score-level fusion