Skip to main content

Advertisement

SpringerLink
Log in

Video-based person re-identification using a novel feature extraction and fusion technique

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Person re-identification has received extensive attention in the academic community. In this paper, a novel multiple feature fusion network (MPFF-Net) is proposed for video-based person re-identification. The proposed network is used to obtain the robust and discriminative feature representation for describing the pedestrian in the video, which contains the hand-crafted and deep-learned parts. First, the image-level features of all consecutive frames are extracted. Then the hand-crafted branch uses these descriptors to obtain the average feature of the video and the information of frame-to-frame differences. The deep-learned branch is based on the bidirectional LSTM (BiLSTM) network. It is responsible for aggregating frame-wise representations of human regions and yielding sequence-level features. Furthermore, the problem of misalignment is taken into account in this branch. Finally, the hand-crafted and deep-learned parts are considered to be complementary, and the fusion of them can help to capture the complete information of the video. Extensive experiments are conducted on the iLIDS-VID, PRID2011 and MARS datasets. The results demonstrate that the proposed algorithm outperforms state-of-the-art video-based re-identification methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Bedagkar Gala A, Shah S (2014) Editor’s choice article: a survey of approaches and trends in person re-identification. Image Vis Comput 32(4):270–286

    Article  Google Scholar 

  2. Boulgouris N V, Hatzinakos D, Plataniotis K N (2005) Gait recognition: a challenging signal processing technology for biometric identification. Signal Process Mag IEEE 22(6):78–90

    Article  Google Scholar 

  3. Cheng D, Gong Y, Zhou S et al (2016) Person re-identification by multi-channel parts-based cnn with improved triplet loss function, pp 1335–1344

  4. Dai J, Zhang P, Wang D, Lu H, Wang H (2019) Video person re-identification by temporal residual learning. IEEE Trans Image Process 28(3):1366–1377

    Article  MathSciNet  Google Scholar 

  5. Engel C, Baumgartner P, Holzmann M et al (2010) Person re-identification by support vector ranking. Aberystwyth, UK, pp 1–11

  6. Gao Y, Ma J, Yuille AL (2017) Semi-supervised sparse representation based classification for face recognition with insufficient labeled samples. IEEE Trans Image Process 26(5):2545–2560

    Article  MathSciNet  Google Scholar 

  7. Gheissari N, Sebastian TB et al (2006) Person reidentification using spatiotemporal appearance, pp 1528–1535

  8. Gong S, Cristani M, Yan S, et al. (2014) Person Re-Identification. Springer Publishing Company, Incorporated

  9. Gray D, Tao H (2008) Viewpoint invariant pedestrian recognition with an ensemble of localized features, pp 262–275

  10. He Z, Jung C, Fu Q, Zhang Z (2018) Deep feature embedding learning for person re-identification based on lifted structured loss. Multimed Tools Appl 78 (5):5863–5880

    Article  Google Scholar 

  11. Hirzer M, Beleznai C, Roth P, et al. (2011) Person re-identification by descriptive and discriminative classification. Scandinavian Conference on Image Analysis 6688:91–102

    Article  Google Scholar 

  12. Huang W, Liang C, Yu Y, Wang Z, Ruan W, Hu R (2018) Video-based person re-identification via self paced weighting. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence, New Orleans, pp 2273–2280

  13. Kan S, Cen Y, He Z, Zhang Z, Zhang L, Wang Y (2019) Supervised deep feature embedding with hand crafted feature. IEEE Trans Image Process 28 (12):5809–5823

    Article  MathSciNet  Google Scholar 

  14. Klaser A, Marszalek M, Schmid C (2008) A spatio-temporal descriptor based on 3d-gradients

  15. Ksibi S, Mejdoub M, Amar C B (2018) Deep salient-gaussian fisher vector encoding of the spatio-temporal trajectory structures for person re-identification. Multimed Tools Appl 78(2):1583–1611

    Article  Google Scholar 

  16. Kviatkovsky I, Adam A, Rivlin E (2013) Color invariants for person reidentification. IEEE Trans Pattern Anal Mach Intell 35(7):1622–1634

    Article  Google Scholar 

  17. Li D, Chen X, Zhang Z, Huang K (2017) Learning deep context-aware features over body and latent parts for person re-identification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 7398–7407

  18. Li T, Sun L, Chong H, Jian G (2018) Person re-identification using salient region matching game. Multimed Tools Appl (1C3) 77(16):21393–21415

    Article  Google Scholar 

  19. Liao S, Hu Y, Zhu X et al (2015) Person re-identification by local maximal occurrence representation and metric learning, pp 2197–2206

  20. Liu C, Gong S, Loy CC et al (2012) Person re-identification: What features are important? European Conference on Computer Vision (ECCV). Berlin, Heidelberg, pp 391–401

  21. Liu K, Ma B, Zhang W et al (2015) A spatio-temporal appearance representation for viceo-based pedestrian re-identification, pp 3810–3818

  22. Liu H, Feng J, Qi M, Jiang J, Yan S (2016) End-to-end comparative attention networks for person re-identification. IEEE Trans Image Process PP(99):1–1

    MATH  Google Scholar 

  23. Liu H, Jie Z, Jayashree K, et al. (2017) Video-based person re-identification with accumulative motion context. IEEE Trans Circ Syst Video Technol PP(99):1–1

    Article  Google Scholar 

  24. Liu Z, Wang Y, Li A (2018) Hierarchical integration of rich features for video-based person re-identification. IEEE Trans Circ Syst Video Technol 29 (12):3646–3659

    Article  Google Scholar 

  25. Liu Y, Song N, Han Y (2019) Multi-cue fusion: Discriminative enhancing for person re-identification. J Vis Commun Image Represent 58:46–52

    Article  Google Scholar 

  26. Luo Y, Liu T, Tao D, Xu C (2015a) Multiview matrix completion for multilabel image classification. IEEE Trans Image Process 24(8):2355–2368

  27. Luo Y, Tao D, Ramamohanarao K, Xu C, Wen Y (2015b) Tensor canonical correlation analysis for multi-view dimension reduction. IEEE Trans Knowl Data Eng 27(11):3111–3124

  28. Luo Y, Wen Y, Tao D, Gui J, Xu C (2016) Large margin multi-modal multi-task feature extraction for image classification. IEEE Trans Image Process 25 (1):414–427

    Article  MathSciNet  Google Scholar 

  29. Ma J, Yong M, Chang L (2018) Infrared and visible image fusion methods and applications: a survey. Inf Fusion 45:153–178

    Article  Google Scholar 

  30. Matsukawa T, Okabe T, Suzuki E et al (2016) Hierarchical gaussian descriptor for person re-identification, pp 1363–1372

  31. McLaughlin N, Rincon J, Miller P (2016) In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1325–1334

  32. Mclaughlin N, Rincon J, Miller P (2016) Recurrent convolutional network for video-based person re-identification, pp 1325–1334

  33. Mignon A, Jurie F (2012) Pcca: A new approach for distance learning from sparse pairwise constraints, pp 2666–2672

  34. Roth PM, Wohlhart P, Hirzer M et al (2012) Large scale metric learning from equivalence constraints, pp 2288–2295

  35. Wang T, Gong S, Zhu X, et al. (2014) Person re-identification by video ranking. European Conference on Computer Visio (ECCV) 8692:688–703

    Google Scholar 

  36. Wang F, Zhang C, Chen S, Ying G, Lv J (2018) Engineering hand-designed and deeply-learned features for person re-identification. Pattern Recognition Letters

  37. Wu S, Chen Y, Li Xea (2016) An enhanced deep feature representation for person re-identification. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV):1–8

  38. Xiao T, Li H, Wea O (2016) Learning deep feature representations with domain guided dropout for person re-identification, pp 1249–1258

  39. Xu S, Cheng Y, Gu K et al (2017) Jointly attentive spatial-temporal pooling networks for video-based person re-identification, pp 4743–4752

  40. Yan Y, Ni B, Song Z, et al. (2016) Person re-identification via recurrent feature aggregation. European Conference on Computer Vision (ECCV) 9910:701–716

    Google Scholar 

  41. You J, Wu A, Li X et al (2016) Top-push video-based person re-identification, pp 1345–1353

  42. Zhang W, He X, Lu W, Qiao H, Li Y (2019) Feature aggregation with reinforcement learning for video-based person re-identification. IEEE Trans Neural Netw Learn Syst 30(12):3847–3852

    Article  Google Scholar 

  43. Zheng W, Gong S, Xiang T (2011) Person re-identification by probabilistic relative distance comparison. USA, pp 649–656

  44. Zheng W, Gong S, Xiang T (2013) Reidentification by relative distance comparison. IEEE Trans Pattern Anal Mach Intell 35(3):653–668

    Article  Google Scholar 

  45. Zheng L, Yang Y, Hauptmann GA (2016a) Person re-identification: Past, present and future

  46. Zheng L, Zhi B, Sun Y et al (2016b) Mars: A video benchmark for large-scale person re-identification, vol 9910

  47. Zheng Z, Zheng L, Yang Y (2019) Pedestrian alignment network for large-scale person re-identification. IEEE Transactions on Circuits and Systems for Video Technology 29(10):3037–3045

    Article  Google Scholar 

  48. Zhou K, Yang Y, Cavallaro A, Xiang T (2019) Omni-scale feature learning for person re-identification. arXiv:190500953

  49. Zhou S, Wang J, Meng D, Liang Y, Gong Y, Zheng N (2019) Discriminative feature learning with foreground attention for person re-identification. IEEE Trans Image Process 28(9):4671–4684

    Article  MathSciNet  Google Scholar 

  50. Zhu X, Jing X, Wu F et al (2016a) Video-based person re-identification by simultaneously learning intra-video and inter-video distance metrics, pp 3552–3558

  51. Zhu X, Jing XY, Wu F, Feng H (2016b) Video-based person re-identification by simultaneously learning intra-video and inter-video distance metrics. IEEE Trans Image Process PP(99):1–1

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61471201 and Grant 61501260, and in part by the Priority Academic Program Development of Jiangsu Higher Education Institutions and in part by Postgraduate Research & Practice Innovation Program of Jiangsu Province KYCX18_0890.

Author information

Authors and Affiliations

  1. Jiangsu Key Lab of Image Processing and Image Communications, Nanjing University of Posts and Telecommunications, Nanjing, China

    Wanru Song, Jieying Zheng, Yahong Wu, Changhong Chen & Feng Liu

Authors
  1. Wanru Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jieying Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yahong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Changhong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Feng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feng Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, W., Zheng, J., Wu, Y. et al. Video-based person re-identification using a novel feature extraction and fusion technique. Multimed Tools Appl 79, 12471–12491 (2020). https://doi.org/10.1007/s11042-019-08432-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08432-0

Keywords

Search

Navigation