Skip to main content
SpringerLink
Log in

CAViT: Contextual Alignment Vision Transformer for Video Object Re-identification

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13674))

Included in the following conference series:

Abstract

Video object re-identification (reID) aims at re-identifying the same object under non-overlapping cameras by matching the video tracklets with cropped video frames. The key point is how to make full use of spatio-temporal interactions to extract more accurate representation. However, there are dilemmas within existing approaches: (1) 3D solutions model the spatio-temporal interaction but are often troubled with the misalignment of adjacent frames, and (2) 2D solutions adopt a divide-and-conquer strategy against the misalignment but cannot take advantage of the spatio-temporal interactions. To address the above problems, we propose a Contextual Alignment Vision Transformer (CAViT) to the spatio-temporal interaction with a 2D solution. It contains a Multi-shape Patch Embedding (MPE) module and a Temporal Shift Attention (TSA) module. MPE is designed to retain spatial semantic information against the misalignment caused by pose, occlusion, or misdetection. TSA is designed to achieve contextual spatial semantic feature alignment and jointly model spatio-temporal clues. We further propose a Residual Position Embedding (RPE) to guide TSA in focusing on the temporal saliency clues. Experimental results on five video person reID datasets demonstrate the superiority of the proposed CAViT. Additionally, the experiment conducted on VVeRI-901-trial also shows the effectiveness of CAViT for the video vehicle reID. Our code is available on https://github.com/KimWu1994/CAViT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aich, A., Zheng, M., Karanam, S., Chen, T., Roy-Chowdhury, A.K., Wu, Z.: Spatio-temporal representation factorization for video-based person re-identification, In: ICCV (2021)

    Google Scholar 

  2. Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095 (2021)

  3. Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)

  4. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV (2020)

    Google Scholar 

  5. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR (2017)

    Google Scholar 

  6. Chen, C.F., Fan, Q., Panda, R.: Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103.14899 (2021)

  7. Chen, G., Rao, Y., Lu, J., Zhou, J.: Temporal coherence or temporal motion: Which is more critical for video-based person re-identification? In: ECCV (2020)

    Google Scholar 

  8. Dehghan, A., Modiri Assari, S., Shah, M.: Gmmcp tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking. In: CVPR (2015)

    Google Scholar 

  9. Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  10. Eom, C., Lee, G., Lee, J., Ham, B.: Video-based person re-identification with spatial and temporal memory networks. In: ICCV (2021)

    Google Scholar 

  11. Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: ICCV (2019)

    Google Scholar 

  12. Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE TPAMI (2009)

    Google Scholar 

  13. Gu, X., Chang, H., Ma, B., Zhang, H., Chen, X.: Appearance-preserving 3d convolution for video-based person re-identification. In: ECCV (2020)

    Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

    Google Scholar 

  15. He, L., Liao, X., Liu, W., Liu, X., Cheng, P., Mei, T.: Fastreid: a pytorch toolbox for real-world person re-identification. arXiv preprint arXiv:2006.02631 (2020)

  16. He, S., Luo, H., Wang, P., Wang, F., Li, H., Jiang, W.: Transreid: Transformer-based object re-identification. arXiv preprint arXiv:2102.04378 (2021)

  17. He, T., Jin, X., Shen, X., Huang, J., Chen, Z., Hua, X.S.: Dense interaction learning for video-based person re-identification supplementary materials. Identities (2021)

    Google Scholar 

  18. Hirzer, M., Beleznai, C., Roth, P.M., Bischof, H.: Person re-identification by descriptive and discriminative classification. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 91–102. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21227-7_9

    Chapter  Google Scholar 

  19. Hou, R., Chang, H., Ma, B., Huang, R., Shan, S.: Bicnet-tks: Learning efficient spatial-temporal representation for video person re-identification. In: CVPR (2021)

    Google Scholar 

  20. Hou, R., Chang, H., Ma, B., Shan, S., Chen, X.: Temporal complementary learning for video person re-identification. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 388–405. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_24

    Chapter  Google Scholar 

  21. Hou, R., Ma, B., Chang, H., Gu, X., Shan, S., Chen, X.: Iaunet: Global context-aware feature learning for person reidentification. IEEE TNNLS (2020)

    Google Scholar 

  22. Hou, R., Ma, B., Chang, H., Gu, X., Shan, S., Chen, X.: Feature completion for occluded person re-identification. IEEE TPAMI (2021)

    Google Scholar 

  23. Zhao, J., Qi, F., G.R., Xu, L.: Vveri-901: Video vehicle re-identification dataset (2020). https://www.graviti.cn/open-datasets/VVeRI901’

  24. Li, C., Zhong, Q., Xie, D., Pu, S.: Collaborative spatiotemporal feature learning for video action recognition. In: CVPR (2019)

    Google Scholar 

  25. Li, J., Wang, J., Tian, Q., Gao, W., Zhang, S.: Global-local temporal representations for video person re-identification. In: ICCV (2019)

    Google Scholar 

  26. Li, J., Zhang, S., Huang, T.: Multi-scale 3D convolution network for video based person re-identification. In: AAAI (2019)

    Google Scholar 

  27. Li, S., Bak, S., Carr, P., Wang, X.: Diversity regularized spatiotemporal attention for video-based person re-identification. In: CVPR (2018)

    Google Scholar 

  28. Li, S.Z.: Markov random field modeling in image analysis. Springer Science & Business Media (2009)

    Google Scholar 

  29. Li, X., Zhou, W., Zhou, Y., Li, H.: Relation-guided spatial attention and temporal refinement for video-based person re-identification. In: AAAI (2020)

    Google Scholar 

  30. Li, Y., He, J., Zhang, T., Liu, X., Zhang, Y., Wu, F.: Diverse part discovery: Occluded person re-identification with part-aware transformer. In: CVPR (2021)

    Google Scholar 

  31. Liao, S., Shao, L.: Transformer-based deep image matching for generalizable person re-identification. NeurIPS Workshops (2021)

    Google Scholar 

  32. Lin, J., Gan, C., Han, S.: Tsm: Temporal shift module for efficient video understanding. In: ICCV (2019)

    Google Scholar 

  33. Liu, C.T., Wu, C.W., Wang, Y.C.F., Chien, S.Y.: Spatially and temporally efficient non-local attention network for video-based person re-identification. arXiv preprint arXiv:1908.01683 (2019)

  34. Liu, J., Zha, Z.J., Wu, W., Zheng, K., Sun, Q.: Spatial-temporal correlation and topology learning for person re-identification in videos. In: CVPR (2021)

    Google Scholar 

  35. Liu, X., Zhang, P., Yu, C., Lu, H., Yang, X.: Watching you: Global-guided reciprocal learning for video-based person re-identification. In: CVPR (2021)

    Google Scholar 

  36. Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. ICCV (2021)

    Google Scholar 

  37. Liu, Z., et al.: Video swin transformer. arXiv preprint arXiv:2106.13230 (2021)

  38. Luo, H., Gu, Y., Liao, X., Lai, S., Jiang, W.: Bag of tricks and a strong baseline for deep person re-identification. In: CVPR Workshops (2019)

    Google Scholar 

  39. Pathak, P., Eshratifar, A.E., Gormish, M.: Video person re-id: Fantastic techniques and where to find them. arXiv preprint arXiv:1912.05295 (2019)

  40. Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3d residual networks. In: ICCV (2017)

    Google Scholar 

  41. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: ICCV (2015)

    Google Scholar 

  42. Wang, T., Gong, S., Zhu, X., Wang, S.: Person re-identification by video ranking. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 688–703. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_45

    Chapter  Google Scholar 

  43. Wang, Y., Zhang, P., Gao, S., Geng, X., Lu, H., Wang, D.: Pyramid spatial-temporal aggregation for video-based person re-identification. In: ICCV (2021)

    Google Scholar 

  44. Weng, X., Kitani, K.: Learning spatio-temporal features with two-stream deep 3d cnns for lipreading. arXiv preprint arXiv:1905.02540 (2019)

  45. Wu, Y., et al.: Adaptive graph representation learning for video person re-identification. IEEE TIP (2020)

    Google Scholar 

  46. Yan, Y., et al.: Learning multi-granular hypergraphs for video-based person re-identification. In: CVPR (2020)

    Google Scholar 

  47. Yang, J., Zheng, W.S., Yang, Q., Chen, Y.C., Tian, Q.: Spatial-temporal graph convolutional network for video-based person re-identification. In: CVPR (2020)

    Google Scholar 

  48. Zhang, H., et al.: Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955 (2020)

  49. Zhang, H., Hao, Y., Ngo, C.W.: Token shift transformer for video classification. In: ACM MM (2021)

    Google Scholar 

  50. Zhang, Z., Lan, C., Zeng, W., Chen, Z.: Multi-granularity reference-aided attentive feature aggregation for video-based person re-identification. In: CVPR (2020)

    Google Scholar 

  51. Zhao, J., Qi, F., Ren, G., Xu, L.: Phd learning: Learning with pompeiu-hausdorff distances for video-based vehicle re-identification. In: CVPR (2021)

    Google Scholar 

  52. Zhao, Y., Shen, X., Jin, Z., Lu, H., Hua, X.s.: Attribute-driven feature disentangling and temporal aggregation for video person re-identification. In: CVPR (2019)

    Google Scholar 

  53. Zheng, L., et al.: MARS: a video benchmark for large-scale Person re-identification. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 868–884. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_52

    Chapter  Google Scholar 

  54. Zhou, Z., Huang, Y., Wang, W., Wang, L., Tan, T.: See the forest for the trees: Joint spatial and temporal recurrent neural networks for video-based person re-identification. In: CVPR (2017)

    Google Scholar 

  55. Zhu, K., et al.: Aaformer: Auto-aligned transformer for person re-identification. arXiv preprint arXiv:2104.00921 (2021)

Download references

Acknowledgments

This research was supported by the National Key R &D Program of China under Grant No.2020YFC2003901, Chinese National Natural Science Foundation Projects #61876178, #61872367, #61976229, #62176256, #62106264 and the InnoHK program.

Author information

Authors and Affiliations

  1. CBSR & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Jinlin Wu, Yang Yang & Zhen Lei

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    Jinlin Wu, Yang Yang & Zhen Lei

  3. Centre for Artificial Intelligence and Robotics, HKISI, CAS, Hong Kong, China

    Jinlin Wu & Zhen Lei

  4. JD Explore Academy, Beijing, China

    Wu Liu & Tao Mei

  5. Longfor Inc., Beijing, China

    Lingxiao He

  6. School of Engineering, Westlake University, Hangzhou, China

    Stan Z. Li

Authors
  1. Jinlin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingxiao He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tao Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Stan Z. Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhen Lei .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wu, J. et al. (2022). CAViT: Contextual Alignment Vision Transformer for Video Object Re-identification. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13674. Springer, Cham. https://doi.org/10.1007/978-3-031-19781-9_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19781-9_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19780-2

  • Online ISBN: 978-3-031-19781-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation