Abstract
Visible-thermal person re-identification (VT-ReID) is a challenging pedestrian retrieval problem in the field of security. Due to the intra-modality variations and cross-modality discrepancy caused by different spectrums, it is difficult to extract discriminative features. Existing works are devoted to projecting different-modality features into a shared space, which has weak discriminability and ignores the contextual relationship. In this paper, a novel dual-level information transfer framework is proposed to reduce the modality discrepancy in image level and feature level for VT-ReID. An auxiliary mix-modality is proposed and a mix-visible-thermal (MVT) learning strategy is built to reduce the discrepancy in image level. Firstly, the mix-modality is generated by a mixup scheme which alleviates the direct transfer. Secondly, under the MVT framework, we use ID loss and hetero center triplet loss to guide feature extraction for visible, thermal, and mixed modalities on a one-stream Network. To enhance the robustness of feature extraction, we introduce a graph information transfer module to transfer information across intra-modality and inter-modality in feature level. We build the agent node for modality by using the modality center, where the agent node aggregates the information of all samples in one modality, and then the information from one modality is transmitted to other modalities through the agent nodes. Extensive experimental results on SYSU-MM01 and RegDB datasets show that our method achieves excellent performance.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig1_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig4_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11063-023-11294-1/MediaObjects/11063_2023_11294_Fig11_HTML.png)
Similar content being viewed by others
References
Zheng L, Yang Y, Hauptmann AG (2016) Person Re-identification: Past. Present Future 14(8):1–20
Dong H, Lu P, Zhong S, Liu C, Ji Y, Gong S (2018) Person re-identification by enhanced local maximal occurrence representation and generalized similarity metric learning. Neurocomputing 307:25–37. https://doi.org/10.1016/j.neucom.2018.04.013
Chen D, Wu P, Jia T, Xu F (2022) HOB-net: high-order block network via deep metric learning for person re-identification. Appl Intell 52:4844–4857. https://doi.org/10.1007/S10489-021-02450-Y/TABLES/4
Yao H, Zhang S, Hong R, Zhang Y, Xu C, Tian Q (2019) Deep Representation Learning With Part Loss for Person Re-Identification. IEEE Trans Image Process 28(6):2860–2871. https://doi.org/10.1109/TIP.2019.2891888
Zhao Y-B, Lin J-W, Xuan Q, Xi X (2019) HPILN: a feature learning framework for cross-modality person re-identification. IET Image Proc 13(14):2897–2904. https://doi.org/10.1049/iet-ipr.2019.0699
Gao G, Shao H, Wu F, Yang M, Yu Y (2021) Leaning compact and representative features for cross-modality person re-identification. World Wide Web. https://doi.org/10.1007/s11280-022-01014-5
Xiang X, Lv N, Zhai M, Abdeen R, El Saddik A (2020) Dual-path part-level method for visible-infrared person re-identification. Neural Process Lett 52(1):313–328. https://doi.org/10.1007/s11063-020-10239-2
Wu A, Zheng W-S, Yu H-X, Gong S, Lai J (2017) RGB-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 5390–5399. https://doi.org/10.1109/ICCV.2017.575
Ye M, Wang Z, Lan X, Yuen PC (2018) Visible thermal person re-identification via dual-constrained top-ranking. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, California, pp 1092–1099. https://doi.org/10.24963/ijcai.2018/152
Liu H, Tan X, Zhou X (2021) Parameter sharing exploration and hetero-center triplet loss for visible-thermal person re-identification. IEEE Trans Multimed 23:4414–4425. https://doi.org/10.1109/TMM.2020.3042080
Fan X, Luo H, Zhang C, Jiang W (2020) Cross-Spectrum Dual-Subspace Pairing for RGB-infrared Cross-Modality Person Re-Identification 14(8):1–13. arXiv preprint arXiv:2003.00213
Zhu Y, Yang Z, Wang L, Zhao S, Hu X, Tao D (2020) Hetero-Center loss for cross-modality person Re-identification. Neurocomputing 386:97–109. https://doi.org/10.1016/j.neucom.2019.12.100
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceedings of the advances in neural information processing systems(NeurIPS), vol 27, pp 2672–2680. https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
Wang Z, Wang Z, Zheng Y, Chuang YY, Satoh S (2019) Learning to reduce dual-level discrepancy for infrared-visible person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 2019-June, pp 618–626. https://doi.org/10.1109/CVPR.2019.00071
Zheng Z, Yang X, Yu Z, Zheng L, Yang Y, Kautz J (2019) Joint Discriminative and Generative Learning for Person Re-Identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 2133–2142. https://doi.org/10.1109/CVPR.2019.00224
Zhang Z, Jiang S, Huang C, Li Y, Xu RYD (2021) RGB-IR cross-modality person ReID based on teacher-student GAN model. Pattern Recogn Lett 150:155–161. https://doi.org/10.1016/j.patrec.2021.07.006
Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2017) mixup: Beyond empirical risk minimization. In: Proceedings of the 6th international conference on learning representations, ICLR 2018—conference track proceedings. arXiv:1710.09412
Cheng D, Gong Y, Zhou S, Wang J, Zheng N (2016) Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 1335–1344. https://doi.org/10.1109/CVPR.2016.149
Liu H, Cheng J, Wang W, Su Y, Bai H (2020) Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. Neurocomputing 398:11–19. https://doi.org/10.1016/j.neucom.2020.01.089
Ye M, Shen J, Lin G, Xiang T, Shao L, Hoi SCH (2021) Deep learning for person re-identification: a survey and outlook. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3054775
Yang X, Liu W, Liu W, Tao D (2021) A survey on canonical correlation analysis. IEEE Trans Knowl Data Eng 33(6):2349–2368. https://doi.org/10.1109/TKDE.2019.2958342
Pedagadi S, Orwell J, Velastin S, Boghossian B (2013) Local fisher discriminant analysis for pedestrian re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 3318–3325. https://doi.org/10.1109/CVPR.2013.426
Yang Y, Yang J, Yan J, Liao S, Yi D, Li SZ (2014) Salient color names for person re-identification. Proc Lect Notes Comput Sci 8689:536–551. https://doi.org/10.1007/978-3-319-10590-1_35
Liu C, Gong S, Loy CC, Lin X (2012) Person re-identification: What features are important? Proc Lect Notes Comput Sci 7583:391–401. https://doi.org/10.1007/978-3-642-33863-2_39
Matsukawa T, Okabe T, Suzuki E, Sato Y (2016) Hierarchical Gaussian descriptor for person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 1363–1372. https://doi.org/10.1109/CVPR.2016.152
An L, Chen X, Yang S, Li X (2017) Person re-identification by multi-hypergraph fusion. IEEE Trans Neural Netw Learn Syst 28(11):2763–2774. https://doi.org/10.1109/TNNLS.2016.2602082
Cheng DS, Cristani M, Stoppa M, Bazzani L, Murino V (2011) Custom pictorial structures for re-identification. In: Procedings of the British machine vision conference 2011, pp 68–16811. https://doi.org/10.5244/C.25.68
Luo H, Jiang W, Zhang X, Fan X, Qian J, Zhang C (2019) AlignedReID++: dynamically matching local information for person re-identification. Pattern Recogn 94:53–61. https://doi.org/10.1016/j.patcog.2019.05.028
Li D, Hu R, Huang W, Li D, Wang X, Hu C (2021) Trajectory association for person re-identification. Neural Process Lett 53(5):3267–3285. https://doi.org/10.1007/s11063-021-10540-8
Chen Y, Zhu X, Gong S (2017) Person re-identification by deep learning multi-scale representations. In: Proceedings of the IEEE international conference on computer vision workshops (ICCVW), pp 2590–2600. https://doi.org/10.1109/ICCVW.2017.304
Luo C, Song C, Zhang Z (2020) Generalizing person re-identification by camera-aware invariance learning and cross-domain mixup. Lect Notes Comput Sci 12360:224–241. https://doi.org/10.1007/978-3-030-58555-6_14
Sun Y, Zheng L, Yang Y, Tian Q, Wang S (2018) Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline). Lect Notes Comput Sci 11208:501–518. https://doi.org/10.1007/978-3-030-01225-0_30
Wang H, Bi X (2021) Person re-identification based on graph relation learning. Neural Process Lett 53:1401–1415. https://doi.org/10.1007/s11063-021-10446-5
Su C, Li J, Zhang S, Xing J, Gao W, Tian Q (2017) Pose-driven deep convolutional model for person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 3980–3989. https://doi.org/10.1109/ICCV.2017.427
Zhang Z, Lan C, Zeng W, Chen Z (2019) Densely semantically aligned person re-identification. In: Proceedings of the IEEE Computer society conference on computer vision and pattern recognition (CVPR), pp 667–676. https://doi.org/10.1109/CVPR.2019.00076
Liu X, Zhao H, Tian M, Sheng L, Shao J, Yi S, Yan J, Wang X (2017) HydraPlus-Net: attentive deep features for pedestrian analysis. In: Proceedings of the international conference on computer vision (ICCV), pp 350–359. https://doi.org/10.1109/ICCV.2017.46
Ning X, Gong K, Li W, Zhang L, Bai X, Tian S (2021) Feature refinement and filter network for person re-identification. IEEE Trans Circuits Syst Video Technol 31(9):3391–3402. https://doi.org/10.1109/TCSVT.2020.3043026
Yang F, Yan K, Lu S, Jia H, Xie X, Gao W (2019) Attention driven person re-identification. Pattern Recogn 86:143–155. https://doi.org/10.1016/j.patcog.2018.08.015
Zhang M, Xin M, Gao C, Wang X, Zhang S (2020) Attention-aware scoring learning for person re-identification. Knowl-Based Syst 203:106154. https://doi.org/10.1016/j.knosys.2020.106154
Wang X, Gao C, Xin M, Zhang S, Zhang M (2021) Topology and channel affinity reinforced global attention for person re-identification. Int J Intell Syst 36(9):5136–5160. https://doi.org/10.1002/int.22506
Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th international conference on learning representations, ICLR 2017—conference track proceedings, pp 1–14. arXiv:1609.02907
Zhang S, Yang Y, Wang P, Liang G, Zhang X, Zhang Y (2021) Attend to the difference: cross-modality person re-identification via contrastive correlation. IEEE Trans Image Process 30(8):8861–8872. https://doi.org/10.1109/TIP.2021.3120881
Ye M, Lan X, Li J, Yuen PC (2018) Hierarchical discriminative learning for visible thermal person re-identification. In: Proceedings of the 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 7501–7508. https://ojs.aaai.org/index.php/AAAI/article/view/12293
Liu Q, Teng Q, Chen H, Li B, Qing L (2022) Dual adaptive alignment and partitioning network for visible and infrared cross-modality person re-identification. Appl Intell 52(1):547–563. https://doi.org/10.1007/S10489-021-02390-7/FIGURES/11
Bai Z, Wang Z, Wang J, Hu D, Ding E (2021) Unsupervised multi-source domain adaptation for person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 12909–12918. https://doi.org/10.1109/CVPR46437.2021.01272
Liu W, Li J, Liu B, Guan W, Zhou Y, Xu C (2021) Unified cross-domain classification via geometric and statistical adaptations. Pattern Recogn 110:107658. https://doi.org/10.1016/j.patcog.2020.107658
Zhang B, Xiao J, Jiao J, Wei Y, Zhao Y (2021) Affinity attention graph neural network for weakly supervised semantic segmentation. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3083269
Li W, Qi K, Chen W, Zhou Y (2021) Unified batch all triplet loss for visible-infrared person re-identification. In: Proceedings of the 2021 international joint conference on neural networks (IJCNN), pp 1–8. https://doi.org/10.1109/IJCNN52387.2021.9533325
Hao Y, Wang N, Li J, Gao X (2019) HSME: Hypersphere manifold embedding for visible thermal person re-identification. Proc AAAI Conf Artif Intell 33:8385–8392. https://doi.org/10.1609/aaai.v33i01.33018385
Dai P, Ji R, Wang H, Wu Q, Huang Y (2018) Cross-modality person re-identification with generative adversarial training. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, California, pp 677–683. https://doi.org/10.24963/ijcai.2018/94
Wang G, Zhang T, Cheng J, Liu S, Yang Y, Hou Z (2019) RGB-infrared cross-modality person re-identification via joint pixel and feature alignment. In: Proceedings of the international conference on computer vision (ICCV), pp 3622–3631. https://doi.org/10.1109/ICCV.2019.00372
Choi S, Lee S, Kim Y, Kim T, Kim C (2020) Hi-CMD: hierarchical cross-modality disentanglement for visible-infrared person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 10254–10263. https://doi.org/10.1109/CVPR42600.2020.01027
Kumar K, Shrimankar DD (2018) F-DES: fast and deep event summarization. IEEE Trans Multimed 20(2):323–334. https://doi.org/10.1109/TMM.2017.2741423
Kumar K, Shrimankar DD, Singh N (2018) Eratosthenes sieve based key-frame extraction technique for event summarization in videos. Multimed Tools Appl 77(6):7383–7404
Kumar K, Shrimankar DD (2018) Deep event learning boost-up approach: DELTA. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5882-z
Kumar A, Singh N, Kumar P, Vijayvergia A, Kumar K (2017) A novel superpixel based color spatial feature for salient object detection. In: 2017 Conference on information and communication technology (CICT), pp 1–5. https://doi.org/10.1109/INFOCOMTECH.2017.8340630
Sharma S, Shivhare SN, Singh N, Kumar K (2019) Computationally efficient ANN model for small-scale problems. Adv Intell Syst Comput. https://doi.org/10.1007/978-981-13-0923-6_37
Kumar K, Mishra A, Dahiya S, Kumar A (2022) A technique for human upper body parts movement tracking. IETE J Res. https://doi.org/10.1080/03772063.2022.2048708
He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 770–778. https://doi.org/10.1109/CVPR.2016.90
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Szegedy C, Wei Liu, Yangqing Jia, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594
Deng Jia, Dong Wei, Socher R, Li Li-Jia, Li Kai, Fei-Fei Li (2009) ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 248–255. https://doi.org/10.1109/CVPRW.2009.5206848
Wu Z, Wu Z (2010) Exploration, visualization, and preprocessing of high-dimensional data. Methods Mole Biol 620:267–284. https://doi.org/10.1007/978-1-60761-580-4_8
Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737
Nguyen D, Hong H, Kim K, Park K (2017) Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17(3):605. https://doi.org/10.3390/s17030605
Lu Y, Wu Y, Liu B, Zhang T, Li B, Chu Q, Yu N (2020) Cross-modality person re-identification with shared-specific feature transfer. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR42600.2020.01339
Tian X, Zhang Z, Lin S, Qu Y, Xie Y, Ma L (2021) Farewell to mutual information: variational distillation for cross-modal person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 1522–1531. https://doi.org/10.1109/CVPR46437.2021.00157
Wu Q, Dai P, Chen J, Lin C-W, Wu Y, Huang F, Zhong B, Ji R (2021) Discover cross-modality nuances for visible-infrared person re-identification. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 4330–4339. https://openaccess.thecvf.com/content/CVPR2021/papers/Wu_Discover_Cross-Modality_Nuances_for_Visible-Infrared_Person_Re-Identification_CVPR_2021_paper.pdf
Ye M, Ruan W, Du B, Shou MZ (2021) Channel augmented joint learning for visible-infrared recognition. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 13547–13556. https://doi.org/10.1109/ICCV48922.2021.01331
Yang M, Huang Z, Hu P, Li T, Lv J, Peng X (2022) Learning with twin noisy labels for visible-infrared person re-identification. Cvpr 2022(1):14308–14317
Zhang Q, Lai C, Liu J, Huang N, Han J (2022) FMCNet : Feature-level modality compensation for visible-infrared person. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR), pp 7349–7358. https://openaccess.thecvf.com/content/CVPR2022/html/Zhang_FMCNet_Feature-Level_Modality_Compensation_for_Visible-Infrared_Person_Re-Identification_CVPR_2022_paper.html
Wei Z, Yang X, Wang N, Gao X (2021) Syncretic modality collaborative learning for visible infrared person re-identification. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 225–234. https://doi.org/10.1109/ICCV48922.2021.00029
Ling Y, Luo Z, Lin Y, Li S (2021) A multi-constraint similarity learning with adaptive weighting for visible-thermal person re-identification. In: Proceedings of the thirtieth international joint conference on artificial intelligence, pp 845–851. https://doi.org/10.24963/ijcai.2021/117
Acknowledgements
This research was funded by the National Natural Science Foundation of China (62002100), and the Key R &D and Promotion Projects in Henan Province (212102210411).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Song, J., Wang, X., Li, K. et al. Dual-Level Information Transfer for Visible-Thermal Person Re-identification. Neural Process Lett 55, 7999–8021 (2023). https://doi.org/10.1007/s11063-023-11294-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-023-11294-1