Hard-Aware Point-to-Set Deep Metric for Person Re-identification

  • Rui Yu
  • Zhiyong Dou
  • Song Bai
  • Zhaoxiang Zhang
  • Yongchao XuEmail author
  • Xiang BaiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11220)


Person re-identification (re-ID) is a highly challenging task due to large variations of pose, viewpoint, illumination, and occlusion. Deep metric learning provides a satisfactory solution to person re-ID by training a deep network under supervision of metric loss, e.g., triplet loss. However, the performance of deep metric learning is greatly limited by traditional sampling methods. To solve this problem, we propose a Hard-Aware Point-to-Set (HAP2S) loss with a soft hard-mining scheme. Based on the point-to-set triplet loss framework, the HAP2S loss adaptively assigns greater weights to harder samples. Several advantageous properties are observed when compared with other state-of-the-art loss functions: (1) Accuracy: HAP2S loss consistently achieves higher re-ID accuracies than other alternatives on three large-scale benchmark datasets; (2) Robustness: HAP2S loss is more robust to outliers than other losses; (3) Flexibility: HAP2S loss does not rely on a specific weight function, i.e., different instantiations of HAP2S loss are equally effective. (4) Generality: In addition to person re-ID, we apply the proposed method to generic deep metric learning benchmarks including CUB-200-2011 and Cars196, and also achieve state-of-the-art results.


Person re-identification Deep metric learning Triplet loss 



This work was supported by National Key R&D Program of China No. 2018YFB1004600, NSFC 61703171, and NSFC 61573160, to Dr. Xiang Bai by the National Program for Support of Top-notch Young Professionals and the Program for HUST Academic Frontier Youth Team. We would also like to thank the reviewers for their helpful comments.

Supplementary material

474218_1_En_12_MOESM1_ESM.pdf (1 mb)
Supplementary material 1 (pdf 1048 KB)


  1. 1.
    Ahmed, E., Jones, M., Marks, T.K.: An improved deep learning architecture for person re-identification. In: CVPR, pp. 3908–3916 (2015)Google Scholar
  2. 2.
    Bai, S., Bai, X., Tian, Q.: Scalable person re-identification on supervised smoothed manifold. In: CVPR, pp. 2530–2539 (2017)Google Scholar
  3. 3.
    Chang, X., Hospedales, T.M., Xiang, T.: Multi-level factorisation net for person re-identification. In: CVPR, pp. 2109–2118 (2018)Google Scholar
  4. 4.
    Chen, W., Chen, X., Zhang, J., Huang, K.: Beyond triplet loss: a deep quadruplet network for person re-identification. In: CVPR, pp. 403–412 (2017)Google Scholar
  5. 5.
    Chen, W., Chen, X., Zhang, J., Huang, K.: A multi-task deep network for person re-identification. In: AAAI, pp. 3988–3994 (2017)Google Scholar
  6. 6.
    Chen, Y.C., Zhu, X., Zheng, W.S., Lai, J.H.: Person re-identification by camera correlation aware feature augmentation. IEEE TPAMI 40(2), 392–408 (2018)CrossRefGoogle Scholar
  7. 7.
    Cheng, D., Gong, Y., Zhou, S., Wang, J., Zheng, N.: Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In: CVPR, pp. 1335–1344 (2016)Google Scholar
  8. 8.
    Ding, S., Lin, L., Wang, G., Chao, H.: Deep feature learning with relative distance comparison for person re-identification. Pattern Recognit. 48(10), 2993–3003 (2015)CrossRefGoogle Scholar
  9. 9.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)Google Scholar
  10. 10.
    Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification (2017). arXiv:1703.07737
  11. 11.
    Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE TPAMI 33(1), 117–128 (2011)CrossRefGoogle Scholar
  12. 12.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)Google Scholar
  13. 13.
    Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: ICCV Workshop, pp. 554–561 (2013)Google Scholar
  14. 14.
    Kumar, V.B., Harwood, B., Carneiro, G., Reid, I., Drummond, T.: Smart mining for deep metric learning. In: ICCV, pp. 2821–2829 (2017)Google Scholar
  15. 15.
    Li, D., Chen, X., Zhang, Z., Huang, K.: Learning deep context-aware features over body and latent parts for person re-identification. In: CVPR, pp. 384–393 (2017)Google Scholar
  16. 16.
    Li, W., Zhao, R., Xiao, T., Wang, X.: Deepreid: Deep filter pairing neural network for person re-identification. In: CVPR, pp. 152–159 (2014)Google Scholar
  17. 17.
    Li, W., Zhu, X., Gong, S.: Person re-identification by deep joint learning of multi-loss classification. In: IJCAI, pp. 2194–2200 (2017)Google Scholar
  18. 18.
    Li, W., Zhu, X., Gong, S.: Harmonious attention network for person re-identification. In: CVPR, pp. 2285–2294 (2018)Google Scholar
  19. 19.
    Lin, J., Ren, L., Lu, J., Feng, J., Zhou, J.: Consistent-aware deep learning for person re-identification in a camera network. In: CVPR, pp. 5771–5780 (2017)Google Scholar
  20. 20.
    Lin, Y., Zheng, L., Zheng, Z., Wu, Y., Yang, Y.: Improving person re-identification by attribute and identity learning (2017). arXiv:1703.07220
  21. 21.
    Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. JMLR 9, 2579–2605 (2008)Google Scholar
  22. 22.
    Manning, C.D., Raghavan, P., Schütze, H., et al.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)CrossRefGoogle Scholar
  23. 23.
    McLaughlin, N., Martinez del Rincon, J., Miller, P.: Recurrent convolutional network for video-based person re-identification. In: CVPR, pp. 1325–1334 (2016)Google Scholar
  24. 24.
    Movshovitz-Attias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss distance metric learning using proxies. In: ICCV, pp. 360–368 (2017)Google Scholar
  25. 25.
    Oh Song, H., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: CVPR, pp. 4004–4012 (2016)Google Scholar
  26. 26.
    Qian, X., Fu, Y., Jiang, Y.G., Xiang, T., Xue, X.: Multi-scale deep learning architectures for person re-identification. In: ICCV, pp. 5399–5408 (2017)Google Scholar
  27. 27.
    Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera tracking. In: ECCV Workshop, pp. 17–35 (2016)Google Scholar
  28. 28.
    Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: CVPR, pp. 815–823 (2015)Google Scholar
  29. 29.
    Schumann, A., Stiefelhagen, R.: Person re-identification by deep learning attribute-complementary information. In: CVPR Workshop, pp. 1435–1443 (2017)Google Scholar
  30. 30.
    Shi, H., Yang, Y., Zhu, X., Liao, S., Lei, Z., Zheng, W., Li, S.Z.: Embedding deep metric for person re-identification: a study against large variations. In: ECCV, pp. 732–748 (2016)Google Scholar
  31. 31.
    Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. In: NIPS, pp. 1857–1865 (2016)Google Scholar
  32. 32.
    Song, H.O., Jegelka, S., Rathod, V., Murphy, K.: Deep metric learning via facility location. In: CVPR, pp. 5382–5390 (2017)Google Scholar
  33. 33.
    Song, H.O., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: CVPR, pp. 4004–4012 (2016)Google Scholar
  34. 34.
    Su, C., Li, J., Zhang, S., Xing, J., Gao, W., Tian, Q.: Pose-driven deep convolutional model for person re-identification. In: ICCV, pp. 3960–3969 (2017)Google Scholar
  35. 35.
    Sun, Y., Zheng, L., Deng, W., Wang, S.: SVDNet for pedestrian retrieval. In: ICCV, pp. 3800–3808 (2017)Google Scholar
  36. 36.
    Varior, R.R., Haloi, M., Wang, G.: Gated siamese convolutional neural network architecture for human re-identification. In: ECCV, pp. 791–808 (2016)Google Scholar
  37. 37.
    Varior, R.R., Shuai, B., Lu, J., Xu, D., Wang, G.: A siamese long short-term memory architecture for human re-identification. In: ECCV, pp. 135–153 (2016)Google Scholar
  38. 38.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)Google Scholar
  39. 39.
    Wang, J., Zhou, F., Wen, S., Liu, X., Lin, Y.: Deep metric learning with angular loss. In: ICCV, pp. 2593–2601 (2017)Google Scholar
  40. 40.
    Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. JMLR 10, 207–244 (2009)zbMATHGoogle Scholar
  41. 41.
    Xiao, T., Li, S., Wang, B., Lin, L., Wang, X.: Joint detection and identification feature learning for person search. In: CVPR, pp. 3415–3424 (2017)Google Scholar
  42. 42.
    Yuan, Y., Yang, K., Zhang, C.: Hard-aware deeply cascaded embedding. In: ICCV, pp. 814–823 (2017)Google Scholar
  43. 43.
    Zhang, L., Xiang, T., Gong, S.: Learning a discriminative null space for person re-identification. In: CVPR, pp. 1239–1248 (2016)Google Scholar
  44. 44.
    Zhao, H., Tian, M., Sun, S., Shao, J., Yan, J., Yi, S., Wang, X., Tang, X.: Spindle net: Person re-identification with human body region guided feature decomposition and fusion. In: CVPR, pp. 1077–1085 (2017)Google Scholar
  45. 45.
    Zhao, L., Li, X., Wang, J., Zhuang, Y.: Deeply-learned part-aligned representations for person re-identification. In: ICCV, pp. 3219–3228 (2017)Google Scholar
  46. 46.
    Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q.: Scalable person re-identification: A benchmark. In: ICCV, pp. 1116–1124 (2015)Google Scholar
  47. 47.
    Zheng, L., Yang, Y., Hauptmann, A.G.: Person re-identification: Past, present and future (2016). arXiv:1610.02984
  48. 48.
    Zheng, Z., Zheng, L., Yang, Y.: Pedestrian alignment network for large-scale person re-identification (2017). arXiv:1707.00408
  49. 49.
    Zheng, Z., Zheng, L., Yang, Y.: Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In: ICCV, pp. 3754–3762 (2017)Google Scholar
  50. 50.
    Zhong, Z., Zheng, L., Cao, D., Li, S.: Re-ranking person re-identification with k-reciprocal encoding. In: CVPR, pp. 1318–1327 (2017)Google Scholar
  51. 51.
    Zhou, J., Yu, P., Tang, W., Wu, Y.: Efficient online local metric adaptation via negative samples for person re-identification. In: ICCV, pp. 2420–2428 (2017)Google Scholar
  52. 52.
    Zhou, S., Wang, J., Wang, J., Gong, Y., Zheng, N.: Point to set similarity based deep feature learning for person re-identification. In: CVPR, pp. 3741–3750 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Huazhong University of Science and TechnologyWuhanChina
  2. 2.Institute of AutomationChinese Academy of SciencesBeijingChina

Personalised recommendations