Advertisement

DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning

Conference paper
  • 648 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12353)

Abstract

Visual similarity plays an important role in many computer vision applications. Deep metric learning (DML) is a powerful framework for learning such similarities which not only generalize from training data to identically distributed test distributions, but in particular also translate to unknown test classes. However, its prevailing learning paradigm is class-discriminative supervised training, which typically results in representations specialized in separating training classes. For effective generalization, however, such an image representation needs to capture a diverse range of data characteristics. To this end, we propose and study multiple complementary learning tasks, targeting conceptually different data relationships by only resorting to the available training samples and labels of a standard DML setting. Through simultaneous optimization of our tasks we learn a single model to aggregate their training signals, resulting in strong generalization and state-of-the-art performance on multiple established DML benchmark datasets.

Keywords

Deep metric learning Generalization Self-supervision 

Notes

Acknowledgements

This work has been supported by hardware donations from NVIDIA (DGX-1), resources from Compute Canada, in part by Bayer AG and the German federal ministry BMWi within the project “KI Absicherung”.

References

  1. 1.
    Bachman, P., Hjelm, R.D., Buchwalter, W.: Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910 (2019)
  2. 2.
    Bautista, M.A., Sanakoyeu, A., Tikhoncheva, E., Ommer, B.: Cliquecnn: deep unsupervised exemplar learning. In: Advances in Neural Information Processing Systems, pp. 3846–3854 (2016)Google Scholar
  3. 3.
    Belghazi, M.I., Rajeswar, S., Mastropietro, O., Rostamzadeh, N., Mitrovic, J., Courville, A.: Hierarchical adversarially learned inference (2018)Google Scholar
  4. 4.
    Berthelot, D., Raffel, C., Roy, A., Goodfellow, I.: Understanding and improving interpolation in autoencoders via an adversarial regularizer (2018)Google Scholar
  5. 5.
    Bhattarai, B., Sharma, G., Jurie, F.: Cp-mtml: coupled projection multi-task metric learning for large scale face retrieval. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4226–4235 (2016)Google Scholar
  6. 6.
    Büchler, U., Brattoli, B., Ommer, B.: Improving spatiotemporal self-supervision by deep reinforcement learning. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)Google Scholar
  7. 7.
    Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: European Conference on Computer Vision (2018)Google Scholar
  8. 8.
    Chawla, N.V., Japkowicz, N., Kotcz, A.: Editorial: special issue on learning from imbalanced data sets. SIGKDD Explor. Newsl. 6, 1–6 (2004)Google Scholar
  9. 9.
    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 (2020)
  10. 10.
    Chen, W., Chen, X., Zhang, J., Huang, K.: Beyond triplet loss: a deep quadruplet network for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  11. 11.
    Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: Interpretable representation learning by information maximizing generative adversarial nets (2016)Google Scholar
  12. 12.
    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: additive angular margin loss for deep face recognition (2018)Google Scholar
  13. 13.
    Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning (2016)Google Scholar
  14. 14.
    Duan, Y., Zheng, W., Lin, X., Lu, J., Zhou, J.: Deep adversarial metric learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  15. 15.
    Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., Courville, A.: Adversarially learned inference (2016)Google Scholar
  16. 16.
    Gautheron, L., Morvant, E., Habrard, A., Sebban, M.: Metric learning from imbalanced data (2019)Google Scholar
  17. 17.
    Ge, W.: Deep metric learning with hierarchical triplet loss. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 269–285 (2018)Google Scholar
  18. 18.
    Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (2018)Google Scholar
  19. 19.
    Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (2010)Google Scholar
  20. 20.
    Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2006)Google Scholar
  21. 21.
    He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning (2019)Google Scholar
  22. 22.
    Hendrycks, D., Mazeika, M., Kadavath, S., Song, D.: Using self-supervised learning can improve model robustness and uncertainty. CoRR abs/1906.12340 (2019). http://arxiv.org/abs/1906.12340
  23. 23.
    Hsu, K., Levine, S., Finn, C.: Unsupervised learning via meta-learning (2018)Google Scholar
  24. 24.
    Hu, J., Lu, J., Tan, Y.: Discriminative deep metric learning for face verification in the wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  25. 25.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (2015)Google Scholar
  26. 26.
    Jacob, P., Picard, D., Histace, A., Klein, E.: Metric learning with horde: high-order regularizer for deep embeddings. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  27. 27.
    Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2011)Google Scholar
  28. 28.
    Kim, W., Goyal, B., Chawla, K., Lee, J., Kwon, K.: Attention-based ensemble for deep metric learning. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)Google Scholar
  29. 29.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2015)Google Scholar
  30. 30.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: Proceedings of the International Conference on Learning Representations (ICLR) (2013)Google Scholar
  31. 31.
    Konno, T., Iwazume, M.: Cavity filling: pseudo-feature generation for multi-class imbalanced data problems in deep learning (2018)Google Scholar
  32. 32.
    Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 554–561 (2013)Google Scholar
  33. 33.
    Lin, X., Duan, Y., Dong, Q., Lu, J., Zhou, J.: Deep variational metric learning. In: The European Conference on Computer Vision (ECCV), September 2018Google Scholar
  34. 34.
    Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: Sphereface: deep hypersphere embedding for face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  35. 35.
    Manning, C., Raghavan, P., Schütze, H.: Introduction to information retrieval. Nat. Lang. Eng. 16(1), 100–103 (2010)zbMATHGoogle Scholar
  36. 36.
    Milbich, T., Ghori, O., Diego, F., Ommer, B.: Unsupervised representation learning by discovering reliable image relations. Pattern Recogn. (PR) 102, 107107 (2020)Google Scholar
  37. 37.
    Milbich, T., Roth, K., Brattoli, B., Ommer, B.: Sharing matters for generalization in deep metric learning. IEEE Trans. Pattern Anal. Mach. Intell. (2020)Google Scholar
  38. 38.
    Misra, I., van der Maaten, L.: Self-supervised learning of pretext-invariant representations (2019)Google Scholar
  39. 39.
    Movshovitz-Attias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss distance metric learning using proxies. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 360–368 (2017)Google Scholar
  40. 40.
    Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 69–84. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46466-4_5CrossRefGoogle Scholar
  41. 41.
    Noroozi, M., Vinjimoor, A., Favaro, P., Pirsiavash, H.: Boosting self-supervised learning via knowledge transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9359–9367 (2018)Google Scholar
  42. 42.
    Oh Song, H., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4004–4012 (2016)Google Scholar
  43. 43.
    Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding (2018), arXiv preprint arXiv:1807.03748
  44. 44.
    Opitz, M., Waltner, G., Possegger, H., Bischof, H.: Deep metric learning with bier: boosting independent embeddings robustly. IEEE Trans. Pattern Anal. Mach. Intell. 42, 276–290 (2018)Google Scholar
  45. 45.
    Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS-W (2017)Google Scholar
  46. 46.
    Pu, J., Jiang, Y.G., Wang, J., Xue, X.: Which looks like which: exploring inter-class relationships in fine-grained visual categorization. In: Proceedings of the IEEE European Conference on Computer Vision (ECCV), pp. 425–440 (2014)Google Scholar
  47. 47.
    Qian, Q., Shang, L., Sun, B., Hu, J., Li, H., Jin, R.: Softtriple loss: deep metric learning without triplet sampling (2019)Google Scholar
  48. 48.
    Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NeuRips (2015)Google Scholar
  49. 49.
    Roth, K., Brattoli, B., Ommer, B.: Mic: mining interclass characteristics for improved metric learning. In: The IEEE International Conference on Computer Vision (ICCV), October 2019Google Scholar
  50. 50.
    Roth, K., Milbich, T., Ommer, B.: Pads: policy-adapted sampling for visual similarity learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)Google Scholar
  51. 51.
    Roth, K., Milbich, T., Sinha, S., Gupta, P., Ommer, B., Cohen, J.P.: Revisiting training strategies and generalization performance in deep metric learning. In: Proceedings of the International Conference on Machine Learning (ICML) (2020)Google Scholar
  52. 52.
    Sanakoyeu, A., Tschernezki, V., Buchler, U., Ommer, B.: Divide and conquer the embedding space for metric learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  53. 53.
    Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)Google Scholar
  54. 54.
    Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. In: Advances in Neural Information Processing Systems, pp. 1857–1865 (2016)Google Scholar
  55. 55.
    Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. arXiv preprint arXiv:1906.05849 (2019)
  56. 56.
    Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle (2015)Google Scholar
  57. 57.
    Verma, V., et al.: Manifold mixup: better representations by interpolating hidden states (2018)Google Scholar
  58. 58.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders (2008)Google Scholar
  59. 59.
    Wagemans, J., et al.: A century of gestalt psychology in visual perception: I. perceptual grouping and figure-ground organization. Psychol. Bull. 138(6), 1172 (2012)Google Scholar
  60. 60.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)Google Scholar
  61. 61.
    Wang, J., Zhou, F., Wen, S., Liu, X., Lin, Y.: Deep metric learning with angular loss. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2593–2601 (2017)Google Scholar
  62. 62.
    Wang, X., Hua, Y., Kodirov, E., Hu, G., Garnier, R., Robertson, N.M.: Ranked list loss for deep metric learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  63. 63.
    Wang, X., Han, X., Huang, W., Dong, D., Scott, M.R.: Multi-similarity loss with general pair weighting for deep metric learning. CoRR abs/1904.06627 (2019), http://arxiv.org/abs/1904.06627
  64. 64.
    Wu, C.Y., Manmatha, R., Smola, A.J., Krahenbuhl, P.: Sampling matters in deep embedding learning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2840–2848 (2017)Google Scholar
  65. 65.
    Wu, Z., Xiong, Y., Yu, S., Lin, D.: Unsupervised feature learning via non-parametric instance-level discrimination (2018)Google Scholar
  66. 66.
    Xuan, H., Souvenir, R., Pless, R.: Deep randomized ensembles for metric learning. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 723–734 (2018)Google Scholar
  67. 67.
    Zhai, A., Wu, H.Y.: Classification is a strong baseline for deep metric learning (2018)Google Scholar
  68. 68.
    Zhao, Y., Jin, Z., Qi, G.J., Lu, H., Hua, X.S.: An adversarial approach to hard triplet generation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 501–517 (2018)Google Scholar
  69. 69.
    Zheng, W., Chen, Z., Lu, J., Zhou, J.: Hardness-aware deep metric learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Heidelberg Collaboratory for Image Processing (HCI), Heidelberg UniversityHeidelbergGermany
  2. 2.Mila, Universite de MontrealMontrealCanada
  3. 3.Vector Institute, Toronto Robotics InstituteTorontoCanada
  4. 4.University of TorontoTorontoCanada
  5. 5.CIFARTorontoCanada

Personalised recommendations