Skip to main content
SpringerLink
Log in

Ensemble of loss functions to improve generalizability of deep metric learning methods

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The success of a Deep metric learning (DML) algorithm greatly depends on its loss function. However, no loss function is perfect and deals only with some aspects of an optimal similarity embedding. Besides, they omit the generalizability of the DML on unseen categories. To address these challenges, we propose novel approaches to combine different losses built on top of a shared deep network. The proposed ensemble of losses enforces the model to extract compatible features with all losses. Since the selected losses are diverse and emphasize different aspects of an optimal embedding, our effective combining method yields a considerable improvement over any individual loss and generalize well on unseen classes. It can optimize each loss function and its weight without imposing an additional hyper-parameter. We evaluate our methods on some popular datasets in a Zero-Shot-Learning setting. The results are very encouraging and show that our methods outperform all baseline losses by a large margin in all datasets. Specifically, the proposed method surpasses the best individual loss on the Cars-196 dataset by 10.37% and 9.54% in terms of Recall@1 and kNN accuracy respectively. Moreover, we develop a novel distance-based compression method that compresses the coefficient and embedding of losses into a single embedding vector. The size of the resulting embedding is identical to each baseline learner. Thus, it is fast as each baseline DML in the evaluation stage. Meanwhile, it outperforms the best individual loss on the Cars-196 dataset by 8.28% and 7.76% in terms of Recall@1 and kNN accuracy respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

Datasets used in the experiments are publicly available and can be downloaded from the following links:

1. Oxford 102 Flowers: https://www.robots.ox.ac.uk/~vgg/data/flowers/102/

2. CUB-200–2011: https://github.com/cyizhuo/CUB-200-2011-dataset

3. CARS-196: https://ai.stanford.edu/~jkrause/cars/car_dataset.html

Notes

  1. Normalized Mutual Information.

  2. Back Propagation.

  3. Content-Based Information Retrieval.

  4. Exponential Moving Average.

  5. Weighted Ensemble of Diverse Losses-DML.

  6. WEDL-Compact.

  7. Normalized Mutual Information.

  8. Multi-Scale Metric Learning.

  9. Few-Shot Learning.

  10. Back Propagation.

  11. Weighted Ensemble Losses DML.

  12. Weighted Ensemble of Diverse Losses-DML.

  13. WEDL-Compact.

  14. Normalized Mutual Information.

  15. General Discriminative Feature Learning.

References

  1. Al-Kaabi K, Monsefi R, Zabihzadeh D (2022) A framework to enhance generalization of deep metric learning methods using general discriminative feature learning and class adversarial neural networks. Appl Intell 1–19

  2. Chen B, Deng W (2019) Energy confused adversarial metric learning for zero-shot image retrieval and clustering. In Proc AAAI Conf Artif Intell. Honolulu, USA, January 27-February 1. AAAI Press, 33(1):8134–8141

  3. Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification, in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1: IEEE, pp 539–546

  4. Elezi I, Seidenschwarz J, Wagner L, Vascon S, Torcinovich A, Pelillo M, Leal-Taixe L (2022) The group loss++: a deeper look into group loss for deep metric learning. IEEE Trans Pattern Anal Mach Intell 45(2):2505–2518

    Article  Google Scholar 

  5. Ge W, Huang W, Dong D, Scott MR (2018) Deep metric learning with hierarchical triplet loss. In: Proc Eur Conf Comp Vis. Munich, Germany, September 8–14. Springer, pp 269–285. https://doi.org/10.1007/978-3-030-01231-1_17

  6. Gonzalez-Zapata J, Reyes-Amezcua I, Flores-Araiza D, Mendez-Ruiz M, Ochoa-Ruiz G, Mendez-Vazquez A (2022) Guided deep metric learning. In: Proc IEEE/CVF Conf Comput Visi Patt Recognit. Nashville, USA, June 19–25. IEEE, pp 1481–1489

  7. Gu G, Ko B, Kim H-G (2021) Proxy synthesis: learning with synthetic classes for deep metric learning. In: Proc AAAI Conf Artif Intell. Vancouver, Canada, February 2-9. AAAI Press, pp 1–8

  8. Hajiabadi H, Babaiyan V, Zabihzadeh D, Hajiabadi M (2020) Combination of loss functions for robust breast cancer prediction. Comput Electr Eng 84. https://doi.org/10.1016/j.compeleceng.2020.106624

  9. Hajiabadi H, Monsefi R, Yazdi HS (2019) Relf: robust regression extended with ensemble loss function. Appl Intell 49:1437–1450. https://doi.org/10.1007/s10489-018-1341-9

    Article  Google Scholar 

  10. Hoffer E, Ailon N (2015) Deep metric learning using triplet network. In: similarity-based pattern recognition. Third international workshop. Copenhagen, Denmark, October 12–14. Springer, pp 84–92.

  11. Jiang W, Huang K, Geng J, Deng X (2020) Multi-scale metric learning for few-shot learning. IEEE Trans Circuits Syst Video Technol 30(12):4454–4465. https://doi.org/10.1109/TCSVT.2020.2995754

    Article  Google Scholar 

  12. Kaya M, Bilge HŞ (2019) Deep metric learning: a survey. Symmetry 11(9):1066. https://doi.org/10.3390/sym11091066

    Article  ADS  Google Scholar 

  13. Kim S, Kim D, Cho M, Kwak S (2020) Proxy anchor loss for deep metric learning, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3238–3247

  14. Krause J, Stark M, Deng J, Fei-Fei L (2013) 3D object representations for fine-grained categorization. In: Proc IEEE Int Conf Comput Vis Workshops. Sydney, Australia, December 2–8. IEEE, pp 554–561. https://doi.org/10.1109/ICCVW.2013.77

  15. Li X, Yu L, Fu CW, Fang M, Heng PA (2020) Revisiting metric learning for few-shot image classification. Neurocomputing 406:49–58

    Article  Google Scholar 

  16. Milbich T, Roth K, Sinha S, Schmidt L, Ghassemi M, Ommer B (2021) Characterizing generalization under out-of-distribution shifts in deep metric learning. Adv Neural Inf Process Syst 34:25006–25018

    Google Scholar 

  17. Movshovitz-Attias Y, Toshev A, Leung TK, Ioffe S, Singh S (2017) No fuss distance metric learning using proxies, in Proceedings of the IEEE International Conference on Computer Vision, pp. 360–368

  18. Ni J, Liu J, Zhang C, Ye D, Ma Z (2017) Fine-grained patient similarity measuring using deep metric learning. In: Proc ACM Conf Inf Knowl Manag. Singapore, November 6–10. ACM, pp 1189–1198

  19. Nilsback M-E, Zisserman A (2008) Automated flower classification over a large number of classes, in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, IEEE, pp 722–729

  20. Oh Song H, Jegelka S, Rathod V, Murphy K (2017) "Deep metric learning via facility location". In: Proc IEEE Conf ComputVis Patt Recognit. Honolulu, USA, July 21–26. IEEE, pp 5382–5390

  21. Oh Song H, Xiang Y, Jegelka S, Savarese S (2016) Deep metric learning via lifted structured feature embedding, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4004–4012

  22. Opitz M, Waltner G, Possegger H, Bischof H (2018) Deep metric learning with bier: boosting independent embeddings robustly. IEEE Trans Pattern Anal Mach Intell 42(2):276–290

    Article  PubMed  Google Scholar 

  23. Qian Q, Shang L, Sun B, Hu J, Li H, Jin R (2019) Softtriple loss: Deep metric learning without triplet sampling. In: Proc IEEE Int Conf ComputVis. Seoul, Korea, October 27–November 2. IEEE/CVF, pp 6450–6458

  24. Rippel O, Paluri M, Dollar P, Bourdev L (2015) Metric learning with adaptive density discrimination, arXiv preprint arXiv:1511.05939

  25. Salman H, Taherinia AH, Zabihzadeh D (2023) Fast and accurate image retrieval using knowledge distillation from multiple deep pre-trained networks. Multimed Tools Appl 1–23

  26. Sohn K (2016) Improved deep metric learning with multi-class n-pair loss objective. In: Adv Neural Inf Proces Syst. Barcelona, Spain, December 5–10, pp 1857–1865

  27. Ustinova E, Lempitsky V (2016) Learning deep embeddings with histogram loss. Adv Neural Inf Process Syst 29:4170–4178

    Google Scholar 

  28. Wah C, Branson S, Welinder P, Perona P, Belongie S (2011) The Caltech-UCSD Birds-200–2011 Dataset, Computation & Neural Systems. Technical Report, CNS-TR-2011–001

  29. Wang J, Song Y, Leung T, Rosenberg C, Wang J, Philbin J, Chen B, Wu Y (2014) Learning fine-grained image similarity with deep ranking. In: Proc IEEE Conf Comput Vis Patt Recognit. Columbus, USA, June 23–28. IEEE, pp 1386–1393

  30. Wang J, Zhou F, Wen S, Liu X, Lin Y (2017) Deep metric learning with angular loss. In: Proc IEEE Int Conf Comput Vis. Venice, Italy, October 22–29. IEEE/CVF, pp 2593–2601

  31. Yao X, She D, Zhang H, Yang J, Cheng M-M, Wang L (2020) Adaptive deep metric learning for affective image retrieval and classification. IEEE Trans Multimedia 23:1640–1653

    Article  Google Scholar 

  32. Yuan T, Deng W, Tang J, Tang Y, Chen B (2019) Signal-to-noise ratio: A robust distance metric for deep metric learning, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4815–4824

  33. Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2017) mixup: Beyond empirical risk minimization, arXiv preprint arXiv:1710.09412

Download references

Acknowledgements

We would like to acknowledge the Machine Learning Lab in the Engineering Faculty of FUM for their kind and technical support.

Author information

Authors and Affiliations

  1. Computer Engineering Department, Hakim Sabzevari University, Sabzevar, Iran

    Davood Zabihzadeh & Seyed Jalaleddin Mousavirad

  2. Computer Engineering Department, Engineering Faculty, Ferdowsi University of Mashhad (FUM), Mashhad, Iran

    Zahraa Alitbi

Authors
  1. Davood Zabihzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zahraa Alitbi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seyed Jalaleddin Mousavirad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Davood Zabihzadeh.

Ethics declarations

Competing interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zabihzadeh, D., Alitbi, Z. & Mousavirad, S.J. Ensemble of loss functions to improve generalizability of deep metric learning methods. Multimed Tools Appl 83, 21525–21549 (2024). https://doi.org/10.1007/s11042-023-16160-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16160-9

Keywords

Search

Navigation