Abstract
The success of a Deep metric learning (DML) algorithm greatly depends on its loss function. However, no loss function is perfect and deals only with some aspects of an optimal similarity embedding. Besides, they omit the generalizability of the DML on unseen categories. To address these challenges, we propose novel approaches to combine different losses built on top of a shared deep network. The proposed ensemble of losses enforces the model to extract compatible features with all losses. Since the selected losses are diverse and emphasize different aspects of an optimal embedding, our effective combining method yields a considerable improvement over any individual loss and generalize well on unseen classes. It can optimize each loss function and its weight without imposing an additional hyper-parameter. We evaluate our methods on some popular datasets in a Zero-Shot-Learning setting. The results are very encouraging and show that our methods outperform all baseline losses by a large margin in all datasets. Specifically, the proposed method surpasses the best individual loss on the Cars-196 dataset by 10.37% and 9.54% in terms of Recall@1 and kNN accuracy respectively. Moreover, we develop a novel distance-based compression method that compresses the coefficient and embedding of losses into a single embedding vector. The size of the resulting embedding is identical to each baseline learner. Thus, it is fast as each baseline DML in the evaluation stage. Meanwhile, it outperforms the best individual loss on the Cars-196 dataset by 8.28% and 7.76% in terms of Recall@1 and kNN accuracy respectively.
Similar content being viewed by others
Data availability
Datasets used in the experiments are publicly available and can be downloaded from the following links:
1. Oxford 102 Flowers: https://www.robots.ox.ac.uk/~vgg/data/flowers/102/
2. CUB-200–2011: https://github.com/cyizhuo/CUB-200-2011-dataset
3. CARS-196: https://ai.stanford.edu/~jkrause/cars/car_dataset.html
Notes
Normalized Mutual Information.
Back Propagation.
Content-Based Information Retrieval.
Exponential Moving Average.
Weighted Ensemble of Diverse Losses-DML.
WEDL-Compact.
Normalized Mutual Information.
Multi-Scale Metric Learning.
Few-Shot Learning.
Back Propagation.
Weighted Ensemble Losses DML.
Weighted Ensemble of Diverse Losses-DML.
WEDL-Compact.
Normalized Mutual Information.
General Discriminative Feature Learning.
References
Al-Kaabi K, Monsefi R, Zabihzadeh D (2022) A framework to enhance generalization of deep metric learning methods using general discriminative feature learning and class adversarial neural networks. Appl Intell 1–19
Chen B, Deng W (2019) Energy confused adversarial metric learning for zero-shot image retrieval and clustering. In Proc AAAI Conf Artif Intell. Honolulu, USA, January 27-February 1. AAAI Press, 33(1):8134–8141
Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification, in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1: IEEE, pp 539–546
Elezi I, Seidenschwarz J, Wagner L, Vascon S, Torcinovich A, Pelillo M, Leal-Taixe L (2022) The group loss++: a deeper look into group loss for deep metric learning. IEEE Trans Pattern Anal Mach Intell 45(2):2505–2518
Ge W, Huang W, Dong D, Scott MR (2018) Deep metric learning with hierarchical triplet loss. In: Proc Eur Conf Comp Vis. Munich, Germany, September 8–14. Springer, pp 269–285. https://doi.org/10.1007/978-3-030-01231-1_17
Gonzalez-Zapata J, Reyes-Amezcua I, Flores-Araiza D, Mendez-Ruiz M, Ochoa-Ruiz G, Mendez-Vazquez A (2022) Guided deep metric learning. In: Proc IEEE/CVF Conf Comput Visi Patt Recognit. Nashville, USA, June 19–25. IEEE, pp 1481–1489
Gu G, Ko B, Kim H-G (2021) Proxy synthesis: learning with synthetic classes for deep metric learning. In: Proc AAAI Conf Artif Intell. Vancouver, Canada, February 2-9. AAAI Press, pp 1–8
Hajiabadi H, Babaiyan V, Zabihzadeh D, Hajiabadi M (2020) Combination of loss functions for robust breast cancer prediction. Comput Electr Eng 84. https://doi.org/10.1016/j.compeleceng.2020.106624
Hajiabadi H, Monsefi R, Yazdi HS (2019) Relf: robust regression extended with ensemble loss function. Appl Intell 49:1437–1450. https://doi.org/10.1007/s10489-018-1341-9
Hoffer E, Ailon N (2015) Deep metric learning using triplet network. In: similarity-based pattern recognition. Third international workshop. Copenhagen, Denmark, October 12–14. Springer, pp 84–92.
Jiang W, Huang K, Geng J, Deng X (2020) Multi-scale metric learning for few-shot learning. IEEE Trans Circuits Syst Video Technol 30(12):4454–4465. https://doi.org/10.1109/TCSVT.2020.2995754
Kaya M, Bilge HŞ (2019) Deep metric learning: a survey. Symmetry 11(9):1066. https://doi.org/10.3390/sym11091066
Kim S, Kim D, Cho M, Kwak S (2020) Proxy anchor loss for deep metric learning, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3238–3247
Krause J, Stark M, Deng J, Fei-Fei L (2013) 3D object representations for fine-grained categorization. In: Proc IEEE Int Conf Comput Vis Workshops. Sydney, Australia, December 2–8. IEEE, pp 554–561. https://doi.org/10.1109/ICCVW.2013.77
Li X, Yu L, Fu CW, Fang M, Heng PA (2020) Revisiting metric learning for few-shot image classification. Neurocomputing 406:49–58
Milbich T, Roth K, Sinha S, Schmidt L, Ghassemi M, Ommer B (2021) Characterizing generalization under out-of-distribution shifts in deep metric learning. Adv Neural Inf Process Syst 34:25006–25018
Movshovitz-Attias Y, Toshev A, Leung TK, Ioffe S, Singh S (2017) No fuss distance metric learning using proxies, in Proceedings of the IEEE International Conference on Computer Vision, pp. 360–368
Ni J, Liu J, Zhang C, Ye D, Ma Z (2017) Fine-grained patient similarity measuring using deep metric learning. In: Proc ACM Conf Inf Knowl Manag. Singapore, November 6–10. ACM, pp 1189–1198
Nilsback M-E, Zisserman A (2008) Automated flower classification over a large number of classes, in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, IEEE, pp 722–729
Oh Song H, Jegelka S, Rathod V, Murphy K (2017) "Deep metric learning via facility location". In: Proc IEEE Conf ComputVis Patt Recognit. Honolulu, USA, July 21–26. IEEE, pp 5382–5390
Oh Song H, Xiang Y, Jegelka S, Savarese S (2016) Deep metric learning via lifted structured feature embedding, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4004–4012
Opitz M, Waltner G, Possegger H, Bischof H (2018) Deep metric learning with bier: boosting independent embeddings robustly. IEEE Trans Pattern Anal Mach Intell 42(2):276–290
Qian Q, Shang L, Sun B, Hu J, Li H, Jin R (2019) Softtriple loss: Deep metric learning without triplet sampling. In: Proc IEEE Int Conf ComputVis. Seoul, Korea, October 27–November 2. IEEE/CVF, pp 6450–6458
Rippel O, Paluri M, Dollar P, Bourdev L (2015) Metric learning with adaptive density discrimination, arXiv preprint arXiv:1511.05939
Salman H, Taherinia AH, Zabihzadeh D (2023) Fast and accurate image retrieval using knowledge distillation from multiple deep pre-trained networks. Multimed Tools Appl 1–23
Sohn K (2016) Improved deep metric learning with multi-class n-pair loss objective. In: Adv Neural Inf Proces Syst. Barcelona, Spain, December 5–10, pp 1857–1865
Ustinova E, Lempitsky V (2016) Learning deep embeddings with histogram loss. Adv Neural Inf Process Syst 29:4170–4178
Wah C, Branson S, Welinder P, Perona P, Belongie S (2011) The Caltech-UCSD Birds-200–2011 Dataset, Computation & Neural Systems. Technical Report, CNS-TR-2011–001
Wang J, Song Y, Leung T, Rosenberg C, Wang J, Philbin J, Chen B, Wu Y (2014) Learning fine-grained image similarity with deep ranking. In: Proc IEEE Conf Comput Vis Patt Recognit. Columbus, USA, June 23–28. IEEE, pp 1386–1393
Wang J, Zhou F, Wen S, Liu X, Lin Y (2017) Deep metric learning with angular loss. In: Proc IEEE Int Conf Comput Vis. Venice, Italy, October 22–29. IEEE/CVF, pp 2593–2601
Yao X, She D, Zhang H, Yang J, Cheng M-M, Wang L (2020) Adaptive deep metric learning for affective image retrieval and classification. IEEE Trans Multimedia 23:1640–1653
Yuan T, Deng W, Tang J, Tang Y, Chen B (2019) Signal-to-noise ratio: A robust distance metric for deep metric learning, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4815–4824
Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2017) mixup: Beyond empirical risk minimization, arXiv preprint arXiv:1710.09412
Acknowledgements
We would like to acknowledge the Machine Learning Lab in the Engineering Faculty of FUM for their kind and technical support.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zabihzadeh, D., Alitbi, Z. & Mousavirad, S.J. Ensemble of loss functions to improve generalizability of deep metric learning methods. Multimed Tools Appl 83, 21525–21549 (2024). https://doi.org/10.1007/s11042-023-16160-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-16160-9