Skip to main content
SpringerLink
Log in

Attention-Guided Optimal Transport for Unsupervised Domain Adaptation with Class Structure Prior

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Unsupervised domain adaptation(UDA) methods based on optimal transport have been successfully used to improve cross-domain classification performance. Optimal transport aligns the distribution of source domain and target domain by minimizing the transport cost. However, the existing works based on optimal transport ignore the class-structure prior information of domains and do not adequately reflect the real data distribution. It always leads to be difficult in distinguishing target instances near the decision boundary. In this paper, we propose an end-to-end Attention-guided Optimal Transport (AOT) framework to achieve better domain adaptation. Concretely, first we introduce a weighted cost matrix based on the self-attention mechanism to reduce the bias caused by minibatch selection in training. It is realized by relating the prediction results in source and target domains. Meanwhile, a Jensen–Shannon divergence (JSD) regularization term is exploited to establish the mutual relationship between the feature space and the label space to achieve more reliable transport plan. Second, in order to enhance the discriminability of domain-invariant features using the class-structure prior, we also develop a pairwise metric learning strategy. It defines the positive/negative pairs by labels and enhances the class-structure prior by coupling feature and label similarities. Finally, we compare the proposed methods with ten SOTA approaches on multiple single source benchmarks and a multi-source benchmarks. The experimental results demonstrate that AOT achieves the best performance for classification tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ajakan H, Germain P, Larochelle H (2017) Domain-adversarial neural networks. J Mach Learn Res 17(1):2096–2030

    MathSciNet  Google Scholar 

  2. Balaji Y, Chellappa R, Feizi S (2020) Robust optimal transport with applications in generative modeling and domain adaptation. Adv Neural Inform Process Syst (NeurIPS) 33:12934–12944

    Google Scholar 

  3. Chen C, Chen Z, Jiang B, et al (2019) Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In: Proceedings of the AAAI conference on artificial intelligence, pp 3296–3303

  4. Chen M, Zhao S, Liu H, et al (2020) Adversarial-learned loss for domain adaptation. In: Proceedings of the AAAI conference on artificial intelligence, pp 3521–3528

  5. Chen W, Hu H (2020) Unsupervised domain adaptation via discriminative classes-center feature learning in adversarial network. Neural Process Lett 52(1):467–483

    Article  MathSciNet  Google Scholar 

  6. Chizat L, Peyré G, Schmitzer B et al (2018) Scaling algorithms for unbalanced transport problems. Math Comput 87:2563–2609

    Article  MathSciNet  Google Scholar 

  7. Courty N, Flamary R, Tuia D et al (2016) Optimal transport for domain adaptation. IEEE Trans Pattern Anal Mach Intell 39(9):1853–1865

    Article  Google Scholar 

  8. Courty N, Flamary R, Habrard A et al (2017) Joint distribution optimal transportation for domain adaptation. Adv Neural Inform Process Syst (NeurIPS) 30:1640–1650

    Google Scholar 

  9. Damodaran BB, Kellenberger B, Flamary R, et al (2018) Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In: European Conference on Computer Vision, pp 467–483

  10. Deng Z, Luo Y, Zhu J (2019) Cluster alignment with a teacher for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9944–9953

  11. Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. Adv Neural Inform Process Syst (NeurIPS) 27:2672–2680

    Google Scholar 

  12. Gretton A, Borgwardt K, Rasch M et al (2012) A kernel two-sample test. J Mach Learn Res 13:723–773

    MathSciNet  Google Scholar 

  13. Hamburger C (2021) T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9991–10,000

  14. HassanPour Zonoozi M, Seydi V (2022) A survey on adversarial domain adaptation. Neural Process Lett. https://doi.org/10.1007/s11063-022-10977-5

    Article  Google Scholar 

  15. Hauptman AG, Guoliang, Lu K (2019) Contrastive adaptation network for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4893–4902

  16. He C, Tan T, Fan X et al (2023) Noise-residual mixup for unsupervised adversarial domain adaptation. Appl Intell 53:3034–3047

    Article  Google Scholar 

  17. Khan S, Guo Y, Ye Y et al (2023) Mini-batch dynamic geometric embedding for unsupervised domain adaptation. Neural Process Lett 34:2063–2080

    Article  Google Scholar 

  18. Li D, Yang Y, Song YZ, et al (2017) Deeper, broader and artier domain generalization. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 5543–5551

  19. Li M, Zhai YM, Luo YW, et al (2020) Enhanced transport distance for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13,933–13,941

  20. Lin J (1991) Divergence measures based on the Shannon entropy. IEEE Trans Inf Theory 37(1):145–151

    Article  MathSciNet  Google Scholar 

  21. Liu YB, Han TT, Gao Z (2019) Pairwise generalization network for cross-domain image recognition. Neural Process Lett 52:1023–1041

    Article  Google Scholar 

  22. Long M, Cao Y, Wang J, et al (2015) Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd international conference on international conference on machine learning, pp 97–105

  23. Long M, Cao Y, Wang J, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 770–778

  24. Long M, Zhu H, Wang J, et al (2017) Deep transfer learning with joint adaptation networks. In: Proceedings of the 34th international conference on machine learning, pp 2208–2217

  25. Long M, Cao Z, Wang J et al (2018) Conditional adversarial domain adaptation. Adv Neural Inform Process Syst (NeurIPS) 31:1647–1657

    Google Scholar 

  26. Pan Y, Yao T, Li Y, et al (2019) Transferrable prototypical networks for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2234–2242

  27. Peng X, Bai Q, Xia X, et al (2019) Moment matching for multi-source domain adaptation. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 1406–1415

  28. Perrot M, Courty N, Flamary R et al (2023) Mapping estimation for discrete optimal transport. Adv Neural Inform Process Syst (NeurIPS) 29:4197–4205

    Google Scholar 

  29. Qian J, Wong WK, Zhang H et al (2022) Joint optimal transport with convex regularization for robust image classification. IEEE Trans Cybern 52(3):1553–1564

    Article  Google Scholar 

  30. Redko I, Habrard A, Sebban M (2017) Theoretical analysis of domain adaptation with optimal transport. In: Machine learning and knowledge discovery in databases, pp 737–753

  31. Saenko K, Kulis B, Fritz M, et al (2010) Adapting visual category models to new domains. In: ECCV, pp 213–226

  32. Saito K, Watanabe K (2018) Maximum classifier discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3723–3732

  33. Shen J, Qu Y, Zhang W, et al (2018) Wasserstein distance guided representation learning for domain adaptation. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 4058–4065

  34. Sun B, Saenko K (2016) Deep coral: correlation alignment for deep domain adaptation. In: European conference on computer vision, pp 443–450

  35. Sun H, Lin L, Liu N, et al (2021) Robust ensembling network for unsupervised domain adaptation. In: PRICAI 2021: Trends in Artificial Intelligence, pp 530–543

  36. Tzeng E, Hoffman J, Zhang N, et al (2014) Deep domain confusion: Maximizing for domain invariance. Preprint at https://arxiv.org/abs/1412.3474

  37. Tzeng E, Hoffman J, Saenko K, et al (2017) Adversarial discriminative domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2962–2971

  38. Venkateswara H, Eusebio J, Chakraborty S, et al (2017) Deep hashing network for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5018–5027

  39. Wang H, Xu M, Ni B, et al (2020) Learning to combine: Knowledge aggregation for multi-source domain adaptation. In: European Conference on Computer Vision(ECCV), pp 727–744

  40. Wang M, Deng W (2018) Deep visual domain adaptation: a survey. Neurocomputing 312:135–153

    Article  Google Scholar 

  41. Wang X, Han X, Huang W (2019) Multi-similarity loss with general pair weighting for deep metric learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5017–5025

  42. Xiao N, Zhang L (2021) Dynamic weighted learning for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15,242–15,251

  43. Xu R, Chen Z, Zuo W, et al (2018) Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3964-3973

  44. Xu R, Chen Z, Zuo W et al (2019) Domain generalization via model-agnostic learning of semantic features. Adv Neural Inform Process Syst (NeurIPS) 32:6450–6461

    Google Scholar 

  45. Xu R, Liu P, Wang L, et al (2020) Reliable weighted optimal transport for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4394–4403

  46. Zhang H, Yang J, Xie J et al (2017) Weighted sparse coding regularized nonconvex matrix regression for robust face recognition. Inf Sci 394–395:1–17

    MathSciNet  Google Scholar 

  47. Zhang H, Qian F, Shang F et al (2022) Global convergence guarantees of (a)gist for a family of nonconvex sparse learning problems. IEEE Trans Cybern 52(5):3276–3288

    Article  Google Scholar 

  48. Zhao H, Zhang S (2018) Adversarial multiple source domain adaptation. Adv Neural Inform Process Syst (NeurIPS) 31:8568–8579

    Google Scholar 

  49. Zhao S, Wang G, Zhang S (2020) Multi-source distilling domain adaptation. National Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence (AAAI) pp 12975–12983

  50. Zhou Q, Wang S, Xing Y (2021) Unsupervised domain adaptation with adversarial distribution adaptation network. Neural Comput Appl 33:7709–7721

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Key Research and Development Program of China (No. 2021YFA1003004)

Author information

Author notes

  1. Ying Li and Yanan Zhu have authors equally contributed to this work.

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Shangda Road, Shanghai, 200444, China

    Ying Li & Yanan Zhu

  2. School of Science, Shanghai University, Shangda Road, Shanghai, 200444, China

    Shihui Ying

Authors
  1. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shihui Ying
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SY and YL conceived the proposed idea and supervised the experimental work. YZ implemented the method and carried out the experiments. All authors discussed the results and contributed to the final manuscript

Corresponding author

Correspondence to Shihui Ying.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Zhu, Y. & Ying, S. Attention-Guided Optimal Transport for Unsupervised Domain Adaptation with Class Structure Prior. Neural Process Lett 55, 12547–12567 (2023). https://doi.org/10.1007/s11063-023-11432-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11432-9

Keywords

Search

Navigation