Skip to main content

A Brief Review of Domain Adaptation

Part of the Transactions on Computational Science and Computational Intelligence book series (TRACOSCI)

Abstract

Classical machine learning assumes that the training and test sets come from the same distributions. Therefore, a model learned from the labeled training data is expected to perform well on the test data. However, this assumption may not always hold in real-world applications where the training and the test data fall from different distributions, due to many factors, e.g., collecting the training and test sets from different sources or having an outdated training set due to the change of data over time. In this case, there would be a discrepancy across domain distributions, and naively applying the trained model on the new dataset may cause degradation in the performance. Domain adaptation is a subfield within machine learning that aims to cope with these types of problems by aligning the disparity between domains such that the trained model can be generalized into the domain of interest. This paper focuses on unsupervised domain adaptation, where the labels are only available in the source domain. It addresses the categorization of domain adaptation from different viewpoints. Besides, it presents some successful shallow and deep domain adaptation approaches that aim to deal with domain adaptation problems.

Keywords

  • Machine learning
  • Transfer learning
  • Domain adaptation
  • Classification
  • Risk minimization

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-71704-9_65
  • Chapter length: 18 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   149.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-71704-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   199.99
Price excludes VAT (USA)

References

  1. H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, Domain-adversarial neural networks (2014). arXiv preprint arXiv:1412.4446

    Google Scholar 

  2. S. Amirian, Z. Wang, T.R. Taha, H.R. Arabnia, Dissection of deep learning with applications in image recognition, In Computational Science and Computational Intelligence; “Artificial Intelligence” (CSCI-ISAI); 2018 International Conference on. IEEE (2018), pp. 1132–1138

    Google Scholar 

  3. E. Asali, F. Shenavarmasouleh, F.G. Mohammadi, P.S. Suresh, H.R. Arabnia, Deepmsrf: a novel deep multimodal speaker recognition framework with feature selection (2020). arXiv preprint arXiv:2007.06809

    Google Scholar 

  4. S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, J.W. Vaughan, A theory of learning from different domains. Mach. Learn. 79(1–2), 151–175 (2010)

    MathSciNet  CrossRef  Google Scholar 

  5. S. Ben-David, J. Blitzer, K. Crammer, F. Pereira, Analysis of representations for domain adaptation, in Advances in Neural Information Processing Systems (2007), pp. 137–144

    Google Scholar 

  6. A. Blum, T. Mitchell, Combining labeled and unlabeled data with co-training, in Proceedings of the Eleventh Annual Conference on Computational Learning Theory (ACM, 1998), pp. 92–100

    Google Scholar 

  7. Z. Bo, S. Zhong-Zhi, Z. Xiao-Fei, Z. Jian-hua, A transfer learning based on canonical correlation analysis across different domains Chinese. Chin. J. Comput. 38(7), 1326–1336 (2015)

    Google Scholar 

  8. L. Bottou, Stochastic gradient descent tricks, in Neural Networks: Tricks of the Trade (Springer, Berlin, 2012), pp. 421–436

    CrossRef  Google Scholar 

  9. K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, D. Krishnan, Unsupervised pixel-level domain adaptation with generative adversarial networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 3722–3731

    Google Scholar 

  10. Z. Cao, M. Long, J. Wang, M.I. Jordan, Partial transfer learning with selective adversarial networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 2724–2732

    Google Scholar 

  11. R. Caruana, Multitask learning. Machine Learn. 28(1), 41–75 (1997)

    MathSciNet  CrossRef  Google Scholar 

  12. M. Chen, Z. Xu, K. Weinberger, F. Sha, Marginalized denoising autoencoders for domain adaptation (2012). arXiv preprint arXiv:1206.4683

    Google Scholar 

  13. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014). arXiv preprint arXiv:1406.1078

    Google Scholar 

  14. G.E. Dahl, D. Yu, L. Deng, A. Acero, Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio Speech Lang. Process. 20(1), 30–42 (2011)

    CrossRef  Google Scholar 

  15. L. Deng, M.L. Seltzer, D. Yu, A. Acero, A.-R. Mohamed, G. Hinton, Binary coding of speech spectrograms using a deep auto-encoder, in Eleventh Annual Conference of the International Speech Communication Association (2010)

    Google Scholar 

  16. A. Eitel, J.T. Springenberg, L. Spinello, M. Riedmiller, W. Burgard, Multimodal deep learning for robust RGB-D object recognition, in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, Piscataway, 2015), pp. 681–687

    Google Scholar 

  17. B. Fernando, A. Habrard, M. Sebban, T. Tuytelaars, Unsupervised visual domain adaptation using subspace alignment, in Proceedings of the IEEE International Conference on Computer Vision (2013), pp. 2960–2967

    Google Scholar 

  18. Y. Ganin, V. Lempitsky, Unsupervised domain adaptation by backpropagation, in International Conference on Machine Learning (2015), pp. 1180–1189

    Google Scholar 

  19. M. Ghifary, W.B. Kleijn, M. Zhang, D. Balduzzi, W. Li, Deep reconstruction-classification networks for unsupervised domain adaptation, in European Conference on Computer Vision (Springer, Berlin, 2016), pp. 597–613

    Google Scholar 

  20. X. Glorot, A. Bordes, Y. Bengio, Domain adaptation for large-scale sentiment classification: A deep learning approach, in ICML (2011)

    Google Scholar 

  21. B. Gong, Y. Shi, F. Sha, K. Grauman, Geodesic flow kernel for unsupervised domain adaptation, in 2012 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2012), pp. 2066–2073

    CrossRef  Google Scholar 

  22. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in Advances in Neural Information Processing Systems (2014), pp. 2672–2680

    Google Scholar 

  23. R. Gopalan, R. Li, R. Chellappa, Domain adaptation for object recognition: an unsupervised approach, in 2011 International Conference on Computer Vision (IEEE, Piscataway, 2011), pp. 999–1006

    Google Scholar 

  24. A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, A.J. Smola, A kernel method for the two-sample-problem, in Advances in Neural Information Processing Systems (2007), pp. 513–520

    Google Scholar 

  25. A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, B. Schölkopf, Covariate shift by kernel mean matching. Dataset Shift Mach. Learn. 3(4), 5 (2009)

    Google Scholar 

  26. W.K. Härdle, M. Müller, S. Sperlich, A. Werwatz, Nonparametric and Semiparametric Models (Springer, Berlin, 2012)

    MATH  Google Scholar 

  27. J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A.A. Efros, T. Darrell, Cycada: cycle-consistent adversarial domain adaptation (2017). arXiv preprint arXiv:1711.03213

    Google Scholar 

  28. H. Hotelling, Relations between two sets of variates, in Breakthroughs in Statistics (Springer, Berlin, 1992), pp. 162–190

    Google Scholar 

  29. J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 7132–7141

    Google Scholar 

  30. J. Huang, A. Gretton, K. Borgwardt, B. Schölkopf, A.J. Smola, Correcting sample selection bias by unlabeled data, in Advances in Neural Information Processing Systems (2007), pp. 601–608

    Google Scholar 

  31. I.-H. Jhuo, D. Liu, D. Lee, S.-F. Chang, Robust visual domain adaptation with low-rank reconstruction, in 2012 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2012), pp. 2168–2175

    CrossRef  Google Scholar 

  32. G. Kang, L. Jiang, Y. Yang, A.G. Hauptmann, Contrastive adaptation network for unsupervised domain adaptation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019), pp. 4893–4902

    Google Scholar 

  33. S. Kullback, R.A. Leibler, On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)

    MathSciNet  CrossRef  Google Scholar 

  34. M.-Y. Liu, O. Tuzel, Coupled generative adversarial networks, in Advances in Neural Information Processing Systems (2016), pp. 469–477

    Google Scholar 

  35. K. Livescu, M. Stoehr, Multi-view learning of acoustic features for speaker recognition, in 2009 IEEE Workshop on Automatic Speech Recognition & Understanding (IEEE, Piscataway, 2009), pp. 82–86

    CrossRef  Google Scholar 

  36. M. Long, Y. Cao, J. Wang, M.I. Jordan, Learning transferable features with deep adaptation networks (2015). arXiv preprint arXiv:1502.02791

    Google Scholar 

  37. M. Long, J. Wang, G. Ding, J. Sun, P.S. Yu, Transfer feature learning with joint distribution adaptation, in Proceedings of the IEEE International Conference on Computer Vision (2013), pp. 2200–2207

    Google Scholar 

  38. M. Mohri, A. Rostamizadeh, A. Talwalkar, Foundations of Machine Learning (MIT Press, Cambridge, 2018)

    MATH  Google Scholar 

  39. K. Muandet, D. Balduzzi, B. Schölkopf, Domain generalization via invariant feature representation, in International Conference on Machine Learning (2013), pp. 10–18

    Google Scholar 

  40. W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S. Yang, Z. Wang, C.-C. Loy, et al., Deepid-net: deformable deep convolutional neural networks for object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 2403–2412

    Google Scholar 

  41. S. Pan, A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    CrossRef  Google Scholar 

  42. S.J. Pan, I.W. Tsang, J.T. Kwok, Q. Yang, Domain adaptation via transfer component analysis. IEEE Trans. Neural Netw. 22(2), 199–210 (2010)

    CrossRef  Google Scholar 

  43. P. Panareda Busto, J. Gall, Open set domain adaptation, in Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 754–763

    Google Scholar 

  44. Z. Pei, Z. Cao, M. Long, J. Wang, Multi-adversarial domain adaptation, in Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  45. N.N. Pise, P. Kulkarni, A survey of semi-supervised learning methods, in 2008 International Conference on Computational Intelligence and Security, vol. 2 (IEEE, Piscataway, 2008), pp. 30–34

    CrossRef  Google Scholar 

  46. V. Ramanishka, A. Das, D.H. Park, S. Venugopalan, L.A. Hendricks, M. Rohrbach, K. Saenko, Multimodal video description, in Proceedings of the 24th ACM International Conference on Multimedia (2016), pp. 1092–1096

    Google Scholar 

  47. A.M. Rush, S. Chopra, J. Weston, A neural attention model for abstractive sentence summarization (2015). arXiv preprint arXiv:1509.00685

    Google Scholar 

  48. K. Saito, S. Yamamoto, Y. Ushiku, T. Harada, Open set domain adaptation by backpropagation, in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 153–168

    Google Scholar 

  49. I.V. Serban, A. Sordoni, Y. Bengio, A. Courville, J. Pineau, Building end-to-end dialogue systems using generative hierarchical neural network models, in Thirtieth AAAI Conference on Artificial Intelligence (2016)

    Google Scholar 

  50. M. Shao, D. Kit, Y. Fu, Generalized transfer subspace learning through low-rank constraint. Int. J. Comput. Vis. 109(1–2), 74–93 (2014)

    MathSciNet  CrossRef  Google Scholar 

  51. H. Shimodaira, Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Stat. Plan. Inference 90(2), 227–244 (2000)

    MathSciNet  CrossRef  Google Scholar 

  52. A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, R. Webb, Learning from simulated and unsupervised images through adversarial training, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 2107–2116

    Google Scholar 

  53. M. Sugiyama, M. Krauledat, K.-R. MÞller, Covariate shift adaptation by importance weighted cross validation. J. Mach. Learn. Res. 8, 985–1005 (2007)

    MATH  Google Scholar 

  54. M. Sugiyama, K.-R. Müller, Input-dependent estimation of generalization error under covariate shift. Stat. Decis. 23(4/2005), 249–279 (2005)

    MathSciNet  MATH  Google Scholar 

  55. M. Sugiyama, K.-R. Müller, Model selection under covariate shift, in International Conference on Artificial Neural Networks (Springer, 2005), pp. 235–240

    Google Scholar 

  56. M. Sugiyama, S. Nakajima, H. Kashima, P.V. Buenau, M. Kawanabe, Direct importance estimation with model selection and its application to covariate shift adaptation, in Advances in Neural Information Processing Systems (2008), pp. 1433–1440

    Google Scholar 

  57. B. Sun, J. Feng, K. Saenko, Correlation alignment for unsupervised domain adaptation, in Domain Adaptation in Computer Vision Applications (Springer, Berlin, 2017), pp. 153–171

    CrossRef  Google Scholar 

  58. B. Sun, K. Saenko, Subspace distribution alignment for unsupervised domain adaptation, in BMVC, vol. 4 (2015), pp. 24–1

    Google Scholar 

  59. Y. Taigman, A. Polyak, L. Wolf, Unsupervised cross-domain image generation (2016). arXiv preprint arXiv:1611.02200

    Google Scholar 

  60. D. Tran, L. Bourdev, R. Fergus, L. Torresani, M. Paluri, Learning spatiotemporal features with 3d convolutional networks, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 4489–4497

    Google Scholar 

  61. E. Tzeng, J. Hoffman, T. Darrell, K. Saenko, Simultaneous deep transfer across domains and tasks, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 4068–4076

    Google Scholar 

  62. E. Tzeng, J. Hoffman, K. Saenko, T. Darrell, Adversarial discriminative domain adaptation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 7167–7176

    Google Scholar 

  63. V. Vapnik, Principles of risk minimization for learning theory, in Advances in Neural Information Processing Systems (1992), pp. 831–838

    Google Scholar 

  64. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, L. Bottou, Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  65. C. Wang, D.M. Blei, Collaborative topic modeling for recommending scientific articles, in Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2011), pp. 448–456

    Google Scholar 

  66. H. Wang, N. Wang, D.-Y. Yeung, Collaborative deep learning for recommender systems, in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015), pp. 1235–1244

    Google Scholar 

  67. K. You, M. Long, Z. Cao, J. Wang, M.I. Jordan, Universal domain adaptation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019), pp. 2720–2729

    Google Scholar 

  68. B. Zadrozny, Learning and evaluating classifiers under sample selection bias, in Proceedings of the Twenty-First International Conference on Machine Learning (2004), p. 114

    Google Scholar 

  69. M.D. Zeiler, D. Krishnan, G.W. Taylor, R. Fergus, Deconvolutional networks, in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2010), pp. 2528–2535

    CrossRef  Google Scholar 

  70. J. Zhang, Z. Ding, W. Li, P. Ogunbona, Importance weighted adversarial nets for partial domain adaptation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 8156–8164

    Google Scholar 

  71. L. Zhang, S. Wang, B. Liu, Deep learning for sentiment analysis: a survey. Wiley Interdiscip. Rev. Data Mining Knowl. Discovery 8(4), e1253 (2018)

    Google Scholar 

  72. X. Zhang, F.X. Yu, S.-F. Chang, S. Wang, Deep transfer network: unsupervised domain adaptation (2015). arXiv preprint arXiv:1503.00591

    Google Scholar 

  73. Z.-Q. Zhao, P. Zheng, S.-T. Xu, X. Wu, Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019)

    CrossRef  Google Scholar 

  74. J.-Y. Zhu, T. Park, P. A.A. Isola, Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 2223–2232

    Google Scholar 

  75. X.J. Zhu, Semi-supervised learning literature survey. Tech. rep., University of Wisconsin-Madison Department of Computer Sciences, 2005

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Abolfazl Farahani .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Farahani, A., Voghoei, S., Rasheed, K., Arabnia, H.R. (2021). A Brief Review of Domain Adaptation. In: Stahlbock, R., Weiss, G.M., Abou-Nasr, M., Yang, CY., Arabnia, H.R., Deligiannidis, L. (eds) Advances in Data Science and Information Engineering. Transactions on Computational Science and Computational Intelligence. Springer, Cham. https://doi.org/10.1007/978-3-030-71704-9_65

Download citation