Advertisement

Deep Domain Adaptation

  • Zhengming Ding
  • Handong Zhao
  • Yun Fu
Chapter
Part of the Advanced Information and Knowledge Processing book series (AI&KP)

Abstract

Learning with limited labeled data is always a challenge in AI problems, and one of promising ways is transferring well-established source domain knowledge to the target domain, i.e., domain adaptation. Recent researches on transfer learning exploit deep structures for discriminative feature representation to tackle cross-domain disparity. However, few of them are able to joint feature learning and knowledge transfer in a unified deep framework. In this chapter, we develop three novel deep domain adaptation approaches for knowledge transfer. First, we propose a Deep Low-Rank Coding framework (DLRC) for transfer learning. The core idea of DLRC is to jointly learn a deep structure of feature representation and transfer knowledge via an iterative structured low-rank constraint, which aims to deal with the mismatch between source and target domains layer by layer. Second, we propose a novel Deep Transfer Low-rank Coding (DTLC) framework to uncover more shared knowledge across source and target in a multi-layer manner. Specifically, we extend traditional low-rank coding with one dictionary to multi-layer dictionaries by jointly building multiple latent common dictionaries shared by two domains. Third, we propose a novel deep model called “Deep Adaptive Exemplar AutoEncoder”, where we build a spectral bisection tree to generate source-target data compositions as the training pairs fed to autoencoders, and impose a low-rank coding regularizer to ensure the transferability of the learned hidden layer.

References

  1. Bahadori MT, Liu Y, Zhang D (2011) Learning with minimum supervision: a general framework for transductive transfer learning. In: IEEE ICDM. IEEE, pp 61–70Google Scholar
  2. Bartels RH, Stewart G (1972) Solution of the matrix equation ax + xb = c [f4]. Commun ACM 15(9):820–826CrossRefGoogle Scholar
  3. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828CrossRefGoogle Scholar
  4. Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: NIPS. MIT Press, Cambridge, pp 153–160Google Scholar
  5. Berry MW (1992) Large-scale sparse singular value computations. Int J Supercomput Appl 6(1):13–49CrossRefGoogle Scholar
  6. Bruzzone L, Marconcini M (2010) Domain adaptation problems: a dasvm classification technique and a circular validation strategy. IEEE Trans Pattern Anal Mach Intell 32(5):770–787CrossRefGoogle Scholar
  7. Cai J-F, Candès EJ, Shen Z (2010) A singular value thresholding algorithm for matrix completion. SIOPT 20(4):1956–1982MathSciNetCrossRefGoogle Scholar
  8. Chen M, Weinberger KQ, Sha F, Bengio Y (2014) Marginalized denoising auto-encoders for nonlinear representations. In: ICML, pp 1476–1484Google Scholar
  9. Chen M, Xu Z, Weinberger K, Sha F (2012) Marginalized denoising autoencoders for domain adaptation. In: ICML, pp 767–774Google Scholar
  10. Cui Z, Chang H, Shan S, Chen X (2014) Generalized unsupervised manifold alignment. In: NIPS, pp 2429–2437Google Scholar
  11. Ding Z, Fu Y (2014) Low-rank common subspace for multi-view learning. In: ICDM, pp 110–119Google Scholar
  12. Ding Z, Nasrabadi NM, Fu Y (2016) Task-driven deep transfer learning for image classification. In: 2016 IEEE international conference on acoustics, speech and signal processing. IEEE, pp 2414–2418Google Scholar
  13. Ding Z, Shao M, Fu Y (2014) Latent low-rank transfer subspace learning for missing modality recognition. In: AAAIGoogle Scholar
  14. Ding Z, Shao M, Fu Y (2015) Deep low-rank coding for transfer learning. In: Proceedings of the 24th international conference on artificial intelligence. AAAI Press, pp 3453–3459Google Scholar
  15. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: Proceedings of The 31st international conference on machine learning, pp 647–655Google Scholar
  16. Duan L, Xu D, Tsang IW (2012) Domain adaptation from multiple sources: a domain-dependent regularization approach. IEEE Trans Neural Netw Learn Syst 23(3):504–518CrossRefGoogle Scholar
  17. Erhan D, Manzagol P-A, Bengio Y, Bengio S, Vincent P (2009) The difficulty of training deep architectures and the effect of unsupervised pre-training. In: AISTATS, pp 153–160Google Scholar
  18. Fernando B, Habrard A, Sebban M, Tuytelaars T (2013) Unsupervised visual domain adaptation using subspace alignment. In: ICCV, pp 2960–2967Google Scholar
  19. Fernando B, Habrard A, Sebban M, Tuytelaars T (2013) Unsupervised visual domain adaptation using subspace alignment. In: IEEE ICCV. IEEE, pp 2960–2967Google Scholar
  20. Gao J, Fan W, Jiang J, Han J (2008) Knowledge transfer via multiple model local structure mapping. In: KDD, pp 283–291Google Scholar
  21. Glorot X, Bordes A, Bengio Y (2011) Domain adaptation for large-scale sentiment classification: a deep learning approach. In: ICML, pp 513–520Google Scholar
  22. Gong B, Shi Y, Sha F, Grauman K (2012) Geodesic flow kernel for unsupervised domain adaptation. In: CVPR, pp 2066–2073Google Scholar
  23. Gong B, Shi Y, Sha F, Grauman K (2012) Geodesic flow kernel for unsupervised domain adaptation. In: IEEE CVPR. IEEE, pp 2066–2073Google Scholar
  24. Gopalan R, Li R, Chellappa R (2011) Domain adaptation for object recognition: an unsupervised approach. In: ICCV, pp 999–1006Google Scholar
  25. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507MathSciNetCrossRefGoogle Scholar
  26. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366CrossRefGoogle Scholar
  27. Hu J, Lu J, Tan Y-P (2015) Deep transfer metric learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 325–333Google Scholar
  28. LeCun YA, Bottou L, Orr GB, Müller K-R (2012) Efficient backprop. Neural networks: tricks of the trade. Springer, Berlin, pp 9–48Google Scholar
  29. Lin Z, Chen M, Ma Y (2010) The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint arXiv:1009.5055
  30. Liu G, Lin Z, Yan S, Sun J, Yu Y, Ma Y (2013) Robust recovery of subspace structures by low-rank representation. TPAMI 35(1):171–184CrossRefGoogle Scholar
  31. Liu G, Lin Z, Yan S, Sun J, Yu Y, Ma Y (2013) Robust recovery of subspace structures by low-rank representation. IEEE TPAMI 35(1):171–184CrossRefGoogle Scholar
  32. Liu G, Lin Z, Yu Y (2010) Robust subspace segmentation by low-rank representation. In: Proceedings of the 27th international conference on machine learning, pp 663–670Google Scholar
  33. Long M, Wang J, Ding G, Pan SJ et al (2014) Adaptation regularization: a general framework for transfer learning. TKDE 26(5):1076–1089Google Scholar
  34. Long M, Wang J, Ding G, Shen D, Yang Q (2014) Transfer learning with graph co-regularization. IEEE TKDE 26(7):1805–1818Google Scholar
  35. Long M, Wang J, Ding G, Shen D, Yang Q (2014) Transfer learning with graph co-regularization. TKDE 26(7):1805–1818Google Scholar
  36. Long M, Ding G, Wang J, Sun J, Guo Y, Yu P (2013) Transfer sparse coding for robust image representation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 407–414Google Scholar
  37. Long M, Ding G, Wang J, Sun J, Guo Y, Yu PS (2013) Transfer sparse coding for robust image representation. In: CVPR, pp 407–414Google Scholar
  38. Long M, Ding G, Wang J, Sun J, Guo Y, Yu PS (2013) Transfer sparse coding for robust image representation. In: IEEE CVPR. IEEE, pp 407–414Google Scholar
  39. Long M, Wang J, Ding G, Sun J, Yu PS (2013) Transfer feature learning with joint distribution adaptation. In: IEEE international conference on computer visionGoogle Scholar
  40. Malo J, Epifanio I, Navarro R, Simoncelli EP (2006) Nonlinear image representation for efficient perceptual coding. IEEE Trans Image Process 15(1):68–80CrossRefGoogle Scholar
  41. Mesnil G, Dauphin Y, Glorot X, Rifai S, Bengio Y, Goodfellow IJ, Lavoie E, Muller X, Desjardins G, Warde-Farley D et al (2012) Unsupervised and transfer learning challenge: a deep learning approach. ICML Transfer Learning Workshop 27:97–110Google Scholar
  42. Nguyen HV, Ho HT, Patel VM, Chellappa R (2013) Joint hierarchical domain adaptation and feature learning. In: TPAMIGoogle Scholar
  43. Ni J, Qiu Q, Chellappa R (2013) Subspace interpolation via dictionary learning for unsupervised domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 692–699Google Scholar
  44. Nocedal J (1980) Updating quasi-newton matrices with limited storage. Math Comput 35(151):773–782MathSciNetCrossRefGoogle Scholar
  45. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE TKDE 22(10):1345–1359Google Scholar
  46. Pan SJ, Tsang IW, Kwok JT, Yang Q (2011) Domain adaptation via transfer component analysis. TNN 22(2):199–210Google Scholar
  47. Riesenhuber M, Poggio T (1999) Hierarchical models of object recognition in cortex. Nat Neurosci 2(11):1019–1025CrossRefGoogle Scholar
  48. Rumelhart DE, Hinton GE, Williams RJ (1988) Learning representations by back-propagating errors. Cognit Model 5:696–699zbMATHGoogle Scholar
  49. Shao M, Kit D, Fu Y (2014) Generalized transfer subspace learning through low-rank constraint. Int J Comput Vis 109(1–2):74–93MathSciNetCrossRefGoogle Scholar
  50. Shao M, Ding Z, Zhao H, Fu Y (2016) Spectral bisection tree guided deep adaptive exemplar autoencoder for unsupervised domain adaptation. In: Proceedings of the thirtieth AAAI conference on artificial intelligence. AAAI Press, pp 2023–2029Google Scholar
  51. Shao M, Kit D, Fu Y (2014) Generalized transfer subspace learning through low-rank constraint. In: IJCV, pp 1–20Google Scholar
  52. Shekhar S, Patel VM, Nguyen HV, Chellappa R (2013) Generalized domain-adaptive dictionaries. In: CVPR, pp 361–368Google Scholar
  53. Shekhar S, Patel V, Nguyen H, Chellappa R (2013) Generalized domain-adaptive dictionaries. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 361–368Google Scholar
  54. Si S, Tao D, Geng B (2010) Bregman divergence-based regularization for transfer subspace learning. IEEE TKDE 22(7):929–942Google Scholar
  55. Wang W, Arora R, Livescu K, Bilmes J (2015) On deep multi-view representation learning. In: Proceedings of the 32nd international conference on machine learning, pp 1083–1092Google Scholar
  56. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Advances in neural information processing systems, pp 3320–3328Google Scholar
  57. Zhang T, Ghanem B, Liu S, Xu C, Ahuja N (2013) Low-rank sparse coding for image classification. In: Proceedings of the IEEE international conference on computer vision, pp 281–288Google Scholar
  58. Zhang Y, Jiang Z, Davis LS (2013) Learning structured low-rank representations for image classification. In: CVPR, pp 676–683Google Scholar
  59. Zhang Q, Li B (2010) Discriminative k-svd for dictionary learning in face recognition. In: 2010 IEEE conference on computer vision and pattern recognition, pp 2691–2698Google Scholar
  60. Zhou JT, Pan SJ, Tsang IW, Yan Y (2014) Hybrid heterogeneous transfer learning through deep learning. In: AAAI, pp 2213–2220Google Scholar
  61. Zhuang F, Luo P, Xiong H, He Q, Xiong Y, Shi Z (2011) Exploiting associations between word clusters and document classes for cross-domain text categorization? Stat Anal Data Min ASA Data Sci J 4(1):100–114MathSciNetCrossRefGoogle Scholar
  62. Zhuang F, Cheng X, Luo P, Pan SJ, He Q (2015) Supervised representation learning: transfer learning with deep autoencoders. In: Proceedings of the 24th international conference on artificial intelligence, pp 4119–4125Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Indiana University-Purdue University IndianapolisIndianapolisUSA
  2. 2.Adobe ResearchSan JoseUSA
  3. 3.Northeastern UniversityBostonUSA

Personalised recommendations