Advertisement

Deep Domain Adaptation for Regression

  • Ankita Singh
  • Shayok ChakrabortyEmail author
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 867)

Abstract

Developing machine learning algorithms under the constraint of limited labeled data has attracted significant attention in the research community in recent years. Domain adaptation or transfer learning algorithms alleviate this challenge by transferring relevant knowledge from a source domain to induce a model for a related target domain, where labeled data are scarce. Further, deep learning algorithms are instrumental in learning informative feature representations from a given dataset and have replaced the need for hand-crafted features. In this chapter, we propose a novel framework, DeepDAR, for domain adaptation for regression applications, using deep convolutional neural networks (CNNs). We formulate a loss function relevant to the research task and exploit the gradient descent algorithm to optimize the loss and train the deep CNN. To the best of our knowledge, domain adaptation for regression applications using deep neural networks has not been explored in the literature. Our extensive empirical studies on two popular regression applications (age estimation and head pose estimation from images) depict the merit of our framework over competing baselines.

Keywords

Domain adaptation Deep learning Regression 

References

  1. 1.
    Aldecoa, R., Marín, I.: Deciphering network community structure by surprise. PLoS ONE 6(9), e24195 (2011)CrossRefGoogle Scholar
  2. 2.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv:1701.07875 (2017)
  3. 3.
    Aytar, Y., Zisserman, A.: Tabula rasa: model transfer for object category detection. In: IEEE International Conference on Computer Vision (ICCV), pp. 2252–2259 (2011)Google Scholar
  4. 4.
    Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoderdecoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 39(12), 2481–2495 (2017)CrossRefGoogle Scholar
  5. 5.
    Bay, H., Tuytelaars, T., VanGool, L.: Surf: speeded up robust features. In: European Conference on Computer Vision (ECCV), pp. 404–417 (2006)CrossRefGoogle Scholar
  6. 6.
    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.: A theory of learning from different domains. Mach. Learn. 79(1–2), 151–175 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Bhattarai, B., Sharma, G., Lechervy, A., Jurie, F.: A joint learning approach for cross domain age estimation. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1901–1905 (2016)Google Scholar
  8. 8.
    Borghi, G., Fabbri, M., Vezzani, R., Calderara, S., Cucchiara, R.: Face-from-depth for head pose estimation on depth images. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2018)Google Scholar
  9. 9.
    Borgwardt, K., Gretton, A., Rasch, M., Kriegel, H., Scholkopf, B., Smola, A.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), e49–e57 (2006)CrossRefGoogle Scholar
  10. 10.
    Bruzzone, L., Marconcini, M.: Domain adaptation problems: a dasvm classification technique and a circular validation strategy. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 32(5), 770–787 (2010)CrossRefGoogle Scholar
  11. 11.
    Cao, B., Pan, S., Zhang, Y., Yeung, D., Yang, Q.: Adaptive transfer learning. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 407–412 (2010)Google Scholar
  12. 12.
    Chapelle, O., Scholkopf, B., Zien, A.: Semi-Supervised Learning. MIT Press (2006)Google Scholar
  13. 13.
    Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. In: British Machine Vision Conference (BMVC) (2014)Google Scholar
  14. 14.
    Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(4), 834–848 (2018)CrossRefGoogle Scholar
  15. 15.
    Cortes, C., Mohri, M.: Domain adaptation in regression. In: International Conference on Algorithmic Learning Theory, pp. 308–323 (2011)zbMATHGoogle Scholar
  16. 16.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 886–893 (2005)Google Scholar
  17. 17.
    Dong, X., Thanou, D., Frossard, P., Vandergheynst, P.: Learning laplacian matrix in smooth graph signal representations. IEEE Trans. Signal Process. 64(23), 6160–6173 (2016)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Duan, L., Tsang, I., Xu, D., Maybank, S.: Domain transfer SVM for video concept detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1375–1381 (2009)Google Scholar
  19. 19.
    Escalera, S., Fabian, J., et al., P.P.: Chalearn looking at people 2015: apparent age and cultural event recognition datasets and results. In: IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 243–251 (2015)Google Scholar
  20. 20.
    Fanelli, G., Dantone, M., GallA, J., Fossati, A., Gool, L.V.: Random forests for real time 3D face analysis. Int. J. Comput. Vis. (IJCV) 101(3), 437–458 (2013)CrossRefGoogle Scholar
  21. 21.
    Fernando, B., Habrard, A., Sebban, M., Tuytelaars, T.: Unsupervised visual domain adaptation using subspace alignment. In: IEEE Inrternational Conference on Computer Vision (ICCV), pp. 2960–2967 (2013)Google Scholar
  22. 22.
    Fu, Y., Huang, T.: Graph embedded analysis for head pose estimation. In: International Conference on Automatic Face and Gesture Recognition (2006)Google Scholar
  23. 23.
    Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. (JMLR) 17(1), 2096–2130 (2016)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Garcke, J., Vanck, T.: Importance weighted inductive transfer learning for regression. In: European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 466–481 (2014)CrossRefGoogle Scholar
  25. 25.
    Geng, X., Zhou, Z., Miles, K.: Automatic age estimation based on facial aging patterns. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 29(12), 2234–2240 (2007)CrossRefGoogle Scholar
  26. 26.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances of Neural Information Processing Systems (NIPS), pp. 2672–2680 (2014)Google Scholar
  27. 27.
    Gopalan, R., Li, R., Chellappa, R.: Domain adaptation for object recognition: an unsupervised approach. In: IEEE International Conference on Computer Vision (ICCV), pp. 999–1006 (2011)Google Scholar
  28. 28.
    Guan, Z., Li, A., Zhu, T.: Local regression transfer learning with applications to users’ psychological characteristics prediction. Brain Inf. 2(3), 145–153 (2015)CrossRefGoogle Scholar
  29. 29.
    Guo, G., Fu, Y., Dyer, C., Huang, T.: Image-based human age estimation by manifold learning and locally adjusted robust regression. IEEE Trans. Image Process. 17(7), 1178–1188 (2008)MathSciNetCrossRefGoogle Scholar
  30. 30.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  31. 31.
    Hu, L., Kan, M., Shan, S., Chen, X.: Duplex generative adversarial network for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1498–1507 (2018)Google Scholar
  32. 32.
    Jamal, A., Namboodiri, V., Deodhare, D., Venkatesh, K.: Deep domain adaptation in action space. In: British Machine Vision Conference (BMVC) (2018)Google Scholar
  33. 33.
    Khosla, A., Sarma, A., Hamid, R.: What makes an image popular? In: International World Wide Web Conference (WWW), pp. 867–876 (2014)Google Scholar
  34. 34.
    Kingma, D., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  35. 35.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems (NIPS) (2012)Google Scholar
  36. 36.
    Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 2267–2273 (2015)Google Scholar
  37. 37.
    Le, T., Nguyen, K., Ho, N., Bui, H., Phung, D.: Theoretical perspective of deep domain adaptation. arXiv:1811.06199 (2019)
  38. 38.
    LeCun, Y., Bottou, L., Orr, G., Muller, K.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp. 9–50 (1998)Google Scholar
  39. 39.
    Liang, Z., Zhang, G., Huang, J., Hu, Q.: Deep learning for healthcare decision making with EMRs. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 556–559 (2014)Google Scholar
  40. 40.
    Liu, M., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances of Neural Information Processing Systems (NIPS), pp 700–708 (2017)Google Scholar
  41. 41.
    Liu, M., Tuzel, O.: Coupled generative adversarial networks. In: Advances of Neural Information Processing Systems (NIPS), pp. 469–477 (2016)Google Scholar
  42. 42.
    Liu, S., Liu, S., Cai, W., Pujol, S., Kikinis, R., Feng, D.: Early diagnosis of Alzheimer’s disease with deep learning. In: IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1015–1018 (2014)Google Scholar
  43. 43.
    Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with deep adaptation networks. In: International Conference on Machine Learning (ICML), pp. 97–105 (2015)Google Scholar
  44. 44.
    Long, M., Zhu, H., Wang, J., Jordan, M.: Unsupervised domain adaptation with residual transfer networks. In: Advances of Neural Information Processing Systems (NIPS), pp. 136–144 (2016)Google Scholar
  45. 45.
    Luu, T., Low, K., Qu, X., Lim, H., Hoon, K.: An individual-specific gait pattern prediction model based on generalized regression neural networks. Gait and Posture 39(1), 443–448 (2014)CrossRefGoogle Scholar
  46. 46.
    Motiian, S., Jones, Q., Iranmanesh, S., Doretto, G.: Few-shot adversarial domain adaptation. In: Advances of Neural Information Processing Systems (NIPS) (2017)Google Scholar
  47. 47.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. International Journal of Computer Vision (IJCV) 42(3), 145–175 (2001)CrossRefGoogle Scholar
  48. 48.
    Pan, S., Tsang, I., Kwok, J., Yang, Q.: Domain adaptation via transfer component analysis. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 1187–1192 (2009)Google Scholar
  49. 49.
    Pan, S., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. (TKDE) 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  50. 50.
    Pan, S., Tsang, I., Kwok, J., Yang, Q.: Domain adaptation via transfer component analysis. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS) 22(2), 199–210 (2011)CrossRefGoogle Scholar
  51. 51.
    Pardoe, D., Stone, P.: Boosting for regression transfer. In: International Conference on Machine Learning (ICML), pp. 863–870 (2010)Google Scholar
  52. 52.
    Peng, K., Wu, Z., Ernst, J.: Zero-shot deep domain adaptation. In: European Conference on Computer Vision (ECCV), pp. 793–810 (2018)CrossRefGoogle Scholar
  53. 53.
    Pinto, H., Almeida, J., Goncalves, M.: Using early view patterns to predict the popularity of youtube videos. In: ACM International Conference on Web Search and Data Mining (WSDM), pp. 365–374 (2013)Google Scholar
  54. 54.
    Puheim, M., Madarasz, L.: Normalization of inputs and outputs of neural network based robotic arm controller in role of inverse kinematic model. In: IEEE International Symposium on Applied Machine Intelligence and Informatics, pp. 35–38 (2014)Google Scholar
  55. 55.
    Qi, F., Yang, X., Xu, C.: A unified framework for multimodal domain adaptation. In: ACM Multimedia Conference (ACM-MM), pp. 429–437 (2018)Google Scholar
  56. 56.
    Ranganathan, H., Venkateswara, H., Chakraborty, S., Panchanathan, S.: Deep active learning for image classification. In: IEEE International Conference on Image Processing (ICIP), pp. 3934–3938 (2017)Google Scholar
  57. 57.
    Ranganathan, H., Venkateswara, H., Chakraborty, S., Panchanathan, S.: Multi-label deep active learning with label correlation. In: IEEE International Conference on Image Processing (ICIP), pp. 3418–3422 (2018)Google Scholar
  58. 58.
    Rothe, R., Timofte, R., VanGool, L.: Deep expectation of real and apparent age from a single image without facial landmarks. Int. J. Comput. Vis. (IJCV) 126(2–4), 144–157 (2018)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Rozantsev, A., Salzmann, M., Fua, P.: Residual parameter transfer for deep domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4339–4348 (2018)Google Scholar
  60. 60.
    Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: European Conference on Computer Vision (ECCV), pp. 213–226 (2010)CrossRefGoogle Scholar
  61. 61.
    Sankaranarayanan, S., Balaji, Y., Castillo, C., Chellappa, R.: Generate to adapt: aligning domains using generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8503–8512 (2018)Google Scholar
  62. 62.
    Sener, O., Song, H., Saxena, A., Savarese, S.: Learning transferrable representations for unsupervised domain adaptation. In: Advances of Neural Information Processing Systems (NIPS), pp. 2118–2126 (2016)Google Scholar
  63. 63.
    Shen, J., Qu, Y., Zhang, W., Yu, Y.: Wasserstein distance guided representation learning for domain adaptation. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 4058–4065 (2018)Google Scholar
  64. 64.
    Sherrah, J., Gong, S.: Fusion of perceptual cues for robust tracking of head pose and position. Pattern Recogn. 34(8), 1565–1572 (2001)CrossRefGoogle Scholar
  65. 65.
    Smola, A., Kondor, R.: Kernels and regularization on graphs. In: Conference on Computational Learning Theory (COLT), pp. 144–158 (2003)CrossRefGoogle Scholar
  66. 66.
    Sriperumbudur, B., Fukumizu, K., Lanckriet, G.: Universality, characteristic kernels and RKHS embedding of measures. J. Mach. Learn. Res. (JMLR) 12, 2389–2410 (2011)Google Scholar
  67. 67.
    Sriperumbudur, B., Gretton, A., Fukumizu, K., Scholkopf, B., Lanckriet, G.: Hilbert space embeddings and metrics on probability measures. J. Mach. Learn. Res. (JMLR) 11, 1517–1561 (2010)Google Scholar
  68. 68.
    Steinwart, I.: On the influence of the kernel on the consistency of support vector machines. J. Mach. Learn. Res. (JMLR) 2, 67–93 (2001)Google Scholar
  69. 69.
    Stevenage, S., Nixon, M., Vince, K.: Visual analysis of gait as a cue to identity. Appl. Cognit. Psychol. 13(6), 513–526 (1999)CrossRefGoogle Scholar
  70. 70.
    Sun, B., Feng, J., Saenko, K.: Return of frustratingly easy domain adaptation. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 2058–2065 (2016)Google Scholar
  71. 71.
    Szabo, G., Huberman, B.: Predicting the popularity of online content. Commun. ACM 53(8), 80–88 (2010)CrossRefGoogle Scholar
  72. 72.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  73. 73.
    Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: IEEE International Conference on Computer Vision (ICCV), pp. 4068–4076 (2015)Google Scholar
  74. 74.
    Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7167–7176 (2017)Google Scholar
  75. 75.
    Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  76. 76.
    Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5018–5027 (2017)Google Scholar
  77. 77.
    Venkateswara, H., Lade, P., Ye, J., Panchanathan, S.: Coupled support vector machines for supervised domain adaptation. In: ACM Multimedia Conference (ACM-MM), pp. 1295–1298 (2015)Google Scholar
  78. 78.
    Volpi, R., Morerio, P., Savarese, S., Murino, V.: Adversarial feature augmentation for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5495–5504 (2018)Google Scholar
  79. 79.
    Winter, D.: The Biomechanics and Motor Control of Human Gait: Normal, Elderly, and Pathological. University of Waterloo Press (1991)Google Scholar
  80. 80.
    How to improve neural network stability and modeling performance with data scaling. https://machinelearningmastery.com/how-to-improve-neural-network-stabilityand-modeling-performance-with-data-scaling/
  81. 81.
    Yamada, M., Sigal, L., Chang, Y.: Domain adaptation for structured regression. Int. J. Comput. Vision 109(1–2), 126–145 (2014)CrossRefGoogle Scholar
  82. 82.
    Yan, K., Zheng, W., Cui, Z., Zong, Y., Zhang, T., Tang, C.: Unsupervised facial expression recognition using domain adaptation based dictionary learning approach. Neurocomputing 319, 84–91 (2018)CrossRefGoogle Scholar
  83. 83.
    Yun, Y., Kim, H., Shin, S., Lee, J., Deshpande, A., Kim, C.: Statistical method for prediction of gait kinematics with gaussian process regression. J. Biomech. 47(1), 186–192 (2014)CrossRefGoogle Scholar
  84. 84.
    Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances of Neural Information Processing Systems (NIPS), pp. 649–657 (2015)Google Scholar
  85. 85.
    Zhou, D., Scholkopf, B.: A regularization framework for learning from graph data. In: International Conference on Machine Learning Workshop (2004)Google Scholar
  86. 86.
    Zhu, R., Sang, G., Zhao, Q.: Discriminative feature adaptation for cross-domain facial expression recognition. In: International Conference on Biometrics (ICB) (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer ScienceFlorida State UniversityTallahasseeUSA

Personalised recommendations