Abstract
With the development of deep learning, more and more researchers adopt deep neural networks for transfer learning. Compared to traditional machine learning, deep transfer learning increases the performance on various tasks. In addition, deep learning can take the vanilla data as the inputs, thus it has two more benefits: automatic feature extraction and end-to-end training. This chapter will introduce the basic of deep transfer learning, including network structure of deep transfer learning, distribution adaptation, structure adaptation, knowledge distillation, and practice.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Borgwardt, K. M., Gretton, A., Rasch, M. J., Kriegel, H.-P., Schölkopf, B., and Smola, A. J. (2006). Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):e49–e57.
Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D. (2016). Domain separation networks. In Advances in Neural Information Processing Systems, pages 343–351.
Cui, S., Wang, S., Zhuo, J., Li, L., Huang, Q., and Tian, Q. (2020). Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3941–3950.
Du, Y., Wang, J., Feng, W., Pan, S., Qin, T., Xu, R., and Wang, C. (2021). AdaRNN: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 402–411.
Ganin, Y. and Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. In ICML, pages 1180–1189.
Ghifary, M., Kleijn, W. B., and Zhang, M. (2014). Domain adaptive neural networks for object recognition. In PRICAI, pages 898–904.
Gong, B., Shi, Y., Sha, F., and Grauman, K. (2012). Geodesic flow kernel for unsupervised domain adaptation. In CVPR, pages 2066–2073.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. (2020). Knowledge distillation: A survey. arXiv preprint arXiv:2006.05525.
Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems (NIPS).
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1735–1780.
Hu, Z., Ma, X., Liu, Z., Hovy, E., and Xing, E. (2016). Harnessing deep neural networks with logic rules. In ACL.
Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105.
Kundu, J. N., Lakkakula, N., and Babu, R. V. (2019). Um-adapt: Unsupervised multi-task adaptation using adversarial cross-task distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1436–1445.
Li, Y., Wang, N., Shi, J., Hou, X., and Liu, J. (2018). Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80:109–117.
Long, M., Cao, Y., Wang, J., and Jordan, M. (2015). Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97–105.
Long, M., Zhu, H., Wang, J., and Jordan, M. I. (2017). Deep transfer learning with joint adaptation networks. In ICML, pages 2208–2217.
Maria Carlucci, F., Porzi, L., Caputo, B., et al. (2017). Autodial: Automatic domain alignment layers. In ICCV, pages 5067–5075.
Nayak, G. K., Mopuri, K. R., Shaj, V., Babu, R. V., and Chakraborty, A. (2019). Zero-shot knowledge distillation in deep networks. arXiv preprint arXiv:1905.08114.
Pan, S. J., Tsang, I. W., Kwok, J. T., and Yang, Q. (2011). Domain adaptation via transfer component analysis. IEEE TNN, 22(2):199–210.
Sun, B., Feng, J., and Saenko, K. (2016). Return of frustratingly easy domain adaptation. In AAAI.
Sun, B. and Saenko, K. (2016). Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pages 443–450.
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., and Darrell, T. (2014). Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S. (2017). Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027.
Wang, J., Chen, Y., Feng, W., Yu, H., Huang, M., and Yang, Q. (2020). Transfer learning with dynamic distribution adaptation. ACM TIST, 11(1):1–25.
Wang, J., Zheng, V. W., Chen, Y., and Huang, M. (2018). Deep transfer learning for cross-domain activity recognition. In proceedings of the 3rd International Conference on Crowd Science and Engineering, pages 1–8.
Yang, Y., Qiu, J., Song, M., Tao, D., and Wang, X. (2020). Distilling knowledge from graph convolutional networks. arXiv preprint arXiv:2003.10477.
Zhou, C., Neubig, G., and Gu, J. (2020). Understanding knowledge distillation in non-autoregressive machine translation. In ICLR.
Zhu, Y., Zhuang, F., Wang, J., Chen, J., Shi, Z., Wu, W., and He, Q. (2019). Multi-representation adaptation network for cross-domain image classification. Neural Networks, 119:214–221.
Zhu, Y., Zhuang, F., Wang, J., Ke, G., Chen, J., Bian, J., Xiong, H., and He, Q. (2020). Deep subdomain adaptation network for image classification. IEEE Transactions on Neural Networks and Learning Systems.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Wang, J., Chen, Y. (2023). Deep Transfer Learning. In: Introduction to Transfer Learning. Machine Learning: Foundations, Methodologies, and Applications. Springer, Singapore. https://doi.org/10.1007/978-981-19-7584-4_9
Download citation
DOI: https://doi.org/10.1007/978-981-19-7584-4_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-7583-7
Online ISBN: 978-981-19-7584-4
eBook Packages: Computer ScienceComputer Science (R0)