Abstract
This article proposes an autoencoder-based domain invariant feature representation learning approach to domain adaptation and the cross-domain text classification problem. Finding domain invariant feature representations is a transfer learning method for transmitting knowledge between source and target domain data. Our method aims to avoid the overfitting of an autoencoder model on source domain training data in a trained embedded feature space using a target regularization technique. We hypothesize that when forcing the semantic similarity of target domain representation to source domain representation by adding the source domain similarity penalty to reconstruction loss during autoencoder training, the penalty is greater when the the domain’s representations separability is. In this work, we contribute to domain adaptation by demonstrating that a regularization technique based on an auxiliary pretrained domain classification model can be used to build robust, shared domain feature representations. Our model achieves a classification accuracy improvement in standard cross-domain sentiment classification tasks over the baseline model in most cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
The code is available at https://github.com/mmichall/atuda-pytorch.
- 3.
We experimented with \(P_n \in \{0.3, 0.5, 0.7, 0.9\}\).
References
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein Generative Adversarial Networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 214–223. PMLR (06–11 Aug 2017)
Ben-David, E., Rabinovitz, C., Reichart, R.: PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models. CoRR abs/2006.09075 (2020)
Ben Zaken, E., Goldberg, Y., Ravfogel, S.: BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1–9. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-short.1
Blitzer, J., Dredze, M., Pereira, F.: Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440–447. Association for Computational Linguistics, Prague, Czech Republic (Jun 2007). www.aclanthology.org/P07-1056
Blitzer, J., McDonald, R., Pereira, F.: Domain Adaptation with Structural Correspondence Learning. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 120–128. Association for Computational Linguistics, Sydney, Australia (2006). www.aclanthology.org/W06-1615
Chen, M., Xu, Z., Weinberger, K.Q., Sha, F.: Marginalized Denoising Autoencoders for Domain Adaptation. In: Proceedings of the 29th International Conference on International Conference on Machine Learning, pp. 1627–1634. ICML’12, Omnipress, Madison, WI, USA (2012)
Clinchant, S., Csurka, G., Chidlovskii, B.: A Domain Adaptation Regularization for Denoising Autoencoders. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 26–31. Association for Computational Linguistics, Berlin, Germany (2016). https://doi.org/10.18653/v1/P16-2005, www.aclanthology.org/P16-2005
Dong, X., de Melo, G.: A Helping Hand: Transfer Learning for Deep Sentiment Analysis. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2524–2534. Association for Computational Linguistics, Melbourne, Australia (Jul 2018). 10.18653/v1/P18-1235, www.aclanthology.org/P18-1235
Ganin, Y., Lempitsky, V.: Unsupervised Domain Adaptation by Backpropagation. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, p. 1180–1189. ICML’15, JMLR.org (2015)
Ganin, Y., et al.: Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res. 17(1), 2096–2030 (2016)
Glorot, X., Bordes, A., Bengio, Y.: Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach. In: Proceedings of the 28th International Conference on International Conference on Machine Learning. p. 513–520. ICML’11, Omnipress, Madison, WI, USA (2011)
Goodfellow, I., et al.: Generative Adversarial Nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc. (2014)
Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., Smola, A.: A Kernel Method for the Two-Sample-Problem. In: Schölkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems, vol. 19. MIT Press (2006)
Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardt, K., Schölkopf, B.: Covariate Shift by Kernel Mean Matching. In: Dataset Shift in Machine Learning. The MIT Press (12 2008). https://doi.org/10.7551/mitpress/9780262170055.003.0008
Han, X., Eisenstein, J.: Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling. In: EMNLP (2019)
Howard, J., Ruder, S.: Universal Language Model Fine-tuning for Text Classification, pp. 328–339 (2018). https://doi.org/10.18653/v1/P18-1031
Jiang, J., Zhai, C.: Instance Weighting for Domain Adaptation in NLP. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 264–271. Association for Computational Linguistics, Prague, Czech Republic (2007)
Li, Z., Wei, Y., Zhang, Y., Yang, Q.: Hierarchical Attention Transfer Network for Cross-Domain Sentiment Classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 no.(1) (2018). https://doi.org/10.1609/aaai.v32i1.12055, www.ojs.aaai.org/index.php/AAAI/article/view/12055
Li, Z., Zhang, Y., Wei, Y., Wu, Y., Yang, Q.: End-to-End Adversarial Memory Network for Cross-domain Sentiment Classification. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 2237–2243 (2017). https://doi.org/10.24963/ijcai.2017/311
Miller, T.: Simplified Neural Unsupervised Domain Adaptation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 414–419. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1039, www.aclanthology.org/N19-1039
Pan, S.J., Ni, X., Sun, J.T., Yang, Q., Chen, Z.: Cross-domain sentiment classification via spectral feature alignment. In: Proceedings of the 19th International Conference on World Wide Web. p. 751–760. WWW ’10, Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1772690.1772767
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191
Peng, M., Zhang, Q., Jiang, Y.g., Huang, X.: Cross-domain sentiment classification with target domain specific information. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2505–2513. Association for Computational Linguistics, Melbourne, Australia (Jul 2018). https://doi.org/10.18653/v1/P18-1233, www.aclanthology.org/P18-1233
Ramponi, A., Plank, B.: Neural Unsupervised Domain Adaptation in NLP–A Survey. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 6838–6855. International Committee on Computational Linguistics, Barcelona, Spain (Online) (2020). https://doi.org/10.18653/v1/2020.coling-main.603
Ramponi, A., Plank, B.: Neural Unsupervised Domain Adaptation in NLP-A Survey. ArXiv:abs/2006.00632 (2020)
Ruder, S., Plank, B.: Strong Baselines for Neural Semi-supervised Learning under Domain Shift. CoRR abs/1804.09530 (2018)
Saito, K., Ushiku, Y., Harada, T.: Asymmetric Tri-training for Unsupervised Domain Adaptation. CoRR abs/1702.08400 (2017)
Shen, J., Qu, Y., Zhang, W., Yu, Y.: Wasserstein Distance Guided Representation Learning for Domain Adaptation. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI’18/IAAI’18/EAAI’18, AAAI Press (2018)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ICML ’08, Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1390156.1390294
Yang, Y., Eisenstein, J.: Fast Easy Unsupervised Domain Adaptation with Marginalized Structured Dropout. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 538–544. Association for Computational Linguistics, Baltimore, Maryland (2014). https://doi.org/10.3115/v1/P14-2088, www.aclanthology.org/P14-2088
Zhuang, F., Cheng, X., Luo, P., Pan, S.J., He, Q.: Supervised representation learning: Transfer learning with deep autoencoders. In: IJCAI (2015)
Ziser, Y., Reichart, R.: Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 238–249. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1022, www.aclanthology.org/D18-1022
Ziser, Y., Reichart, R.: Pivot based language modeling for improved neural domain adaptation. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1241–1251. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-1112, www.aclanthology.org/N18-1112
Ziser, Y., Reichart, R.: Task refinement learning for improved accuracy and stability of unsupervised domain adaptation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5895–5906. Association for Computational Linguistics, Florence, Italy (2019). https://doi.org/10.18653/v1/P19-1591, www.aclanthology.org/P19-1591
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Perełkiewicz, M., Poświata, R., Kierzkowski, J. (2023). Unsupervised Representation Learning: Target Regularization for Cross-Domain Sentiment Classification. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2023. Lecture Notes in Computer Science(), vol 14125. Springer, Cham. https://doi.org/10.1007/978-3-031-42505-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-42505-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42504-2
Online ISBN: 978-3-031-42505-9
eBook Packages: Computer ScienceComputer Science (R0)