Abstract
Word segmentation can contribute to improve the results of natural language processing tasks on several problem domains, including social media sentiment analysis, source code summarization and neural machine translation. Taking the English language as a case study, we fine-tune a Transformer architecture which has been trained through the Pre-trained Distillation (PD) algorithm, while comparing it to previous experiments with recurrent neural networks. We organize datasets and resources from multiple application domains under a unified format, and demonstrate that our proposed architecture has competitive performance and superior cross-domain generalization in comparison with previous approaches for word segmentation in Western languages.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
Standard Generalized Markup Language is a metalanguage through which it is possible to define markup languages for documents.
- 5.
- 6.
- 7.
- 8.
References
Sentiment140 - a Twitter sentiment analysis tool (2009). http://help.sentiment140.com/for-students
EMNLP 2017 - second conference on machine translation (WMT17) (2017). http://www.statmt.org/wmt17/. Accessed 17 May 2020
Celebi, A., Ozgur, A.: Segmenting hashtags and analyzing their grammatical structure. J. Assoc. Inf. Sci. Technol. 69 (2017). https://doi.org/10.1002/asi.23989
Chen, X., Shi, Z., Qiu, X., Huang, X.: Adversarial multi-criteria learning for Chinese word segmentation (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Doval, Y., Gómez-Rodríguez, C.: Comparing neural- and N-gram-based language models for word segmentation. J. Assoc. Inf. Sci. Technol. 70(2), 187–197 (2018). 10/gfs6rd, https://doi.org/10.1002/asi.24082, 105
Huang, W., Cheng, X., Chen, K., Wang, T., Chu, W.: Toward fast and accurate neural Chinese word segmentation with multi-criteria learning. arXiv:1903.04190 [cs], March 2019. http://arxiv.org/abs/1903.04190, 97.9
Inuzuka, M.A., Rocha, A.S., Nascimento, H.A.D.: Segmentation of words written in the Latin alphabet: a systematic review. In: Quaresma, P., Vieira, R., Aluísio, S., Moniz, H., Batista, F., Gonçalves, T. (eds.) PROPOR 2020. LNCS (LNAI), vol. 12037, pp. 291–302. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-41505-1_28
Lei, J.: Bert
(2019). Accessed 17 May 2020Li, J., Du, Q., Shi, K., He, Y., Wang, X., Xu, J.: Helpful or not? An investigation on the feasibility of identifier splitting via CNN-BiLSTM-CRF (2018)
Li, X., Meng, Y., Sun, X., Han, Q., Yuan, A., Li, J.: Is word segmentation necessary for deep learning of Chinese representations? arXiv preprint arXiv:1905.05526 (2019)
Ma, J., Ganchev, K., Weiss, D.: State-of-the-art Chinese word segmentation with Bi-LSTMs. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4902–4908. Association for Computational Linguistics, Brussels, October–November 2018. https://doi.org/10.18653/v1/D18-1529, https://www.aclweb.org/anthology/D18-1529
Maddela, M., Xu, W., Preoţiuc-Pietro, D.: Multi-task pairwise neural ranking for hashtag segmentation (2019)
Morabia, K., Bhanu Murthy, N.L., Malapati, A., Samant, S.: SEDTWik: segmentation-based event detection from tweets using Wikipedia. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 77–85. Association for Computational Linguistics, Minneapolis, June 2019. https://doi.org/10.18653/v1/N19-3011, https://www.aclweb.org/anthology/N19-3011
Pan, Y., Li, X., Yang, Y., Dong, R.: Morphological word segmentation on agglutinative languages for neural machine translation (2020)
Rhodes, D.: Conditional random field Latin word segmenter (2013)
Saenger, P.: Space Between Words: The Origins of Silent Reading. Stanford University Press, Palo Alto (1997)
Shi, X., Huang, H., Jian, P., Guo, Y., Wei, X., Tang, Y.-K.: Neural Chinese word segmentation as sequence to sequence translation. In: Cheng, X., Ma, W., Liu, H., Shen, H., Feng, S., Xie, X. (eds.) SMP 2017. CCIS, vol. 774, pp. 91–103. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-6805-8_8
Souza, F., Nogueira, R., Lotufo, R.: Portuguese named entity recognition using BERT-CRF. arXiv:1909.10649 [cs], February 2020. zSCC: NoCitationData[s0]
Sproat, R.W., Shih, C., Gale, W., Chang, N.: A stochastic finite-state word-segmentation algorithm for Chinese. Comput. Linguist. 22(3), 377–404 (1996). https://www.aclweb.org/anthology/J96-3004
Turc, I., Chang, M.W., Lee, K., Toutanova, K.: Well-read students learn better: on the importance of pre-training compact models (2019)
Wang, B., Pan, B., Li, X., Li, B.: Towards evaluating the robustness of Chinese BERT classifiers (2020)
Yang, H.: BERT meets Chinese word segmentation (2019)
Ye, Y., Zhang, Y., Li, W., Qiu, L., Sun, J.: Improving cross-domain Chinese word segmentation with word embeddings. In: Proceedings of the 2019 Conference of the North, pp. 2726–2735 (2019). https://doi.org/10.18653/v1/N19-1279, http://arxiv.org/abs/1903.01698, arXiv: 1903.01698
Zhang, Y., Clark, S.: Syntactic processing using the generalized perceptron and beam search. Comput. Linguist. 37(1), 105–151 (2011). https://doi.org/10.1162/coli_a_00037, https://www.aclweb.org/anthology/J11-1005
(2019). Accessed 17 May 2020Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Rodrigues, R.C., Rocha, A.S., Inuzuka, M.A., do Nascimento, H.A.D. (2020). Domain Adaptation of Transformers for English Word Segmentation. In: Cerri, R., Prati, R.C. (eds) Intelligent Systems. BRACIS 2020. Lecture Notes in Computer Science(), vol 12319. Springer, Cham. https://doi.org/10.1007/978-3-030-61377-8_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-61377-8_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61376-1
Online ISBN: 978-3-030-61377-8
eBook Packages: Computer ScienceComputer Science (R0)