Abstract
This paper describes our system designed for the NLPCC 2016 shared task on word segmentation on micro-blog texts (i.e., Weibo). We treat word segmentation as a character-wise sequence labeling problem, and explore two directions to enhance our CRF-based baseline. First, we employ a large-scale external lexicon for constructing extra lexicon features in the model, which is proven to be extremely useful. Second, we exploit two heterogeneous datasets, i.e., Penn Chinese Treebank 7 (CTB7) and People Daily (PD) to help word segmentation on Weibo. We adopt two mainstream approaches, i.e., the guide-feature based approach and the recently proposed coupled sequence labeling approach. We combine the above techniques in different ways and obtain four well-performing models. Finally, we merge the outputs of the four models and obtain the final results via Viterbi-based re-decoding. On the test data of Weibo, our proposed approach outperforms the baseline by \(95.63-94.24=1.39\%\) in terms of F1 score. Our final system rank the first place among five participants in the open track in terms of F1 score, and is also the best among all 28 submissions. All codes, experiment configurations, and the external lexicon are released at http://hlt.suda.edu.cn/~zhli.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Please refer to http://zhangkaixu.github.io/bibpage/cws.html for a long list of related papers.
- 2.
We are very grateful for their kind sharing. Their dictionary is composed of several word lists, the SogouW word dictionary (http://www.sogou.com/labs/resource/w.php), and a few lists on different domains (finance, sports, and entertainment) from the lexicon sharing website of Sogou (http://pinyin.sogou.com/dict/).
References
Chang, P.C., Galley, M., Manning., C.D.: Optimizing Chinese word segmentation for machine translation performance. In: ACL 2008 Third Workshop on Statistical Machine Translation (2008)
Chao, J., Li, Z., Chen, W., Zhang, M.: Exploiting heterogeneous annotations for weibo word segmentation and POS tagging. In: Li, J., Ji, H., Zhao, D., Feng, Y. (eds.) NLPCC 2015. LNCS (LNAI), vol. 9362, pp. 495–506. Springer, Heidelberg (2015). doi:10.1007/978-3-319-25207-0_46
Jiang, W., Huang, L., Liu, Q.: Automatic adaptation of annotation standards: Chinese word segmentation and POS tagging - a case study. In: Proceedings of ACL, pp. 522–530 (2009)
Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: probabilistic models for segmenting and labeling sequence data (2001)
Li, Z., Chao, J., Zhang, M.: Fast coupled sequence labeling on heterogeneous annotations via context-aware pruning. In: Proceedings of EMNLP (2016)
Li, Z., Chao, J., Zhang, M., Chen, W.: Coupled sequence labeling on heterogeneous annotations: POS tagging as a case study. In: Proceedings of ACL (2015)
Li, Z., Che, W., Liu, T.: Exploiting multiple treebanks for parsing with quasisynchronous grammar. In: ACL, pp. 675–684 (2012)
Qiu, X., Qian, P., Shi, Z.: Overview of the NLPCC-ICCPOL 2016 shared task: Chinese word segmentation for micro-blog texts. In: Proceedings of the Fifth Conference on Natural Language Processing and Chinese Computing & the Twenty Fourth International Conference on Computer Processing of Oriental Languages (2016)
Qiu, X., Zhao, J., Huang, X.: Joint Chinese word segmentation and POS tagging on heterogeneous annotated corpora with multiple task learning. In: Proceedings of EMNLP, pp. 658–668 (2013)
Sagae, K., Lavie, A.: Parser combination by reparsing. In: Proceedings of NAACL, pp. 129–132 (2006)
Sun, W.: A stacked sub-word model for joint Chinese word segmentation and part-of-speech tagging. In: Proceedings of ACL, pp. 1385–1394 (2011)
Sun, W., Wan, X.: Reducing approximation and estimation errors for Chinese lexical processing with heterogeneous annotations. In: Proceedings of ACL, pp. 232–241 (2012)
Xue, N., et al.: Chinese word segmentation as character tagging. Comput. Linguist. Chin. Lang. Process. 8(1), 29–48 (2003)
Yu, Z., Dai, X.-Y., Shen, S., Huang, S., Chen, J.: Word segmentation of micro blogs with bagging. In: Li, J., Ji, H., Zhao, D., Feng, Y. (eds.) NLPCC 2015. LNCS (LNAI), vol. 9362, pp. 573–580. Springer, Heidelberg (2015). doi:10.1007/978-3-319-25207-0_54
Zhang, M., Deng, Z., Che, W., Liu, T.: Combining statistical model and dictionary for domain adaption of Chinese word segmentation. J. Chin. Inf. Process. 26, 8–12 (2012)
Zhang, M., Zhang, Y., Che, W., Liu, T.: Character-level Chinese dependency parsing. In: Proceedings of ACL, pp. 1326–1336 (2014)
Acknowledgments
The authors would like to thank the anonymous reviewers for the helpful comments. This work was supported by National Natural Science Foundation of China (Grant No. 61502325, 61432013) and the Natural Science Foundation of the Jiangsu Higher Education Institutions of China (No. 15KJB520031).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Xia, Q., Li, Z., Chao, J., Zhang, M. (2016). Word Segmentation on Micro-Blog Texts with External Lexicon and Heterogeneous Data. In: Lin, CY., Xue, N., Zhao, D., Huang, X., Feng, Y. (eds) Natural Language Understanding and Intelligent Applications. ICCPOL NLPCC 2016 2016. Lecture Notes in Computer Science(), vol 10102. Springer, Cham. https://doi.org/10.1007/978-3-319-50496-4_64
Download citation
DOI: https://doi.org/10.1007/978-3-319-50496-4_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-50495-7
Online ISBN: 978-3-319-50496-4
eBook Packages: Computer ScienceComputer Science (R0)