Abstract
Named Entity Recognition (NER) is a subtask of information extraction in Natural Language Processing (NLP) field and thus being wildly studied. Currently Recurrent Neural Network (RNN) has become a popular way to do NER task, but it needs a lot of train data. The lack of labeled train data is one of the hard problems and traditional co-training strategy is a way to alleviate it. In this paper, we consider this situation and focus on doing NER with co-training using RNN and two probability statistic models i.e. Hidden Markov Model (HMM) and Conditional Random Field (CRF). We proposed a modified RNN model by redefining its activation function. Compared to traditional sigmoid function, our new function avoids saturation to some degree and makes its output scope very close to [0, 1], thus improving recognition accuracy. Our experiments are conducted ATIS benchmark. First, supervised learning using those models are compared when using different train data size. The experimental results show that it is not necessary to use whole data, even small part of train data can also get good performance. Then, we compare the results of our modified RNN with original RNN. 0.5% improvement is obtained. Last, we compare the co-training results. HMM and CRF get higher improvement than RNN after co-training. Moreover, using our modified RNN in co-training, their performances are improved further.
This research project is supported by the National Social Science Foundation of China (Grant No:15BGL048), National Natural Science Foundation of China (Grant No:61602353, 61303029), 863 Program (2015AA015403), Hubei Province Science and Technology Support Project (2015BAA072).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wahiba, B.A.K.: Named entity recognition using web document corpus. CoRR abs/1102.5728 (2011)
Lishuang, L., Liuke, J., Zhenchao, J., et al.: Biomedical named entity recognition based on extended Recurrent Neural Networks. In: BIBM, pp. 649–652 (2015)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Eleventh Conference on Computational Learning Theory, pp. 92–100 (1998)
Li, L., Fan, W., Huang, D., et al.: Boosting performance of gene mention tagging system by hybrid methods. J. Biomed. Inform. 45(1), 156–164 (2012)
Padmaja, S., Utpal, S., Jugal, K.: Named entity recognition in Assamese using CRFS and rules. In: IALP, pp. 15–18 (2014)
Tang, Z., Lingang, J., Yang, L., et al.: CRFs based parallel biomedical named entity recognition algorithm employing MapReduce framework. Cluster Comput. 18(2), 493–505 (2015)
Ki-Joong, L., Young-Sook, H., Kim, S., et al.: Biomedical named entity recognition using two-phase model based on SVMs. J. Biomed. Inform. 37(6), 436–447 (2004)
Gayen, V., Sarkar, K.: An HMM based named entity recognition system for indian languages: the JU system at ICON 2013. CoRR abs/1405.7397 (2014)
Sladojevic, S., Arsenovic, M., Anderia, A., et al.: Deep neural networks based recognition of plant diseases by leaf image classification. Comp. Int. Neurosc. 2016(6), 1–11 (2016)
Janosek, M., Voln, E., Kotyrba, M.: Knowledge discovery in dynamic data using neural networks. Cluster Comput. 18(4), 1411–1421 (2015)
Chollampatt, S., Kaveh, T., Hwee, T.N.: Neural network translation models for grammatical error correction. In: IJCAI, pp. 2768–2774 (2016)
Collobert, R., Weston, J., Bottou, L., et al.: Natural language processing (almost) from scratch. Mach. Learn. Res. 12, 2493–2537 (2011)
Dingxin, S., Lishuang, L., Liuke, J., et al.: Biomedical named entity recognition based on recurrent neural networks with different extended methods. IJDMB 16(1), 17–31 (2016)
Chiu, J.P.C., Nichols, E.: Named entity recognition with bidirectional LSTM-CNNs. TACL 4, 357–370 (2016)
Hoon, C., Sung, J.L., Jeon, G.P.: Deep neural network using trainable activation functions. In: IJCNN, pp. 348–352 (2016)
Anhao, X., Qingwei, Z., Yonghong, Y.: Speeding up deep neural networks in speech recognition with piecewise quantized sigmoidal activation function. IEICE Trans. 99-D(10), 2558–2561 (2016)
Liew, S.S., Khalil-Hani, M., Bakhteri, R.: Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing 216, 718–734 (2016)
Tsendsuren, M., Meijing, L., Unil, Y., et al.: An active co-training algorithm for biomedical named-entity recognition. JIPS 8(4), 575–588 (2012)
Li, Y., Huang, H., Zhao, X., Shi, S.: Named entity recognition based on bilingual co-training. In: Liu, P., Su, Q. (eds.) CLSW 2013. LNCS (LNAI), vol. 8229, pp. 480–489. Springer, Heidelberg (2013). doi:10.1007/978-3-642-45185-0_50
Qikang, W., Tao, C., Ruifeng, X., et al.: Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks. In: Database (2016)
Mikolov, T., Kara_t, M., Burget, L., et al.: Recurrent neural network based language model. In: INTERSPEECH, pp. 1045–1048 (2010)
Mesnil, G., He, X., Deng, L., et al.: Investigation of recurrent neural network architectures and learning methods for spoken language understanding. In: INTERSPEECH, pp. 3771–3775 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Sun, Y., Li, L., Xie, Z., Xie, Q., Li, X., Xu, G. (2017). Co-training an Improved Recurrent Neural Network with Probability Statistic Models for Named Entity Recognition. In: Candan, S., Chen, L., Pedersen, T., Chang, L., Hua, W. (eds) Database Systems for Advanced Applications. DASFAA 2017. Lecture Notes in Computer Science(), vol 10178. Springer, Cham. https://doi.org/10.1007/978-3-319-55699-4_33
Download citation
DOI: https://doi.org/10.1007/978-3-319-55699-4_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-55698-7
Online ISBN: 978-3-319-55699-4
eBook Packages: Computer ScienceComputer Science (R0)