Co-training an Improved Recurrent Neural Network with Probability Statistic Models for Named Entity Recognition

  • Yueqing Sun
  • Lin LiEmail author
  • Zhongwei Xie
  • Qing Xie
  • Xin Li
  • Guandong Xu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10178)


Named Entity Recognition (NER) is a subtask of information extraction in Natural Language Processing (NLP) field and thus being wildly studied. Currently Recurrent Neural Network (RNN) has become a popular way to do NER task, but it needs a lot of train data. The lack of labeled train data is one of the hard problems and traditional co-training strategy is a way to alleviate it. In this paper, we consider this situation and focus on doing NER with co-training using RNN and two probability statistic models i.e. Hidden Markov Model (HMM) and Conditional Random Field (CRF). We proposed a modified RNN model by redefining its activation function. Compared to traditional sigmoid function, our new function avoids saturation to some degree and makes its output scope very close to [0, 1], thus improving recognition accuracy. Our experiments are conducted ATIS benchmark. First, supervised learning using those models are compared when using different train data size. The experimental results show that it is not necessary to use whole data, even small part of train data can also get good performance. Then, we compare the results of our modified RNN with original RNN. 0.5% improvement is obtained. Last, we compare the co-training results. HMM and CRF get higher improvement than RNN after co-training. Moreover, using our modified RNN in co-training, their performances are improved further.


Named entity recognition Co-training Recurrent neural network Probability statistic model Natural language processing 


  1. 1.
    Wahiba, B.A.K.: Named entity recognition using web document corpus. CoRR abs/1102.5728 (2011)Google Scholar
  2. 2.
    Lishuang, L., Liuke, J., Zhenchao, J., et al.: Biomedical named entity recognition based on extended Recurrent Neural Networks. In: BIBM, pp. 649–652 (2015)Google Scholar
  3. 3.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  4. 4.
    Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Eleventh Conference on Computational Learning Theory, pp. 92–100 (1998)Google Scholar
  5. 5.
    Li, L., Fan, W., Huang, D., et al.: Boosting performance of gene mention tagging system by hybrid methods. J. Biomed. Inform. 45(1), 156–164 (2012)CrossRefGoogle Scholar
  6. 6.
    Padmaja, S., Utpal, S., Jugal, K.: Named entity recognition in Assamese using CRFS and rules. In: IALP, pp. 15–18 (2014)Google Scholar
  7. 7.
    Tang, Z., Lingang, J., Yang, L., et al.: CRFs based parallel biomedical named entity recognition algorithm employing MapReduce framework. Cluster Comput. 18(2), 493–505 (2015)CrossRefGoogle Scholar
  8. 8.
    Ki-Joong, L., Young-Sook, H., Kim, S., et al.: Biomedical named entity recognition using two-phase model based on SVMs. J. Biomed. Inform. 37(6), 436–447 (2004)CrossRefGoogle Scholar
  9. 9.
    Gayen, V., Sarkar, K.: An HMM based named entity recognition system for indian languages: the JU system at ICON 2013. CoRR abs/1405.7397 (2014)Google Scholar
  10. 10.
    Sladojevic, S., Arsenovic, M., Anderia, A., et al.: Deep neural networks based recognition of plant diseases by leaf image classification. Comp. Int. Neurosc. 2016(6), 1–11 (2016)CrossRefGoogle Scholar
  11. 11.
    Janosek, M., Voln, E., Kotyrba, M.: Knowledge discovery in dynamic data using neural networks. Cluster Comput. 18(4), 1411–1421 (2015)CrossRefGoogle Scholar
  12. 12.
    Chollampatt, S., Kaveh, T., Hwee, T.N.: Neural network translation models for grammatical error correction. In: IJCAI, pp. 2768–2774 (2016)Google Scholar
  13. 13.
    Collobert, R., Weston, J., Bottou, L., et al.: Natural language processing (almost) from scratch. Mach. Learn. Res. 12, 2493–2537 (2011)zbMATHGoogle Scholar
  14. 14.
    Dingxin, S., Lishuang, L., Liuke, J., et al.: Biomedical named entity recognition based on recurrent neural networks with different extended methods. IJDMB 16(1), 17–31 (2016)CrossRefGoogle Scholar
  15. 15.
    Chiu, J.P.C., Nichols, E.: Named entity recognition with bidirectional LSTM-CNNs. TACL 4, 357–370 (2016)Google Scholar
  16. 16.
    Hoon, C., Sung, J.L., Jeon, G.P.: Deep neural network using trainable activation functions. In: IJCNN, pp. 348–352 (2016)Google Scholar
  17. 17.
    Anhao, X., Qingwei, Z., Yonghong, Y.: Speeding up deep neural networks in speech recognition with piecewise quantized sigmoidal activation function. IEICE Trans. 99-D(10), 2558–2561 (2016)Google Scholar
  18. 18.
    Liew, S.S., Khalil-Hani, M., Bakhteri, R.: Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing 216, 718–734 (2016)Google Scholar
  19. 19.
    Tsendsuren, M., Meijing, L., Unil, Y., et al.: An active co-training algorithm for biomedical named-entity recognition. JIPS 8(4), 575–588 (2012)Google Scholar
  20. 20.
    Li, Y., Huang, H., Zhao, X., Shi, S.: Named entity recognition based on bilingual co-training. In: Liu, P., Su, Q. (eds.) CLSW 2013. LNCS (LNAI), vol. 8229, pp. 480–489. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-45185-0_50 CrossRefGoogle Scholar
  21. 21.
    Qikang, W., Tao, C., Ruifeng, X., et al.: Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks. In: Database (2016)Google Scholar
  22. 22.
    Mikolov, T., Kara_t, M., Burget, L., et al.: Recurrent neural network based language model. In: INTERSPEECH, pp. 1045–1048 (2010)Google Scholar
  23. 23.
    Mesnil, G., He, X., Deng, L., et al.: Investigation of recurrent neural network architectures and learning methods for spoken language understanding. In: INTERSPEECH, pp. 3771–3775 (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yueqing Sun
    • 1
  • Lin Li
    • 1
    Email author
  • Zhongwei Xie
    • 1
  • Qing Xie
    • 1
  • Xin Li
    • 2
  • Guandong Xu
    • 3
  1. 1.School of Computer Science and TechnlogyWuhan University of TechnologyWuhanChina
  2. 2.iFLYTEK Big Data Research InstituteHefeiChina
  3. 3.School of SoftwareUniversity of Technology SydneyUltimoAustralia

Personalised recommendations