Skip to main content
SpringerLink
Log in

Targeted BERT Pre-training and Fine-Tuning Approach for Entity Relation Extraction

  • Conference paper
  • First Online:
Data Science (ICPCSEE 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1452))

  • 1105 Accesses

Abstract

Entity relation extraction (ERE) is an important task in the field of information extraction. With the wide application of pre-training language model (PLM) in natural language processing (NLP), using PLM has become a brand new research direction of ERE. In this paper, BERT is used to extracting entity-relations, and a separated pipeline architecture is proposed. ERE was decomposed into entity-relation classification sub-task and entity-pair annotation sub-task. Both sub-tasks conduct the pre-training and fine-tuning independently. Combining dynamic and static masking, new Verb-MLM and Entity-MLM BERT pre-training tasks were put forward to enhance the correlation between BERT pre-training and Targeted NLP downstream task-ERE. Inter-layer sharing attention mechanism was added to the model, sharing the attention parameters according to the similarity of the attention matrix. Contrast experiment on the SemEavl 2010 Task8 dataset demonstrates that the new MLM task and inter-layer sharing attention mechanism improve the performance of BERT on the entity relation extraction effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Rink, B., Harabagiu, S.: Utd: classifying semantic relations by combining lexical and semantic resources. In: Proceedings of the 5th International Workshop on Semantic Evaluation, pp. 256–259 (2010)

    Google Scholar 

  2. Zelenko, D., Aone, C., Richardella, A.: Kernel methods for relation extraction. J. Mach. Learn. Res. 3, 1083–1106 (2003)

    Google Scholar 

  3. Fu, T.J., Li, P.H., Ma, W.Y.: GraphRel: modeling text as relational graphs for joint entity and relation extraction. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.1409–1418 (2019)

    Google Scholar 

  4. Han, X., Yu, P., Liu, Z., et al.: Hierarchical relation extraction with coarse-to-fine grained attention. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 2236–2245 (2018)

    Google Scholar 

  5. Jiang, X., Wang, Q., Li, P., et al.: Relation extraction with multi-instance multi-label convolutional neural networks. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1471–1480 (2016)

    Google Scholar 

  6. Lin, Y., Shen, S., Liu, Z., et al.: Neural relation extraction with selective attention over instances. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2124–2133 (2016)

    Google Scholar 

  7. Liu, C., Sun, W., Chao, W., Che, W.: Convolution neural network for relation extraction. In: Motoda, H., Wu, Z., Cao, L., Zaiane, O., Yao, M., Wang, W. (eds.) ADMA 2013. LNCS (LNAI), vol. 8347, pp. 231–242. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-53917-6_21

    Chapter  Google Scholar 

  8. Nguyen, T.H., Grishman, R.: Relation extraction: perspective from convolutional neural networks. In: Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 39–48 (2015)

    Google Scholar 

  9. Zeng, D., Liu, K., Lai, S., et al.: Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 2335–2344 (2014)

    Google Scholar 

  10. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)

  11. Devlin, J., Chang, M.W., Lee, K., et al.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  12. Mintz, M., Bills, S., Snow, R., et al.: Distant supervision for relation extraction without labeled data. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003–1011 (2009)

    Google Scholar 

  13. Zeng, D., Liu, K., Chen, Y., et al.: Distant supervision for relation extraction via piecewise convolutional neural networks. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1753–1762 (2015)

    Google Scholar 

  14. Ji, G., Liu, K., He, S., et al.: Distant supervision for relation extraction with sentence-level attention and entity descriptions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)

    Google Scholar 

  15. Du, J., Han, J., Way, A., et al.: Multi-level structured self-attentions for distantly supervised relation extraction. arXiv preprint arXiv:1809.00699 (2018)

  16. Mikolov, T., Chen, K., Corrado, G., et al.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  17. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  18. Peters, M.E., Neumann, M., Iyyer, M., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)

  19. Radford, A., Narasimhan, K., Salimans, T., et al.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  20. Lan, Z., Chen, M., Goodman, S., et al.: Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)

  21. Zhang, Z., Han, X., Liu, Z., et al.: ERNIE: enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129 (2019)

  22. Dehghani, M., Gouws, S., Vinyals, O., et al.: Universal transformers. arXiv preprint arXiv:1807.03819 (2018)

  23. Han, X., Zhu, H., Yu, P., et al.: Fewrel: a large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv preprint arXiv:1810.10147 (2018). Iris Hendrickx

  24. Hendrickx, I., Kim, S.N., Kozareva, Z., et al.: Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint arXiv:1911.10422 (2019)

Download references

Acknowledgment

The writing of this paper has received support and help from many people. Thanks to Professor Zhao Qiu, the corresponding author of the paper. This paper is supported by Hainan Province High level talent project of basic and applied basic research plan (Natural Science Field) in 2019 (No. 2019RC100), Haikou City Key Science and Technology Plan Project (2020–049), Hainan Province Key Research and Development Project (ZDYF2020018).

Author information

Authors and Affiliations

  1. School of Computer Science and Cyberspace Security, Hainan University, HaiKou, China

    Chao Li & Zhao Qiu

Authors
  1. Chao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhao Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. North University of China, Taiyuan, China

    Jianchao Zeng

  2. North University of China, Taiyuan, China

    Pinle Qin

  3. Northeast Forestry University, Harbin, China

    Weipeng Jing

  4. Harbin University of Science and Technology, Harbin, China

    Xianhua Song

  5. National Academy of Guo Ding Institute of Data Science, Beijing, China

    Zeguang Lu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, C., Qiu, Z. (2021). Targeted BERT Pre-training and Fine-Tuning Approach for Entity Relation Extraction. In: Zeng, J., Qin, P., Jing, W., Song, X., Lu, Z. (eds) Data Science. ICPCSEE 2021. Communications in Computer and Information Science, vol 1452. Springer, Singapore. https://doi.org/10.1007/978-981-16-5943-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-5943-0_10

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-5942-3

  • Online ISBN: 978-981-16-5943-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation