Abstract
Document-level relation extraction (Doc-RE) aims to classify relations between entities spread over multiple sentences. When one entity is paired with separate entities, the importance of its mentions varies, which means the entity representation should be different. However, most of the previous RE models failed to make the relation classification entity-pair aware effectively. To that end, we propose a novel adaptation to simultaneously utilize the encoder and decoder of the sequence-to-sequence (Seq2Seq) pre-trained model BART in a non-generative manner to tackle the Doc-RE task. The encoder encodes the document to get the entity-aware contextualized mention representation. The decoder uses a non-causal self-attention mechanism and masked cross-attention to model the interactions between the entity pair under consideration explicitly. By doing so, we can fully take advantage of the pre-trained model in the encoder and decoder sides. And experiments in three Doc-RE datasets show that our model can not only take more advantage of BART, but surpass various BERT and RoBERTa based models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
The DocRED corpus provides each fact with evidence sentences which indicate the sentences used to infer this fact.
- 3.
We first average the attention scores along the head dimension.
References
Christopoulou, F., Miwa, M., Ananiadou, S.: Connecting the dots: document-level neural relation extraction with edge-oriented graphs. In: EMNLP-IJCNLP (2019)
Dai, D., Ren, J., Zeng, S., Chang, B., Sui, Z.: Coarse-to-fine entity representations for document-level relation extraction. CoRR abs/2012.02507 (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
Huang, Q., Zhu, S., Feng, Y., Ye, Y., Lai, Y., Zhao, D.: Three sentences are all you need: local path enhanced document relation extraction. In: ACL/IJCNLP (2021)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL (2020)
Li, J., et al.: Biocreative V CDR task corpus: a resource for chemical disease relation extraction. Database J. Biol, Databases Curation (2016)
Li, S., Ji, H., Han, J.: Document-level event argument extraction by conditional generation. CoRR abs/2104.05919 (2021). https://arxiv.org/abs/2104.05919
Liu, Y., et al.: Roberta: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019). http://arxiv.org/abs/1907.11692
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
Nan, G., Guo, Z., Sekulic, I., Lu, W.: Reasoning with latent structure refinement for document-level relation extraction. In: ACL (2020)
Paolini, G., et al.: Structured prediction as translation between augmented natural languages. In: ICLR (2021)
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., Huang, X.: Pre-trained models for natural language processing: a survey. CoRR abs/2003.08271 (2020)
Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune BERT for text classification? In: CCL (2019)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of NeuIPS (2017)
Wu, T., Li, X., et al.: Curriculum-meta learning for order-robust continual relation extraction. In: Proceedings of AAAI (2021)
Wu, Y., Luo, R., Leung, H.C.M., Ting, H., Lam, T.W.: RENET: a deep learning approach for extracting gene-disease associations from literature. In: RECOMB (2019)
Xu, B., Wang, Q., Lyu, Y., Zhu, Y., Mao, Z.: Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. In: Proceedings of AAAI (2021)
Xu, W., Chen, K., Zhao, T.: Document-level relation extraction with reconstruction. In: Proceedings of AAAI (2021)
Yao, Y., et al.: Docred: a large-scale document-level relation extraction dataset. In: ACL (2019)
Ye, D., Lin, Y., Du, J., Liu, Z., Sun, M., Liu, Z.: Coreferential reasoning learning for language representation. arXiv:2004.06870 (2020)
Ye, H., et al.: Contrastive triple extraction with generative transformer. In: Proceedings of AAAI (2021)
Zeng, S., Wu, Y., Chang, B.: SIRE: separate intra- and inter-sentential reasoning for document-level relation extraction. In: Findings of ACL/IJCNLP (2021)
Zeng, S., Xu, R., Chang, B., Li, L.: Double graph based reasoning for document-level relation extraction. In: EMNLP 2020 (2020)
Zhang, N., et al.: Document-level relation extraction as semantic segmentation. In: IJCAI (2021)
Zhang, Z., et al.: Document-level relation extraction with dual-tier heterogeneous graph. In: Proceedings of COLING (2020)
Zhou, W., Huang, K., Ma, T., Huang, J.: Document-level relation extraction with adaptive thresholding and localized context pooling. In: Proceedings of AAAI (2021)
Acknowledgements
We would like to thank the anonymous reviewers for their insightful comments. We also thank the developers of fastNLP and fitlog. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0108702) and National Natural Science Foundation of China (No. 62022027).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yan, H. et al. (2022). BART-Reader: Predicting Relations Between Entities via Reading Their Document-Level Context Information. In: Lu, W., Huang, S., Hong, Y., Zhou, X. (eds) Natural Language Processing and Chinese Computing. NLPCC 2022. Lecture Notes in Computer Science(), vol 13551. Springer, Cham. https://doi.org/10.1007/978-3-031-17120-8_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-17120-8_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-17119-2
Online ISBN: 978-3-031-17120-8
eBook Packages: Computer ScienceComputer Science (R0)
