Abstract
Relation Extraction (RE) task aims to discover the semantic relation that holds between two entities and contributes to many applications such as knowledge graph construction and completion. Reinforcement Learning (RL) has been widely used for RE task and achieved SOTA results, which are mainly designed with rewards to choose the optimal actions during the training procedure, to improve RE’s performance, especially for low-resource conditions. Recent work has shown that offline or online RL can be flexibly formulated as a sequence understanding problem and solved via approaches similar to large-scale pre-training language modeling. To strengthen the ability for understanding the semantic signals interactions among the given text sequence, this paper leverages Transformer architecture for RL-based RE methods, and proposes a generic framework called Transformer Enhanced RL (TERL) towards RE task. Unlike prior RL-based RE approaches that usually fit value functions or compute policy gradients, TERL only outputs the best actions by utilizing a masked Transformer. Experimental results show that the proposed TERL framework can improve many state-of-the-art RL-based RE methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Other transformer architecture is also applicable.
References
Chen, L., et al.: Decision transformer: reinforcement learning via sequence modeling. In: NeurIPS (2021)
Chen, Y., Su, J., Wei, W.: Multi-granularity textual adversarial attack with behavior cloning. In: EMNLP (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
Gardent, C., Shimorina, A., Narayan, S., Perez-Beltrachini, L.: Creating training corpora for NLG micro-planners. In: ACL (2017)
Gormley, M.R., Yu, M., Dredze, M.: Improved relation extraction with feature-rich compositional embedding models. In: EMNLP (2015)
Guo, J., Fan, Y., Ai, Q., Croft, W.B.: A deep relevance matching model for ad-hoc retrieval. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (2016)
Guo, Z., Nan, G., Lu, W., Cohen, S.B.: Learning latent forests for medical relation extraction. In: IJCAI (2020)
Hendrickx, I., et al.: SemEval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. In: *SEMEVAL (2010)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., Weld, D.S.: Knowledge-based weak supervision for information extraction of overlapping relations. In: ACL (2011)
Hu, X., Ma, F., Liu, C., Zhang, C., Wen, L., Yu, P.S.: Semi-supervised relation extraction via incremental meta self-training. In: EMNLP (2021)
Hu, X., et al.: Gradient imitation reinforcement learning for low resource relation extraction. In: EMNLP (2021)
Hung, C.C., et al.: Optimizing agent behavior over long time scales by transporting value. Nat. Commun. 10, 5223 (2019)
Janner, M., Li, Q., Levine, S.: Reinforcement learning as one big sequence modeling problem. arXiv abs/2106.02039 (2021)
Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP (2014)
Lee, K., Laskin, M., Srinivas, A., Abbeel, P.: Sunrise: a simple unified framework for ensemble learning in deep reinforcement learning. In: ICML (2021)
Li, Q., Ji, H.: Incremental joint extraction of entity mentions and relations. In: ACL (2014)
Li, Z., Sun, Y., Tang, S., Zhang, C., Ma, H.: Reinforcement learning with dual attention guided graph convolution for relation extraction. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 946–953 (2021)
Lin, H., Yan, J., Qu, M., Ren, X.: Learning dual retrieval module for semi-supervised relation extraction. In: The World Wide Web Conference (2019)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
Narasimhan, K., Yala, A., Barzilay, R.: Improving information extraction by acquiring external evidence with reinforcement learning. In: EMNLP (2016)
Parisotto, E., Salakhutdinov, R.: Efficient transformers in reinforcement learning using actor-learner distillation. arXiv abs/2104.01655 (2021)
Parisotto, E., et al.: Stabilizing transformers for reinforcement learning. In: ICML (2020)
Qin, P., Xu, W., Wang, W.Y.: Robust distant supervision relation extraction via deep reinforcement learning. In: ACL (2018)
Radford, A., Narasimhan, K.: Improving language understanding by generative pre-training (2018)
Ramesh, A., et al.: Zero-shot text-to-image generation. arXiv abs/2102.12092 (2021)
Ren, X., et al.: Cotype: joint extraction of typed entities and relations with knowledge bases. In: Proceedings of the 26th International Conference on World Wide Web (2017)
Riedel, S., Yao, L., McCallum, A.: Modeling relations and their mentions without labeled text. In: ECML/PKDD (2010)
Rosenberg, C., Hebert, M., Schneiderman, H.: Semi-supervised self-training of object detection models. In: 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION 2005), vol. 1, pp. 29–36 (2005)
Takanobu, R., Zhang, T., Liu, J., Huang, M.: A hierarchical framework for relation extraction with reinforcement learning. In: AAAI (2018)
Takanobu, R., Zhang, T., Liu, J., Huang, M.: A hierarchical framework for relation extraction with reinforcement learning. arXiv abs/1811.03925 (2019)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: NIPS (2017)
Vaswani, A., et al.: Attention is all you need. arXiv abs/1706.03762 (2017)
Wang, S., Zhang, Y., Che, W., Liu, T.: Joint extraction of entities and relations based on a novel graph scheme. In: IJCAI (2018)
Wang, Y., Zhang, H.: BIRL: bidirectional-interaction reinforcement learning framework for joint relation and entity extraction. In: DASFAA (2021)
Xiong, R., et al.: On layer normalization in the transformer architecture. In: ICML (2020)
Xue, F., Sun, A., Zhang, H., Chng, E.S.: GDPNet: refining latent multi-view graph for relation extraction. arXiv abs/2012.06780 (2020)
Yu, X., Lam, W.: Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach. In: COLING (2010)
Zambaldi, V.F., et al.: Deep reinforcement learning with relational inductive biases. In: ICLR (2019)
Zeng, D., Liu, K., Chen, Y., Zhao, J.: Distant supervision for relation extraction via piecewise convolutional neural networks. In: EMNLP (2015)
Zeng, X., He, S., Zeng, D., Liu, K., Liu, S., Zhao, J.: Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. In: EMNLP/IJCNLP (2019)
Zeng, X., Zeng, D., He, S., Liu, K., Zhao, J.: Extracting relational facts by an end-to-end neural model with copy mechanism. In: ACL (2018)
Zhang, Y., Zhong, V., Chen, D., Angeli, G., Manning, C.D.: Position-aware attention and supervised data improve slot filling. In: EMNLP (2017)
Zheng, Q., Zhang, A., Grover, A.: Online decision transformer. In: ICML (2022)
Zheng, S., Wang, F., Bao, H., Hao, Y., Zhou, P., Xu, B.: Joint extraction of entities and relations based on a novel tagging scheme. arXiv abs/1706.05075 (2017)
Zhou, X., Liu, L., Luo, X., Chen, H., Qing, L., He, X.: Joint entity and relation extraction based on reinforcement learning. IEEE Access 7, 125688–125699 (2019)
Acknowledgements
We thank anonymous reviewers for valuable comments. This work is funded by: the National Natural Science Foundation of China (No. U19B2026, 62106243, U22B2601).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, Y., Shi, T., Ouyang, X., Guo, D. (2023). TERL: Transformer Enhanced Reinforcement Learning for Relation Extraction. In: Sun, M., et al. Chinese Computational Linguistics. CCL 2023. Lecture Notes in Computer Science(), vol 14232. Springer, Singapore. https://doi.org/10.1007/978-981-99-6207-5_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-6207-5_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-6206-8
Online ISBN: 978-981-99-6207-5
eBook Packages: Computer ScienceComputer Science (R0)