Best from Top k Versus Top 1: Improving Distant Supervision Relation Extraction with Deep Reinforcement Learning

  • Yaocheng Gui
  • Qian Liu
  • Tingming Lu
  • Zhiqiang GaoEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11441)


Distant supervision relation extraction is a promising approach to find new relation instances from large text corpora. Most previous works employ the top 1 strategy, i.e., predicting the relation of a sentence with the highest confidence score, which is not always the optimal solution. To improve distant supervision relation extraction, this work applies the best from top k strategy to explore the possibility of relations with lower confidence scores. We approach the best from top k strategy using a deep reinforcement learning framework, where the model learns to select the optimal relation among the top k candidates for better predictions. Specifically, we employ a deep Q-network, trained to optimize a reward function that reflects the extraction performance under distant supervision. The experiments on three public datasets - of news articles, Wikipedia and biomedical papers - demonstrate that the proposed strategy improves the performance of traditional state-of-the-art relation extractors significantly. We achieve an improvement of 5.13% in average F\(_1\)-score over four competitive baselines.


Distant supervision Relation extraction Deep reinforcement learning Deep Q-networks 



This work is partially funded by the National Science Foundation of China under Grant 61170165, Grant 61702279, Grant 61602260, and Grant 61502095.


  1. 1.
    Feng, J., Huang, M., Zhao, L., Yang, Y., Zhu, X.: Reinforcement learning for relation classification from noisy data. In: Proceedings of AAAI 2018 (2018)Google Scholar
  2. 2.
    Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., Weld, D.S.: Knowledge-based weak supervision for information extraction of overlapping relations. In: Proceedings of ACL 2011, pp. 541–550 (2011)Google Scholar
  3. 3.
    Koch, M., Gilmer, J., Soderland, S., Weld, D.S.: Type-aware distantly supervised relation extraction with linked arguments. In: Proceedings of EMNLP 2014, pp. 1891–1901 (2014)Google Scholar
  4. 4.
    Lin, Y., Shen, S., Liu, Z., Luan, H., Sun, M.: Neural relation extraction with selective attention over instances. In: Proceedings of ACL 2016, pp. 2124–2133 (2016)Google Scholar
  5. 5.
    Ling, X., Weld, D.S.: Fine-grained entity recognition. In: Proceedings AAAI 2012, vol. 12, pp. 94–100 (2012)Google Scholar
  6. 6.
    Lockard, C., Dong, X.L., Einolghozati, A., Shiralkar, P.: CERES: distantly supervised relation extraction from the semi-structured web. In: Proceedings of VLDB 2018, pp. 1084–1096 (2018)Google Scholar
  7. 7.
    Lourentzou, I., Alba, A., Coden, A., Gentile, A.L., Gruhl, D., Welch, S.: Mining relations from unstructured content. In: Phung, D., Tseng, V.S., Webb, G.I., Ho, B., Ganji, M., Rashidi, L. (eds.) PAKDD 2018. LNCS, vol. 10938, pp. 363–375. Springer, Cham (2018). Scholar
  8. 8.
    Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: Proceedings of ACL 2009, pp. 1003–1011 (2009)Google Scholar
  9. 9.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  10. 10.
    Narasimhan, K., Yala, A., Barzilay, R.: Improving information extraction by acquiring external evidence with reinforcement learning. In: Proceedings of EMNLP 2016, pp. 2355–2365 (2016)Google Scholar
  11. 11.
    Pyysalo, S., et al.: BioInfer: a corpus for information extraction in the biomedical domain. BMC Bioinform. 8(1), 50 (2007)CrossRefGoogle Scholar
  12. 12.
    Qin, P., Xu, W., Wang, W.Y.: Robust distant supervision relation extraction via deep reinforcement learning. In: Proceedings of ACL 2018, pp. 2137–2147 (2018)Google Scholar
  13. 13.
    Riedel, S., Yao, L., McCallum, A.: Modeling relations and their mentions without labeled text. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS, vol. 6323, pp. 148–163. Springer, Heidelberg (2010). Scholar
  14. 14.
    Ritter, A., Zettlemoyer, L., Etzioni, O., et al.: Modeling missing data in distant supervision for information extraction. Trans. Assoc. Comput. Linguist. 1, 367–378 (2013)CrossRefGoogle Scholar
  15. 15.
    Surdeanu, M., Tibshirani, J., Nallapati, R., Manning, C.D.: Multi-instance multi-label learning for relation extraction. In: Proceedings of EMNLP 2012, pp. 455–465 (2012)Google Scholar
  16. 16.
    Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)CrossRefGoogle Scholar
  17. 17.
    Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)zbMATHGoogle Scholar
  18. 18.
    Zeng, D., Liu, K., Chen, Y., Zhao, J.: Distant supervision for relation extraction via piecewise convolutional neural networks. In: Proceedings of EMNLP 2015, pp. 1753–1762 (2015)Google Scholar
  19. 19.
    Zeng, X., He, S., Liu, K., Zhao, J.: Large scaled relation extraction with reinforcement learning. In: Proceedings of AAAI 2018 (2018)Google Scholar
  20. 20.
    Zhou, G., Su, J., Jie, Z., Zhang, M.: Exploring various knowledge in relation extraction. In: Proceedings of ACL 2005, pp. 427–434 (2005)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yaocheng Gui
    • 1
  • Qian Liu
    • 2
  • Tingming Lu
    • 1
  • Zhiqiang Gao
    • 1
    Email author
  1. 1.School of Computer Science and EngineeringSoutheast UniversityNanjingChina
  2. 2.School of Computer Science and TechnologyNanjing University of Posts and TelecommunicationsNanjingChina

Personalised recommendations