Skip to main content

Improving Open Information Extraction with Distant Supervision Learning

Neural Processing Letters volume 53, pages 3287–3306 (2021)Cite this article

Abstract

Open information extraction (Open IE), as one of the essential applications in the area of Natural Language Processing (NLP), has gained great attention in recent years. As a critical technology for building Knowledge Bases (KBs), it converts unstructured natural language sentences into structured representations, usually expressed in the form of triples. Most conventional open information extraction approaches leverage a series of manual pre-defined extraction patterns or learn patterns from labeled training examples, which requires a large number of human resources. Additionally, many Natural Language Processing tools are involved, which leads to error accumulation and propagation. With the rapid development of neural networks, neural-based models can minimize the error propagation problem, but it also faces the problem of data-hungry in supervised learning. Especially, they leverage existing Open IE tools to generate training data, and it causes data quality issues. In this paper, we employ a distant supervision learning approach to improve the Open IE task. We conduct extensive experiments by employing two popular sequence-to-sequence models (RNN and Transformer) and a large benchmark data set to demonstrate the performance of our approach.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Availability of data and materials

The experimental data we used, including the experimental test data, are open and transparent.

Notes

  1. https://github.com/allenai/openie-standalone.

  2. https://github.com/dair-iitd/OpenIE-standalone.

  3. https://github.com/knowitall/openie.

  4. https://dumps.wikimedia.org/enwiki/20200620/.

  5. https://lod-cloud.net/dataset/wikidata.

  6. https://github.com/keras-team/keras.

  7. https://github.com/CyberZHG/keras-transformer.

  8. https://github.com/gabrielStanovsky/oie-benchmark.

References

  1. Angeli G, Premkumar MJJ, Manning CD (2015) Leveraging linguistic structure for open domain information extraction. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, The Association for Computer Linguistics, pp 344–354. https://doi.org/10.3115/v1/p15-1034

  2. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: Bengio Y, LeCun Y (eds) Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, arXiv:1409.0473

  3. Cetto M, Niklaus C, Freitas A, Handschuh S (2018) Graphene: Semantically-linked propositions in open information extraction. In: Bender EM, Derczynski L, Isabelle P (eds) Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20–26, 2018, Association for Computational Linguistics, pp 2300–2311. https://www.aclweb.org/anthology/C18-1195/

  4. Cho K, van Merrienboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25–29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, ACL, pp 1724–1734. https://doi.org/10.3115/v1/d14-1179,

  5. Christensen J, Mausam SS, Etzioni O (2010) Semantic role labeling for open information extraction. In: Proceedings of the NAACL HLT 2010 First International Workshop on Formalisms and Methodology for Learning by Reading, Association for Computational Linguistics, USA, FAM-LbR ’10, p 52–60

  6. Corro LD, Gemulla R (2013) Clausie: clause-based open information extraction. In: Schwabe D, Almeida VAF, Glaser H, Baeza-Yates R, Moon SB (eds) Proceedings of the 22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro, Brazil, May 13–17, 2013, International World Wide Web Conferences Steering Committee/ACM, pp 355–366. https://doi.org/10.1145/2488388.2488420

  7. Cui L, Wei F, Zhou M (2018) Neural open information extraction. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15–20, 2018, Volume 2: Short Papers, Association for Computational Linguistics, pp 407–413, 10.18653/v1/P18-2065. https://www.aclweb.org/anthology/P18-2065/

  8. Etzioni O, Banko M, Soderland S, Weld DS (2008) Open information extraction from the web. Commun ACM 51(12):68–74. https://doi.org/10.1145/1409360.1409378

    Article  Google Scholar 

  9. Etzioni O, Fader A, Christensen J, Soderland S, Mausam (2011) Open information extraction: The second generation. In: Walsh T (ed) IJCAI 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, July 16–22, 2011, IJCAI/AAAI, pp 3–10. https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-012

  10. Fader A, Soderland S, Etzioni O (2011) Identifying relations for open information extraction. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, 27–31 July 2011, John McIntyre Conference Centre, Edinburgh, UK, A meeting of SIGDAT, a Special Interest Group of the ACL, ACL, pp 1535–1545. https://www.aclweb.org/anthology/D11-1142/

  11. Gashteovski K, Gemulla R, Corro LD (2017) Minie: Minimizing facts in open information extraction. In: Palmer M, Hwa R, Riedel S (eds) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9–11, 2017, Association for Computational Linguistics, pp 2630–2640. https://doi.org/10.18653/v1/d17-1278

  12. Gehring J, Auli M, Grangier D, Dauphin YN (2017) A convolutional encoder model for neural machine translation. In: Barzilay R, Kan M (eds) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, Volume 1: Long Papers, Association for Computational Linguistics, pp 123–135. https://doi.org/10.18653/v1/P17-1012

  13. Hermann KM, Kociský T, Grefenstette E, Espeholt L, Kay W, Suleyman M, Blunsom P (2015) Teaching machines to read and comprehend. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7–12, 2015, Montreal, Quebec, Canada, pp 1693–1701. https://proceedings.neurips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html

  14. Hill F, Bordes A, Chopra S, Weston J (2016) The goldilocks principle: Reading children’s books with explicit memory representations. In: Bengio Y, LeCun Y (eds) Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings. arXiv:1511.02301

  15. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  16. Hoffmann R, Zhang C, Ling X, Zettlemoyer LS, Weld DS (2011) Knowledge-based weak supervision for information extraction of overlapping relations. In: Lin D, Matsumoto Y, Mihalcea R (eds) The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19–24 June, 2011, Portland, Oregon, USA, The Association for Computer Linguistics, pp 541–550. https://www.aclweb.org/anthology/P11-1055/

  17. Kenter T, de Rijke M (2015) Short text similarity with word embeddings. In: Bailey J, Moffat A, Aggarwal CC, de Rijke M, Kumar R, Murdock V, Sellis TK, Yu JX (eds) Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM 2015, Melbourne, VIC, Australia, October 19–23, 2015, ACM, pp 1411–1420. https://dl.acm.org/citation.cfm?id=2806475

  18. Lei K, Zhang B, Liu Y, Deng Y, Zhang D, Shen Y (2018) A knowledge graph based solution for entity discovery and linking in open-domain questions. CoRR arXiv:1812.01889

  19. Li C, Zhang Z, Lee WS, Lee GH (2018) Convolutional sequence to sequence model for human dynamics. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018, IEEE Computer Society, pp 5226–5234. https://doi.org/10.1109/CVPR.2018.00548.http://openaccess.thecvf.com/content_cvpr_2018/html/Li_Convolutional_Sequence_to_CVPR_2018_paper.html

  20. Li G, Wu CH, Vijay-Shanker K (2017) Noise reduction methods for distantly supervised biomedical relation extraction. In: Cohen KB, Demner-Fushman D, Ananiadou S, Tsujii J (eds) BioNLP 2017, Vancouver, Canada, August 4, 2017, Association for Computational Linguistics, pp 184–193. https://doi.org/10.18653/v1/W17-2323

  21. Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Màrquez L, Callison-Burch C, Su J, Pighin D, Marton Y (eds) Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, The Association for Computational Linguistics, pp 1412–1421. https://doi.org/10.18653/v1/d15-1166

  22. Mausam (2016) Open information extraction systems and downstream applications. In: Kambhampati S (ed) Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9–15 July 2016, IJCAI/AAAI Press, pp 4074–4077. http://www.ijcai.org/Abstract/16/604

  23. Mausam, Schmitz M, Soderland S, Bart R, Etzioni O (2012) Open language learning for information extraction. In: Tsujii J, Henderson J, Pasca M (eds) Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL 2012, July 12–14, 2012, Jeju Island, Korea, ACL, pp 523–534. https://www.aclweb.org/anthology/D12-1048/

  24. Min B, Grishman R, Wan L, Wang C, Gondek D (2013) Distant supervision for relation extraction with an incomplete knowledge base. In: Vanderwende L, III HD, Kirchhoff K (eds) Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9–14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, The Association for Computational Linguistics, pp 777–782. https://www.aclweb.org/anthology/N13-1095/

  25. Nallapati R, Zhou B, dos Santos CN, Gülçehre Ç, Xiang B (2016) Abstractive text summarization using sequence-to-sequence rnns and beyond. In: Goldberg Y, Riezler S (eds) Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11–12, 2016, ACL, pp 280–290. https://doi.org/10.18653/v1/k16-1028

  26. Pal H, Mausam (2016) Demonyms and compound relational nouns in nominal open IE. In: Pujara J, Rocktäschel T, Chen D, Singh S (eds) Proceedings of the 5th Workshop on Automated Knowledge Base Construction, AKBC@NAACL-HLT 2016, San Diego, CA, USA, June 17, 2016, The Association for Computer Linguistics, pp 35–39. https://doi.org/10.18653/v1/w16-1307

  27. Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) Squad: 100,000+ questions for machine comprehension of text. In: Su J, Carreras X, Duh K (eds) Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, The Association for Computational Linguistics, pp 2383–2392. https://doi.org/10.18653/v1/d16-1264

  28. Riedel S, Yao L, McCallum A (2010) Modeling relations and their mentions without labeled text. In: Balcázar JL, Bonchi F, Gionis A, Sebag M (eds) Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2010, Barcelona, Spain, September 20–24, 2010, Proceedings, Part III, Springer, Lecture Notes in Computer Science, vol 6323, pp 148–163. https://doi.org/10.1007/978-3-642-15939-8_10

  29. Roth B, Barth T, Wiegand M, Klakow D (2013) A survey of noise reduction methods for distant supervision. In: Suchanek FM, Riedel S, Singh S, Talukdar PP (eds) Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC@CIKM 13, San Francisco, California, USA, October 27–28, 2013, ACM, pp 73–78. https://doi.org/10.1145/2509558.2509571

  30. Rush AM, Chopra S, Weston J (2015) A neural attention model for abstractive sentence summarization. In: Màrquez L, Callison-Burch C, Su J, Pighin D, Marton Y (eds) Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17–21, 2015, The Association for Computational Linguistics, pp 379–389. https://doi.org/10.18653/v1/d15-1044

  31. Saha S, Pal H, Mausam (2017) Bootstrapping for numerical open IE. In: Barzilay R, Kan M (eds) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, Vol 2: Short Papers, Association for Computational Linguistics, pp 317–323. https://doi.org/10.18653/v1/P17-2050

  32. Santoro A, Raposo D, Barrett DGT, Malinowski M, Pascanu R, Battaglia PW, Lillicrap T (2017) A simple neural network module for relational reasoning. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp 4967–4976. https://proceedings.neurips.cc/paper/2017/hash/e6acf4b0f69f6f6e60e9a815938aa1ff-Abstract.html

  33. Sarhan I, Spruit MR (2019) Contextualized word embeddings in a neural open information extraction model. In: Métais E, Meziane F, Vadera S, Sugumaran V, Saraee M (eds) Natural Language Processing and Information Systems: 24th International Conference on Applications of Natural Language to Information Systems, NLDB 2019, Salford, UK, June 26–28, 2019, Proceedings, Springer, Lecture Notes in Computer Science, vol 11608, pp 359–367. https://doi.org/10.1007/978-3-030-23281-8_31

  34. Schneider R, Oberhauser T, Klatt T, Gers FA, Löser A (2017) Analysing errors of open information extraction systems. CoRR arXiv:1707.07499

  35. See A, Liu PJ, Manning CD (2017) Get to the point: Summarization with pointer-generator networks. In: Barzilay R, Kan M (eds) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, Volume 1: Long Papers, Association for Computational Linguistics, pp 1073–1083. https://doi.org/10.18653/v1/P17-1099

  36. Stanovsky G, Dagan I (2016) Creating a large benchmark for open information extraction. In: Su J, Carreras X, Duh K (eds) Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, The Association for Computational Linguistics, pp 2300–2305. https://doi.org/10.18653/v1/d16-1252

  37. Stanovsky G, Ficler J, Dagan I, Goldberg Y (2016) Getting more out of syntax with props. CoRR arXiv:1603.01648

  38. Sun M, Li X, Wang X, Fan M, Feng Y, Li P (2019) Logician: a unified end-to-end neural approach for open-domain information extraction. CoRR arXiv:1904.12535

  39. Surdeanu M, Tibshirani J, Nallapati R, Manning CD (2012) Multi-instance multi-label learning for relation extraction. In: Tsujii J, Henderson J, Pasca M (eds) Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL 2012, July 12–14, 2012, Jeju Island, Korea, ACL, pp 455–465. https://www.aclweb.org/anthology/D12-1042/

  40. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8–13 2014, Montreal, Quebec, Canada, pp 3104–3112, https://proceedings.neurips.cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html

  41. Takamatsu S, Sato I, Nakagawa H (2012) Reducing wrong labels in distant supervision for relation extraction. In: The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8–14, 2012, Jeju Island, Korea, Vol 1: Long Papers, The Association for Computer Linguistics, pp 721–729. https://www.aclweb.org/anthology/P12-1076/

  42. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, pp 5998–6008. http://papers.nips.cc/paper/7181-attention-is-all-you-need

  43. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp 5998–6008. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

  44. Weston J, Bordes A, Chopra S, Mikolov T (2016) Towards ai-complete question answering: a set of prerequisite toy tasks. In: Bengio Y, LeCun Y (eds) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings. arXiv:1502.05698

  45. Wingfield A, Stine-Morrow EA (2000) Language and speech. The Handbook of Aging and Cognition pp 359–416

  46. Wu F, Weld DS (2010) Open information extraction using wikipedia. In: Hajic J, Carberry S, Clark S (eds) ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11–16, 2010, Uppsala, Sweden, The Association for Computer Linguistics, pp 118–127. https://www.aclweb.org/anthology/P10-1013/

  47. Wu Y, Schuster M (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR arXiv:1609.08144

  48. Xiong C, Merity S, Socher R (2016) Dynamic memory networks for visual and textual question answering. In: Balcan M, Weinberger KQ (eds) Proceedings of the 33rd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19–24, 2016, JMLR.org, JMLR Workshop and Conference Proceedings, vol 48, pp 2397–2406. http://proceedings.mlr.press/v48/xiong16.html

  49. Yates A, Banko M, Broadhead M, Cafarella MJ, Etzioni O, Soderland S (2007) Textrunner: Open information extraction on the web. In: Sidner CL, Schultz T, Stone M, Zhai C (eds) Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, April 22–27, 2007, Rochester, New York, USA, The Association for Computational Linguistics, pp 25–26. https://www.aclweb.org/anthology/N07-4013/

  50. Zhang Y, Fang Y, Xiao W (2017) Deep keyphrase generation with a convolutional sequence to sequence model. In: 4th International Conference on Systems and Informatics, ICSAI 2017, Hangzhou, China, November 11–13, 2017, IEEE, pp 1477–1485. https://doi.org/10.1109/ICSAI.2017.8248519

Download references

Funding

This paper was partially supported by NSFC grant U1866602, 61772157.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150001, China

    Jiabao Han & Hongzhi Wang

Authors
  1. Jiabao Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongzhi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jiabao Han is responsible for this paper design and experimentation and partial writing. Hongzhi Wang is responsible for the writing and review of the paper.

Corresponding author

Correspondence to Hongzhi Wang.

Ethics declarations

Conflicts of interest/Competing interests

We declare that we have no known competing financial interests or personal relationsships that could have appeared to influence the work reported in this paper.

Code availability

The code for the experiment will be uploaded to Github later

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Han, J., Wang, H. Improving Open Information Extraction with Distant Supervision Learning. Neural Process Lett 53, 3287–3306 (2021). https://doi.org/10.1007/s11063-021-10548-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10548-0

Keywords

  • Distant supervision learning
  • Open information extraction
  • Neural network
  • Sequence-to-sequence model