Skip to main content
SpringerLink
Log in

Concept Extraction Using Pointer–Generator Networks and Distant Supervision for Data Augmentation

  • Conference paper
  • First Online:
Knowledge Engineering and Knowledge Management (EKAW 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12387))

Abstract

Concept extraction is crucial for a number of downstream applications. However, surprisingly enough, straightforward single token/nominal chunk–concept alignment or dictionary lookup techniques such as DBpedia Spotlight still prevail. We propose a generic open domain-oriented extractive model that is based on distant supervision of a pointer–generator network leveraging bidirectional LSTMs and a copy mechanism and that is able to cope with the out-of-vocabulary phenomenon. The model has been trained on a large annotated corpus compiled specifically for this task from 250K Wikipedia pages, and tested on regular pages, where the pointers to other pages are considered as ground truth concepts. The outcome of the experiments shows that our model significantly outperforms standard techniques and, when used on top of DBpedia Spotlight, further improves its performance. The experiments furthermore show that the model can be readily ported to other datasets on which it equally achieves a state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We adopt Halliday’s notion of classifying nominal group as definition of a concept.

  2. 2.

    https://wiki.dbpedia.org/develop/datasets/dbpedia-version-2016-10.

  3. 3.

    https://opennlp.apache.org/.

  4. 4.

    We use a similar layout as in [30] for easier comparison of our extension with the original model.

  5. 5.

    https://wordnet.princeton.edu/.

  6. 6.

    Wikipedia does not contain self-links, therefore the concept “Grundy County” in a text from the self-titled page is not a link.

  7. 7.

    Henceforth, we refer to the link snippet-based annotation of the pages as a sparse gold standard annotation since it covers by far not all concepts encountered in a page. Our distant supervision-based annotation is referred to as dense annotation since it (supposedly) covers all concepts on a given page. As usual, distant supervision-based annotation is also referred to as weak since it is an automatic annotation.

  8. 8.

    https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html.

  9. 9.

    FRED [11] was not used as baseline as it is not scalable enough for the task: its REST service has a strong limitation on a number of possible requests per day, and it fails on processing long sentences (approximately more than 40 tokens).

  10. 10.

    https://github.com/glample/tagger

  11. 11.

    https://github.com/kyzhouhzau/BERT-NER.

  12. 12.

    https://github.com/ufal/acl2019_nested_ner.

References

  1. Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: an easy-to-use framework for state-of-the-art NLP. In: Proceeding of NAACL (2019)

    Google Scholar 

  2. Al-Zaidy, R., Caragea, C., Giles, C.L.: Bi-LSTM-CRF sequence labeling for keyphrase extraction from scholarly documents. In: Proceeding of WWW (2019)

    Google Scholar 

  3. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations (2015)

    Google Scholar 

  4. Cheng, X., Roth, D.: Relational inference for wikification. In: Proceeding of the EMNLP, pp. 1787–1796 (2013)

    Google Scholar 

  5. Cordeiro, S., Ramisch, C., Villavicencio, A.: UFRGS&LIF at semeval-2016 task 10: rule-based MWE identification and predominant-supersense tagging. In: Proceeding of SemEval-2016, pp. 910–917 (2016)

    Google Scholar 

  6. Daiber, J., Jakob, M., Hokamp, C., Mendes, P.: Improving efficiency and accuracy in multilingual entity extraction. In: Proceeding of the 9th International Conference on Semantic Systems (I-Semantics) (2013)

    Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceeding of the NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  8. Fader, A., Soderland, S., Etzioni, O.: Identifying relations for open information extraction. In: Proceeding of the EMNLP, pp. 1535–1545 (2011)

    Google Scholar 

  9. Fillmore, C., Baker, C.: Frame semantics for text understanding. In: Proceeding of the JWordNet and Other Lexical Resources Workshop at NAACL (2001)

    Google Scholar 

  10. Frege, G.: Ueber Sinn und Bedeutung. Zeitschrift fuer Philosophie und philosophische Kritik 100, 25–50 (1892)

    Google Scholar 

  11. Gangemi, A., Presutti, V., Reforgiato Recupero, D., Nuzzolese, A., Draicchio, F., Mongiovì, M.: Semantic web machine reading with fred. Semant. Web 8(6), 873–893 (2017)

    Article  Google Scholar 

  12. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)

    Article  Google Scholar 

  13. Gu, J., Lu, Z., Li, H., Li, V.O.: Incorporating copying mechanism in sequence-to-sequence learning. In: Proceeding of the ACL, pp. 1631–1640 (2016)

    Google Scholar 

  14. Habibi, M., Weber, L., Neves, M., Wiegandt, D.L., Leser, U.: Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33(14), i37–i48 (2017)

    Article  Google Scholar 

  15. Hailu, N.G.: Investigation of Traditional and Deep Neural Sequence Models for Biomedical Concept Recognition. Ph.D. thesis, University of Colorado (2019)

    Google Scholar 

  16. Halliday, M.: Halliday’s Introduction to Functional Grammar. Routledge, London & New York (2013)

    Book  Google Scholar 

  17. Hasibi, F., Balog, K., Bratsberg, S.: Entity linking in queries: tasks and evaluation. In: Proceeding International Conference on The Theory of Information Retrieval, pp. 171–180. ACM (2015)

    Google Scholar 

  18. Honnibal, M., Montani, I.: spaCy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing (2017). https://spacy.io/

  19. Klein, G., Kim, Y., Deng, Y., Nguyen, V., Senellart, J., Rush, A.: Opennmt: neural machine translation toolkit. In: Proceeding of the 13th Conference of the AMTA, vol. 1, pp. 177–184 (2018)

    Google Scholar 

  20. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. Proc. NAACL-HLT (2016)

    Google Scholar 

  21. Lin, Y., Michel, J.B., Aiden Lieberman, E., Orwant, J., Brockman, W., Petrov, S.: Syntactic annotations for the Google books NGram corpus. In: Proceeding of the ACL 2012 System Demonstrations, pp. 169–174, July 2012

    Google Scholar 

  22. Logeswaran, L., Chang, M.W., Lee, K., Toutanova, K., Devlin, J., Lee, H.: Zero-shot entity linking by reading entity descriptions. Proc. ACL, 3449–3460, July 2019

    Google Scholar 

  23. Luong, T., Pham, H., Manning, C.: Effective approaches to attention-based neural machine translation. Proc. EMNLP, 1412–1421 (2015)

    Google Scholar 

  24. Mausam, Schmitz, M., Soderland, S., Bart, R., Etzioni, O.: Open language learning for information extraction. In: Proceeding of the 2012 Joint EMNLP and CoNLL Conferences, pp. 523–534 (2012)

    Google Scholar 

  25. Meng, R., Zhao, S., Han, S., He, D., Brusilovsky, P., Chi, Y.: Deep keyphrase generation. Proc. ACL, 582–592 (2017)

    Google Scholar 

  26. Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: Proceeding of the ACL, pp. 1003–1011 (2009)

    Google Scholar 

  27. Petrucci, G., Rospocher, M., Ghidini, C.: Expressive ontology learning as neural machine translation. J. Web Semant. 52, 66–82 (2018)

    Article  Google Scholar 

  28. Piccinno, F., Ferragina, P.: From TagME to WAT: a new entity annotator. In: Proceeding of the First International Workshop on Entity Recognition and Disambiguation, pp. 55–62. ERD ’14, ACM, New York, NY, USA (2014)

    Google Scholar 

  29. Schenkel, R., Suchanek, F., Kasneci, G.: Yawn: a semantically annotated wikipedia xml corpus. Datenbanksysteme in Business, Technologie und Web, -12 (2007)

    Google Scholar 

  30. See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. Proc. ACL, 1073–1083 (2017)

    Google Scholar 

  31. Shang, J., Liu, J., Jiang, M., Ren, X., Voss, C., Han, J.: Automated phrase mining from massive text corpora. IEEE Trans. Knowl. Data Eng. 30(10), 1825–1837 (2018)

    Article  Google Scholar 

  32. Straková, J., Straka, M., Hajic, J.: Neural architectures for nested NER through linearization. Proc. ACL, 5326–5331 (2019)

    Google Scholar 

  33. Tulkens, S., Šuster, S., Daelemans, W.: Unsupervised concept extraction from clinical text through semantic composition. J. Biomed. Inform. 91, 103–120 (2019)

    Article  Google Scholar 

  34. Woods, W.A.: Conceptual Indexing: A Better Way to Organize Knowledge. Technical Report SMLI, TR97-61, Sun Microsystems Laboratories (1997)

    Google Scholar 

  35. Yosef, M., Hoffart, J., Bordino, I., Spaniol, M., Weikum, G.: Aida: an online tool for accurate disambiguation of named entities in text and tables. Proc. VLDB Endowment 4(12), 1450–1453 (2011)

    Article  Google Scholar 

  36. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., Liu, Q.: ERNIE: enhanced language representation with informative entities. Proc. ACL, 1441–1451 (2019)

    Google Scholar 

Download references

Acknowledgments

The work presented in this paper has been supported by the European Commission within its H2020 Research Programme under the grant numbers 700024, 700475, 779962, 786731, 825079, and 870930.

Author information

Authors and Affiliations

  1. NLP Group, Pompeu Fabra University, Roc Boronat 138, Barcelona, Spain

    Alexander Shvets & Leo Wanner

  2. Catalan Institute for Research and Advanced Studies (ICREA), Barcelona, Spain

    Leo Wanner

Authors
  1. Alexander Shvets
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leo Wanner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Shvets .

Editor information

Editors and Affiliations

  1. University of Cape Town, Cape Town, South Africa

    C. Maria Keet

  2. Institute of Data Science, Maastricht University, Maastricht, The Netherlands

    Michel Dumontier

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shvets, A., Wanner, L. (2020). Concept Extraction Using Pointer–Generator Networks and Distant Supervision for Data Augmentation. In: Keet, C.M., Dumontier, M. (eds) Knowledge Engineering and Knowledge Management. EKAW 2020. Lecture Notes in Computer Science(), vol 12387. Springer, Cham. https://doi.org/10.1007/978-3-030-61244-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61244-3_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61243-6

  • Online ISBN: 978-3-030-61244-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation