Skip to main content

Drilling Lexico-Semantic Knowledge in Portuguese from BERT

  • Conference paper
  • First Online:
Computational Processing of the Portuguese Language (PROPOR 2022)

Abstract

We compiled a set of patterns that denote lexico-semantic relations in Portuguese, created templates where one argument is fixed and the other is replaced by a mask, and then used BERT for predicting the latter. For most relations and measures, when assessed in a test of lexico-semantic analogies, BERT predictions outperformed those of earlier methods computed from static word embeddings, either with a pattern alone or with a combination of patterns. There is still a large margin of progression, but this suggests that BERT can be used as a source of lexico-semantic knowledge in Portuguese.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/NLP-CISUC/PT-LexicalSemantics/blob/master/Patterns/BERT_patterns_for_TALES.txt.

  2. 2.

    https://github.com/NLP-CISUC/PT-LexicalSemantics/tree/master/TALESv1.1.

  3. 3.

    Even though both the models and the methods applied are significantly different, we note that the BERT model used was pre-trained in a corpus with nearly twice the number of tokens as the GloVe model used. This is still relevant for less experienced users, or when nor time nor resources are available for training of new models, limiting the choice to what is available out-of-the-box.

  4. 4.

    TALES is organised in files where the source (first column) is related to the target (second). So, in the Hyponymy file, hyponymyOf(source, target) holds, and consequently hypernymOf(target, source). Since we are predicting the targets, this file is used for assessing hypernym prediction.

  5. 5.

    BERT predictions are always ranked according to a score also given by the model.

  6. 6.

    https://huggingface.co/pierreguillou/gpt2-small-portuguese.

  7. 7.

    Patterns for GPT available from https://github.com/NLP-CISUC/PT-LexicalSemantics/blob/master/Patterns/GPT_patterns_for_TALES.txt.

References

  1. Bouraoui, Z., Camacho-Collados, J., Schockaert, S.: Inducing relational knowledge from BERT. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7456–7463. AAAI Press (2020)

    Google Scholar 

  2. Cederberg, S., Widdows, D.: Using LSA and noun coordination information to improve the recall and precision of automatic hyponymy extraction. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 111–118 (2003)

    Google Scholar 

  3. Chang, H.S., Wang, Z., Vilnis, L., McCallum, A.: Distributional inclusion vector embedding for unsupervised hypernymy detection. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long Papers), pp. 485–495 (2018)

    Google Scholar 

  4. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186. ACL (2019)

    Google Scholar 

  5. Drozd, A., Gladkova, A., Matsuoka, S.: Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In: Proceedings the 26th International Conference on Computational Linguistics: Technical papers COLING 2016, pp. 3519–3530. COLING 2016 (2016)

    Google Scholar 

  6. Emerson, P.: The original Borda count and partial voting. Soc. Choice Welfare 40(2), 353–358 (2013)

    Article  MathSciNet  Google Scholar 

  7. Ettinger, A.: What BERT is not: lessons from a new suite of psycholinguistic diagnostics for language models. Trans. ACL 8, 34–48 (2020)

    Google Scholar 

  8. Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database (Language, Speech, and Communication). The MIT Press, London (1998)

    MATH  Google Scholar 

  9. de Freitas, M.C., Quental, V.: Subsídios para a elaboração automática de taxonomias. In: Anais do XXVII Congresso da SBC, pp. 1585–1594 (2007)

    Google Scholar 

  10. Gladkova, A., Drozd, A., Matsuoka, S.: Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In: Procs of NAACL 2016 Student Research Workshop, pp. 8–15. ACL (2016)

    Google Scholar 

  11. Goldberg, Y.: Assessing BERT’s syntactic abilities. arXiv:1901.05287 (2019)

  12. Gonçalo Oliveira, H.: A survey on Portuguese lexical knowledge bases: contents, comparison and combination. Information 9(2), 34 (2018)

    Google Scholar 

  13. Gonçalo Oliveira, H., Costa, H., Gomes, P.: Extracção de conhecimento léxico-semântico a partir de resumos da Wikipédia. In: Proceedings of INFORUM 2010, Simpósio de Informática. Braga, Portugal (September 2010)

    Google Scholar 

  14. Gonçalo Oliveira, H., Sousa, T., Alves, A.: TALES: test set of Portuguese lexical-semantic relations for assessing word embeddings. In: Procs of ECAI 2020 Workshop on Hybrid Intelligence for Natural Language Processing Tasks (HI4NLP 2020). CEUR Workshop Proceedings, vol. 2693, pp. 41–47. CEUR-WS.org (2020)

    Google Scholar 

  15. Gonçalo Oliveira, H., Aguiar, F.S.d.S., Rademaker, A.: On the utility of word embeddings for enriching OpenWordNet-PT. In: Proceedings of 3rd Conference on Language, Data and Knowledge (LDK 2021). OASIcs, vol. 93, pp. 21:1–21:13. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl, Germany (2021)

    Google Scholar 

  16. Hartmann, N.S., et al.: Portuguese word embeddings: evaluating on word analogies and natural language tasks. In: Proceedings of 11th Brazilian Symposium in Information and Human Language Technology (STIL 2017), pp. 122–131 (2017)

    Google Scholar 

  17. Hearst, M.A.: Automatic acquisition of hyponyms from large text corpora. In: Proceedings of 14th Conference on Computational Linguistics, pp. 539–545. COLING 1992, Association for Computational Linguistics, Morristown, NJ, USA (1992)

    Google Scholar 

  18. Markov, I., Mamede, N., Baptista, J.: Automatic identification of whole-part relations in Portuguese. In: Proceedings of 3rd Symposium on Languages, Applications and Technologies. OASICS, vol. 38, pp. 225–232. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2014)

    Google Scholar 

  19. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of Workshop track of ICLR (2013)

    Google Scholar 

  20. Paes, G.E.: Detecção de Hiperônimos com BERT e Padrões de Hearst. Master’s thesis, Universidade Federal de Mato Grosso do Sul (2021)

    Google Scholar 

  21. Pantel, P., Pennacchiotti, M.: Espresso: leveraging generic patterns for automatically harvesting semantic relations. In: Procs of 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pp. 113–120. ACL Press, Sydney, Australia (2006)

    Google Scholar 

  22. Petroni, F., et al.: Language models as knowledge bases? In: Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473. ACL (2019)

    Google Scholar 

  23. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)

    Google Scholar 

  24. Snow, R., Jurafsky, D., Ng, A.: Learning syntactic patterns for automatic hypernym discovery. Adv. Neural Inf. Process. Syst. 17, 1297–1304 (2005)

    Google Scholar 

  25. Souza, F., Nogueira, R., Lotufo, R.: BERTimbau: pretrained BERT models for Brazilian Portuguese. In: Cerri, R., Prati, R.C. (eds.) BRACIS 2020. LNCS (LNAI), vol. 12319, pp. 403–417. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was funded by the project POWER (grant number POCI-01-0247-FEDER-070365), co-financed by the European Regional Development Fund (FEDER), through Portugal 2020 (PT2020), and by the Competitiveness and Internationalization Operational Programme (COMPETE 2020); and by national funds through FCT, within the scope of the project CISUC (UID/CEC/00326/2020) and by European Social Fund, through the Regional Operational Program Centro 2020. It is also based upon work from COST Action CA18209 Nexus Linguarum, supported by COST (European Cooperation in Science and Technology). http://www.cost.eu/.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hugo Gonçalo Oliveira .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gonçalo Oliveira, H. (2022). Drilling Lexico-Semantic Knowledge in Portuguese from BERT. In: Pinheiro, V., et al. Computational Processing of the Portuguese Language. PROPOR 2022. Lecture Notes in Computer Science(), vol 13208. Springer, Cham. https://doi.org/10.1007/978-3-030-98305-5_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98305-5_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98304-8

  • Online ISBN: 978-3-030-98305-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics