Skip to main content

SORBET: A Siamese Network for Ontology Embeddings Using a Distance-Based Regression Loss and BERT

  • Conference paper
  • First Online:
The Semantic Web – ISWC 2023 (ISWC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14265))

Included in the following conference series:

  • 2016 Accesses

Abstract

Ontology embedding methods have been popular in recent years, especially when it comes to representation learning algorithms for solving ontology-related tasks. Despite the impact of large language models on knowledge graphs’ related tasks, there has been less focus on adapting these models to construct ontology embeddings that are both semantically relevant and faithful to the ontological structure. In this paper, we present a novel ontology embedding method that encodes ontology classes into a pre-trained SBERT through random walks and then fine-tunes the embeddings using a distance-based regression loss. We benchmark our algorithm on four different datasets across two tasks and show the impact of transfer learning and our distance-based loss on the quality of the embeddings. Our results show that SORBET outperform state-of-the-art ontology embedding techniques for the performed tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/filyp/autocorrect.

  2. 2.

    https://oaei.ontologymatching.org/2022/.

  3. 3.

    https://foodon.org/.

  4. 4.

    http://geneontology.org/ontology/ accessed on the 2020-09-08.

  5. 5.

    https://github.com/KRR-Oxford/OWL2Vec-Star.

  6. 6.

    https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens.

  7. 7.

    https://github.com/Lama-West/SORBET_ISWC23.

References

  1. Ontology Matching 2021. Proceedings of the 16th International Workshop on Ontology Matching Co-located with the 20th International Semantic Web Conference (ISWC 2021), CEUR Workshop Proceedings, vol. 3063. CEUR-WS.org (2021)

    Google Scholar 

  2. Ontology Matching 2022. Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) Co-located with the 21th International Semantic Web Conference (ISWC 2022), Hangzhou, Virtual Conference, 23 October 2022, CEUR Workshops Proceedings, vol. 3324. CEUR-WS.org (2022)

    Google Scholar 

  3. Ashburner, M., et al.: Gene ontology: tool for the unification of biology. Nat. Genet. 25(1), 25–29 (2000). https://doi.org/10.1038/75556

    Article  Google Scholar 

  4. Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Adv. Neural Inf. Process. Syst. 26 (2013)

    Google Scholar 

  5. Chen, J., Hu, P., Jimenez-Ruiz, E., Holter, O.M., Antonyrajah, D., Horrocks, I.: OWL2Vec*: embedding of OWL ontologies. Mach. Learn. 110(7), 1813–1845 (2021). https://doi.org/10.1007/s10994-021-05997-6

    Article  MathSciNet  MATH  Google Scholar 

  6. Chen, J., Jiménez-Ruiz, E., Horrocks, I., Antonyrajah, D., Hadian, A., Lee, J.: Augmenting ontology alignment by semantic embedding and distant supervision. In: Verborgh, R., et al. (eds.) ESWC 2021. LNCS, vol. 12731, pp. 392–408. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77385-4_23

  7. Consortium, T.G.O.: The gene ontology resource: enriching a gold mine. Nucl. Acids Res. 49(D1), D325–D334 (12 2020). https://doi.org/10.1093/nar/gkaa1113

  8. Dooley, D.M., et al.: Foodon: a harmonized food ontology to increase global food traceability, quality control and data integration. NPJ Sci. Food 2(1), 23 (2018). https://doi.org/10.1038/s41538-018-0032-6

    Article  Google Scholar 

  9. Efeoglu, S.: Graphmatcher: A graph representation learning approach for ontology matching. In: Ontology Matching 2022 : Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) Co-located with the 21th International Semantic Web Conference (ISWC 2022), Hangzhou, Virtual Conference, 23 October 2022 [2], pp. 174–180 (2022)

    Google Scholar 

  10. Gosselin, F., Zouaq, A.: Sebmatcher results for OAEI 2022. In: Ontology Matching 2022: Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) Co-located with the 21th International Semantic Web Conference (ISWC 2022), Hangzhou, Virtual Conference, 23 October 2022 [2], pp. 202–209 (2022)

    Google Scholar 

  11. Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864 (2016)

    Google Scholar 

  12. Gutiérrez-Basulto, V., Schockaert, S.: From knowledge graph embedding to ontology embedding? an analysis of the compatibility between vector space representations and rules. In: International Conference on Principles of Knowledge Representation and Reasoning (2018)

    Google Scholar 

  13. He, Y., Chen, J., Antonyrajah, D., Horrocks, I.: Bertmap: a bert-based ontology alignment system. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 5684–5691 (2022)

    Google Scholar 

  14. Iyer, V., Agarwal, A., Kumar, H.: VeeAlign: multifaceted context representation using dual attention for ontology alignment. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 10780–10792. Association for Computational Linguistics, Punta Cana (2021). https://doi.org/10.18653/v1/2021.emnlp-main.842

  15. Knorr, L., Portisch, J.: Fine-tom matcher results for OAEI 2021. In: Ontology Matching 2021: Proceedings of the 16th International Workshop on Ontology Matching Co-located with the 20th International Semantic Web Conference (ISWC 2021) [1], pp. 144–151 (2021)

    Google Scholar 

  16. Kolyvakis, P., Kalousis, A., Smith, B., Kiritsis, D.: Biomedical ontology alignment: an approach based on representation learning. J. Biomed. Semant. 9(1), 1–20 (2018)

    Article  Google Scholar 

  17. Kossack, D., Borg, N., Knorr, L., Portisch, J.: Tom matcher results for OAEI 2021. In: Ontology Matching 2021: Proceedings of the 16th International Workshop on Ontology Matching co-located with the 20th International Semantic Web Conference (ISWC 2021) [1], pp. 193–198 (2021)

    Google Scholar 

  18. Kulmanov, M., Liu-Wei, W., Yan, Y., Hoehndorf, R.: El embeddings: geometric construction of models for the description logic el ++. Int. Joint Conf. Artif. Intell. (2019)

    Google Scholar 

  19. Li, C., Li, A., Wang, Y., Tu, H., Song, Y.: A survey on approaches and applications of knowledge representation learning. In: 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC), pp. 312–319. IEEE (2020)

    Google Scholar 

  20. Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29 (2015)

    Google Scholar 

  21. Liu, W., et al.: K-bert: enabling language representation with knowledge graph. In: AAAI Conference on Artificial Intelligence (2019). https://api.semanticscholar.org/CorpusID:202583325

  22. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, 2–4 May 2013, Workshop Track Proceedings (2013)

    Google Scholar 

  23. Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710 (2014)

    Google Scholar 

  24. Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using siamese bert-networks. In: Conference on Empirical Methods in Natural Language Processing (2019)

    Google Scholar 

  25. Ristoski, P., Rosati, J., Di Noia, T., De Leone, R., Paulheim, H.: Rdf2vec: Rdf graph embeddings and their applications. Semant. Web 10(4), 721–752 (2019)

    Article  Google Scholar 

  26. Smaili, F.Z., Gao, X., Hoehndorf, R.: Onto2vec: joint vector-based representation of biological entities and their ontology-based annotations. Bioinformatics 34(13), i52–i60 (2018)

    Article  Google Scholar 

  27. Smaili, F.Z., Gao, X., Hoehndorf, R.: Opa2vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction. Bioinformatics 35(12), 2133–2140 (2019)

    Article  Google Scholar 

  28. Sousa, G., Lima, R., Trojahn, C.: An eye on representation learning in ontology matching. In: Ontology Matching 2022: Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) Co-located with the 21th International Semantic Web Conference (ISWC 2022), Hangzhou, Virtual Conference, October 23, 2022 [2], pp. 49–60 (2022)

    Google Scholar 

  29. Sun, T., et al.: Colake: contextualized language and knowledge embedding. arXiv preprint arXiv:2010.00309 (2020)

  30. Wang, X., Gao, T., Zhu, Z., Liu, Z., Li, J.Z., Tang, J.: Kepler: a unified model for knowledge embedding and pre-trained language representation. Trans. Assoc. Comput. Linguist. 9, 176–194 (2019)

    Article  Google Scholar 

  31. Wu, J., Lv, J., Guo, H., Ma, S.: Daeom: a deep attentional embedding approach for biomedical ontology matching. Appl. Sci. 10, 7909 (2020)

    Article  Google Scholar 

  32. Yanardag, P., Vishwanathan, S.: Deep graph kernels. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1365–1374 (2015)

    Google Scholar 

Download references

Acknowledgment

This research has been funded by the NSERC Discovery Grant Program. The authors acknowledge support from Compute Canada for providing computational resources.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Francis Gosselin .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gosselin, F., Zouaq, A. (2023). SORBET: A Siamese Network for Ontology Embeddings Using a Distance-Based Regression Loss and BERT. In: Payne, T.R., et al. The Semantic Web – ISWC 2023. ISWC 2023. Lecture Notes in Computer Science, vol 14265. Springer, Cham. https://doi.org/10.1007/978-3-031-47240-4_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47240-4_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47239-8

  • Online ISBN: 978-3-031-47240-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics