Skip to main content
SpringerLink
Log in

Binarized Knowledge Graph Embeddings

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11437))

Included in the following conference series:

Abstract

Tensor factorization has become an increasingly popular approach to knowledge graph completion (KGC), which is the task of automatically predicting missing facts in a knowledge graph. However, even with a simple model like CANDECOMP/PARAFAC (CP) tensor decomposition, KGC on existing knowledge graphs is impractical in resource-limited environments, as a large amount of memory is required to store parameters represented as 32-bit or 64-bit floating point numbers. This limitation is expected to become more stringent as existing knowledge graphs, which are already huge, keep steadily growing in scale. To reduce the memory requirement, we present a method for binarizing the parameters of the CP tensor decomposition by introducing a quantization function to the optimization problem. This method replaces floating point–valued parameters with binary ones after training, which drastically reduces the model size at run time. We investigate the trade-off between the quality and size of tensor factorization models for several KGC benchmark datasets. In our experiments, the proposed method successfully reduced the model size by more than an order of magnitude while maintaining the task performance. Moreover, a fast score computation technique can be developed with bitwise operations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Following [11, 14], for each triple \((e_i,e_j,r_k)\) observed in the training set, we added its inverse triple \((e_j,e_i,r_k^{-1})\) also in the training set.

  2. 2.

    As the original CP model has much larger memory consumption than B-CP, we did not test model ensemble with the CP model in our experiments.

  3. 3.

    https://datalab.snu.ac.kr/haten2/.

  4. 4.

    https://github.com/KokiKishimoto/cp_decomposition.git.

References

  1. Bengio, Y., LĂ©onard, N., Courville, A.C.: Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR abs/1308.3432 (2013). http://arxiv.org/abs/1308.3432

  2. Bollacker, K.D., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, 10–12 June 2008, pp. 1247–1250 (2008). https://doi.org/10.1145/1376616.1376746

  3. Bordes, A., Chopra, S., Weston, J.: Question answering with subgraph embeddings. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar, 25–29 October 2014, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 615–620 (2014). http://aclweb.org/anthology/D/D14/D14-1067.pdf

  4. Bordes, A., Usunier, N., García-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of A meeting held 5–8 December 2013, Lake Tahoe, Nevada, United States, pp. 2787–2795 (2013). http://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data

  5. Courbariaux, M., Bengio, Y., David, J.: Binaryconnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, Montreal, Quebec, Canada, 7–12 December 2015, pp. 3123–3131 (2015). http://papers.nips.cc/paper/5647-binaryconnect-training-deep-neural-networks-with-binary-weights-during-propagations

  6. Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2D knowledge graph embeddings. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2–7 February 2018, New Orleans, Louisiana, USA (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17366

  7. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010, pp. 249–256 (2010). http://www.jmlr.org/proceedings/papers/v9/glorot10a.html

  8. Guo, Y.: A survey on methods and theories of quantized neural networks. CoRR abs/1808.04752 (2018). http://arxiv.org/abs/1808.04752

  9. Hayashi, K., Shimbo, M.: On the equivalence of holographic and complex embeddings for link prediction. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, 30 July–4 August, Volume 2: Short Papers, pp. 554–559 (2017). https://doi.org/10.18653/v1/P17-2088

  10. Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys 6(1), 164–189 (1927)

    Article  Google Scholar 

  11. Kazemi, S.M., Poole, D.: Simple embedding for link prediction in knowledge graphs. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, 3–8 December 2018. pp. 4289–4300 (2018). http://papers.nips.cc/paper/7682-simple-embedding-for-link-prediction-in-knowledge-graphs

  12. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009). https://doi.org/10.1137/07070111X

    Article  MathSciNet  MATH  Google Scholar 

  13. Kolda, T.G., Bader, B.W., Kenny, J.P.: Higher-order web link analysis using multilinear algebra. In: Proceedings of the 5th IEEE International Conference on Data Mining (ICDM 2005), Houston, Texas, USA, 27–30 November 2005, pp. 242–249 (2005). https://doi.org/10.1109/ICDM.2005.77

  14. Lacroix, T., Usunier, N., Obozinski, G.: Canonical tensor decomposition for knowledge base completion. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 July 2018, pp. 2869–2878 (2018). http://proceedings.mlr.press/v80/lacroix18a.html

  15. Lam, M.: Word2bits - quantized word vectors. CoRR abs/1803.05651 (2018). http://arxiv.org/abs/1803.05651

  16. Liu, H., Wu, Y., Yang, Y.: Analogical inference for multi-relational embeddings. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, pp. 2168–2178 (2017). http://proceedings.mlr.press/v70/liu17d.html

  17. Ma, Y., Crook, P.A., Sarikaya, R., Fosler-Lussier, E.: Knowledge graph inference for spoken dialog systems. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, 19–24 April 2015, pp. 5346–5350 (2015). https://doi.org/10.1109/ICASSP.2015.7178992

  18. Nickel, M., Rosasco, L., Poggio, T.A.: Holographic embeddings of knowledge graphs. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, USA, 12–17 February 2016, pp. 1955–1961 (2016). http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12484

  19. Nickel, M., Tresp, V.: Logistic tensor factorization for multi-relational data. CoRR abs/1306.2084 (2013). http://arxiv.org/abs/1306.2084

  20. Nickel, M., Tresp, V., Kriegel, H.: A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, 28 June–2 July 2011, pp. 809–816 (2011)

    Google Scholar 

  21. Palumbo, E., Rizzo, G., Troncy, R., Baralis, E., Osella, M., Ferro, E.: An empirical comparison of knowledge graph embeddings for item recommendation. In: Proceedings of the First Workshop on Deep Learning for Knowledge Graphs and Semantic Technologies (DL4KGS) Co-located with the 15th Extended Semantic Web Conerence (ESWC 2018), Heraklion, Crete, Greece, 4 June 2018, pp. 14–20 (2018). http://ceur-ws.org/Vol-2106/paper2.pdf

  22. Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-net: imagenet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part IV. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_32

    Chapter  Google Scholar 

  23. Socher, R., Chen, D., Manning, C.D., Ng, A.Y.: Reasoning with neural tensor networks for knowledge base completion. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Proceedings of a meeting held 5–8 December 2013, Lake Tahoe, Nevada, United States, pp. 926–934 (2013). http://papers.nips.cc/paper/5028-reasoning-with-neural-tensor-networks-for-knowledge-base-completion

  24. Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff, Alberta, Canada, 8–12 May 2007, pp. 697–706 (2007). https://doi.org/10.1145/1242572.1242667

  25. Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., Bouchard, G.: Complex embeddings for simple link prediction. In: Proceedings of the 33rd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, 19–24 June 2016, pp. 2071–2080 (2016). http://jmlr.org/proceedings/papers/v48/trouillon16.html

  26. Yang, B., Yih, W., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. CoRR abs/1412.6575 (2014). http://arxiv.org/abs/1412.6575

Download references

Author information

Authors and Affiliations

  1. Osaka University, Osaka, Japan

    Koki Kishimoto, Katsuhiko Hayashi, Genki Akai & Kazunori Komatani

  2. Nara Institute of Science and Technology, Nara, Japan

    Masashi Shimbo

  3. RIKEN Center for Advanced Intelligence Project, Tokyo, Japan

    Katsuhiko Hayashi & Masashi Shimbo

Authors
  1. Koki Kishimoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Katsuhiko Hayashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Genki Akai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Masashi Shimbo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kazunori Komatani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Koki Kishimoto .

Editor information

Editors and Affiliations

  1. University of Strathclyde, Glasgow, UK

    Leif Azzopardi

  2. Bauhaus Universität Weimar, Weimar, Germany

    Benno Stein

  3. Universität Duisburg-Essen, Duisburg, Germany

    Norbert Fuhr

  4. GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany

    Philipp Mayr

  5. Delft University of Technology, Delft, The Netherlands

    Claudia Hauff

  6. University of Twente, Enschede, The Netherlands

    Djoerd Hiemstra

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kishimoto, K., Hayashi, K., Akai, G., Shimbo, M., Komatani, K. (2019). Binarized Knowledge Graph Embeddings. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds) Advances in Information Retrieval. ECIR 2019. Lecture Notes in Computer Science(), vol 11437. Springer, Cham. https://doi.org/10.1007/978-3-030-15712-8_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-15712-8_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-15711-1

  • Online ISBN: 978-3-030-15712-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation