WebBrain: Joint Neural Learning of Large-Scale Commonsense Knowledge

  • Jiaqiang Chen
  • Niket Tandon
  • Charles Darwis Hariman
  • Gerard de MeloEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9981)


Despite the emergence and growth of numerous large knowledge graphs, many basic and important facts about our everyday world are not readily available on the Web. To address this, we present WebBrain, a new approach for harvesting commonsense knowledge that relies on joint learning from Web-scale data to fill gaps in the knowledge acquisition. We train a neural network model to learn relations based on large numbers of textual patterns found on the Web. At the same time, the model learns vector representations of general word semantics. This joint approach allows us to generalize beyond the explicitly extracted information. Experiments show that we can obtain representations of words that reflect their semantics, yet also allow us to capture conceptual relationships and commonsense knowledge.


Semantic Similarity Stochastic Gradient Descent Relation Prediction Word Representation Word Vector 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)zbMATHGoogle Scholar
  2. 2.
    Bizer, C., Lehmann, J., Kobilarov, G., Auer, S., Becker, C., Cyganiak, R., Hellmann, S.: DBpedia - a crystallization point for the web of data. Web Semant. 7(3), 154–165 (2009)CrossRefGoogle Scholar
  3. 3.
    Bollegala, D., Maehara, T., Kawarabayashi, K.: Embedding semantic relations into word representations. In: Proceedings of IJCAI, pp. 1222–1228 (2015)Google Scholar
  4. 4.
    Bordes, A., Weston, J., Collobert, R., Bengio, Y.: Learning structured embeddings of knowledge bases. In: Proceedings of AAAI (2011)Google Scholar
  5. 5.
    Bordes, A., Glorot, X., Weston, J., Bengio, Y.: Joint learning of words and meaning representations for open-text semantic parsing. In: Proceedings of AISTATS (2012)Google Scholar
  6. 6.
    Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Adv. Neural Inf. Process. Syst. 26, 2787–2795 (2013)Google Scholar
  7. 7.
    Bruni, E., Tran, N.K., Baroni, M.: Multimodal distributional semantics. J. Artif. Int. Res. 49(1), 1–47 (2014)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Chen, J., Tandon, N., de Melo, G.: Neural word representations from large-scale commonsense knowledge. In: Proceedings of WI/IAT 2015 (2015)Google Scholar
  9. 9.
    Espinosa, J.A., Lieberman, H.: EventNet: inferring temporal relations between commonsense events. In: Gelbukh, A., de Albornoz, Á., Terashima-Marín, H. (eds.) MICAI 2005. LNCS (LNAI), vol. 3789, pp. 61–69. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Fellbaum, C.: WordNet: An Electronic Lexical Database. The MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  11. 11.
    Finkelstein, L., Evgenly, G., Yossi, M., Ehud, R., Zach, S., Gadi, W., Eytan, R.: Placing search in context: the concept revisited. In: Proceedings of WWW (2001)Google Scholar
  12. 12.
    Havasi, C., Speer, R., Alonso, J.: ConceptNet 3: a flexible, multilingual semantic network for common sense knowledge. In: Proceedings of RANLP (2007)Google Scholar
  13. 13.
    Hearst, M.A.: Automatic acquisition of hyponyms from large text corpora. In: Proceedings of COLING, pp. 539–545 (1992)Google Scholar
  14. 14.
    Hill, F., Korhonen, A.: Learning abstract concept embeddings from multi-modal data: since you probably can’t see what I mean. In: Proceedings of EMNLP, pp. 255–265 (2014)Google Scholar
  15. 15.
    Jenatton, R., Roux, N.L., Bordes, A., Obozinski, G.R.: A latent factor model for highly multi-relational data. Adv. Neural Inf. Process. Syst. 25, 3167–3175 (2012)Google Scholar
  16. 16.
    Jiang, X., Huang, Y., Nickel, M., Tresp, V.: Combining information extraction, deductive reasoning and machine learning for relation prediction. In: Simperl, E., Cimiano, P., Polleres, A., Corcho, O., Presutti, V. (eds.) ESWC 2012. LNCS, vol. 7295, pp. 164–178. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  17. 17.
    Krompaß, D., Baier, S., Tresp, V.: Type-constrained representation learning in knowledge graphs. In: Proceedings of ISWC (2015)Google Scholar
  18. 18.
    Lenat, D.B.: CYC: a large-scale investment in knowledge infrastructure. Commun. ACM 38, 33–38 (1995)CrossRefGoogle Scholar
  19. 19.
    Lin, Y., Liu, Z., Luan, H., Sun, M., Rao, S., Liu, S.: Modeling relation paths for representation learning of knowledge bases. In: Proceedings of EMNLP (2015)Google Scholar
  20. 20.
    Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: Proceedings AAAI 2015, AAAI Press (2015)Google Scholar
  21. 21.
    Liu, H., Singh, P.: ConceptNet– a practical commonsense reasoning tool-kit. BT Technol. J. 22(4), 211–226 (2004)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26, pp. 3111–3119 (2013)Google Scholar
  23. 23.
    Nickel, M., Tresp, V., Kriegel, H.P.: A three-way model for collective learning on multi-relational data. In: Proceedings of ICML (2011)Google Scholar
  24. 24.
    Niles, I., Pease, A.: Towards a standard upper ontology. In: Proceedings of FOIS (2001)Google Scholar
  25. 25.
    Pantel, P., Pennacchiotti, M.: Espresso: leveraging generic patterns for automatically harvesting semantic relations. In: Proceedings of COLING/ACL 2006 (2006)Google Scholar
  26. 26.
    Paulheim, H.: Knowledge graph refinement: a survey of approaches and evaluation methods. Semantic Web (2016)Google Scholar
  27. 27.
    Socher, R., Chen, D., Manning, C.D., Ng, A.: Reasoning with neural tensor networks for knowledge base completion. Adv. Neural Inf. Process. Syst. 26, 926–934 (2013)Google Scholar
  28. 28.
    Speer, R., Havasi, C., Lieberman, H.: Analogyspace: reducing the dimensionality of common sense knowledge. In: Proceedings of AAAI, AAAI Press (2008)Google Scholar
  29. 29.
    Sutskever, I., Tenenbaum, J.B., Salakhutdinov, R.R.: Modelling relational data using Bayesian clustered tensor factorization. Adv. Neural Inf. Process. Syst. 22, 1821–1828 (2009)Google Scholar
  30. 30.
    Tandon, N., de Melo, G., Suchanek, F.M., Weikum, G.: WebChild: harvesting and organizing commonsense knowledge from the web. In: Proceedings of WSDM (2014)Google Scholar
  31. 31.
    Tandon, N., de Melo, G., Weikum, G.: Deriving a Web-scale common sense fact database. In: Proceedings of AAAI 2011, AAAI Press, Palo Alto, CA, USA (2011)Google Scholar
  32. 32.
    Tandon, N., Weikum, G., de Melo, G., De, A.: Lights, camera, action: Knowledge extraction from movie scripts. In: Proceedings of WWW (2015)Google Scholar
  33. 33.
    Toutanova, K., Chen, D., Pantel, P., Poon, H., Choudhury, P., Gamon, M.: Representing text for joint embedding of text and knowledge bases. In: Proceedings of EMNLP, ACL (2015)Google Scholar
  34. 34.
    Wang, Z., Zhang, J., Feng, J., Chen, Z.: Knowledge graph and text jointly embedding. In: Proceedings of EMNLP, pp. 1591–1601 (2014)Google Scholar
  35. 35.
    Wang, Z., Zhang, J., Feng, J., Chen, Z.: Knowledge graph embedding by translating on hyperplanes. In: Proceedings of AAAI 2014, pp. 1112–1119 (2014)Google Scholar
  36. 36.
    Xie, R., Liu, Z., Jia, J., Luan, H., Sun, M.: Representation learning of knowledge graphs with entity descriptions. In: Proceedings of AAAI (2016)Google Scholar
  37. 37.
    Yu, M., Dredze, M.: Improving lexical embeddings with semantic knowledge. In: Proceedings of ACL 2014 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Jiaqiang Chen
    • 1
  • Niket Tandon
    • 2
  • Charles Darwis Hariman
    • 3
  • Gerard de Melo
    • 4
    Email author
  1. 1.IIIS, Tsinghua UniversityBeijingChina
  2. 2.Allen Institute for Artificial IntelligenceSeattleUSA
  3. 3.Max Planck Institute for InformaticsSaarbrückenGermany
  4. 4.Rutgers UniversityPiscatawayUSA

Personalised recommendations