Mining Relations from Unstructured Content

  • Ismini Lourentzou
  • Alfredo Alba
  • Anni Coden
  • Anna Lisa GentileEmail author
  • Daniel Gruhl
  • Steve Welch
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10938)


Extracting relations from unstructured Web content is a challenging task and for any new relation a significant effort is required to design, train and tune the extraction models. In this work, we investigate how to obtain suitable results for relation extraction with modest human efforts, relying on a dynamic active learning approach. We propose a method to reliably generate high quality training/test data for relation extraction - for any generic user-demonstrated relation, starting from a few user provided examples and extracting valuable samples from unstructured and unlabeled Web content. To this extent we propose a strategy which learns how to identify the best order to human-annotate data, maximizing learning performance early in the process. We demonstrate the viability of the approach (i) against state of the art datasets for relation extraction as well as (ii) a real case study identifying text expressing a causal relation between a drug and an adverse reaction from user generated Web content.


  1. 1.
    Adel, H., Roth, B., Schütze, H.: Comparing convolutional neural networks to traditional models for slot filling. In: NAACL-HLT (2016)Google Scholar
  2. 2.
    Alba, A., Coden, A., Gentile, A.L., Gruhl, D., Ristoski, P., Welch, S.: Language agnostic dictionary extraction. In: ISWC (ISWC-PD-Industry). CEUR Workshop Proceedings, vol. 1963 (2017)Google Scholar
  3. 3.
    Angeli, G., Tibshirani, J., Wu, J., Manning, C.D.: Combining distant and partial supervision for relation extraction. In: EMNLP, pp. 1556–1567 (2014)Google Scholar
  4. 4.
    Arora, S., Liang, Y., Ma, T.: A simple but tough-to-beat baseline for sentence embeddings. In: ICLR (2017)Google Scholar
  5. 5.
    Augenstein, I., Maynard, D., Ciravegna, F.: Distantly supervised web relation extraction for knowledge base population. Semant. Web 7(4), 335–349 (2016)CrossRefGoogle Scholar
  6. 6.
    Bengio, Y.: Curriculum learning. In: ICML (2009)Google Scholar
  7. 7.
    Bunescu, R.C., Mooney, R.J.: A shortest path dependency kernel for relation extraction. In: HLT/EMNLP, pp. 724–731. ACL (2005)Google Scholar
  8. 8.
    Culotta, A., Sorensen, J.: Dependency tree kernels for relation extraction. In: ACL, p. 423. ACL (2004)Google Scholar
  9. 9.
    Donmez, P., Carbonell, J.G., Bennett, P.N.: Dual strategy active learning. In: Kok, J.N., Koronacki, J., Mantaras, R.L., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS (LNAI), vol. 4701, pp. 116–127. Springer, Heidelberg (2007). Scholar
  10. 10.
    Fu, L., Grishman, R.: An efficient active learning framework for new relation types. In: IJCNLP, pp. 692–698 (2013)Google Scholar
  11. 11.
    Gal, Y., Islam, R., Ghahramani, Z.: Deep bayesian active learning with image data. In: ICML (2017)Google Scholar
  12. 12.
    Gentile, A.L., Zhang, Z., Augenstein, I., Ciravegna, F.: Unsupervised wrapper induction using linked data. In: K-CAP, pp. 41–48. ACM (2013)Google Scholar
  13. 13.
    Hendrickx, I., Kim, S.N., Kozareva, Z., Nakov, P., Ó Séaghdha, D., Padó, S., Pennacchiotti, M., Romano, L., Szpakowicz, S.: Semeval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. In: DEW Workshop, pp. 94–99. ACL (2009)Google Scholar
  14. 14.
    Hsu, W., Lin, H.: Active learning by learning. In: Bonet, B., Koenig, S. (eds.) AAAI, pp. 2659–2665. AAAI Press (2015)Google Scholar
  15. 15.
    Huang, S.J., Jin, R., Zhou, Z.H.: Active learning by querying informative and representative examples. In: NIPS, pp. 892–900 (2010)Google Scholar
  16. 16.
    Ji, G., Liu, K., He, S., Zhao, J.: Distant supervision for relation extraction with sentence-level attention and entity descriptions. In: AAAI, pp. 3060–3066 (2017)Google Scholar
  17. 17.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  18. 18.
    Lewis, D.D., Catlett, J.: Heterogeneous uncertainty sampling for supervised learning. In: ICML, pp. 148–156 (1994)CrossRefGoogle Scholar
  19. 19.
    Liu, M.X.C.: Semantic relation classification via hierarchical recurrent neural network with attention. In: COLING (2016)Google Scholar
  20. 20.
    Mooney, R.J., Bunescu, R.C.: Subsequence kernels for relation extraction. In: NIPS, pp. 171–178 (2006)Google Scholar
  21. 21.
    Morgan, N., Bourlard, H.: Generalization and parameter estimation in feedforward nets: some experiments. In: NIPS, pp. 630–637 (1990)Google Scholar
  22. 22.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML, pp. 807–814 (2010)Google Scholar
  23. 23.
    Nguyen, H.T., Smeulders, A.: Active learning using pre-clustering. In: ICML. ACM (2004)Google Scholar
  24. 24.
    Nguyen, T.H., Grishman, R.: Relation extraction: perspective from convolutional neural networks. In: VS@ HLT-NAACL, pp. 39–48 (2015)Google Scholar
  25. 25.
    Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP, vol. 14, pp. 1532–1543 (2014)Google Scholar
  26. 26.
    Ratner, A.J., Sa, C.D., Wu, S., Selsam, D., Ré, C.: Data programming: creating large training sets, quickly. In: NIPS, pp. 3567–3575 (2016)Google Scholar
  27. 27.
    Roth, B., Barth, T., Wiegand, M., Klakow, D.: A survey of noise reduction methods for distant supervision. In: AKBC, pp. 73–78. ACM (2013)Google Scholar
  28. 28.
    Settles, B.: Active learning literature survey. Univ. Wis. Madison 52(55–66), 11 (2010)Google Scholar
  29. 29.
    Stanovsky, G., Gruhl, D., Mendes, P.: Recognizing mentions of adverse drug reaction in social media using knowledge-infused recurrent models. In: EACL, pp. 142–151. ACL (2017)Google Scholar
  30. 30.
    Sterckx, L., Demeester, T., Deleu, J., Develder, C.: Using active learning and semantic clustering for noise reduction in distant supervision. In: AKBC at NIPS, pp. 1–6 (2014)Google Scholar
  31. 31.
    Vu, N.T., Adel, H., Gupta, P., et al.: Combining recurrent and convolutional neural networks for relation classification. In: NAACL-HLT, pp. 534–539 (2016)Google Scholar
  32. 32.
    Zelenko, D., Aone, C., Richardella, A.: Kernel methods for relation extraction. J. Mach. Learn. Res. 3, 1083–1106 (2003)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Zeng, D., Liu, K., Lai, S., Zhou, G., Zhao, J., et al.: Relation classification via convolutional deep neural network. In: COLING, pp. 2335–2344 (2014)Google Scholar
  34. 34.
    Zhao, S., Grishman, R.: Extracting relations with integrated information using kernel methods. In: ACL, pp. 419–426. ACL (2005)Google Scholar
  35. 35.
    Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., Xu, B.: Attention-based bidirectional long short-term memory networks for relation classification. In: ACL - Short Papers, vol. 2, pp. 207–212 (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Ismini Lourentzou
    • 1
  • Alfredo Alba
    • 3
  • Anni Coden
    • 2
  • Anna Lisa Gentile
    • 3
    Email author
  • Daniel Gruhl
    • 3
  • Steve Welch
    • 3
  1. 1.University of Illinois at Urbana - ChampaignChampaignUSA
  2. 2.IBM Watson Research LaboratoryNew YorkUSA
  3. 3.IBM Research AlmadenSan JoseUSA

Personalised recommendations