The VLDB Journal

, Volume 24, Issue 1, pp 143–167

A unified framework for approximate dictionary-based entity extraction

  • Dong Deng
  • Guoliang Li
  • Jianhua Feng
  • Yi Duan
  • Zhiguo Gong
Regular Paper

DOI: 10.1007/s00778-014-0367-9

Cite this article as:
Deng, D., Li, G., Feng, J. et al. The VLDB Journal (2015) 24: 143. doi:10.1007/s00778-014-0367-9

Abstract

Dictionary-based entity extraction identifies predefined entities (e.g., person names or locations) from documents. A recent trend for improving extraction recall is to support approximate entity extraction, which finds all substrings from documents that approximately match entities in a given dictionary. Existing methods to address this problem support either token-based similarity (e.g., Jaccard Similarity) or character-based dissimilarity (e.g., Edit Distance). It calls for a unified method to support various similarity/dissimilarity functions, since a unified method can reduce the programing efforts, the hardware requirements, and the manpower. In this paper, we propose a unified framework to support various similarity/dissimilarity functions, such as jaccard similarity, cosine similarity, dice similarity, edit similarity, and edit distance. Since many real-world applications have high-performance requirement for approximate entity extraction on data streams (e.g., Twitter), we focus on devising efficient algorithms to achieve high performance. We find that many substrings in documents have overlaps, and we can utilize the shared computation across the overlaps to avoid unnecessary redundant computation. To this end, we propose efficient filtering algorithms and develop effective pruning techniques. Experimental results show our method achieves high performance and outperforms state-of-the-art studies significantly.

Keywords

Approximate entity extraction Unified framework Filtering algorithms Pruning techniques 

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Dong Deng
    • 1
  • Guoliang Li
    • 1
  • Jianhua Feng
    • 1
  • Yi Duan
    • 2
  • Zhiguo Gong
    • 3
  1. 1.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  2. 2.School of SoftwareBeihang UniversityBeijingChina
  3. 3.University of MacauMacauChina

Personalised recommendations