Semantic Duplicate Identification with Parsing and Machine Learning

  • Sven Hartrumpf
  • Tim vor der Brück
  • Christian Eichhorn
Conference paper

DOI: 10.1007/978-3-642-15760-8_12

Part of the Lecture Notes in Computer Science book series (LNCS, volume 6231)
Cite this paper as:
Hartrumpf S., vor der Brück T., Eichhorn C. (2010) Semantic Duplicate Identification with Parsing and Machine Learning. In: Sojka P., Horák A., Kopeček I., Pala K. (eds) Text, Speech and Dialogue. TSD 2010. Lecture Notes in Computer Science, vol 6231. Springer, Berlin, Heidelberg

Abstract

Identifying duplicate texts is important in many areas like plagiarism detection, information retrieval, text summarization, and question answering. Current approaches are mostly surface-oriented (or use only shallow syntactic representations) and see each text only as a token list. In this work however, we describe a deep, semantically oriented method based on semantic networks which are derived by a syntactico-semantic parser. Semantically identical or similar semantic networks for each sentence of a given base text are efficiently retrieved by using a specialized index. In order to detect many kinds of paraphrases the semantic networks of a candidate text are varied by applying inferences: lexico-semantic relations, relation axioms, and meaning postulates. Important phenomena occurring in difficult duplicates are discussed. The deep approach profits from background knowledge, whose acquisition from corpora is explained briefly. The deep duplicate recognizer is combined with two shallow duplicate recognizers in order to guarantee a high recall for texts which are not fully parsable. The evaluation shows that the combined approach preserves recall and increases precision considerably in comparison to traditional shallow methods.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Sven Hartrumpf
    • 1
  • Tim vor der Brück
    • 1
  • Christian Eichhorn
    • 2
  1. 1.Intelligent Information and Communication Systems (IICS)FernUniversität in HagenHagenGermany
  2. 2.Lehrstuhl Informatik 1, Technische Universität DortmundDortmundGermany

Personalised recommendations