How to find the nearest by evaluating only few? Clustering techniques used to improve the efficiency of an Information Retrieval system based on Distributional Semantics

Summary

The first objective of this contribution is to give a description of our textual information retrieval system based on distributional semantics. The central idea of the approach is to represent the retrievable units and the user queries in a unified way as projections in a vector space of pertinent terms. The projections are derived from a co-occurrence matrix computed on large reference (textual) corpora collecting the distributional semantic information. A similarity computation based on the cosine measure is then used to characterize the semantic proximity between queries and documents.

Retrieval effectiveness can be further improved by the use of relevance feedback techniques. A simple feedback method where document relevance is interactively integrated to the original query will also be presented and evaluated.

Although our first experiments lead to quite promising results, one major drawback of our IR system in its original form is that the satisfaction of a query requires the evaluation of the similarities between that query and all the documents in the textual base. Therefore, the second objective of this contribution is to investigate how clustering techniques can be applied to the textual database in order to retrieve the documents satisfying a query through a partial exploration of the base. A tentative solution based on hierarchical clustering will be suggested.