On the evaluation of Geographic Information Retrieval systems

Evaluation framework and case study
  • Damien Palacio
  • Guillaume Cabanac
  • Christian Sallaberry
  • Gilles Hubert
Article

DOI: 10.1007/s00799-011-0070-z

Cite this article as:
Palacio, D., Cabanac, G., Sallaberry, C. et al. Int J Digit Libr (2010) 11: 91. doi:10.1007/s00799-011-0070-z

Abstract

Search engines for Digital Libraries allow users to retrieve documents according to their contents. They process documents without differentiating the manifold aspects of information. Spatial and temporal dimensions are particularly dismissed. These dimensions are, however, of great interest for users of search engines targeting either the Web or specialized Digital Libraries. Recent studies reported that nearly 20% queries convey spatial and temporal information in addition to topical information. These three dimensions were referred to as parts of “geographic information.” In the literature, search engines handling those dimensions are called “Geographic Information Retrieval (GIR) systems.” Although several initiatives for evaluating GIR systems were undertaken, none was concerned with evaluating these three dimensions altogether. In this article, we address this issue by designing an evaluation framework, usefulness of which is highlighted through a case study involving a test collection and a GIR system. This framework allowed the comparison of our GIR system to state-of-the-art topical approaches. We also performed experiments for measuring performance improvement stemming from each dimension or their combination. We show that combining the three dimensions yields improvement in effectiveness (+73.9%) over a common topical baseline. Moreover, rather than conveying redundancy, the three dimensions complement each other.

Keywords

Geographic Information Retrieval Effectiveness measurement Evaluation framework Case study 

Abbreviations

ACF

Absolute calendar feature

AP

Average precision

ASF

Absolute spatial feature

CF

Calendar feature

DCG

Definite clause grammar

GIR

Geographic information retrieval

GIS

Geographic information system

IE

Information extraction

IR

Information retrieval

IV

Information visualization

MAP

Mean average precision

NDCG

Normalized discounted cumulative gain

NER

Named entity recognition

NLP

Natural language processing

RCF

Relative calendar feature

RSF

Relative spatial feature

SF

Spatial feature

TF

Term frequency

TFc

Continuous term frequency

Supplementary material

799_2011_70_MOESM1_ESM.zip (3.4 mb)
ESM 1 (ZIP 3511 kb)

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Damien Palacio
    • 1
  • Guillaume Cabanac
    • 2
  • Christian Sallaberry
    • 1
  • Gilles Hubert
    • 2
  1. 1.Université de Pau et des Pays de l’Adour, LIUPPA ÉA 3000Pau cedexFrance
  2. 2.Université de Toulouse, IRIT UMR 5505 CNRSToulouse cedex 9France

Personalised recommendations