Measuring Effectiveness of Geographic IR Systems in Digital Libraries

Evaluation Framework and Case Study
  • Damien Palacio
  • Guillaume Cabanac
  • Christian Sallaberry
  • Gilles Hubert
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6273)


Common search engines process users’ queries (i.e., information needs) by retrieving documents from pre-built term-based indexes. For digital libraries, such approaches are limited regarding particular contexts, such as specialized collections (e.g., cultural heritage collections) or specific retrieval criteria (e.g., multidimensional criteria). In this paper, we consider Information Retrieval systems exploiting geographic dimensions: spatial, temporal, and topical dimensions. Our contribution is twofold as we propose a Geographic Information Retrieval system evaluation framework and test the following hypothesis: combining spatial and temporal dimensions along with the topical dimension improves the effectiveness of Information Retrieval systems.


Information Retrieval Digital Library Evaluation Framework Test Collection Result List 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sautter, G., Böhm, K., Padberg, F., Tichy, W.F.: Empirical Evaluation of Semi-automated XML Annotation of Text Documents with the GoldenGATE Editor. In: Kovács, L., Fuhr, N., Meghini, C. (eds.) ECDL 2007. LNCS, vol. 4675, pp. 357–367. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  2. 2.
    Gaio, M., Sallaberry, C., Etcheverry, P., Marquesuzaa, C., Lesbegueries, J.: A global process to access documents’ contents from a geographical point of view. J. Vis. Lang. Comput. 19(1), 3–23 (2008)CrossRefGoogle Scholar
  3. 3.
    Sanderson, M., Kohler, J.: Analyzing Geographic Queries. In: SIGIR-GIR 2004: Workshop on Geographic Information Retrieval at SIGIR (2004)Google Scholar
  4. 4.
    Gan, Q., Attenberg, J., Markowetz, A., Suel, T.: Analysis of geographic queries in a search engine log. In: LocWeb 2008: 1st Int. Workshop on Location and the Web, pp. 49–56. ACM, New York (2008)CrossRefGoogle Scholar
  5. 5.
    Jones, R., Zhang, W.V., Rey, B., Jhala, P., Stipp, E.: Geographic intention and modification in web search. Int. J. Geogr. Inf. Sci. 22(3), 229–246 (2008)CrossRefGoogle Scholar
  6. 6.
    Kanhabua, N., Nørvåg, K.: Temporal Language Models for Determining Time of Non-timestamped Documents. In: Christensen-Dalsgaard, B., Castelli, D., Ammitzbøll Jurik, B., Lippincott, J. (eds.) ECDL 2008. LNCS, vol. 5173, pp. 358–370. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  7. 7.
    Liesaputra, V., Witten, I.H., Bainbridge, D.: Searching in a Book. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds.) ECDL 2009. LNCS, vol. 5714, pp. 442–446. Springer, Heidelberg (2009)Google Scholar
  8. 8.
    Usery, E.L.: A feature-based geographic information system model. Photogramm. Eng. Rem. Sens. 62(7), 833–838 (1996)Google Scholar
  9. 9.
    Le Parc-Lacayrelle, A., Gaio, M., Sallaberry, C.: La composante temps dans l’information géographique textuelle. Document Numérique 10(2), 129–148 (2007)CrossRefGoogle Scholar
  10. 10.
    Sallaberry, C., Baziz, M., Lesbegueries, J., Gaio, M.: Towards an IE and IR System Dealing with Spatial Information in Digital Libraries – Evaluation Case Study. In: ICEIS 2007: 9th Int. Conference on Enterprise Information Systems, pp. 190–197 (2007)Google Scholar
  11. 11.
    Gey, F.C., Larson, R.R., Sanderson, M., Joho, H., Clough, P., Petras, V.: GeoCLEF 2005: The CLEF 2005 Cross-Language Geographic Information Retrieval Track Overview. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 908–919. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  12. 12.
    Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)Google Scholar
  13. 13.
    Verhagen, M., Gaizauskas, R., Schilder, F., Hepple, M., Moszkowicz, J., Pustejovsky, J.: The TempEval challenge: identifying temporal relations in text. Lang. Resour. Eval. 43(2), 161–179 (2009)CrossRefGoogle Scholar
  14. 14.
    Bucher, B., Clough, P., Joho, H., Purves, R., Syed, A.K.: Geographic IR Systems: Requirements and Evaluation. In: ICC 2005: 22nd Int. Cartographic Conference (2005) (CDROM)Google Scholar
  15. 15.
    Peters, C.: Introduction. In: Peters, C. (ed.) CLEF 2000. LNCS, vol. 2069, pp. 1–6. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  16. 16.
    Perea-Ortega, J.M., García-Cumbreras, M.A., García-Vega, M., Ureña-López, L.A.: Comparing Several Textual Information Retrieval Systems for the Geographical Information Retrieval Task. In: Kapetanios, E., Sugumaran, V., Spiliopoulou, M. (eds.) NLDB 2008. LNCS, vol. 5039, pp. 142–147. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  17. 17.
    Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Johnson, D.: Terrier Information Retrieval Platform. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 517–519. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  18. 18.
    Harman, D.K.: The TREC Test Collections. In: [12], ch. 2, pp. 21–53Google Scholar
  19. 19.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (July 2008)zbMATHGoogle Scholar
  20. 20.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)CrossRefGoogle Scholar
  21. 21.
    Hull, D.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In: SIGIR 1993: 16th Annual Int. SIGIR Conference, pp. 329–338. ACM Press, New York (1993)CrossRefGoogle Scholar
  22. 22.
    Clough, P., Joho, H., Purves, R.: Judging the Spatial Relevance of Documents for GIR. In: Lalmas, M., MacFarlane, A., Rüger, S.M., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds.) ECIR 2006. LNCS, vol. 3936, pp. 548–552. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  23. 23.
    Martins, B., Silva, M.J., Andrade, L.: Indexing and ranking in Geo-IR systems. In: GIR 2005: Workshop on Geographic Information Retrieval, pp. 31–34. ACM, New York (2005)CrossRefGoogle Scholar
  24. 24.
    Jones, C.B., Purves, R.: GIR 2005 ACM Workshop on Geographical Information Retrieval. SIGIR Forum 40(1), 34–37 (2006)Google Scholar
  25. 25.
    Larson, R.R.: Geographic Information Retrieval and Digital Libraries. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds.) ECDL 2009. LNCS, vol. 5714, pp. 461–464. Springer, Heidelberg (2009)Google Scholar
  26. 26.
    Bilhaut, F., Charnois, T., Enjalbert, P., Mathet, Y.: Geographic reference analysis for geographic document querying. In: HLT-NAACL 2003: Workshop on Analysis of Geographic References, pp. 55–62. ACL, Morristown (2003)CrossRefGoogle Scholar
  27. 27.
    Palacio, D., Sallaberry, C., Gaio, M.: Normalizing Spatial Information to Improve Geographical Information Indexing and Retrieval in Digital Libraries. In: ISGIS 2010: Joint Int. Conference on Theory, Data Handling and Modelling in GeoSpatial Information Science Proceedings (to appear, 2010)Google Scholar
  28. 28.
    Fox, E.A., Shaw, J.A.: Combination of Multiple Searches. In: Harman, D.K. (ed.) TREC-1: 1st Text REtrieval Conference, Gaithersburg, MD, USA, pp. 243–252. NIST (February 1993)Google Scholar
  29. 29.
    Hubert, G., Mothe, J.: An adaptable search engine for multimodal information retrieval. J. Am. Soc. Inf. Sci. Technol. 60(8), 1625–1634 (2009)CrossRefGoogle Scholar
  30. 30.
    Lee, J.H.: Analyses of Multiple Evidence Combination. In: SIGIR 1997: 20th Annual Int. SIGIR Conference, pp. 267–276. ACM Press, New York (1997)CrossRefGoogle Scholar
  31. 31.
    Woodruff, A.G., Plaunt, C.: Gipsy: automated geographic indexing of text documents. J. Am. Soc. Inf. Sci. 45(9), 645–655 (1994)CrossRefGoogle Scholar
  32. 32.
    Vaid, S., Jones, C.B., Joho, H., Sanderson, M.: Spatio-textual Indexing for Geographical Search on the Web. In: Bauzer Medeiros, C., Egenhofer, M.J., Bertino, E. (eds.) SSTD 2005. LNCS, vol. 3633, pp. 218–235. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  33. 33.
    Lieberman, M.D., Samet, H., Sankaranarayanan, J., Sperling, J.: STEWARD: Architecture of a Spatio-Textual Search Engine. In: GIS 2007: 15th Annual ACM Int. Symposium on Advances in Geographic Information Systems, pp. 1–8. ACM, New York (2007)CrossRefGoogle Scholar
  34. 34.
    Pfoser, D., Efentakis, A., Hadzilacos, T., Karagiorgou, S., Vasiliou, G.: Providing Universal Access to History Textbooks: A Modified GIS Case. In: Carswell, J.D., Fotheringham, A.S., McArdle, G. (eds.) W2GIS 2009. LNCS, vol. 5886, pp. 87–102. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  35. 35.
    Manguinhas, H., Martins, B., Borbinha, J., Siabato, W.: The DIGMAP Geo-Temporal Web Gazetteer Gervice. e-Perimetron: Int. Web J. Sci. Technol. Affined Hist. Cartogr. Maps 4(1), 9–24 (2009)Google Scholar
  36. 36.
    Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensional relevance: A new aggregation criterion. In: ECIR 2009: 31th European Conference on IR Research on Advances in Information Retrieval, pp. 264–275. Springer, Heidelberg (2009)Google Scholar
  37. 37.
    Farah, M., Vanderpooten, D.: An outranking approach for information retrieval. Inf. Retr. 11(4), 315–334 (2008)CrossRefGoogle Scholar
  38. 38.
    Yager, R.R.: On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans. Syst. Man Cybern. 18(1), 183–190 (1988)zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Damien Palacio
    • 1
  • Guillaume Cabanac
    • 2
  • Christian Sallaberry
    • 1
  • Gilles Hubert
    • 2
  1. 1.LIUPPA ÉA 3000Université de Pau et des Pays de l’AdourPau cedex
  2. 2.IRIT UMR 5505 CNRSUniversité de ToulouseToulouse cedex 9

Personalised recommendations