On the evaluation of Geographic Information Retrieval systems

Evaluation framework and case study
  • Damien Palacio
  • Guillaume Cabanac
  • Christian Sallaberry
  • Gilles Hubert
Article

Abstract

Search engines for Digital Libraries allow users to retrieve documents according to their contents. They process documents without differentiating the manifold aspects of information. Spatial and temporal dimensions are particularly dismissed. These dimensions are, however, of great interest for users of search engines targeting either the Web or specialized Digital Libraries. Recent studies reported that nearly 20% queries convey spatial and temporal information in addition to topical information. These three dimensions were referred to as parts of “geographic information.” In the literature, search engines handling those dimensions are called “Geographic Information Retrieval (GIR) systems.” Although several initiatives for evaluating GIR systems were undertaken, none was concerned with evaluating these three dimensions altogether. In this article, we address this issue by designing an evaluation framework, usefulness of which is highlighted through a case study involving a test collection and a GIR system. This framework allowed the comparison of our GIR system to state-of-the-art topical approaches. We also performed experiments for measuring performance improvement stemming from each dimension or their combination. We show that combining the three dimensions yields improvement in effectiveness (+73.9%) over a common topical baseline. Moreover, rather than conveying redundancy, the three dimensions complement each other.

Keywords

Geographic Information Retrieval Effectiveness measurement Evaluation framework Case study 

Abbreviations

ACF

Absolute calendar feature

AP

Average precision

ASF

Absolute spatial feature

CF

Calendar feature

DCG

Definite clause grammar

GIR

Geographic information retrieval

GIS

Geographic information system

IE

Information extraction

IR

Information retrieval

IV

Information visualization

MAP

Mean average precision

NDCG

Normalized discounted cumulative gain

NER

Named entity recognition

NLP

Natural language processing

RCF

Relative calendar feature

RSF

Relative spatial feature

SF

Spatial feature

TF

Term frequency

TFc

Continuous term frequency

Supplementary material

799_2011_70_MOESM1_ESM.zip (3.4 mb)
ESM 1 (ZIP 3511 kb)

References

  1. 1.
    Purday J.: Think culture: Europeana.eu from concept to construction. Electron. Libr. 27(6), 919–937 (2009). doi:10.1108/02640470911004039 CrossRefGoogle Scholar
  2. 2.
    Usery E.L.: A feature-based geographic information system model. Photogramm. Eng. Rem. Sens. 62(7), 833–838 (1996) ISSN 0099-1112Google Scholar
  3. 3.
    Sanderson, M., Kohler, J.: Analyzing geographic queries. In: SIGIR-GIR’04: Proceedings of the Workshop on Geographic Information Retrieval at SIGIR (2004)Google Scholar
  4. 4.
    Asadi, S., Chang, C.-Y., Zhou, X., Diederich, J.: Searching the World Wide Web for local services and facilities: a review on the patterns of location-based queries. In: Fan, W., Wu, Z., Yang, J. (eds.) WAIM’05: Proceedings of the 6th International Conference on Web-Age Information Management, LNCS, vol. 3739, pp. 91–101. Springer (2005). doi:10.1007/11563952_9
  5. 5.
    Souza, L.A., Davis, C.A., Jr., Borges, K.A.V., Delboni, T.M., Laender, A.H.F.: The role of gazetteers in geographic knowledge discovery on the Web. In: LA-WEB’05: Proceedings of the 3rd Latin American Web Congress, pp. 157–165. IEEE Computer Society, October 2005. doi:10.1109/LAWEB.2005.38
  6. 6.
    Gan, Q., Attenberg, J., Markowetz, A., Suel, T.: Analysis of geographic queries in a search engine log. In: LocWeb’08: Proceedings of the First International Workshop on Location and the Web, pp. 49–56. New York, NY, USA, 2008. ACM. doi:10.1145/1367798.1367806
  7. 7.
    Jones R., Zhang W.V., Rey B., Jhala P., Stipp E.: Geographic intention and modification in web search. Int. J. Geogr. Inf. Sci. 22(3), 229–246 (2008). doi:10.1080/13658810701626186 CrossRefGoogle Scholar
  8. 8.
    Gaio M., Sallaberry C., Etcheverry P., Marquesuzaa C., Lesbegueries J.: A global process to access documents’ contents from a geographical point of view. J. Vis. Lang. Comput. 19(1), 3–23 (2008). doi:10.1016/j.jvlc.2007.08.010 CrossRefGoogle Scholar
  9. 9.
    LeParc-Lacayrelle A., Gaio M., Sallaberry C.: La composante temps dans l’information géographique textuelle. Document Numérique 10(2), 129–148 (2007). doi:10.3166/dn.10.2.129-148 CrossRefGoogle Scholar
  10. 10.
    Sallaberry, C., Baziz, M., Lesbegueries, J., Gaio, M.: Towards an IE and IR system dealing with spatial information in digital libraries—evaluation case study. In: Cardoso, J., Cordeiro, J., Filipe, J. (eds.) ICEIS’07: Proceedings of the 9th International Conference on Enterprise Information Systems, pp. 190–197, 2007Google Scholar
  11. 11.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to information retrieval. Cambridge University Press, July 2008Google Scholar
  12. 12.
    Gaizauskas R.J.: Recent advances in computational terminology edited by Didier Bourigault, Christian Jacquemin, and Marie-Claude L’Homme. Comput. Linguist. 29(2), 328–332 (2003). doi:10.1162/coli.2003.29.2.328 CrossRefGoogle Scholar
  13. 13.
    Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V.: GATE: an architecture for development of robust HLT applications. In: ACL’02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 168–175, Morristown, NJ, USA, 2002. ACL. doi:10.3115/1073083.1073112
  14. 14.
    Bilhaut, F., Widlöcher, A.: Linguastream: an integrated environment for computational linguistics experimentation. In: EACL’06: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations, pp. 95–98, Morristown, NJ, USA, 2006. ACL. doi:10.1.1.134.4336
  15. 15.
    Liu, Z., Gibbon, D.C., Shahraray, B.: Multimedia content acquisition and processing in the MIRACLE system. In: CCNC’06: Proceedings of the 3rd IEEE Conference on Consumer Communications and Networking, pp. 272–276, January 2006. doi:10.1109/CCNC.2006.1593030
  16. 16.
    Sagot B., Boullier P.: SxPipe 2: architecture pour le traitement pré syntaxique de corpus bruts. Traitement Automatique des Langues 49(2), 155–188 (2008)Google Scholar
  17. 17.
    Ferruci D., Lally A.: UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat. Lang. Eng., 10(3–4), 327–348 (2004). doi:10.1017/S1351324904003523 CrossRefGoogle Scholar
  18. 18.
    Spärck Jones, K.S.: A statistical interpretation of term specificity and its application in retrieval. J. Doc., 28(1), 11–21 (1972). ISSN 0022-0418. doi:10.1108/eb026526
  19. 19.
    Robertson S.E., Spärck Jones K.S.: Relevance weighting of search terms. J. Am. Soc. Inf. Sci. 27(3), 129–146 (1976). doi:10.1002/asi.4630270302 CrossRefGoogle Scholar
  20. 20.
    Salton G., Wong A., Yang C.S.: A vector space model for automatic indexing. Commun. ACM 18(11), 613–620 (1975). doi:10.1145/361219.361220 MATHCrossRefGoogle Scholar
  21. 21.
    Larson, R.R., Frontiera, P.: Spatial ranking methods for geographic information retrieval (GIR) in digital libraries. In: ECDL’04: Proceedings of the 8th European Conference on Digital Libraries, LNCS, 3232, pp. 45–56. Springer, 2004. ISBN 3-540-23013-0. doi:10.1007/978-3-540-30230-8_5
  22. 22.
    Andogah, G.: Geographically Constrained Information Retrieval. PhD thesis, University of Groningen, Netherlands, May 2010Google Scholar
  23. 23.
    Alonso, O.R.: Temporal information retrieval. PhD thesis, University of California, USA, May 2008Google Scholar
  24. 24.
    Kalczynski, P.J., Chou, A.: Temporal document retrieval model for business news archives. Inf. Process. Manage, 41(3), 635–650, 2005. ISSN 0306-4573. doi:10.1016/j.ipm.2004.01.002 Google Scholar
  25. 25.
    Woodruff, A.G., Plaunt, C.: Gipsy: automated geographic indexing of text documents. J. Am. Soc. Inf. Sci., 45(9): 645–655, 1994. ISSN 0002-8231. doi:10.1002/(SICI)1097-4571(199410)45:9<645::AID-ASI2>3.0.CO;2-8
  26. 26.
    Bilhaut, F., Charnois, T., Enjalbert, P., Mathet, Y.: Geographic reference analysis for geographic document querying. In: HLT- NAACL’03: Proceedings of the Workshop on Analysis of Geographic References, pp. 55–62, Morristown, NJ, USA, 2003. ACL. doi:10.3115/1119394.1119403
  27. 27.
    Vaid, S., Jones, C.B., Joho, H., Sanderson, M.: Spatio-textual indexing for geographical search on the Web. In: SSTD’05: Proceedings of the 9th International Symposium on Spatial and Temporal Databases, LNCS, vol. 3633, pp. 218–235. Springer, 2005. doi:10.1007/11535331_13
  28. 28.
    Valcartier. GRID—geospatial retrieval of indexed document. Technical report, R&D pour la défense, Canada (2006)Google Scholar
  29. 29.
    Martins, B., Borbinha, J., Pedrosa, G., Gil, J., Freire, N.: Geographically-aware information retrieval for collections of digitized historical maps. In: GIR’07: Proceedings of the 4th ACM workshop on Geographical Information Retrieval, pp. 39–42, New York, NY, USA, 2007. ACM. doi:10.1145/1316948.1316959
  30. 30.
    Chen, Y.-F.R., Di Fabbrizio, G., Gibbon, D., Jora, S., Renger, B., Wei, B.: Geotracker: geospatial and temporal RSS navigation. In: WWW’07: Proceedings of the 16th International Conference on World Wide Web, pp. 41–50, New York, NY, USA, 2007. ACM. doi:10.1145/1242572.1242579
  31. 31.
    Lieberman, M.D., Samet, H., Sankaranarayanan, J., Sperling, J.: STEWARD: architecture of a Spatio-Textual search engine. In: GIS’07: Proceedings of the 15th Annual ACM International Symposium on Advances in Geographic Information Systems, pp. 1–8, New York, NY, USA, 2007. ACM. doi:10.1145/1341012.1341045
  32. 32.
    Pfoser, D., Efentakis, A., Hadzilacos, T., Karagiorgou, S., Vasiliou, G.: Providing universal access to history textbooks: a modified GIS case. In: W2GIS’09: Proceedings of the 9th International Symposium on Web and Wireless Geographical Information Systems, LNCS, vol. 5886, pp. 87–102, 2009. doi:10.1007/978-3-642-10601-9_7
  33. 33.
    Buscaldi, D., Rosso, P.: Geooreka: enhancing web searches with geographical information. In: De Antonellis, V., Castano, S., Catania, B., Guerrini, G. (eds.) SEBD’09: Proceedings of the Seventeenth Italian Symposium on Advanced Database Systems, pp. 205–212. Edizioni Seneca, 2009Google Scholar
  34. 34.
    García-Cumbreras, M.Á., Perea-Ortega, J.M., García-Vega, M., Ureña-López, L.A.: Information retrieval with geographical references. Relevant documents filtering vs. query expansion. Inf. Process. Manage. 45(5), 605–614 (2009). ISSN 0306-4573. doi:10.1016/j.ipm.2009.04.006
  35. 35.
    Strötgen, J., Gertz, M., Popov, P.: Extraction and exploration of spatio-temporal information in documents. In: GIR’10: Proceedings of the 6th Workshop on Geographic Information Retrieval, pp. 16:1–16:8, New York, NY, USA, 2010. ACM. doi:10.1145/1722080.1722101
  36. 36.
    Brisaboa N., Luaces M., Places , Á. , Seco D.: Exploiting geographic references of documents in a geographical information retrieval system using an ontology-based index. GeoInformatica 14(3), 307–331 (2010). doi:10.1007/s10707-010-0106-3 CrossRefGoogle Scholar
  37. 37.
    Stokes, N., Li, Y., Moffat, A., Rong, J. An empirical study of the effects of NLP components on Geographic IR performance. Int. J. Geogr. Inf. Sci. 22(3), 247–264, 2008. ISSN 1365-8816. doi:10.1080/13658810701626210
  38. 38.
    Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: a high performance and scalable information retrieval platform. In: OSIR’06: Proceedings of ACM SIGIR’06 Workshop on Open Source Information Retrieval (2006)Google Scholar
  39. 39.
    Clough, P., Joho, H., Purves, R.: Judging the spatial relevance of documents for GIR. In: ECIR’06: Proceedings of the 28th European Conference on IR Research, LNCS, vol. 3936, pp. 548–552. Springer, 2006. doi:10.1007/11735106_62
  40. 40.
    Martins, B., Silva, M.J., Andrade, L.: Indexing and ranking in Geo-IR systems. In: GIR’05: Proceedings of the 2005 Workshop on Geographic Information Retrieval, pp. 31–34, New York, NY, USA, 2005. ACM. doi:10.1145/1096985.1096993
  41. 41.
    Jones, C.B., Purves, R.: GIR’05 2005 ACM workshop on geographical information retrieval. SIGIR Forum, 40(1), 34–37 (2006). ISSN 0163-5840. doi:10.1145/1147197.1147202
  42. 42.
    Larson, R.R.: Geographic information retrieval and digital libraries. In: ECDL’09: Proceedings of the 13th European Conference on Digital Libraries, LNCS, vol. 5714, pp. 461–464. Springer, 2009. doi:10.1007/978-3-642-04346-8_59
  43. 43.
    Widlöcher, A., Bilhaut, F.: La plate-forme LinguaStream: un outil d’exploration linguistique sur corpus. In: TALN’05: Actes de la 12e Conférence sur le Traitement Automatique du Langage Naturel (2005)Google Scholar
  44. 44.
    Kergosien, E., Kamel, M., Sallaberry, C., Bessagnet, M.-N., Aussenac-Gilles, N., Gaio, M.: Construction automatique d’ontologie et enrichissement à partir de ressources externes. In: JFO’09: Actes des 3e Journées Francophones sur les Ontologies, pp. 11–20, 2009Google Scholar
  45. 45.
    Sallaberry, C., Gaio, M., Palacio, D., Lesbegueries, J.: Fuzzying GIS topological functions for GIR needs. In: GIR’08: Proceeding of the 2nd international Workshop on Geographic Information Retrieval, pp. 1–8, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-253-5. doi:10.1145/1460007.1460008
  46. 46.
    Chrisman N.R.: Deficiencies of sheets and tiles: building sheetless databases. Int. J. Geogr. Inf. Sci. 4(2), 157–167 (1990). doi:10.1080/02693799008941537 CrossRefGoogle Scholar
  47. 47.
    Palacio, D., Sallaberry, C., Gaio, M.: Normalizing spatial information to improve geographical information indexing and retrieval in digital libraries. In: ISGIS’10: Proceedings of the Joint International Conference on Theory, Data Handling and Modelling in GeoSpatial Information Science proceedings, pp. 229–234 (2010)Google Scholar
  48. 48.
    Palacio, D., Sallaberry, C., Gaio, M.: Normalizing spatial information to better combine criteria in geographical information retrieval. In: ECIR-GIIW’09: Proceeding of the International Workshop on Geographic Information on the Internet, pp. 37–48, 2009Google Scholar
  49. 49.
    Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: Harman, D.K. (ed.) TREC-1: Proceedings of the First Text REtrieval Conference, pp. 243–252, Gaithersburg, MD, USA, February 1993. NISTGoogle Scholar
  50. 50.
    Hubert G., Mothe J.: An adaptable search engine for multimodal information retrieval. J. Am. Soc. Inf. Sci. Technol., 60(8), 1625–1634 (2009). doi:10.1002/asi.21091 CrossRefGoogle Scholar
  51. 51.
    Lee, J.H.: Analyses of multiple evidence combination. In: SIGIR’97: Proceedings of the 20th Annual International ACM SIGIR Conference, pp. 267–276, New York, NY, USA, 1997. ACM Press. doi:10.1145/258525.258587
  52. 52.
    de Borda, J.-C.: Mémoire sur les élections au scrutin. In: Histoire de l’Académie Royale des Sciences, Paris (1781)Google Scholar
  53. 53.
    de Condorcet, M.: Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris (1785)Google Scholar
  54. 54.
    Saari, D.: Which is better: the Condorcet or Borda winner? Soc. Choice Welfare 26(1), 107–129, 2006. ISSN 0176-1714. doi:10.1007/s00355-005-0046-2
  55. 55.
    Aslam, J.A., Montague, M.: Models for metasearch. In: SIGIR’01: Proceedings of the 24th Annual International ACM SIGIR Conference, pp. 276–284, New York, NY, USA, 2001. ACM. doi:10.1145/383952.384007
  56. 56.
    Montague, M., Aslam, J.A.: Condorcet fusion for improved retrieval. In: CIKM’02: Proceedings of the 11th International Conference on Information and Knowledge Management, pp. 538–548, New York, NY, USA, 2002. ACM. doi:10.1145/584792.584881
  57. 57.
    Cabanac, G., Hubert, G., Boughanem, M., Chrisment, C.: Tie-breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A.F. (eds.) CLEF’10: Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, LNCS, vol. 6360, pp. 112–123. Springer, 2010. doi:10.1007/978-3-642-15998-5_13
  58. 58.
    Farah, M., Vanderpooten, D.: An outranking approach for rank aggregation in information retrieval. In: SIGIR’07: Proceedings of the 30th Annual International ACM SIGIR Conference, pp. 591–598, New York, NY, USA, 2007. ACM. doi:10.1145/1277741.1277843
  59. 59.
    Liu, B.: Information Retrieval and Web Search. In: Web Data Mining, Data-Centric Systems and Applications, chapter 6, pp. 183–236. Springer, 2007. ISBN 978-3-540-37882-2. doi:10.1007/978-3-540-37882-2_6
  60. 60.
    Cleverdon, C.W.: Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. ASLIB Cranfield Research Project, ASLIB, Cranfield, UK, October 1962Google Scholar
  61. 61.
    Robertson, S.: On the history of evaluation in IR. J. Inf. Sci. 34(4), 439–456, 2008. doi:10.1177/0165551507086989 Google Scholar
  62. 62.
    Sanderson, M.: Test collection based evaluation of information retrieval systems. Found. Trends Inf. Retr. 4(4):247–375, 2010. ISSN 1554-0669. doi:10.1561/1500000009
  63. 63.
    Harman, D.K.: The TREC test collections. In: Voorhees and Harman [67], chapter 2, pp. 21–53Google Scholar
  64. 64.
    Buckley, C., Voorhees, E.M.: Evaluating evaluation measure stability. In: SIGIR’00: Proceedings of the 23rd International ACM SIGIR Conference, pp. 33–40, New York, NY, USA, 2000. ACM. doi:10.1145/345508.345543
  65. 65.
    Voorhees, E.M.: The philosophy of information retrieval evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF’01: Proceedings of the Second Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 2406, pp. 355–370. Springer, 2002. doi:10.1007/3-540-45691-0_34
  66. 66.
    Harman, D.K. (ed.): TREC-1: Proceedings of the First Text REtrieval Conference, Gaithersburg, MD, USA, February 1993. NISTGoogle Scholar
  67. 67.
    Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA, USA, 2005Google Scholar
  68. 68.
    Kando, N., Kuriyama, K., Nozue, T., Eguchi, K., Kato, H., Hidaka, S.: Overview of IR Tasks at the First NTCIR Workshop. In: Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, pp. 11–44. NACSIS, 1999Google Scholar
  69. 69.
    Kando, N.: Evaluation of information access technologies at the NTCIR workshop. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) Comparative Evaluation of Multilingual Information Access Systems, LNCS, vol. 3237, pp. 197–221. Springer, 2004. doi:10.1007/978-3-540-30222-3_4
  70. 70.
    Peters, C. (ed.): CLEF’01: Proceedings of the 2nd Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 2069. Springer, 2001. doi:10.1007/3-540-44645-1
  71. 71.
    Peters C., Braschler M.: European Research Letter—Cross- Language System Evaluation: the CLEF Campaigns. J. Am. Soc. Inf. Sci. Technol. 52(12), 1067–1072 (2001). doi:10.1002/asi.1164 CrossRefGoogle Scholar
  72. 72.
    Fuhr, N., Gövert, N., Kazai, G., Lalmas, M. (eds.): INEX’02: Proceedings of the First Workshop of the INitiative for the Evaluation of XML Retrieval (INEX), 2002Google Scholar
  73. 73.
    Kazai G., Lalmas M., Fuhr N., Fuhr N., Fuhr N.: A report on the first year of the initiative for the evaluation of XML retrieval. J. Am. Soc. Inf. Sci. Technol. 55(6), 551–556 (2004). doi:10.1002/asi.10386 CrossRefGoogle Scholar
  74. 74.
    Verhagen M., Gaizauskas R., Schilder F., Hepple M., Moszkowicz J., Pustejovsky J.: The TempEval challenge: identifying temporal relations in text. Lang. Resour. Eval. 43(2), 161–179 (2009). doi:10.1007/s10579-009-9086-z CrossRefGoogle Scholar
  75. 75.
    Bucher, B., Clough, P., Joho, H., Purves, R., Syed, A.K.: Geographic IR systems: Requirements and evaluation. In: ICC’05: Proceedings of the 22nd International Cartographic Conference. Global Congressos, 2005. CDROMGoogle Scholar
  76. 76.
    Gey, F.C., Larson, R.R., Sanderson, M., Joho, H., Clough, P., Petras, V.: GeoCLEF’05: the CLEF 2005 cross-language geographic information retrieval track overview. In: CLEF’05: Proceedings of the 5th Workshop on Cross-Language Evalution Forum, LNCS, vol. 4022, pp. 908–919. Springer, 2006. doi:10.1007/11878773_101
  77. 77.
    Perea-Ortega, J.M., García-Cumbreras, M.A., García-Vega, M., Ureña-López, L. A. Comparing several textual information retrieval systems for the geographical information retrieval task. In: NLDB’08: Proceedings of the 13th International Conference on Natural Language and Information Systems, pp. 142–147, Berlin, Heidelberg, 2008. Springer-Verlag. doi:10.1007/978-3-540-69858-6_15
  78. 78.
    Ogilvie, P., Callan, J.P.: Experiments using the Lemur toolkit. In: TREC’01: Proceedings of the 9th Text REtrieval Conference, Gaithersburg, MD, USA, February 2001. NISTGoogle Scholar
  79. 79.
    Gospodnetić, O., Hatcher, E.: Lucene in Action. Manning Publications (2005)Google Scholar
  80. 80.
    Gey, F., Larson, R., Kando, N., Machado, J., Sakai, T.: NTCIR-GeoTime Overview: Evaluating Geographic and Temporal Search. In: NTCIR’10: Proceedings of the 8th NTCIR Workshop, pp. 147–153, Tockyo, Japan, 2010. NIIGoogle Scholar
  81. 81.
    Santos, D., Cabral, L.: GikiCLEF: Expectations and Lessons Learned. In: Peters, C., Di Nunzio, G., Kurimo, M., Mostefa, D., Penas, A., Roda, G. (eds.) CLEF’09: Proceedings of the 9th Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 6241, pp. 212–222. Springer Berlin/Heidelberg, 2010. doi:10.1007/978-3-642-15754-7_23
  82. 82.
    Palacio, D., Cabanac, G., Sallaberry, C., Hubert, G.: Measuring effectiveness of geographic IR systems in digital libraries: evaluation framework and case study. In: ECDL’10: Proceedings of the 14th European Conference on Research and Advanced Technology for Digital Libraries, LNCS, vol. 6273, pp. 340–351. Springer, 2010. doi:10.1007/978-3-642-15464-5_34
  83. 83.
    Järvelin K., Kekäläinen J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002). doi:10.1145/582415.582418 CrossRefGoogle Scholar
  84. 84.
    Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: SIGIR’93: Proceedings of the 16th Annual International ACM SIGIR Conference, pp. 329–338, New York, NY, USA, 1993. ACM Press. doi:10.1145/160688.160758
  85. 85.
    da Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensional relevance: A new aggregation criterion. In: ECIR’09: Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval, pp. 264–275, Berlin, Heidelberg (2009). Springer-Verlag. doi:10.1007/978-3-642-00958-7_25
  86. 86.
    Farah M., Vanderpooten D.: An outranking approach for information retrieval. Inf. Retr. 11(4), 315–334 (2008). doi:10.1007/s10791-008-9046-z CrossRefGoogle Scholar
  87. 87.
    Andogah, G., Bouma, G.: Relevance measures using geographic scopes and types. In: CLEF’07: Proceedings of the 7th Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 5152, pp. 794–801 (2008). doi:10.1007/978-3-540-85760-0_100
  88. 88.
    Yager R.R.: On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans. Syst. Man Cybern. 18(1), 183–190 (1988). doi:10.1109/21.87068 MathSciNetMATHCrossRefGoogle Scholar
  89. 89.
    Boughanem, M., Loiseau, Y., Prade, H.: Refining aggregation functions for improving document ranking in information retrieval. In: SUM’07: Proceedings of the 1st International Conference on Scalable Uncertainty Management, pp. 255–267, Berlin, Heidelberg, 2007. Springer. doi:10.1007/978-3-540-75410-7_19
  90. 90.
    Labreuche C., Grabisch M.: The Choquet integral for the aggregation of interval scales in multicriteria decision making. Fuzzy Sets Syst. 137(1), 11–26 (2003). doi:10.1016/S0165-0114(02)00429-3 MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Damien Palacio
    • 1
  • Guillaume Cabanac
    • 2
  • Christian Sallaberry
    • 1
  • Gilles Hubert
    • 2
  1. 1.Université de Pau et des Pays de l’Adour, LIUPPA ÉA 3000Pau cedexFrance
  2. 2.Université de Toulouse, IRIT UMR 5505 CNRSToulouse cedex 9France

Personalised recommendations