International Journal on Digital Libraries

, Volume 11, Issue 2, pp 91–109 | Cite as

On the evaluation of Geographic Information Retrieval systems

Evaluation framework and case study
  • Damien Palacio
  • Guillaume CabanacEmail author
  • Christian Sallaberry
  • Gilles Hubert


Search engines for Digital Libraries allow users to retrieve documents according to their contents. They process documents without differentiating the manifold aspects of information. Spatial and temporal dimensions are particularly dismissed. These dimensions are, however, of great interest for users of search engines targeting either the Web or specialized Digital Libraries. Recent studies reported that nearly 20% queries convey spatial and temporal information in addition to topical information. These three dimensions were referred to as parts of “geographic information.” In the literature, search engines handling those dimensions are called “Geographic Information Retrieval (GIR) systems.” Although several initiatives for evaluating GIR systems were undertaken, none was concerned with evaluating these three dimensions altogether. In this article, we address this issue by designing an evaluation framework, usefulness of which is highlighted through a case study involving a test collection and a GIR system. This framework allowed the comparison of our GIR system to state-of-the-art topical approaches. We also performed experiments for measuring performance improvement stemming from each dimension or their combination. We show that combining the three dimensions yields improvement in effectiveness (+73.9%) over a common topical baseline. Moreover, rather than conveying redundancy, the three dimensions complement each other.


Geographic Information Retrieval Effectiveness measurement Evaluation framework Case study 



Absolute calendar feature


Average precision


Absolute spatial feature


Calendar feature


Definite clause grammar


Geographic information retrieval


Geographic information system


Information extraction


Information retrieval


Information visualization


Mean average precision


Normalized discounted cumulative gain


Named entity recognition


Natural language processing


Relative calendar feature


Relative spatial feature


Spatial feature


Term frequency


Continuous term frequency


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material (3.4 mb)
ESM 1 (ZIP 3511 kb)


  1. 1.
    Purday J.: Think culture: from concept to construction. Electron. Libr. 27(6), 919–937 (2009). doi: CrossRefGoogle Scholar
  2. 2.
    Usery E.L.: A feature-based geographic information system model. Photogramm. Eng. Rem. Sens. 62(7), 833–838 (1996) ISSN 0099-1112Google Scholar
  3. 3.
    Sanderson, M., Kohler, J.: Analyzing geographic queries. In: SIGIR-GIR’04: Proceedings of the Workshop on Geographic Information Retrieval at SIGIR (2004)Google Scholar
  4. 4.
    Asadi, S., Chang, C.-Y., Zhou, X., Diederich, J.: Searching the World Wide Web for local services and facilities: a review on the patterns of location-based queries. In: Fan, W., Wu, Z., Yang, J. (eds.) WAIM’05: Proceedings of the 6th International Conference on Web-Age Information Management, LNCS, vol. 3739, pp. 91–101. Springer (2005). doi:
  5. 5.
    Souza, L.A., Davis, C.A., Jr., Borges, K.A.V., Delboni, T.M., Laender, A.H.F.: The role of gazetteers in geographic knowledge discovery on the Web. In: LA-WEB’05: Proceedings of the 3rd Latin American Web Congress, pp. 157–165. IEEE Computer Society, October 2005. doi:
  6. 6.
    Gan, Q., Attenberg, J., Markowetz, A., Suel, T.: Analysis of geographic queries in a search engine log. In: LocWeb’08: Proceedings of the First International Workshop on Location and the Web, pp. 49–56. New York, NY, USA, 2008. ACM. doi:
  7. 7.
    Jones R., Zhang W.V., Rey B., Jhala P., Stipp E.: Geographic intention and modification in web search. Int. J. Geogr. Inf. Sci. 22(3), 229–246 (2008). doi: CrossRefGoogle Scholar
  8. 8.
    Gaio M., Sallaberry C., Etcheverry P., Marquesuzaa C., Lesbegueries J.: A global process to access documents’ contents from a geographical point of view. J. Vis. Lang. Comput. 19(1), 3–23 (2008). doi: CrossRefGoogle Scholar
  9. 9.
    LeParc-Lacayrelle A., Gaio M., Sallaberry C.: La composante temps dans l’information géographique textuelle. Document Numérique 10(2), 129–148 (2007). doi: CrossRefGoogle Scholar
  10. 10.
    Sallaberry, C., Baziz, M., Lesbegueries, J., Gaio, M.: Towards an IE and IR system dealing with spatial information in digital libraries—evaluation case study. In: Cardoso, J., Cordeiro, J., Filipe, J. (eds.) ICEIS’07: Proceedings of the 9th International Conference on Enterprise Information Systems, pp. 190–197, 2007Google Scholar
  11. 11.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to information retrieval. Cambridge University Press, July 2008Google Scholar
  12. 12.
    Gaizauskas R.J.: Recent advances in computational terminology edited by Didier Bourigault, Christian Jacquemin, and Marie-Claude L’Homme. Comput. Linguist. 29(2), 328–332 (2003). doi: CrossRefGoogle Scholar
  13. 13.
    Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V.: GATE: an architecture for development of robust HLT applications. In: ACL’02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 168–175, Morristown, NJ, USA, 2002. ACL. doi:
  14. 14.
    Bilhaut, F., Widlöcher, A.: Linguastream: an integrated environment for computational linguistics experimentation. In: EACL’06: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations, pp. 95–98, Morristown, NJ, USA, 2006. ACL. doi: Scholar
  15. 15.
    Liu, Z., Gibbon, D.C., Shahraray, B.: Multimedia content acquisition and processing in the MIRACLE system. In: CCNC’06: Proceedings of the 3rd IEEE Conference on Consumer Communications and Networking, pp. 272–276, January 2006. doi:
  16. 16.
    Sagot B., Boullier P.: SxPipe 2: architecture pour le traitement pré syntaxique de corpus bruts. Traitement Automatique des Langues 49(2), 155–188 (2008)Google Scholar
  17. 17.
    Ferruci D., Lally A.: UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat. Lang. Eng., 10(3–4), 327–348 (2004). doi: CrossRefGoogle Scholar
  18. 18.
    Spärck Jones, K.S.: A statistical interpretation of term specificity and its application in retrieval. J. Doc., 28(1), 11–21 (1972). ISSN 0022-0418. doi:
  19. 19.
    Robertson S.E., Spärck Jones K.S.: Relevance weighting of search terms. J. Am. Soc. Inf. Sci. 27(3), 129–146 (1976). doi: CrossRefGoogle Scholar
  20. 20.
    Salton G., Wong A., Yang C.S.: A vector space model for automatic indexing. Commun. ACM 18(11), 613–620 (1975). doi: CrossRefGoogle Scholar
  21. 21.
    Larson, R.R., Frontiera, P.: Spatial ranking methods for geographic information retrieval (GIR) in digital libraries. In: ECDL’04: Proceedings of the 8th European Conference on Digital Libraries, LNCS, 3232, pp. 45–56. Springer, 2004. ISBN 3-540-23013-0. doi:
  22. 22.
    Andogah, G.: Geographically Constrained Information Retrieval. PhD thesis, University of Groningen, Netherlands, May 2010Google Scholar
  23. 23.
    Alonso, O.R.: Temporal information retrieval. PhD thesis, University of California, USA, May 2008Google Scholar
  24. 24.
    Kalczynski, P.J., Chou, A.: Temporal document retrieval model for business news archives. Inf. Process. Manage, 41(3), 635–650, 2005. ISSN 0306-4573. doi: CrossRefGoogle Scholar
  25. 25.
    Woodruff, A.G., Plaunt, C.: Gipsy: automated geographic indexing of text documents. J. Am. Soc. Inf. Sci., 45(9): 645–655, 1994. ISSN 0002-8231. doi:<645::AID-ASI2>3.0.CO;2-8
  26. 26.
    Bilhaut, F., Charnois, T., Enjalbert, P., Mathet, Y.: Geographic reference analysis for geographic document querying. In: HLT- NAACL’03: Proceedings of the Workshop on Analysis of Geographic References, pp. 55–62, Morristown, NJ, USA, 2003. ACL. doi:
  27. 27.
    Vaid, S., Jones, C.B., Joho, H., Sanderson, M.: Spatio-textual indexing for geographical search on the Web. In: SSTD’05: Proceedings of the 9th International Symposium on Spatial and Temporal Databases, LNCS, vol. 3633, pp. 218–235. Springer, 2005. doi:
  28. 28.
    Valcartier. GRID—geospatial retrieval of indexed document. Technical report, R&D pour la défense, Canada (2006)Google Scholar
  29. 29.
    Martins, B., Borbinha, J., Pedrosa, G., Gil, J., Freire, N.: Geographically-aware information retrieval for collections of digitized historical maps. In: GIR’07: Proceedings of the 4th ACM workshop on Geographical Information Retrieval, pp. 39–42, New York, NY, USA, 2007. ACM. doi:
  30. 30.
    Chen, Y.-F.R., Di Fabbrizio, G., Gibbon, D., Jora, S., Renger, B., Wei, B.: Geotracker: geospatial and temporal RSS navigation. In: WWW’07: Proceedings of the 16th International Conference on World Wide Web, pp. 41–50, New York, NY, USA, 2007. ACM. doi:
  31. 31.
    Lieberman, M.D., Samet, H., Sankaranarayanan, J., Sperling, J.: STEWARD: architecture of a Spatio-Textual search engine. In: GIS’07: Proceedings of the 15th Annual ACM International Symposium on Advances in Geographic Information Systems, pp. 1–8, New York, NY, USA, 2007. ACM. doi:
  32. 32.
    Pfoser, D., Efentakis, A., Hadzilacos, T., Karagiorgou, S., Vasiliou, G.: Providing universal access to history textbooks: a modified GIS case. In: W2GIS’09: Proceedings of the 9th International Symposium on Web and Wireless Geographical Information Systems, LNCS, vol. 5886, pp. 87–102, 2009. doi:
  33. 33.
    Buscaldi, D., Rosso, P.: Geooreka: enhancing web searches with geographical information. In: De Antonellis, V., Castano, S., Catania, B., Guerrini, G. (eds.) SEBD’09: Proceedings of the Seventeenth Italian Symposium on Advanced Database Systems, pp. 205–212. Edizioni Seneca, 2009Google Scholar
  34. 34.
    García-Cumbreras, M.Á., Perea-Ortega, J.M., García-Vega, M., Ureña-López, L.A.: Information retrieval with geographical references. Relevant documents filtering vs. query expansion. Inf. Process. Manage. 45(5), 605–614 (2009). ISSN 0306-4573. doi:
  35. 35.
    Strötgen, J., Gertz, M., Popov, P.: Extraction and exploration of spatio-temporal information in documents. In: GIR’10: Proceedings of the 6th Workshop on Geographic Information Retrieval, pp. 16:1–16:8, New York, NY, USA, 2010. ACM. doi:
  36. 36.
    Brisaboa N., Luaces M., Places , Á. , Seco D.: Exploiting geographic references of documents in a geographical information retrieval system using an ontology-based index. GeoInformatica 14(3), 307–331 (2010). doi: CrossRefGoogle Scholar
  37. 37.
    Stokes, N., Li, Y., Moffat, A., Rong, J. An empirical study of the effects of NLP components on Geographic IR performance. Int. J. Geogr. Inf. Sci. 22(3), 247–264, 2008. ISSN 1365-8816. doi:
  38. 38.
    Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: a high performance and scalable information retrieval platform. In: OSIR’06: Proceedings of ACM SIGIR’06 Workshop on Open Source Information Retrieval (2006)Google Scholar
  39. 39.
    Clough, P., Joho, H., Purves, R.: Judging the spatial relevance of documents for GIR. In: ECIR’06: Proceedings of the 28th European Conference on IR Research, LNCS, vol. 3936, pp. 548–552. Springer, 2006. doi:
  40. 40.
    Martins, B., Silva, M.J., Andrade, L.: Indexing and ranking in Geo-IR systems. In: GIR’05: Proceedings of the 2005 Workshop on Geographic Information Retrieval, pp. 31–34, New York, NY, USA, 2005. ACM. doi:
  41. 41.
    Jones, C.B., Purves, R.: GIR’05 2005 ACM workshop on geographical information retrieval. SIGIR Forum, 40(1), 34–37 (2006). ISSN 0163-5840. doi:
  42. 42.
    Larson, R.R.: Geographic information retrieval and digital libraries. In: ECDL’09: Proceedings of the 13th European Conference on Digital Libraries, LNCS, vol. 5714, pp. 461–464. Springer, 2009. doi:
  43. 43.
    Widlöcher, A., Bilhaut, F.: La plate-forme LinguaStream: un outil d’exploration linguistique sur corpus. In: TALN’05: Actes de la 12e Conférence sur le Traitement Automatique du Langage Naturel (2005)Google Scholar
  44. 44.
    Kergosien, E., Kamel, M., Sallaberry, C., Bessagnet, M.-N., Aussenac-Gilles, N., Gaio, M.: Construction automatique d’ontologie et enrichissement à partir de ressources externes. In: JFO’09: Actes des 3e Journées Francophones sur les Ontologies, pp. 11–20, 2009Google Scholar
  45. 45.
    Sallaberry, C., Gaio, M., Palacio, D., Lesbegueries, J.: Fuzzying GIS topological functions for GIR needs. In: GIR’08: Proceeding of the 2nd international Workshop on Geographic Information Retrieval, pp. 1–8, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-253-5. doi:
  46. 46.
    Chrisman N.R.: Deficiencies of sheets and tiles: building sheetless databases. Int. J. Geogr. Inf. Sci. 4(2), 157–167 (1990). doi: CrossRefGoogle Scholar
  47. 47.
    Palacio, D., Sallaberry, C., Gaio, M.: Normalizing spatial information to improve geographical information indexing and retrieval in digital libraries. In: ISGIS’10: Proceedings of the Joint International Conference on Theory, Data Handling and Modelling in GeoSpatial Information Science proceedings, pp. 229–234 (2010)Google Scholar
  48. 48.
    Palacio, D., Sallaberry, C., Gaio, M.: Normalizing spatial information to better combine criteria in geographical information retrieval. In: ECIR-GIIW’09: Proceeding of the International Workshop on Geographic Information on the Internet, pp. 37–48, 2009Google Scholar
  49. 49.
    Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: Harman, D.K. (ed.) TREC-1: Proceedings of the First Text REtrieval Conference, pp. 243–252, Gaithersburg, MD, USA, February 1993. NISTGoogle Scholar
  50. 50.
    Hubert G., Mothe J.: An adaptable search engine for multimodal information retrieval. J. Am. Soc. Inf. Sci. Technol., 60(8), 1625–1634 (2009). doi: CrossRefGoogle Scholar
  51. 51.
    Lee, J.H.: Analyses of multiple evidence combination. In: SIGIR’97: Proceedings of the 20th Annual International ACM SIGIR Conference, pp. 267–276, New York, NY, USA, 1997. ACM Press. doi:
  52. 52.
    de Borda, J.-C.: Mémoire sur les élections au scrutin. In: Histoire de l’Académie Royale des Sciences, Paris (1781)Google Scholar
  53. 53.
    de Condorcet, M.: Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris (1785)Google Scholar
  54. 54.
    Saari, D.: Which is better: the Condorcet or Borda winner? Soc. Choice Welfare 26(1), 107–129, 2006. ISSN 0176-1714. doi:
  55. 55.
    Aslam, J.A., Montague, M.: Models for metasearch. In: SIGIR’01: Proceedings of the 24th Annual International ACM SIGIR Conference, pp. 276–284, New York, NY, USA, 2001. ACM. doi:
  56. 56.
    Montague, M., Aslam, J.A.: Condorcet fusion for improved retrieval. In: CIKM’02: Proceedings of the 11th International Conference on Information and Knowledge Management, pp. 538–548, New York, NY, USA, 2002. ACM. doi:
  57. 57.
    Cabanac, G., Hubert, G., Boughanem, M., Chrisment, C.: Tie-breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A.F. (eds.) CLEF’10: Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, LNCS, vol. 6360, pp. 112–123. Springer, 2010. doi:
  58. 58.
    Farah, M., Vanderpooten, D.: An outranking approach for rank aggregation in information retrieval. In: SIGIR’07: Proceedings of the 30th Annual International ACM SIGIR Conference, pp. 591–598, New York, NY, USA, 2007. ACM. doi:
  59. 59.
    Liu, B.: Information Retrieval and Web Search. In: Web Data Mining, Data-Centric Systems and Applications, chapter 6, pp. 183–236. Springer, 2007. ISBN 978-3-540-37882-2. doi:
  60. 60.
    Cleverdon, C.W.: Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. ASLIB Cranfield Research Project, ASLIB, Cranfield, UK, October 1962Google Scholar
  61. 61.
    Robertson, S.: On the history of evaluation in IR. J. Inf. Sci. 34(4), 439–456, 2008. doi: CrossRefGoogle Scholar
  62. 62.
    Sanderson, M.: Test collection based evaluation of information retrieval systems. Found. Trends Inf. Retr. 4(4):247–375, 2010. ISSN 1554-0669. doi:
  63. 63.
    Harman, D.K.: The TREC test collections. In: Voorhees and Harman [67], chapter 2, pp. 21–53Google Scholar
  64. 64.
    Buckley, C., Voorhees, E.M.: Evaluating evaluation measure stability. In: SIGIR’00: Proceedings of the 23rd International ACM SIGIR Conference, pp. 33–40, New York, NY, USA, 2000. ACM. doi:
  65. 65.
    Voorhees, E.M.: The philosophy of information retrieval evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF’01: Proceedings of the Second Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 2406, pp. 355–370. Springer, 2002. doi:
  66. 66.
    Harman, D.K. (ed.): TREC-1: Proceedings of the First Text REtrieval Conference, Gaithersburg, MD, USA, February 1993. NISTGoogle Scholar
  67. 67.
    Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA, USA, 2005Google Scholar
  68. 68.
    Kando, N., Kuriyama, K., Nozue, T., Eguchi, K., Kato, H., Hidaka, S.: Overview of IR Tasks at the First NTCIR Workshop. In: Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, pp. 11–44. NACSIS, 1999Google Scholar
  69. 69.
    Kando, N.: Evaluation of information access technologies at the NTCIR workshop. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) Comparative Evaluation of Multilingual Information Access Systems, LNCS, vol. 3237, pp. 197–221. Springer, 2004. doi:
  70. 70.
    Peters, C. (ed.): CLEF’01: Proceedings of the 2nd Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 2069. Springer, 2001. doi:
  71. 71.
    Peters C., Braschler M.: European Research Letter—Cross- Language System Evaluation: the CLEF Campaigns. J. Am. Soc. Inf. Sci. Technol. 52(12), 1067–1072 (2001). doi: CrossRefGoogle Scholar
  72. 72.
    Fuhr, N., Gövert, N., Kazai, G., Lalmas, M. (eds.): INEX’02: Proceedings of the First Workshop of the INitiative for the Evaluation of XML Retrieval (INEX), 2002Google Scholar
  73. 73.
    Kazai G., Lalmas M., Fuhr N., Fuhr N., Fuhr N.: A report on the first year of the initiative for the evaluation of XML retrieval. J. Am. Soc. Inf. Sci. Technol. 55(6), 551–556 (2004). doi: CrossRefGoogle Scholar
  74. 74.
    Verhagen M., Gaizauskas R., Schilder F., Hepple M., Moszkowicz J., Pustejovsky J.: The TempEval challenge: identifying temporal relations in text. Lang. Resour. Eval. 43(2), 161–179 (2009). doi: CrossRefGoogle Scholar
  75. 75.
    Bucher, B., Clough, P., Joho, H., Purves, R., Syed, A.K.: Geographic IR systems: Requirements and evaluation. In: ICC’05: Proceedings of the 22nd International Cartographic Conference. Global Congressos, 2005. CDROMGoogle Scholar
  76. 76.
    Gey, F.C., Larson, R.R., Sanderson, M., Joho, H., Clough, P., Petras, V.: GeoCLEF’05: the CLEF 2005 cross-language geographic information retrieval track overview. In: CLEF’05: Proceedings of the 5th Workshop on Cross-Language Evalution Forum, LNCS, vol. 4022, pp. 908–919. Springer, 2006. doi:
  77. 77.
    Perea-Ortega, J.M., García-Cumbreras, M.A., García-Vega, M., Ureña-López, L. A. Comparing several textual information retrieval systems for the geographical information retrieval task. In: NLDB’08: Proceedings of the 13th International Conference on Natural Language and Information Systems, pp. 142–147, Berlin, Heidelberg, 2008. Springer-Verlag. doi:
  78. 78.
    Ogilvie, P., Callan, J.P.: Experiments using the Lemur toolkit. In: TREC’01: Proceedings of the 9th Text REtrieval Conference, Gaithersburg, MD, USA, February 2001. NISTGoogle Scholar
  79. 79.
    Gospodnetić, O., Hatcher, E.: Lucene in Action. Manning Publications (2005)Google Scholar
  80. 80.
    Gey, F., Larson, R., Kando, N., Machado, J., Sakai, T.: NTCIR-GeoTime Overview: Evaluating Geographic and Temporal Search. In: NTCIR’10: Proceedings of the 8th NTCIR Workshop, pp. 147–153, Tockyo, Japan, 2010. NIIGoogle Scholar
  81. 81.
    Santos, D., Cabral, L.: GikiCLEF: Expectations and Lessons Learned. In: Peters, C., Di Nunzio, G., Kurimo, M., Mostefa, D., Penas, A., Roda, G. (eds.) CLEF’09: Proceedings of the 9th Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 6241, pp. 212–222. Springer Berlin/Heidelberg, 2010. doi:
  82. 82.
    Palacio, D., Cabanac, G., Sallaberry, C., Hubert, G.: Measuring effectiveness of geographic IR systems in digital libraries: evaluation framework and case study. In: ECDL’10: Proceedings of the 14th European Conference on Research and Advanced Technology for Digital Libraries, LNCS, vol. 6273, pp. 340–351. Springer, 2010. doi:
  83. 83.
    Järvelin K., Kekäläinen J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002). doi: CrossRefGoogle Scholar
  84. 84.
    Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: SIGIR’93: Proceedings of the 16th Annual International ACM SIGIR Conference, pp. 329–338, New York, NY, USA, 1993. ACM Press. doi:
  85. 85.
    da Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensional relevance: A new aggregation criterion. In: ECIR’09: Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval, pp. 264–275, Berlin, Heidelberg (2009). Springer-Verlag. doi:
  86. 86.
    Farah M., Vanderpooten D.: An outranking approach for information retrieval. Inf. Retr. 11(4), 315–334 (2008). doi: CrossRefGoogle Scholar
  87. 87.
    Andogah, G., Bouma, G.: Relevance measures using geographic scopes and types. In: CLEF’07: Proceedings of the 7th Workshop of the Cross-Language Evaluation Forum, LNCS, vol. 5152, pp. 794–801 (2008). doi:
  88. 88.
    Yager R.R.: On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans. Syst. Man Cybern. 18(1), 183–190 (1988). doi: MathSciNetCrossRefGoogle Scholar
  89. 89.
    Boughanem, M., Loiseau, Y., Prade, H.: Refining aggregation functions for improving document ranking in information retrieval. In: SUM’07: Proceedings of the 1st International Conference on Scalable Uncertainty Management, pp. 255–267, Berlin, Heidelberg, 2007. Springer. doi:
  90. 90.
    Labreuche C., Grabisch M.: The Choquet integral for the aggregation of interval scales in multicriteria decision making. Fuzzy Sets Syst. 137(1), 11–26 (2003). doi: MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Damien Palacio
    • 1
  • Guillaume Cabanac
    • 2
    Email author
  • Christian Sallaberry
    • 1
  • Gilles Hubert
    • 2
  1. 1.Université de Pau et des Pays de l’Adour, LIUPPA ÉA 3000Pau cedexFrance
  2. 2.Université de Toulouse, IRIT UMR 5505 CNRSToulouse cedex 9France

Personalised recommendations