Knowledge and Information Systems

, Volume 18, Issue 2, pp 199–211 | Cite as

Consensus-based evaluation framework for distributed information retrieval systems

Regular Paper


Multi-agent systems have been attacking the challenges of information retrieval tasks on distributed environment. In this paper, we propose a consensus choice selection method based framework to evaluate the performance of cooperative information retrieval tasks of the multiple agents. Thereby, two well-known measurements, precision and recall, are extended to handle consensual closeness (i.e., local and global consensus) between the sets of retrieved results. We show that in a motivating example the proposed criteria are prone to solve the rigidity problem of classical precision and recall. More importantly, the retrieved results can be ranked with respect to the consensual score, and the ranking mechanism has been verified to be more reasonable.


Information retrieval Multi-agent systems Evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. Addison-Wesley, ReadingGoogle Scholar
  2. 2.
    Borlund P (2003) The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Inform Res 8(3)Google Scholar
  3. 3.
    Buckland M, Gey F (1994) The relationship between recall and precision. J Am Soc Inform Sci 45(1): 12–19CrossRefGoogle Scholar
  4. 4.
    Callan JP (2000) Distributed information retrieval, Chap. 5. Advances in Information Retrieval. Kluwer, Dordrecht, pp 127–150Google Scholar
  5. 5.
    Callan JP et al (1995) Searching distributed collections with inference networks. In: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’95), New York. ACM Press, New York, pp 21–28Google Scholar
  6. 6.
    Chen H-H et al (2003) Overview of CLIR task at the third NTCIR workshop. In: Proceedings of the 3rd NTCIR workshop on research in information retrieval, automatic text summarization and question answering (NTCIR-3)Google Scholar
  7. 7.
    Corrie B, Storey M-AD (2007) Toward understanding the importance of gesture in distributed scientific collaboration. Knowl Inform Syst 13(2): 143–171CrossRefGoogle Scholar
  8. 8.
    Coulouris G et al (1996) Distributed systems—concepts and design. Addison-Wesley, ReadingGoogle Scholar
  9. 9.
    Crestani F, Wu S (2006) Testing the cluster hypothesis in distributed information retrieval. Inform Process Manage 42(5): 1137–1150CrossRefGoogle Scholar
  10. 10.
    de Kretser O et al (1998) Methodologies for distributed information retrieval. In: Proceedings of the 18th International Conference on Distributed Computing Systems, Amsterdam, pp 66–73Google Scholar
  11. 11.
    Ellis D (1998) The dilemma of measurement in information retrieval research. J Am Soc Inform Sci 47(1): 23–36CrossRefGoogle Scholar
  12. 12.
    Fu L et al (2005) CQE: a collaborative querying environment. In: Marlino M, Sumner T, III FMS (eds) Proceedings of the 2005 ACM/IEEE Joint Conference on Digital Libraries (JCDL 2005). ACM Press, New York, p 378Google Scholar
  13. 13.
    Gordon M, Kochen M (1989) Recall-precision trade-off: a derivation. J Am Soc Inform Sci 40(3): 145–151CrossRefGoogle Scholar
  14. 14.
    Gravano L, Garcia-Molina H (1995) Generalizing GlOSS to Vector-Space Databases and Broker Hierarchies. In: Proceedings of the 21th International Conference on Very Large Data Bases (VLDB ’95), San Francisco. Morgan Kaufmann, Menlo Park, pp 78–89Google Scholar
  15. 15.
    Herrera-Viedma E et al (2004) Incorporating filtering techniques in a fuzzy linguistic multi-agent model for information gathering on the web. Fuzzy Sets Syst 148(1): 61–83MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Huhns MN, Singh MP (1997) Agents on the web: agents are everywhere!. IEEE Internet Comput 1(1): 87CrossRefGoogle Scholar
  17. 17.
    Jung JJ (2007) Ontological framework based on contextual mediation for collaborative information retrieval. Inform Retr 10(1): 85–109CrossRefGoogle Scholar
  18. 18.
    Krichel T (2007) Information retrieval performance measures for a current awareness report composition aid. Inform Process Manage 43(4): 1030–1043CrossRefGoogle Scholar
  19. 19.
    Lee RST, Liu JNK (2004) iJADE Web-Miner: an intelligent agent framework for Internet shopping. IEEE Trans Knowl Data Eng 16(4): 461–473CrossRefMathSciNetGoogle Scholar
  20. 20.
    Minkov E et al (2006) NER systems that suit user’s preferences: adjusting the recall-precision trade-off for entity extraction. In: Moore RC, Bilmes JA, Chu-Carroll J, Sanderson M (eds) Proceedings of the Human Language Technology Conference of the North American Chapter of the Association of Computational LinguisticsGoogle Scholar
  21. 21.
    Nguyen NT (2002) Consensus system for solving conflicts in distributed systems. Inform Sci 147(1–4): 91–122MATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Nguyen NT et al (2006) A consensus-based multi-agent approach for information retrieval in Internet. In: Alexandrov VN, van Albada GD, Sloot PMA, Dongarra J (eds) Proceedings of the 6th International Conference Computational Science (ICCS 2006). Lecture Notes in Computer Science, vol 3993. Springer, Berlin, pp 208–215Google Scholar
  23. 23.
    Oates T et al (1997) Cooperative information-gathering: a distributed problem-solving approach. IEE Proc Softw 144(1): 72–88CrossRefGoogle Scholar
  24. 24.
    Palopoli L et al (2005) A graph-based approach for extracting terminological properties from information sources with heterogeneous formats. Knowl Inform Syst 8(4): 462–497CrossRefGoogle Scholar
  25. 25.
    Sakai T (2007) On the reliability of information retrieval metrics based on graded relevance. Inform Process Manage 43(2): 531–548CrossRefGoogle Scholar
  26. 26.
    Sandelin R (2007) Basic of Consensus.
  27. 27.
    Simeoni F et al (2007a) A grid-based infrastructure for distributed retrieval. In: Kovács L, Fuhr N, Meghini C (eds) Proceedings of the 11th European Conference on Digital Libraries (ECDL 2007), Budapest, Hungary, September 16–21. Lecture Notes in Computer Science, vol 4675. Springer, Berlin, pp 161–173Google Scholar
  28. 28.
    Simeoni F et al (2007b) The DILIGENT framework for distributed information retrieval. In: Kraaij W, de Vries AP, Clarke CLA, Fuhr N, Kando N (eds) Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), Amsterdam, July 23–27, pp 781–782Google Scholar
  29. 29.
    Voorhees EM et al (1994) The collection fusion problem. In: TRECGoogle Scholar
  30. 30.
    Wu S, Crestani F (2003) Distributed information retrieval: a multi-objective resource selection approach. Int J Uncertain Fuzziness Knowl Based Syst 11(1): 83–99MATHCrossRefGoogle Scholar
  31. 31.
    Zudilova-Seinstra EV (2007) On the role of individual human abilities in the design of adaptive user interfaces for scientific problem solving environments. Knowl Inform Syst 13(2): 243–270CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2008

Authors and Affiliations

  1. 1.Department of Computer EngineeringYeungnam UniversityGyeonsanSouth Korea

Personalised recommendations