An Effectiveness Measure for Ambiguous and Underspecified Queries

  • Charles L. A. Clarke
  • Maheedhar Kolla
  • Olga Vechtomova
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5766)


Building upon simple models of user needs and behavior, we propose a new measure of novelty and diversity for information retrieval evaluation. We combine ideas from three recently proposed effectiveness measures in an attempt to achieve a balance between the complexity of genuine users needs and the simplicity required for feasible evaluation.


Result List Binary Property Binary Relevance Feasible Evaluation Genuine User 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems 27, 1–27 (2008)CrossRefGoogle Scholar
  2. 2.
    Clarke, C.L., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkann, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 659–666 (2008)Google Scholar
  3. 3.
    Agrawal, R., Gollapudi, S., Halverson, A., Ieong, S.: Diversifying search results. In: Proceedings of the Second ACM International Conference on Web Search and Data Mining, pp. 5–14 (2009)Google Scholar
  4. 4.
    Spärck Jones, K., Robertson, S.E., Sanderson, M.: Ambiguous requests: Implications for retrieval tests. SIGIR Forum 41(2), 8–17 (2007)CrossRefGoogle Scholar
  5. 5.
    Lin, J., Demner-Fushman, D.: Will pyramids built of nuggets topple over? In: Proceedings of the Human Language Technology Conference, 383–390 (2006)Google Scholar
  6. 6.
    Nenkova, A., Passonneau, R., McKeown, K.: The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing 4(2) (2007)Google Scholar
  7. 7.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20(4), 422–446 (2002)CrossRefGoogle Scholar
  8. 8.
    Chen, H., Karger, D.R.: Less is more: Probabilistic models for retrieving fewer relevant documents. In: 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 429–436 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Charles L. A. Clarke
    • 1
  • Maheedhar Kolla
    • 1
  • Olga Vechtomova
    • 1
  1. 1.University of WaterlooCanada

Personalised recommendations