Modeling Expected Utility of Multi-session Information Distillation

  • Yiming Yang
  • Abhimanyu Lad
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5766)

Abstract

An open challenge in information distillation is the evaluation and optimization of the utility of ranked lists with respect to flexible user interactions over multiple sessions. Utility depends on both the relevance and novelty of documents, and the novelty in turn depends on the user interaction history. However, user behavior is non-deterministic. We propose a new probabilistic framework for stochastic modeling of user behavior when browsing multi-session ranked lists, and a novel approximation method for efficient computation of the expected utility over numerous user-interaction patterns. Using this framework, we present the first utility-based evaluation over multi-session search scenarios defined on the TDT4 corpus of news stories, using a state-of-the-art information distillation system. We demonstrate that the distillation system obtains a 56.6% utility enhancement by combining multi-session adaptive filtering with novelty detection and utility-based optimization of system parameters for optimal ranked list lengths.

Keywords

Multi-session distillation utility evaluation based both on novelty and relevance stochastic modeling of user browsing behavior 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Allan, J., Wade, C., Bolivar, A.: Retrieval and novelty detection at the sentence level. In: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pp. 314–321 (2003)Google Scholar
  2. 2.
    Babko-Malaya, O.: Annotation of Nuggets and Relevance in GALE Distillation Evaluation. In: Proceedings LREC 2008 (2008)Google Scholar
  3. 3.
    Buckley, C., Voorhees, E.M.: Retrieval system evaluation. TREC: Experiment and Evaluation in Information Retrieval, 53–75 (2005)Google Scholar
  4. 4.
    Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 659–666 (2008)Google Scholar
  5. 5.
    He, D., Brusilovsky, P., Ahn, J., Grady, J., Farzan, R., Peng, Y., Yang, Y., Rogati, M.: An evaluation of adaptive filtering in the context of realistic task-based information exploration. In: Information Processing and Management (2008)Google Scholar
  6. 6.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) (4), 422–446 (2002)CrossRefGoogle Scholar
  7. 7.
    Järvelin, K., Price, S., Delcambre, L., Nielsen, M.L.: Discounted cumulated gain based evaluation of multiple-query IR sessions. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 4–15. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  8. 8.
    Moffat, A., Zobel, J.: Rank-Biased Precision for Measurement of Retrieval Effectiveness. ACM Transactions on Information Systems, 1–27 (2008)Google Scholar
  9. 9.
    Robertson, S.: A new interpretation of average precision. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 689–690 (2008)Google Scholar
  10. 10.
    Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: A language model-based serach engine for complex queries. In: Proceedings of the International Conference on Intelligence Analysis (2004)Google Scholar
  11. 11.
    White, J.V., Hunter, D., Goldstein, J.D.: Statistical Evaluation of Information Distillation Systems. In: Proceedings of the Sixth International Language Resources and Evaluation LREC, vol. 8Google Scholar
  12. 12.
    Yang, Y., Lad, A., Lao, N., Harpale, A., Kisiel, B., Rogati, M.: Utility-based information distillation over temporally sequenced documents. In: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 31–38 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Yiming Yang
    • 1
  • Abhimanyu Lad
    • 1
  1. 1.Language Technologies InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations