AutoEval: An Evaluation Methodology for Evaluating Query Suggestions Using Query Logs

  • M-Dyaa Albakour
  • Udo Kruschwitz
  • Nikolaos Nanas
  • Yunhyong Kim
  • Dawei Song
  • Maria Fasli
  • Anne De Roeck
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6611)

Abstract

User evaluations of search engines are expensive and not easy to replicate. The problem is even more pronounced when assessing adaptive search systems, for example system-generated query modification suggestions that can be derived from past user interactions with a search engine. Automatically predicting the performance of different modification suggestion models before getting the users involved is therefore highly desirable. AutoEval is an evaluation methodology that assesses the quality of query modifications generated by a model using the query logs of past user interactions with the system. We present experimental results of applying this methodology to different adaptive algorithms which suggest that the predicted quality of different algorithms is in line with user assessments. This makes AutoEval a suitable evaluation framework for adaptive interactive search engines.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Boldi, P., Bonchi, F., Castillo, C., Donato, D., Vigna, S.: Query suggestions using query-flow graphs. In: Proceedings of WSCD 2009, Barcelona, pp. 56–63 (2009)Google Scholar
  2. 2.
    Craswell, N., Szummer, M.: Random Walks on the Click Graph. In: Proceedings of SIGIR 2007, Amsterdam, pp. 239–246 (2007)Google Scholar
  3. 3.
    Dignum, S., Kruschwitz, U., Fasli, M., Kim, Y., Song, D., Cervino, U., De Roeck, A.: Incorporating Seasonality into Search Suggestions Derived from Intranet Query Logs. In: Proceedings of WI 2010, Toronto, pp. 425–430 (2010)Google Scholar
  4. 4.
    Fonseca, B.M., Golgher, P.B., de Moura, E.S., Ziviani, N.: Using association rules to discover search engines related queries. In: Proceedings of the First Latin American Web Congress, Santiago, pp. 66–71 (2003)Google Scholar
  5. 5.
    Joachims, T.: Evaluating retrieval performance using clickthrough data. In: Franke, J., Nakhaeizadeh, G., Renz, I. (eds.) Text Mining, pp. 79–96. Springer, Heidelberg (2003)Google Scholar
  6. 6.
    Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Proceedings of SIGIR 2005, Salvador, pp. 154–161 (2005)Google Scholar
  7. 7.
    Kruschwitz, U.: Intelligent Document Retrieval: Exploiting Markup Structure. The Information Retrieval Series, vol. 17. Springer, Heidelberg (2005)MATHGoogle Scholar
  8. 8.
    Nanas, N., Kruschwitz, U., Albakour, M.-D., Fasli, M., Song, D., Kim, Y., Cervino, U., De Roeck, A.: A Methodology for Simulated Experiments in Interactive Search. In: Proceedings of the SIGIR 2010 SimInt Workshop, Geneva, pp. 23–24 (2010)Google Scholar
  9. 9.
    Nanas, N., Roeck, A.: Autopoiesis, the immune system, and adaptive information filtering. Natural Computing: an International Journal 8(2), 387–427 (2009)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Sanderson, M., Croft, B.: Deriving concept hierarchies from text. In: Proceedings of SIGIR 1999, Berkeley, CA, pp. 206–213 (1999)Google Scholar
  11. 11.
    Soboroff, I., Nicholas, C., Cahan, P.: Ranking retrieval systems without relevance judgments. In: Proceedings of SIGIR 2001, New Orleans, pp. 66–73 (2001)Google Scholar
  12. 12.
    Zhang, J., Kamps, J.: A search log-based approach to evaluation. In: Lalmas, M., Jose, J., Rauber, A., Sebastiani, F., Frommholz, I. (eds.) ECDL 2010. LNCS, vol. 6273, pp. 248–260. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • M-Dyaa Albakour
    • 1
  • Udo Kruschwitz
    • 1
  • Nikolaos Nanas
    • 2
  • Yunhyong Kim
    • 3
  • Dawei Song
    • 3
  • Maria Fasli
    • 1
  • Anne De Roeck
    • 4
  1. 1.University of EssexColchesterUK
  2. 2.Centre for Research and TechnologyThessalyGreece
  3. 3.Robert Gordon UniversityAberdeenUK
  4. 4.Open UniversityMilton KeynesUK

Personalised recommendations