Abstract
User evaluations of search engines are expensive and not easy to replicate. The problem is even more pronounced when assessing adaptive search systems, for example system-generated query modification suggestions that can be derived from past user interactions with a search engine. Automatically predicting the performance of different modification suggestion models before getting the users involved is therefore highly desirable. AutoEval is an evaluation methodology that assesses the quality of query modifications generated by a model using the query logs of past user interactions with the system. We present experimental results of applying this methodology to different adaptive algorithms which suggest that the predicted quality of different algorithms is in line with user assessments. This makes AutoEval a suitable evaluation framework for adaptive interactive search engines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Boldi, P., Bonchi, F., Castillo, C., Donato, D., Vigna, S.: Query suggestions using query-flow graphs. In: Proceedings of WSCD 2009, Barcelona, pp. 56–63 (2009)
Craswell, N., Szummer, M.: Random Walks on the Click Graph. In: Proceedings of SIGIR 2007, Amsterdam, pp. 239–246 (2007)
Dignum, S., Kruschwitz, U., Fasli, M., Kim, Y., Song, D., Cervino, U., De Roeck, A.: Incorporating Seasonality into Search Suggestions Derived from Intranet Query Logs. In: Proceedings of WI 2010, Toronto, pp. 425–430 (2010)
Fonseca, B.M., Golgher, P.B., de Moura, E.S., Ziviani, N.: Using association rules to discover search engines related queries. In: Proceedings of the First Latin American Web Congress, Santiago, pp. 66–71 (2003)
Joachims, T.: Evaluating retrieval performance using clickthrough data. In: Franke, J., Nakhaeizadeh, G., Renz, I. (eds.) Text Mining, pp. 79–96. Springer, Heidelberg (2003)
Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Proceedings of SIGIR 2005, Salvador, pp. 154–161 (2005)
Kruschwitz, U.: Intelligent Document Retrieval: Exploiting Markup Structure. The Information Retrieval Series, vol. 17. Springer, Heidelberg (2005)
Nanas, N., Kruschwitz, U., Albakour, M.-D., Fasli, M., Song, D., Kim, Y., Cervino, U., De Roeck, A.: A Methodology for Simulated Experiments in Interactive Search. In: Proceedings of the SIGIR 2010 SimInt Workshop, Geneva, pp. 23–24 (2010)
Nanas, N., Roeck, A.: Autopoiesis, the immune system, and adaptive information filtering. Natural Computing: an International Journal 8(2), 387–427 (2009)
Sanderson, M., Croft, B.: Deriving concept hierarchies from text. In: Proceedings of SIGIR 1999, Berkeley, CA, pp. 206–213 (1999)
Soboroff, I., Nicholas, C., Cahan, P.: Ranking retrieval systems without relevance judgments. In: Proceedings of SIGIR 2001, New Orleans, pp. 66–73 (2001)
Zhang, J., Kamps, J.: A search log-based approach to evaluation. In: Lalmas, M., Jose, J., Rauber, A., Sebastiani, F., Frommholz, I. (eds.) ECDL 2010. LNCS, vol. 6273, pp. 248–260. Springer, Heidelberg (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Albakour, MD. et al. (2011). AutoEval: An Evaluation Methodology for Evaluating Query Suggestions Using Query Logs. In: Clough, P., et al. Advances in Information Retrieval. ECIR 2011. Lecture Notes in Computer Science, vol 6611. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20161-5_60
Download citation
DOI: https://doi.org/10.1007/978-3-642-20161-5_60
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20160-8
Online ISBN: 978-3-642-20161-5
eBook Packages: Computer ScienceComputer Science (R0)