Abstract
This study analyzes causes for successful and unsuccessful search performance in an academic search test collection. Based on a component-level evaluation setting presented in a parallel paper, analyses of the recall base and the semantic heterogeneity of queries and documents were used for performance prediction. The study finds that neither the recall base, query specificity nor ambiguity can predict the overall search performance or identify badly performing queries. A detailed query analysis finds patterns for negative effects (e.g. non-content-bearing terms in topics), but none are overly significant.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
https://www.lemurproject.org/lemur.php, last accessed: 04-30-2017.
- 2.
http://trec.nist.gov/trec_eval, last accessed: 04-30-2017.
- 3.
T: Generational differences on the Internet; D: Find documents describing the significance of the Internet for communication and the differences in how people of different ages utilize it.; N: Relevant documents describe the significance and function of the Internet for communication in different areas of society (private life, work life, politics, etc.) and the generation gap in using various means of communication.
- 4.
Note that the number of data points becomes smaller with every fixed factor such as DV + CV documents. Thus, the amount of data per query for testing SCS is smaller than in other calculations in this article. Still, the data is comparable to the amount of data used by [9].
References
Buckley, C.: Why current IR engines fail. Inf. Retr. 12(6), 652–665 (2009)
Carmel, D., Yom-Tov, E.: Estimating the query difficulty for information retrieval. Synth. Lect. Inf. Concepts Retr. Serv. 2(1), 1–89 (2010)
Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: SIGIR 2002, pp. 299–306. ACM (2002)
Dietz, F., Petras, V.: A component-level analysis of an academic search test collection. Part I: system and collection configurations. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 16–28. Springer, Cham (2017). doi:10.1007/978-3-319-65813-1_2
Ferro, N., Harman, D.: CLEF 2009: Grid@CLEF pilot track overview. In: Peters, C., di Nunzio, G.M., Kurimo, M., Mandl, T., Mostefa, D., Peñas, A., Roda, G. (eds.) CLEF 2009. LNCS, vol. 6241, pp. 552–565. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15754-7_68
Hanbury, A., Müller, H.: Automated component–level evaluation: present and future. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A. (eds.) CLEF 2010. LNCS, vol. 6360, pp. 124–135. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15998-5_14
Hauff, C.: Predicting the Effectiveness of Queries and Retrieval Systems. Ph.D. thesis, University of Twente, Netherlands (2010)
Hauff, C., Azzopardi, L., Hiemstra, D.: The combination and evaluation of query performance prediction methods. In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 301–312. Springer, Heidelberg (2009). doi:10.1007/978-3-642-00958-7_28
He, B., Ounis, I.: Query performance prediction. Inf. Syst. 31(7), 585–594 (2006)
Katz, G., Shtock, A., Kurland, O., Shapira, B., Rokach, L.: Wikipedia-based query performance prediction. In: SIGIR 2014, pp. 1235–1238. ACM (2014)
Kluck, M., Stempfhuber, M.: Domain-specific track CLEF 2005: overview of results and approaches, remarks on the assessment analysis. In: Peters, C., et al. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 212–221. Springer, Heidelberg (2006). doi:10.1007/11878773_25
Li, X., Schijvenaars, B.J., de Rijke, M.: Investigating queries and search failures in academic search. Inf. Process. Manag. 53(3), 666–683 (2017). http://doi.org/10.1016/j.ipm.2017.01.005
Lykke, M., Larsen, B., Lund, H., Ingwersen, P.: Developing a test collection for the evaluation of integrated search. In: Gurrin, C., He, Y., Kazai, G., Kruschwitz, U., Little, S., Roelleke, T., Rüger, S., van Rijsbergen, K. (eds.) ECIR 2010. LNCS, vol. 5993, pp. 627–630. Springer, Heidelberg (2010). doi:10.1007/978-3-642-12275-0_63
Mothe, J., Tanguy, L.: Linguistic features to predict query difficulty. In: Predicting Query Difficulty Workshop. SIGIR 2005, pp. 7–10 (2005)
Robertson, S.E.: Understanding inverse document frequency: on theoretical arguments for IDF. J. Doc. 60, 503–520 (2004)
Robertson, S.E., Sparck Jones, K.: Relevance weighting of search terms. In: Willett, P. (ed.) Document Retrieval Systems, pp. 143–160. Taylor Graham (1988)
Scholer, F., Williams, H.E., Turpin, A.: Query association surrogates for web search: research articles. J. Am. Soc. Inf. Sci. Technol. 55(7), 637–650 (2004)
Voorhees, E.M.: Overview of the TREC 2003 robust retrieval track. In: TREC, pp. 69–77 (2003)
Voorhees, E.M.: The TREC robust retrieval track. ACM SIGIR Forum 39, 11–20 (2005)
Yom-Tov, E., Fine, S., Carmel, D., Darlow, A.: Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In: SIGIR 2005, pp. 512–519. ACM (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Dietz, F., Petras, V. (2017). A Component-Level Analysis of an Academic Search Test Collection.. In: Jones, G., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2017. Lecture Notes in Computer Science(), vol 10456. Springer, Cham. https://doi.org/10.1007/978-3-319-65813-1_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-65813-1_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-65812-4
Online ISBN: 978-3-319-65813-1
eBook Packages: Computer ScienceComputer Science (R0)