Abstract
Selecting suitable topics in order to assess system effectiveness is a crucial part of any benchmark, particularly those for retrieval systems. This includes establishing a range of example search requests (or topics) in order to test various aspects of the retrieval systems under evaluation. In order to assist with selecting topics, we present a measure of topic difficulty for cross-language image retrieval. This measure has enabled us to ground the topic generation process within a methodical and reliable framework for ImageCLEF 2005. This document describes such a measure for topic difficulty, providing concrete examples for every aspect of topic complexity and an analysis of topics used in the ImageCLEF 2003, 2004 and 2005 ad-hoc task.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Clough, P., Sanderson, M., Reid, N.: The Eurovision St Andrews Photographic Collection (ESTA), Image CLEF Report, University of Sheffield, UK (February 2003)
Grubinger, M., Leung, C.: A Benchmark for Performance Calibration in Visual Informa-tion Search. In: Proceedings of 2003 Conference of Visual Information Systems, Miami, Florida, USA, September 24-26, pp. 414–419 (2003)
Leung, C., Ip, H.: Benchmarking for Content-Based Visual Information Search. In: Laurini, R. (ed.) VISUAL 2000. LNCS, vol. 1929, pp. 442–456. Springer, Heidelberg (2000)
Müller, H., Müller, W., Squire, D., Marchand-Millet, S., Pun, T.: Performance Evaluation in Content Based Image Retrieval: Overview and Proposals. Pattern Recognition Letters 22(5), 563–601 (2001)
Clough, P., Sanderson, M.: The CLEF 2003 Cross Language Image Retrieval Track. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 581–593. Springer, Heidelberg (2004)
Clough, P., Müller, H., Sanderson, M.: Overview of the CLEF cross-language image re-trieval track (ImageCLEF) 2004. In: Peters, C., Clough, P.D., Jones, G.J.F., Gonzalo, J., Kluck, M., Magnini, B. (eds.) Multilingual Information Access for Text, Speech and Im-ages: Result of the fifth CLEF evaluation campaign. LNCS, Springer, England (2005)
Armitage, L., Enser, P.: Analysis of User Need in Image Archives. Journal of Information Science 23(4), 287–299 (1997)
Bagga, A., Biermann, A.: Analysing the complexity of a domain with respect to an infor-mation extraction task. In: Proceedings of the tenth International Conference on Research on Computational Linguistics (ROCLING X), August 1997, pp. 175–194 (1997)
Niyogi, P.: The Informational Complexity of Learning from Examples. PhD Thesis, MIT (1996)
Barton, E., Berwick, R., Ristad, E.: Computational Complexity and Natural Language. The MIT Press, Cambridge (1987)
Ristad, E.: The Language Complexity Games. MIT Press, Cambridge (1993)
Flank, S.: Sentences vs. Phrases: Syntactic Complexity in Multimedia Information Retrieval. In: NAACL-ANLP 2000 Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems (2000)
Pollard, S., Biermann, A.: A Measure of Semantic Complexity for Natural Language Systems. In: NAACL-ANLP 2000 Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems (2000)
Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting Query Performance. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Develop-ment in Information Retrieval 2002, pp. 299–306. ACM Press, New York (2002)
Amati, G., Carpineto, C., Romano, G.: Query Difficulty, Robustness, and Selective Application of Query Expansion. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 127–137. Springer, Heidelberg (2004)
Yom-Tov, E., Fine, S., Carmel, D., Darlow, A.: Learning to Estimate Query Difficulty. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2005, ACM Press, New York (2005)
Clough, P., Müller, H., Deselaers, T., Grubinger, M., Lehmann, T., Jensen, J., Hersh, W.: The CLEF 2005 Cross–Language Image Retrieval Track. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 535–557. Springer, Heidelberg (2006)
Clough, P.D., Sanderson, M.: Assessing Translation Quality for Cross Language Image Retrieval. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 594–610. Springer, Heidelberg (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Grubinger, M., Leung, C., Clough, P. (2006). Linguistic Estimation of Topic Difficulty in Cross-Language Image Retrieval. In: Peters, C., et al. Accessing Multilingual Information Repositories. CLEF 2005. Lecture Notes in Computer Science, vol 4022. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11878773_61
Download citation
DOI: https://doi.org/10.1007/11878773_61
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-45697-1
Online ISBN: 978-3-540-45700-8
eBook Packages: Computer ScienceComputer Science (R0)