Abstract
Approaches to multimedia search often evolve from existing approaches with strong average precision. However, work on search evaluation shows that average precision does not always capture effectiveness in terms of satisfying user needs because it ignores the diversity of search results. This paper investigates whether search approaches with diverse results have been neglected within the multimedia retrieval research agenda due the fact that they are overshadowed by search approaches with strong average precision. To this end, we compare 361 search approaches applied on the TrecVid benchmarks between 2005 and 2007. We motivate two criteria based on measure correlation and statistical equivalence to estimate whether search approaches with diverse results have been neglected. We show that hypothesized effect indeed occurs in the above examined collections. As a consequence, the research community would benefit from reconsidering existing approaches in the light of diversity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Agrawal, R., Gollapudi, S., Halverson, A., Ieong, S.: Diversifying search results. In: WSDM 2009: Proceedings of the Second ACM International Conference on Web Search and Data Mining, pp. 5–14. ACM, New York (2009)
Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, pp. 659–666. ACM, New York (2008)
Clarke, C.L.A., Craswell, N., Soboroff, I., Voorhees, E.: Overview of the TREC 2011 Web Track. In: Twentieth Text Retrieval Conference (TREC 2011) The Proceedings (2011)
Fowlkes, E.B., Mallows, C.L.: A method for comparing two hierarchical clusterings. Journal of the American Statistical Association 78(383), 553–569 (1983)
Goffman, W.: A searching procedure for information retrieval. Information Storage and Retrieval 2(2), 73–78 (1964)
Paramita, M.L., Sanderson, M., Clough, P.: Developing a test collection to support diversity analysis. In: Proceedings of Redundancy, Diversity, and Interdependence Document Relevance Workshop held at ACM SIGIR, pp. 39–45 (2009)
Sanderson, M., Paramita, M.L., Clough, P., Kanoulas, E.: Do user preferences and evaluation measures line up? In: SIGIR 2010: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 555–562. ACM, New York (2010) ISBN 978-1-4503-0153-4
Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and trecvid. In: MIR 2006: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321–330. ACM Press, New York (2006)
van Leuken, R.H., Garcia, L., Olivares, X., van Zwol, R.: Visual diversification of image search results. In: WWW 2009: Proceedings of the 18th International Conference on World Wide Web, pp. 341–350. ACM, New York (2009)
Xu, Y., Yin, H.: Novelty and topicality in interactive information retrieval. Journal of the American Society for Information Science and Technology 59(2), 201–215 (2008)
Zhai, C.X., Cohen, W.W., Lafferty, J.: Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: SIGIR 2003: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, pp. 10–17. ACM, New York (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Aly, R., Trieschnigg, D., McGuinness, K., O’Connor, N.E., de Jong, F. (2014). Average Precision: Good Guide or False Friend to Multimedia Search Effectiveness?. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds) MultiMedia Modeling. MMM 2014. Lecture Notes in Computer Science, vol 8326. Springer, Cham. https://doi.org/10.1007/978-3-319-04117-9_22
Download citation
DOI: https://doi.org/10.1007/978-3-319-04117-9_22
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-04116-2
Online ISBN: 978-3-319-04117-9
eBook Packages: Computer ScienceComputer Science (R0)