Large Scale Evaluations of Multimedia Information Retrieval: The TRECVid Experience
Information Retrieval is a supporting technique which underpins a broad range of content-based applications including retrieval, filtering, summarisation, browsing, classification, clustering, automatic linking, and others. Multimedia information retrieval (MMIR) represents those applications when applied to multimedia information such as image, video, music, etc. In this presentation and extended abstract we are primarily concerned with MMIR as applied to information in digital video format. We begin with a brief overview of large scale evaluations of IR tasks in areas such as text, image and music, just to illustrate that this phenomenon is not just restricted to MMIR on video. The main contribution, however, is a set of pointers and a summarisation of the work done as part of TRECVid, the annual benchmarking exercise for video retrieval tasks.
KeywordsInformation Retrieval Search Task Automatic Speech Recognition Video Retrieval Topic Detection
Unable to display preview. Download preview PDF.
- 1.The TREC Website. Last visited (April 2005), Available at http://trec.nist.gov
- 2.Kazai, G., Lalmas, M., de Vries, A.P.: The Overlap Problem in Content-Oriented XML Retrieval Evaluation. In: Proceedings of the 27th annual international conference on Research and development in information retrieval, SIGIR 2004, Sheffield, United Kingdom, pp. 72–79. ACM Press, New York (2004)CrossRefGoogle Scholar
- 3.MIREX 2005: 2nd Annual Music Information Retrieval Evaluation eXchange. Last visited (April 2005), Available at http://www.music-ir.org/evaluation/MIREX/
- 4.The Benchathlon Network: Home of CBIR Benchmarking. Last visited (April 2005), Available at http://www.benchathlon.net/
- 5.TREC Video Retrieval Evaluation Online Proceedings. Last visited (April 2005), Available at http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html
- 6.TREC Video Retrieval Evaluation Bibliography. Last visited (April 2005), Available at http://www-nlpir.nist.gov/projects/t01v/trecvid.bibliography.txt
- 7.Smeaton, A.F., Kraaij, W., Over, P.: The TREC Video Retrieval Evaluation (TRECVID): A Case Study and Status Report. In: Proceedings of RIAO 2004 - Coupling Approaches, Coupling Media and Coupling Languages for Information Retrieval, Avignon, France (April 26-28, 2004)Google Scholar
- 8.Kraaij, W., Smeaton, A.F., Over, P., Arlandis, J.: TRECVID 2004 - An Overview. In: Proceedings of TRECVid 2004, National Institute for Standards and Technology. Last visited (April 2005), http://www-nlpir.nist.gov/projects/tvpubs/tvpapers04/tv4overview.pdf
- 9.Fiscus, J.G., Doddington, G.R.: Topic Detection and Tracking Evaluation Overview. In: Topic Detection and Tracking: Event-Based Information Organization, pp. 17–31. Kluwer Academic Publishers, Norwell (2002)Google Scholar