A Test Collection for the Evaluation of Content-Based Image Retrieval Algorithms—A User and Task-Based Approach
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Content-based image retrieval (CBIR) algorithms have been seen as a promising access method for digital photograph collections. Unfortunately, we have very little evidence of the usefulness of these algorithms in real user needs and contexts. In this paper, we introduce a test collection for the evaluation of CBIR algorithms. In the test collection, the performance testing is based on photograph similarity perceived by end-users in the context of realistic illustration tasks and environment. The building process and the characteristics of the resulting test collection are outlined, including a typology of similarity criteria expressed by the subjects judging the similarity of photographs. A small-scale study on the consistency of similarity assessments is presented. A case evaluation of two CBIR algorithms is reported. The results show clear correlation between the subjects' similarity assessments and the functioning of feature parameters of the tested algorithms.
Armitage L and Enser P (1997) Analysis of user needs in image archives. Journal of Information Science, 23: 287-299.
Belongie S, Carson C, Greenspan H and Malik J (1998) Color-and texture-based image segmentation using the expectation-maximization algorithm and its application to content-based image retrieval. In: Sixth International Conference on Computer Vision, pp. 675-682. Available URL: http://www.cs.berkeley.edu/projects/vision/ publications.html.
Das M, Manmatha R, Greenspan H and Malik J (1999) Indexing flowers by color names using domain knowledgedriven segmentation. IEEE Intelligent Systems, 14(5): 24-343.
Del Bimbo A (1999) Visual Information Retrieval. Morgan Kaufmann, San Francisco.
Eakins J (1996) Automatic image content retrieval-are we getting anywhere? In: Proceedings of the Third International Conference on Electronic Library and Visual Information Research, De Montfort University, Milton Keynes, May 1996, pp. 123-135.
Enser P (1995) Pictorial information retrieval. Journal of Documentation, 51: 126-170.
Forsyth DA and Fleck MM (1997) Body plans. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition Puerto Rico, June 1997.
Forsyth DA, Malik J, Fleck MM, Greenspan H, Leung T, Belongie S, Carson C and Bregler C (1996) Finding pictures of objects in large collections of images. In: Proceedings of the International Workshop on Object Recognition, Cambridge, April 1996. Available URL: http://www.cs.berkeley.edu/~daf/.
Gong Y (1998) Intelligent Image Databases: Towards Advanced Image Retrieval. Kluwer Academic Publishers, Boston.
Gudivada V and Raghavan V (1997) Modeling and retrieving images by content. Information Processing & Management, 33(4): 427-452.
Gupta A and Jain R (1997) Visual information retrieval. Communications of the ACM, 40: 71-79.
Harman D (1993) The First Text Retrieval Conference (TREC-1). National Institute of Standards and Technology, Gaithersburg. (NIST Special Publication 500-207).
Iivonen M (1995) Consistency in the selection of search concepts and search terms. Information Processing & Management, 31(2): 173-190.
Keister L (1994) User Types and queries: Impact on image access systems. In: Fidel R, Hahn T, Rasmussen E and Smith P, Eds., Challenges in Indexing Electronic Text and Images. Learned Information, Medford, New Jersey, pp. 7-22.
Kekäläinen J (1999) The effects of query complexity, expansion and structure on retrieval performance in propabilistic text retrieval. Doctoral Thesis, University of Tampere, Tampere. Acta Electronica Universitatis Tamperensis, ISBN 951-44-4596-1.
Markkula M and Sormunen E (1998) Searching for photos-journalists' practices in pictorial IR. In: Eakins J, Harper D and Jose J, Eds., The Challenge of Image Retrieval. Electronic Workshops in Computing (eWIC), 1998. URL: http://www.ewic.org.uk/ewic/workshop/view.cfm/CIR-98.
Markkula M and Sormunen E (2000) End-user searching challenges indexing practices in the digital photograph archive. Information Retrieval, 1(4): 259-285.
Panofsky E (1970) Meaning in the Visual Arts. Penguin, London.
Picard R, Minka T and Szummer M (1996) Modeling user subjectivity in image libraries. M.I.T. Media Laboratory, Perceptual Computing Section Technical Report No. 382, 1996. (also IEEE Int. Conf. On Image Proc., Lausanne, Sept. 1996). Available at URL: http://picard.www.media.mit.edu/cgi-bin/tr pagemaker.
Rasmussen E (1997) Indexing images. In: Williams M, Ed., Annual Reviewof Information Science and Technology 32. Information Today, Medford, New Jersey, pp. 169-196.
Saracevic T (1984) Measuring the degree of agreement between searchers. In: Flood B, Witiak J and Hogan H, Eds., ASIS 84: Proceedings of the American Society for Information Science 47th Annual Meeting, Vol. 28. White Plans, Knowledge Industry Publications, NY, pp. 227-230.
Shatford S (1986) Analyzing the subject of a picture: A theoretical approach. Cataloguing and Classification Quarterly, 6: 39-62.
Sormunen E (2000) A method for measuring wide range performance of boolean queries in full-text databases. Doctoral Thesis, University of Tampere, Tampere. Acta Electronica Universitatis Tamperensis, ISBN: 951-44-4732-8, 231 p. URL: http://acta.uta.fi/pdf/951-44-4732-8.pdf.
Sormunen E, Markkula M and Järvelin K (1999) The perceived similarity of photos-seeking a solid basis for the evaluation of content-based retrieval algorithms. In: Draper S. et al., Eds., Mira 99: Evaluating Interactive Information Retrieval. Glasgow, UK, 4-16 April, 1999. Electronic Workshops in Computing. URL: http://www.ewic.org.uk/ewic/workshop/fetch.cfm/MIRA-99/.
Swain MJ and Ballard H (1991) Color indexing. International Journal of Computer Vision, 7(1): 11-32.
Tague-Sutcliffe J (1992) The pragmatics of information retrieval experimentation, revisited. Information Processing & Management, 28(4): 467-490.
Tico M, Haverinen T and Kuosmanen P (1999a) An unsupervised method of rough color image segmentation. In: Proceedings of the 33th Asilomar Conference on Signals, Systems and Computers, Vol. 1. Pacific Grove, California, Oct. 24-27, 1999, pp. 58-62.
Tico M, Haverinen T and Kuosmanen P (2000) A method of color histogram creation for image retrieval. In: Proceedings of the Nordic Signal Processing Symposium (NORSIG'2000), Kolmarden, Sweden, June 13-15, 2000, pp. 157-160.
Tico M and Kuosmanen P (1999b) An efficient sparse data filtering method for image histogram comparison. In: Proceedings of the 11th Scandinavian Conference on Image Analysis (SCIA'99), Kangerlussuaq, Greenland, June 1999, pp. 715-722.
Voorhees E and Harman D (1997) The Fifth Text REtrieval Conference (TREC-5). National Institute of Standards and Technology, Gaithersburg. (NIST Special Publication 500-238).
- A Test Collection for the Evaluation of Content-Based Image Retrieval Algorithms—A User and Task-Based Approach
Volume 4, Issue 3-4 , pp 275-293
- Cover Date
- Print ISSN
- Online ISSN
- Kluwer Academic Publishers
- Additional Links
- content-based image retrieval
- test collections
- user needs
- Industry Sectors
- Author Affiliations
- 1. Department of Information Studies, University of Tampere, Finland
- 2. Digital Media Institute, Tampere, University of Technology, Finland
- 3. Department of Information Services, University of Tampere, Finland