Multimedia Tools and Applications

, Volume 35, Issue 3, pp 241–258 | Cite as

Evaluation of content-based image descriptors by statistical methods



Evaluation of visual information retrieval systems is usually performed by executing test queries and computing recall- and precision-like measures based on predefined media collections and ground truth information. This process is complex and time consuming. For the evaluation of feature transformations (transformation of visual media objects to feature vectors) it would be desirable to have simpler methods available as well. In this paper we introduce a supplementary evaluation procedure for features that is founded on statistical data analysis. A second novelty is that we make use of the existing visual MPEG-7 descriptors to judge the characteristics of feature transformations. The proposed procedure is divided into four steps: (1) feature extraction, (2) merging with MPEG-7 data and normalisation, (3) statistical data analysis and (4) visualisation and interpretation. Three types of statistical methods are used for evaluation: (1) univariate description (moments, etc.), (2) identification of similarities between feature elements (e.g. cluster analysis) and (3) identification of dependencies between variables (e.g. factor analysis). Statistical analysis provides beneficial insights into the structure of features that can be exploited for feature redesign. Application and advantages of the proposed approach are shown in a number of toy examples.


Evaluation Statistical data analysis Feature design Visual information retrieval MPEG-7 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bober M (2001) MPEG-7 visual shape descriptors. Special issue on MPEG-7. IEEE Trans Circuits Syst Video Technol 11(6):716–719CrossRefGoogle Scholar
  2. 2.
    Breiteneder C, Eidenberger H (1999) Content-based image retrieval of coats of arms. In: Proc. IEEE International Workshop on Multimedia Signal Processing, Helsingör, pp 91–96Google Scholar
  3. 3.
    Chang SF, Sikora T, Puri A (2001) Overview of the MPEG-7 standard. Special issue on MPEG-7. IEEE Trans Circuits Syst Video Technol 11(6):688–695CrossRefGoogle Scholar
  4. 4.
    Computer Vision Image Library, (last visited 2005–03–29)
  5. 5.
    Cowan G (1998) Statistical data analysis. Oxford University Press, Oxford, UKGoogle Scholar
  6. 6.
    Del Bimbo A (1999) Visual information retrieval. Morgan Kaufmann, San FranciscoGoogle Scholar
  7. 7.
    Edwards JD, Riley KJ, Eakins JP (2003) A technique for mapping irregular sized vectors applied to image collections. In: Proc. SPIE visual communications and image processing conference, vol 5150. Lugano, pp 467–475Google Scholar
  8. 8.
    Eidenberger H (2003) How good are the visual MPEG-7 Features? In: Proc. SPIE visual communications and image processing conference, vol 5150. Lugano, pp 476–488Google Scholar
  9. 9.
    Eidenberger H (2004) A new method for visual descriptor evaluation. In: Proc. SPIE storage and retrieval methods and applications for multimedia, vol 5307. San Jose, pp 145–157Google Scholar
  10. 10.
    Eidenberger H (2004) Statistical analysis of the MPEG-7 image descriptors. In: ACM Multimedia Systems Journal, vol 10, no 2. Springer, Berlin Heidelberg New York, pp 84–97Google Scholar
  11. 11.
    Eidenberger H, Breiteneder C (2003) VizIR—a framework for visual information retrieval. J Vis Lang Comput, Elsevier 14(5):443–469CrossRefGoogle Scholar
  12. 12.
    Fuhr N (2001) Information retrieval methods for multimedia objects. In: Veltkamp RC, Burkhardt H, Kriegel, HP (eds) State-of-the-art in content-based image and video retrieval. Kluwer, Boston, pp 191–212Google Scholar
  13. 13.
    Izquierdo E, Casas JR, Leonardi R, Migliorati P, O’Connor NE, Kompatsiaris I, Strintzis MG (2003) Advanced content-based semantic scene analysis and information retrieval: the SCHEMA project. In: Proc. workshop on image analysis for multimedia interactive services, London, pp 519–528Google Scholar
  14. 14.
    Jain AK, Murty MN, Flynn PJ (1999) Data clustering: a review. ACM Comput Surv 31(3):264–323CrossRefGoogle Scholar
  15. 15.
    Kohonen T (1990) The self-organizing map. Proc IEEE 78(9):1464–1480CrossRefGoogle Scholar
  16. 16.
    Kohonen T, Oja E, Simula O, Visa A, Kangas J (1996) Engineering applications of the self-organizing map. Proc IEEE 84(10):1358–1384CrossRefGoogle Scholar
  17. 17.
    Koikkalainen P, Oja E (1990) Self-organising hierarchical feature maps. In: Proc. neural networks conference. San Diego, pp 279–284Google Scholar
  18. 18.
    Lew MS (ed) (2003) Principles of visual information retrieval. Springer, Berlin Heidelberg New YorkGoogle Scholar
  19. 19.
    Loehlin JC (2001) Latent variable models: an introduction to factor. Path and structural analysis, 3rd edn. Erlbaum, Mahwah, NJGoogle Scholar
  20. 20.
    Manjunath BS, Ohm JR, Vasudevan VV, Yamada A (2001) Color and texture descriptors. Special issue on MPEG-7. IEEE Trans Circuits Syst Video Technol 11(6):703–715CrossRefGoogle Scholar
  21. 21.
    Manjunath BS, Salembier P, Sikora T (2002) Introduction to MPEG-7. Wiley, San FranciscoGoogle Scholar
  22. 22.
    Marques O, Furht B (2002) Content-based image and video retrieval. Kluwer, BostonMATHGoogle Scholar
  23. 23.
    Müller H, Müller W, Squire DMcG, Marchand-Maillet S, Pun T (2001) Performance evaluation in content-based image retrieval: overview and proposals. Special issue on image and video indexing. Pattern Recogn Lett 22(5):593–601MATHCrossRefGoogle Scholar
  24. 24.
    Smeaton AF, Over P, Kraaij W (2004) TRECVID: evaluating the effectiveness of information retrieval tasks on digital video. In: Proceedings ACM Multimedia Conference, New York, pp 652–655Google Scholar
  25. 25.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  26. 26.
    University of California Berkeley Corel Dataset Website, (last visited 2005–03–29)
  27. 27.
    Vesanto J, Himberg J, Alhoniemi E, Parhankangas J (1999) Self-organizing map in Matlab: the SOM toolbox. In: Proc. Matlab DSP Conference, Espoo, pp 35–40Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Institute of Software Technology and Interactive SystemsVienna University of TechnologyViennaAustria

Personalised recommendations