Evaluation of Localized Semantics: Data, Methodology, and Experiments

  • Kobus Barnard
  • Quanfu Fan
  • Ranjini Swaminathan
  • Anthony Hoogs
  • Roderic Collins
  • Pascale Rondot
  • John Kaufhold


We present a new data set of 1014 images with manual segmentations and semantic labels for each segment, together with a methodology for using this kind of data for recognition evaluation. The images and segmentations are from the UCB segmentation benchmark database (Martin et al., in International conference on computer vision, vol. II, pp. 416–421, 2001). The database is extended by manually labeling each segment with its most specific semantic concept in WordNet (Miller et al., in Int. J. Lexicogr. 3(4):235–244, 1990). The evaluation methodology establishes protocols for mapping algorithm specific localization (e.g., segmentations) to our data, handling synonyms, scoring matches at different levels of specificity, dealing with vocabularies with sense ambiguity (the usual case), and handling ground truth regions with multiple labels. Given these protocols, we develop two evaluation approaches. The first measures the range of semantics that an algorithm can recognize, and the second measures the frequency that an algorithm recognizes semantics correctly. The data, the image labeling tool, and programs implementing our evaluation strategy are all available on-line (

We apply this infrastructure to evaluate four algorithms which learn to label image regions from weakly labeled data. The algorithms tested include two variants of multiple instance learning (MIL), and two generative multi-modal mixture models. These experiments are on a significantly larger scale than previously reported, especially in the case of MIL methods. More specifically, we used training data sets up to 37,000 images and training vocabularies of up to 650 words.

We found that one of the mixture models performed best on image annotation and the frequency correct measure, and that variants of MIL gave the best semantic range performance. We were able to substantively improve the performance of MIL methods on the other tasks (image annotation and frequency correct region labeling) by providing an appropriate prior.


Image annotation Region labeling Ground truth data Segmetation Image semantics WordNet 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Agarwal, S., Awan, A., & Roth, D. (2002). The UIUC image database for car detection. Available from
  2. Agarwal, S., Awan, A., & Roth, D. (2004). Learning to detect objects in images via a sparse, part-based representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(11), 1475–1490. CrossRefGoogle Scholar
  3. Agirre, E., & Rigau, G. (1995). A proposal for word sense disambiguation using conceptual distance. In 1st international conference on recent advances in natural language processing, Velingrad. Google Scholar
  4. Andrews, S., & Hofmann, T. (2004). Multiple instance learning via disjunctive programming boosting. In Advances in neural information processing systems (NIPS 16). Google Scholar
  5. Andrews, S., Hofmann, T., & Tsochantaridis, I. (2002a). Multiple instance learning with generalized support vector machines. Menlo Park: AAAI. Google Scholar
  6. Andrews, S., Tsochantaridis, I., & Hofmann, T. (2002b). Advances in neural information processing systems: Vol. 15. Support vector machines for multiple-instance learning, Vancouver, BC. Google Scholar
  7. Barnard, K., & Forsyth, D. (2001). Learning the semantics of words and pictures. In International conference on computer vision (Vol. II, pp. 408–415). Google Scholar
  8. Barnard, K., Duygulu, P., & Forsyth, D. (2001). Clustering art. In IEEE conference on computer vision and pattern Recognition (Vol. II, pp. 434–441), Hawaii. Google Scholar
  9. Barnard, K. et al. (2003a). Matching Words and Pictures. Journal of Machine Learning Research, 3, 1107–1135. zbMATHCrossRefGoogle Scholar
  10. Barnard, K., Duygulu, P., Raghavendra, K. G., Gabbur, P., & Forsyth, D. (2003b). The effects of segmentation and feature choice in a translation model of object recognition. In IEEE conference on computer vision and pattern recognition (Vol. II, pp. 675–682), Madison, WI. Google Scholar
  11. Berg, A. C., Berg, T. L., & Malik, J. (2005). Shape matching and object recognition using low distortion correspondence. In CVPR. Google Scholar
  12. Carbonetto, P., Freitas, N. D., & Barnard, K. (2004). A statistical model for general contextual object recognition. In European conference on computer vision. Google Scholar
  13. Chang, C.-C., & Lin, C.-J. (2001). LIBSVM—a library for support vector machines. Available from
  14. Chen, Y., & Wang, J. Z. (2004). Image categorization by learning and reasoning with regions. Journal of Machine Learning Research, 5, 913–939. Google Scholar
  15. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B (Methodological), 39(1), 1–38. zbMATHMathSciNetGoogle Scholar
  16. Fei-Fei, L., Fergus, R., & Perona, P. (2004). Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. In Workshop on generative-model based vision. Washington, DC. Google Scholar
  17. Fellbaum, C., Miller, P. G. A., Tengi, R., & Wakefield, P. (1998). WordNet—a lexical database for English. Available from
  18. Fergus, R., & Perona, P. (1998). The Caltech database. Available from
  19. Fergus, R., Perona, P., & Zisserman, A. (2003). Object class recognition by unsupervised scale-invariant learning. In IEEE conference on computer vision and pattern recognition. Madison, WI. Google Scholar
  20. Gabbur, P. (2003). Quantitative evaluation of feature sets, segmentation algorithms, and color constancy algorithms using word prediction. Masters Thesis, University of Arizona, Tucson, AZ. Google Scholar
  21. Gale, W., Church, K., & Yarowsky, D. (1992). One sense per discourse. In DARPA workshop on speech and natural language (pp. 233–237). New York. Google Scholar
  22. Jeon, J., Lavrenko, V., & Manmatha, R. (2003). Automatic image annotation and retrieval using cross-media relevance models. In SIGIR. Google Scholar
  23. Jonker, R., & Volgenant, A. (1987). A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38, 325–340. zbMATHCrossRefMathSciNetGoogle Scholar
  24. Karov, Y., & Edelman, S. (1998). Similarity-based word sense disambiguation. Computational Linguistics, 24(1), 41–59. Google Scholar
  25. Lavrenko, V., Manmatha, R., & Jeon, J. (2003). A model for learning the semantics of pictures. In NIPS. Google Scholar
  26. Leibe, B., & Schiele, B. (2003). The TU Darmstadt database. Available from
  27. Maron, O. (1998). Learning from ambiguity. PhD thesis, Massachusetts Institute of Technology. Google Scholar
  28. Maron, O., & Lozano-Perez, T. (1998). A framework for multiple-instance learning. In Neural information processing systems. Cambridge: MIT Press. Google Scholar
  29. Maron, O., & Ratan, A. L. (1998). Multiple-instance learning for natural scene classification. In The fifteenth international conference on machine learning. Google Scholar
  30. Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International conference on computer vision (Vol. II, pp. 416–421). Google Scholar
  31. Mihalcea, R., & Moldovan, D. (1998). Word sense disambiguation based on semantic density. In COLING/ACL workshop on usage of wordnet in natural language processing systems. Montreal. Google Scholar
  32. Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990). Introduction to WordNet: an on-line lexical database. International Journal of Lexicography, 3(4), 235–244. CrossRefGoogle Scholar
  33. Opelt, A., & Pinz, A. (2004). TU Graz-02 database. Available from
  34. Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(9), 888–905. Google Scholar
  35. Tao, Q., & Scott, S. (2004). A faster algorithm for generalized multiple-instance learning. In Seventeenth annual FLAIRS conference (pp. 550–555), Miami Beach, FL. Google Scholar
  36. Tao, Q., Scott, S., Vinodchandran, N. V., Osugi, T. T., & Mueller, B. (2004a). An extended kernel for generalized multiple-instance learning. In IEEE international conference on tools with artificial intelligence. Google Scholar
  37. Tao, Q., Scott, S. D., & Vinodchandran, N. V. (2004b). SVM-based generalized multiple-instance learning via approximate box counting. In International conference on machine learning (pp. 779–806). Banff, AB, Canada. Google Scholar
  38. Torralba, A., Murphy, K. P., & Freeman, W. T. (2003). The MIT-CSAIL database of objects and scenes. Available from
  39. Torralba, A., Murphy, K. P., & Freeman, W. T. (2004). Sharing features: efficient boosting procedures for multiclass object detection. In IEEE conference on computer vision and pattern recognition (Vol. II, pp. 762–769), Washington, DC. Google Scholar
  40. Traupman, J., & Wilensky, R. (2003). Experiments in improving unsupervised word sense disambiguation. Computer Science Division, University of California Berkeley. Google Scholar
  41. Vivarelli, F., & Williams, C. K. I. (1997). Using Bayesian neural networks to classify segmented images. In IEE international conference on artificial neural networks. Google Scholar
  42. Weber, M., Welling, M., & Perona, P. (2000). Unsupervised learning of models for recognition. In D. Vernon (Ed.), 6th European conference on computer vision (pp. 18–32). Google Scholar
  43. Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In 33rd conference on applied natural language processing. Cambridge: ACL. Google Scholar
  44. Zhang, Q., & Goldman, S. A. (2001). EM-DD: an improved multiple-instance learning technique. In Neural information processing Systems. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • Kobus Barnard
    • 1
  • Quanfu Fan
    • 1
  • Ranjini Swaminathan
    • 1
  • Anthony Hoogs
    • 2
  • Roderic Collins
    • 2
  • Pascale Rondot
    • 3
  • John Kaufhold
    • 4
  1. 1.Computer Science DepartmentThe University of ArizonaTucsonUSA
  2. 2.GE Global ResearchSchenectadyUSA
  3. 3.AeronauticsLockheed Martin Corp.Ft. WorthUSA
  4. 4.Advanced Concepts Business UnitSAIC Corp.McLeanUSA

Personalised recommendations