Dataset Issues in Object Recognition

  • J. Ponce
  • T. L. Berg
  • M. Everingham
  • D. A. Forsyth
  • M. Hebert
  • S. Lazebnik
  • M. Marszalek
  • C. Schmid
  • B. C. Russell
  • A. Torralba
  • C. K. I. Williams
  • J. Zhang
  • A. Zisserman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4170)


Appropriate datasets are required at all stages of object recognition research, including learning visual models of object and scene categories, detecting and localizing instances of these models in images, and evaluating the performance of recognition algorithms. Current datasets are lacking in several respects, and this paper discusses some of the lessons learned from existing efforts, as well as innovative ways to obtain very large and diverse annotated datasets. It also suggests a few criteria for gathering future datasets.


Object Recognition Image Retrieval Background Feature Object Class Recognition Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agarwal, S., Roth, D.: Learning a Sparse Representation for Object Detection. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 113–127. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  2. 2.
    Barnard, K., Duyguly, P., Forsyth, D.: Clustering art. In: Proc. IEEE Conf. Comp. Vision Patt. Recog., vol. II, pp. 435–439 (2001)Google Scholar
  3. 3.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D., Jordan, M.: Matching words and pictures. Journal of Machine Learning Research 3, 1107–1135 (2003)zbMATHCrossRefGoogle Scholar
  4. 4.
    Berg, A., Berg, T.L., Malik, J.: Shape matching and object recognition using low distortion correspondence. In: Proc. IEEE Conf. Comp. Vision Patt. Recog., vol. II, pp. 435–439 (2005)Google Scholar
  5. 5.
    Berg, A.C.: Ph.D thesis (to appear)Google Scholar
  6. 6.
    Berg, T.L., Berg, A.C., Edwards, J., Forsyth, D.: Who’s in the picture? In: Proc. Neural Inf. Proc. Syst. (2004)Google Scholar
  7. 7.
    Berg, T.L., Forsyth, D.: Animals on the Web. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. (2006)Google Scholar
  8. 8.
    Everingham, M., Zisserman, A., Williams, C., Van Gool, L., Allan, M., Bishop, C., Chapelle, O., Dalal, N., Deselaers, T., Dorko, G., Duffner, S., Eichhorn, J., Farquhar, J., Fritz, M., Garcia, C., Griffiths, T., Jurie, F., Keysers, D., Koskela, M., Laaksonen, J., Larlus, D., Leibe, B., Meng, H., Ney, H., Schiele, B., Schmid, C., Seemann, E., Shawe-Taylor, J., Storkey, A., Szedmak, S., Triggs, B., Ulusoy, I., Viitaniemi, V., Zhang, J.: The 2005 PASCAL visual object classes challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS, vol. 3944, pp. 117–176. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  9. 9.
    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. Workshop on Generative-Model Based Vision (2004)Google Scholar
  10. 10.
    Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scale-invariant learning. In: Proc. IEEE Conf. Comp. Vision Patt. Recog., vol.II, pp. 264–271 (2003)Google Scholar
  11. 11.
    Fergus, R., Perona, P., Zisserman, A.: A visual category filter for Google images. In: Proc. Europ. Conf. Comp. Vision (2004)Google Scholar
  12. 12.
    Fergus, R., Fei-Fei, L., Perona, P., Zisserman, A.: Learning object categories from Google’s image search. In: Proc. Int. Conf. Comp. Vision (2005)Google Scholar
  13. 13.
    Giridharan, I., Duygulu, P., Feng, S., Ircing, P., Khudanpur, S., Klakow, D., Krause, M., Manmatha, R., Nock, H., Petkova, D., Pytlik, B., Virga, P.: Joint visual-text modeling for automatic retrieval of multimedia documents. In: Proc. ACM Multimedia Conference (2005)Google Scholar
  14. 14.
    Grauman, K., Darrell, T.: The pyramid match kernel: Discriminative classification with sets of image features.MIT-CSAIL-TR-2006-020 (2006) Updated version of the ICCV 2005 paper with the same title, featuring the improved results shown in Fig. 4Google Scholar
  15. 15.
    Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Machine Learning 43, 177–196 (2001)CrossRefGoogle Scholar
  16. 16.
    Hoiem, D., Efros, A., Hebert, M.: Geometric context from a single image. In: Proc. Int. Conf. Comp. Vision (2005)Google Scholar
  17. 17.
    Holub, A., Welling, M., Perona, P.: Combining generative models and fisher kernels for object class recognition. In: Proc. Int. Conf. Comp. Vision (2005)Google Scholar
  18. 18.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. (2005)Google Scholar
  19. 19.
    Li, J., Wang, J.: Automatic linguistic indexing of pictures by a statistical modeling approach. PAMI 25(9), 1075–1088 (2003)Google Scholar
  20. 20.
    Müller, H., Marchand-Maillet, S., Pun, T.: The Truth about Corel - Evaluation in Image Retrieval. In: Lew, M., Sebe, N., Eakins, J.P. (eds.) CIVR 2002. LNCS, vol. 2383, p. 38. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  21. 21.
    Mutch, J., Lowe, D.: Multiclass object recognition using sparse, localized features. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. (2006)Google Scholar
  22. 22.
    Ommer, B., Buhmann, J.M.: Learning compositional categorization models. In: Proc. Europ. Conf. Comp. Vision (2006)Google Scholar
  23. 23.
    Philips, P., Newton, E.: Meta-analysis of face recognition algorithms. In: Int. Conf. on Automatic Face and Gesture Recognition (2002)Google Scholar
  24. 24.
    Rubner, Y., Tomasi, C., Guibas, L.: The Earth Mover’s distance as a metric for image retrieval. International Journal of Computer Vision 40(2), 99–121 (2000)zbMATHCrossRefGoogle Scholar
  25. 25.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. Technical report. MIT, AI Lab Memo AIM-2005-025 (2005)Google Scholar
  26. 26.
    Serre, T., Wolf, L., Poggio, T.: Object recognition with features inspired by visual cortex. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. (2005)Google Scholar
  27. 27.
    Stork, D.: The open mind initiative. IEEE Intelligent Systems and Their Applications 14(3), 19–20 (1999)Google Scholar
  28. 28.
    Torralba, A.: Contextual priming for object detection. International Journal of Computer Vision 53(2), 153–167 (2003)CrossRefGoogle Scholar
  29. 29.
    von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proc. ACM Conf. Hum. Factors Comp. Syst. (CHI) (2004)Google Scholar
  30. 30.
    von Ahn, L., Liu, R., Blum, M.: Peekaboom: A game for locating objects in images. In: Proc. ACM Conf. Hum. Factors Comp. Syst. (CHI) (2006)Google Scholar
  31. 31.
    Wang, G., Zhang, Y., Fei-Fei, L.: Using dependent regions for object categorization in a generative framework. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. (2006)Google Scholar
  32. 32.
    Yanai, K., Barnard, K.: Probabilistic web image gathering. In: Workshop on MIR (2005)Google Scholar
  33. 33.
    Zhang, H., Berg, A.C., Maire, M., Malik, J.: SVM-KNN: Discriminative nearest neighbor classification for visual category recognition. In: Proc. IEEE Conf. Comp. Vision Patt. Recog. (2006)Google Scholar
  34. 34.
    Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local features and kernels for classification of texture and object categories: An in-depth study.Technical Report RR-5737, INRIA Rhône-Alpes (2005)Google Scholar
  35. 35.
  36. 36.
  37. 37.

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • J. Ponce
    • 1
    • 2
  • T. L. Berg
    • 3
  • M. Everingham
    • 4
  • D. A. Forsyth
    • 1
  • M. Hebert
    • 5
  • S. Lazebnik
    • 1
  • M. Marszalek
    • 6
  • C. Schmid
    • 6
  • B. C. Russell
    • 7
  • A. Torralba
    • 7
  • C. K. I. Williams
    • 8
  • J. Zhang
    • 6
  • A. Zisserman
    • 4
  1. 1.University of Illinois at Urbana-ChampaignUSA
  2. 2.Ecole Normale SupérieureParisFrance
  3. 3.University of California at BerkeleyUSA
  4. 4.Oxford UniversityUK
  5. 5.Carnegie Mellon UniversityPittsburghUSA
  6. 6.INRIA Rhône-AlpesGrenobleFrance
  7. 7.MITCambridgeUSA
  8. 8.University of EdinburghEdinburghUK

Personalised recommendations