A New Biologically Inspired Color Image Descriptor

  • Jun Zhang
  • Youssef Barhomi
  • Thomas Serre
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7576)


We describe a novel framework for the joint processing of color and shape information in natural images. A hierarchical non-linear spatio-chromatic operator yields spatial and chromatic opponent channels, which mimics processing in the primate visual cortex. We extend two popular object recognition systems (i.e., the Hmax hierarchical model of visual processing and a sift-based bag-of-words approach) to incorporate color information along with shape information. We further use the framework in combination with the gist algorithm for scene categorization as well as the Berkeley segmentation algorithm. In all cases, the proposed approach is shown to outperform standard grayscale/shape-based descriptors as well as alternative color processing schemes on several datasets.


image descriptor color Hmax sift bag-of-words gist object recognition scene categorization segmentation 


  1. 1.
    Hurlber, A.C.: The Computation of Color. Dissertation, Massachusetts Institute of Technology (1989)Google Scholar
  2. 2.
    Wurm, L.H., Legge, G.E., Isenberg, L.M., Luebker, A.: Color improves object recognition in normal and low vision. Journal of Experimental Psychology: Human Perception and Performance 19, 899–911 (1993)CrossRefGoogle Scholar
  3. 3.
    Shapley, R., Hawken, M.: Color in the cortex: single- and double-opponent cells. Vision Research 51, 701–717 (2011)CrossRefGoogle Scholar
  4. 4.
    Land, E.H., McCann, J.J.: Lightness and retinex theory. Journal of the Optical Society of America 61, 1–11 (1971)CrossRefGoogle Scholar
  5. 5.
    Bosch, A., Zisserman, A., Munoz, X.: Scene classification using a hybrid generative/discriminative approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 712–727 (2008)CrossRefGoogle Scholar
  6. 6.
    van de Sande, K.E.A., Gevers, T., Snoek, C.G.M.: Evaluating color descriptors for object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 1582–1596 (2010)CrossRefGoogle Scholar
  7. 7.
    Burghouts, G.J., Geusebroek, J.M.: Performance evaluation of local colour invariants. Computer Vision and Image Understanding 113, 48–62 (2009)CrossRefGoogle Scholar
  8. 8.
    van de Weijer, J., Gevers, T., Smeulders, A.W.: Robust photometric invariant features from the color tensor. IEEE Transactions on Image Processing 15, 118–127 (2006)CrossRefGoogle Scholar
  9. 9.
    Brown, M., Susstrunk, S.: Multi-spectral SIFT for scene category recognition. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 177–184 (2011)Google Scholar
  10. 10.
    van de Weijer, J., Schmid, C.: Coloring Local Feature Extraction. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 334–348. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Tang, J., Miller, S., Singh, A., Abbeel, P.: A textured object recognition pipeline for color and depth image data. In: International Conference on Robotics and Automation (2012)Google Scholar
  12. 12.
    Gevers, T., Stokman, H.M.G.: Robust histogram construction from color invariants for object recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 113–118 (2004)CrossRefGoogle Scholar
  13. 13.
    Serre, T., Wolf, L., Bileschi, S.M., Riesenhuber, M., Poggio, T.: Robust object recognition with cortex-like mechanisms. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 411–426 (2007)CrossRefGoogle Scholar
  14. 14.
    Nilsback, M.E., Zisserman, A.: A visual vocabulary for flower classification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1447–1454 (2006)Google Scholar
  15. 15.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge (VOC 2007) Results (2007),
  16. 16.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42, 145–175 (2001)zbMATHCrossRefGoogle Scholar
  17. 17.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 898–916 (2010)CrossRefGoogle Scholar
  18. 18.
    Lennie, P., Krauskopf, J., Sclar, G.: Chromatic mechanisms in striate cortex of macaque. The Journal of Neuroscience 10, 649–669 (1990)Google Scholar
  19. 19.
    Conway, B.R.: Spatial structure of cone inputs to color cells in alert macaque primary visual cortex (V-1). The Journal of Neuroscience 21, 2768–2783 (2001)Google Scholar
  20. 20.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2169–2178 (2006)Google Scholar
  21. 21.
    Heeger, D.J.: Normalization of cell responses in cat striate cortex. Visual Neuroscience 9, 181–197 (1992)CrossRefGoogle Scholar
  22. 22.
    Carandini, M., Heeger, D.J., Movshon, J.A.: Linearity and normalization in simple cells of the macaque primary visual cortex. The Journal of Neuroscience 17, 8621–8644 (1997)Google Scholar
  23. 23.
    Johnson, E.N., Hawken, M.J., Shapley, R.: The orientation selectivity of color-responsive neurons in macaque V1. The Journal of Neuroscience 28, 8096–8106 (2008)CrossRefGoogle Scholar
  24. 24.
    Solomon, S.G., Lennie, P.: Chromatic gain controls in visual cortical neurons. The Journal of Neuroscience 25, 4779–4792 (2005)CrossRefGoogle Scholar
  25. 25.
    van de Weijer, J., Gevers, T., Geusebroek, J.M.: Edge and corner detection by photometric quasi-invariants. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 625–630 (2005)CrossRefGoogle Scholar
  26. 26.
    Geusebroek, J.M., van den Boomgaard, R., Smeulders, A.W., Geerts, H.: Color invariance. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 1338–1350 (2001)CrossRefGoogle Scholar
  27. 27.
    van Gemert, J., Geusebroek, J.M., Veenman, C.J., Smeulders, A.W.: Kernel Codebooks for Scene Categorization. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 696–709. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  28. 28.
    Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 530–549 (2004)CrossRefGoogle Scholar
  29. 29.
    van de Weijer, J., Schmid, C.: Applying color names to image description. In: International Conference on Image Processing, pp. 493–496 (2007)Google Scholar
  30. 30.
    Khan, F.S., van de Weijer, J., Vanrell, M.: Modulating shape features by color attention for object recognition. International Journal of Computer Vision 98, 49–64 (2011)CrossRefGoogle Scholar
  31. 31.
    Vigo, D.A.R., Khan, F.S., van de Weijer, J., Gevers, T.: The impact of color on bag-of-words based object recognition. In: International Conference on Pattern Recognition, pp. 1549–1553 (2010)Google Scholar
  32. 32.
    Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: Indian Conference on Computer Vision Graphics Image Processing, pp. 722–729 (2008)Google Scholar
  33. 33.
    Varma, M., Ray, D.: Learning the discriminative power-invariance trade-off. In: IEEE International Conference on Computer Vision, pp. 1–8 (2007)Google Scholar
  34. 34.
    Gehler, P.V., Nowozin, S.: On feature combination for multiclass object classification. In: IEEE International Conference on Computer Vision, pp. 221–228 (2009)Google Scholar
  35. 35.
    van de Sande, K.E., Gevers, T., Snoek, C.G.: Color descriptors for object category recognition. In: European Conference on Color in Graphics, Imaging and Vision, pp. 378–381 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Jun Zhang
    • 1
    • 2
  • Youssef Barhomi
    • 1
  • Thomas Serre
    • 1
  1. 1.Department of Cognitive Linguistic & Psychological Sciences, Institute for Brain SciencesBrown UniversityProvidenceUSA
  2. 2.School of Computer & InformationHefei University of TechnologyHefeiChina

Personalised recommendations