International Journal of Computer Vision

, Volume 35, Issue 3, pp 245–268 | Cite as

Spatial Color Indexing and Applications

  • Jing Huang
  • S. Ravi Kumar
  • Mandar Mitra
  • Wei-Jing Zhu
  • Ramin Zabih


We define a new image feature called the color correlogram and use it for image indexing and comparison. This feature distills the spatial correlation of colors and when computed efficiently, turns out to be both effective and inexpensive for content-based image retrieval. The correlogram is robust in tolerating large changes in appearance and shape caused by changes in viewing position, camera zoom, etc. Experimental evidence shows that this new feature outperforms not only the traditional color histogram method but also the recently proposed histogram refinement methods for image indexing/retrieval. We also provide a technique to cut down the storage requirement of the correlogram so that it is the same as that of histograms, with only negligible performance penalty compared to the original correlogram.

We also suggest the use of color correlogram as a generic indexing tool to tackle various problems arising from image retrieval and video browsing. We adapt the correlogram to handle the problems of image subregion querying, object localization, object tracking, and cut detection. Experimental results again suggest that the color correlogram is more effective than the histogram for these applications, with insignificant additional storage or processing cost.

image indexing image features content-based image retrieval model-based object recognition spatial correlation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Boresczky, J.S. and Rowe, L.A. 1996. A comparison of video shot boundary detection techniques. In Storage & Retrieval for Image and Video Databases IV; Proc. SPIE 2670, pp. 170–179.Google Scholar
  2. Brock-Gunn, S.A. and Ellis, T.J. 1992. Using color templates for target identification and tracking. In Proc. British Machine Vision Conference, pp. 207–216.Google Scholar
  3. 1995. Content-based image retrieval systems, IEEE Computer.Google Scholar
  4. Cox, I.J., Matt L. Miller, Stephen M. Omohundro, and Peter N. Yianilas, 1996. PicHunter: Bayesian relevance feedback for image retrieval. In Intl. Conf. on Pattern Recognition, Vienna, Austria.Google Scholar
  5. Ennesser, F. and Medioni, G. 1995. Finding waldo, or focus of attention using local color information. IEEE Trans. onPattern Analysis and Machine Intelligence, 17(8).Google Scholar
  6. Enser, P.G.B. 1993. Query analysis in a visual information retrieval context. J. Document and Text Management, 1:25–52.Google Scholar
  7. Fleck, M.M., Forsyth, D.A., and Bregler, C. 1996. Finding naked people. In European Conf. on Computer Vision, Vol. 2, pp. 590–602.Google Scholar
  8. Flickner, M. et al. 1995. Query by image and video content: The QBIC system. IEEE Computer, 28(9):23–32.Google Scholar
  9. Forsyth, D.A. et al. 1996. Finding pictures of objects in large collections of images. In Proc. Intl. Workshop on Object Recognition, Cambridge.Google Scholar
  10. Funt, B. and Finlayson, G. 1995. Color constant color indexing. IEEE Trans. on Pattern Analysis and Machine Intelligence, 17:522–529.Google Scholar
  11. Gong, Y. et al. 1994. An image database system with content capturing and fast image indexing abilities. In Intl. Conf. on Multimedia Comp & Systems, pp. 121–130.Google Scholar
  12. Gong, Y. 1998. Intelligent Image Databases: Towards Advanced Image Retrieval. Kluwer Academic Publishers.Google Scholar
  13. Grimson, W.L. and Lozano-Pérez, T. 1987. Localising overlapping parts by searching the interpretation tree. IEEE Trans. on Pattern Analysis and Machine Intelligence, 9:469–482.Google Scholar
  14. Hafner, J., Sawhney, H., Equitz, W., Flickner, M., and Niblack, W. 1995. Efficient color histogram indexing for quadratic form distance functions. IEEE Trans. on Pattern Analysis and Machine Intelligence, 17(7):729–736.Google Scholar
  15. Hampapur, A., Jain, R., and Weymouth, T. 1994. Digital video indexing in multimedia systems. In Proc. AAAI-94 Workshop on Indexing and Reuse in Multimedia Systems.Google Scholar
  16. Haralick, R.M. 1979. Statistical and structural approaches to texture. Proc. of IEEE, 67(5):786–804.Google Scholar
  17. Haussler, D. 1992. Decision theoretic generalization of the PAC model for neural net and other learning applications. Information and Computation, 100:78–150.Google Scholar
  18. Hsu, W., Chua, T.S., and Pung, H.K. 1995. An integrated colorspatial approach to content-based image retrieval. In Proc. 3rd ACM Multimedia Conf., pp. 305–313.Google Scholar
  19. Huang, J., Kumar, S.R., and Mitra, M. 1997. Combining supervised learning with color correlograms for content-based image retrieval. In Proc. 5th ACM Multimedia Conf., pp. 325–334.Google Scholar
  20. Huang, J., Kumar, S.R., Mitra, M., Zhu, W.J., and Zabih, R. 1997. Image indexing using color correlograms. In Proc. 16th IEEE Conf. on Computer Vision and Pattern Recognition, pp. 762–768.Google Scholar
  21. Huang, J., Kumar, S.R., Mitra, M., and Zhu, W.J. 1998. Spatial color indexing and applications. In Proc. 8th Intl. Conf. on Computer Vision.Google Scholar
  22. Huttenlocher, D.P. and Ullman, S. 1986. Object recognition using alignment. In Proc. Intl. Conf. on Computer Vision, pp. 102–111.Google Scholar
  23. Huttenlocher, D.P., Klanderman, G.A., and Rucklidge, W.J. 1993. Comparing images using the Hausdorff distance. IEEE Trans. Pattern Analysis and Machine Intelligence, 15:850–863.Google Scholar
  24. Huttenlocher, D.P., Lilien, R.H., and Olson, C.F. 1996. Object recognition using subspace methods. In Proc. European Conf. on Computer Vision, pp. 536–545.Google Scholar
  25. Jacobs, D.W., Weinshall, D., and Gdalyahu, Y. 1998. Condensing image databases when retrieval is based on non-metric distances. In Proc. Intl. Conf. on Computer Vision.Google Scholar
  26. Marr, D. and Nishihara, H.K. 1978. Representation and recognition of the spatial organization of three-dimensional shapes. Proc. Royal Soc. Lond. B., 200:269–294.Google Scholar
  27. Margalit, A. and Rosenfeld, A. 1990. Using probabilistic domain knowledge to reduce the expected computational cost of template matching. Computer Vision; Graphics; and Image Processing, 51:219–234.Google Scholar
  28. Matas, J., Marik, R., and Kittler, J. 1995. On representation and matching of multi-colored objects. In Proc. IEEE 5th Intl. Conf. on Computer Vision, pp. 726–732.Google Scholar
  29. Murase, H. and Nayar, S.K. 1995. Visual learning and recognition of 3-D objects from appearance. Intl. Journal of Computer Vision, 14:5–24.Google Scholar
  30. Ogle, V. and Stonebraker, M. 1995. Chabot: Retrieval from a relational database of images. IEEE Computer, 28(9):40–48.Google Scholar
  31. Pass, G. and Zabih, R. 1996. Histogram refinement for content-based image retrieval. In IEEE Workshop on Applications of Computer Vision, pp. 96–102.Google Scholar
  32. Pass, G. and Zabih, R. 1999. Comparing images using joint histograms. In Journal of Multimedia Systems, 7(3):234–240.Google Scholar
  33. Pentland, A., Picard, R., and Sclaroff, S. 1996. Photobook: Contentbased manipulation of image databases. Intl. Journal of Computer Vision, 18(3):233–254.Google Scholar
  34. Rao, R.P. and Ballard, D. 1995. Object indexing using an iconic sparse distributed memory. In Proc. IEEE 5th Intl. Conf. on Computer Vision, pp. 24–31.Google Scholar
  35. Rickman, R. and Stonham, J. 1996. Content-based image retrieval using color tuple histograms. SPIE Proc., 2670:2–7.Google Scholar
  36. Roberts, L.G. 1965. Machine perception of three-dimensional solids. In Optical and Electro-Optical Information Processing. MIT Press.Google Scholar
  37. Rousseeuw, P.J. and Leroy, A.M. 1987. Robust Regression and Outlier Detection. John Wiley & Sons.Google Scholar
  38. Slater, D. and Healey, G. 1995. Combining color and geometric information for the illumination invariant recognition of 3-D objects. In Proc. IEEE 5th Intl. Conf. on Computer Vision, pp. 563–568.Google Scholar
  39. Smith, J. and Chang, S.-F. 1996. Tools and techniques for color image retrieval, SPIE Proc., 2670:1630–1639.Google Scholar
  40. Stricker, M. and Swain, M. 1994. The capacity of color histogram indexing. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 704–708.Google Scholar
  41. Stricker, M. and Dimai, A. 1996. Color indexing with weak spatial constraints, SPIE Proc., 2670:29–40.Google Scholar
  42. Swain, M. and Ballard, D. 1991. Color indexing. In Intl. Journal of Computer Vision, 7(1):11–32.Google Scholar
  43. Syeda-Mahmood, T.F. 1997. Data and model-driven selection using color regions. In Intl. Journal of Computer Vision, 21(1/2): 9–36.Google Scholar
  44. Syeda-Mahmood, T.F. and Cheng, Y.-Q. 1996. Indexing colored surfaces in images. In Intl. Conf. on Pattern Recognition.Google Scholar
  45. Upton, G.J. and Fingleton, B. 1985. Spatial Data Analysis by Example. Vol. 1. John Wiley & Sons.Google Scholar
  46. Vinod, V.V., Murase, H., and Hashizume, C. 1996. Focused color intersection with efficient searching for object detection and image retrieval. IEEE Proc. Multimedia, pp. 229–233.Google Scholar
  47. Wan, X. and Jay Kuo, C.-C. 1996. Color distribution analysis and quantization for image retrieval. SPIE Proc., 2670:8–16.Google Scholar
  48. Yeo, B.-L. and Liu, B. 1995. Rapid scene analysis on compressed videos. In IEEE Trans. Circuits Syst. Video Technology, 5, 6: 533–544.Google Scholar

Copyright information

© Kluwer Academic Publishers 1999

Authors and Affiliations

  • Jing Huang
    • 1
  • S. Ravi Kumar
    • 1
  • Mandar Mitra
    • 1
  • Wei-Jing Zhu
    • 1
  • Ramin Zabih
    • 1
  1. 1.Cornell UniversityIthaca

Personalised recommendations