Improvement the Bag of Words Image Representation Using Spatial Information

  • Mohammad Mehdi Farhangi
  • Mohsen Soryani
  • Mahmood Fathy
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 177)


Bag of visual words (BOW) model is an effective way to represent images in order to classify and detect their contents. However, this type of representation suffers from the fact that, it does not contain any spatial information. In this paper we propose a novel image representation which adds two types of spatial information. The first type which is the spatial locations of the words in the image is added using the spatial pyramid matching approach. The second type is the spatial relation between words. To explore this information a binary tree structure which models the is-a relationships in the vocabulary is constructed from the visual words. This approach is a simple and computationally effective way for modeling the spatial relations of the visual words which shows improvement on the visual classification performance. We evaluated our method on visual classification of two known data sets, namely 15 natural scenes and Caltech-101.


BOW Representation Spatial Information N-gram Model Spatial Pyramid Matching 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: Proc. ICCV (2003)Google Scholar
  2. 2.
    Fei-Fei, L., Perona, P.: A Bayesian Hierarchical Model for Learning Natural Scene Categories. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (2005)Google Scholar
  3. 3.
    Lowe, K.D.: Distinctive Image Features from Scale-Invariant Keypoints. J. of Computer Vision 2(60), 91–110 (2004)CrossRefGoogle Scholar
  4. 4.
    Harris, C., Stephens, M.: A combined corner and edge detector. In: Proc. Alvey Vision Conf., pp. 147–151 (1988)Google Scholar
  5. 5.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded Up Robust Features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  6. 6.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. In: Proc. CVPR 2003, Madison, WI, pp. 257–263 (June 2003)Google Scholar
  7. 7.
    van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Trans. Pattern Analysis and Machine Intelligence 32(7), 1271–1283 (2010)CrossRefGoogle Scholar
  8. 8.
    Jiang, Y.G., Yang, J., Ngo, C.W.: Representation Of KeyPoint-Based Semantic Concept Detection: A Comprehensive Study. IEEE Trans. Multimedia 2(1), 42–53 (2010)CrossRefGoogle Scholar
  9. 9.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2169–2178 (2006)Google Scholar
  10. 10.
    Wu, L., Li, M., Li, Z., Ma, W.-Y., Yu, N.: Visual language modeling for image classification. In: ACM Multimedia Workshop on Multimedia Information Retrieval, pp. 115–124 (2007)Google Scholar
  11. 11.
    Mei, L., Kweon, I., Hua, X.: Contextual Bag-of-Words for Visual Categorization. IEEE Trans. Circuits and Systems for Video Technology 21(4), 381–392 (2011)CrossRefGoogle Scholar
  12. 12.
    Jiang, Y.G., Ngo, C.W.: Bag-of-visual-words expansion using visual relatedness for video indexing. In: ACM Conf. on Research & Development on Information Retrieval, pp. 769–770 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Mohammad Mehdi Farhangi
    • 1
  • Mohsen Soryani
    • 1
  • Mahmood Fathy
    • 1
  1. 1.Department of Computer EngineeringIran University of Science and TechnologyTehranIran

Personalised recommendations