Using Geotags to Derive Rich Tag-Clouds for Image Annotation

  • Dhiraj JoshiEmail author
  • Jiebo Luo
  • Jie Yu
  • Phoury Lei
  • Andrew Gallagher


Geotagging has become popular for many multimedia applications. In this chapter, we present an integrated and intuitive system for location-driven tag suggestion, in the form of tag-clouds, for geotagged photos. Potential tags from multiple sources are extracted and weighted. Sources include points of interest (POI) tags from a public Geographic Names Information System (GNIS) database, community tags from Flickr® pictures, and personal tags shared through users’ own, family, and friends’ photo collections. To increase the effectiveness of GNIS POI tags, bags of place-name tags are first retrieved, clustered, and then re-ranked using a combined tf-idf and spatial distance criteria. The community tags from photos taken in the vicinity of the input geotagged photo are ranked according to distance and visual similarity to the input photo. Personal tags from other personally related photos inherently carry a significant weight due more to their high relevance than to both the generic place-name tags and community tags, and are ranked by weights that decay over time and distance differences. Finally, a rich set of the most relevant location-driven tags is presented to the user in the form of individual tag clouds under the three mentioned source categories. The tag clouds act as intuitive suggestions for tagging an input image. We also discuss quantitative and qualitative findings from a user study that we conducted. Evaluation has revealed the respective benefits of the three categories toward the effectiveness of the integrated tag suggestion system.


Query Image Landmark Recognition Geographical Information System Database Geotagged Photo Probabilistic Latent Semantic Analysis Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Ahern, S., Davis, M., Eckles, D., King, S., Naaman, M., Nair, R., Spasojevic, M., Yang, J.: Zonetag: Designing context aware mobile media capture to increase participation. In: Proceedings of Workshop on Pervasive Image Capture and Sharing (2006) Google Scholar
  2. 2.
    Ames, M., Naaman, M.: Why we tag: Motivations for annotation in mobile and online media. In: Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (2007) Google Scholar
  3. 3.
    Cao, L., Luo, J., Kautz, H., Huang, T.: Annotating collections of geotagged photos using hierarchical event and scene models. In: Proceedings of IEEE CVPR (2008) Google Scholar
  4. 4.
    Cao, L., Luo, J., Huang, T.S.: Annotating photo collections by label propagation according to multiple proximity cues. In: Proceedings of ACM Multimedia (2008) Google Scholar
  5. 5.
    Cao, L., Yu, J., Luo, J., Huang, T.S.: Enhancing semantic and geographic annotation of Web images via logistic canonical correlation regression. In: Proceedings of ACM Multimedia (2009) Google Scholar
  6. 6.
    Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002) CrossRefGoogle Scholar
  7. 7.
    Crandall, D., Backstrom, L., Huttenlocher, D., Kleinberg, J.: Mapping the world’s photos. In: Proceedings of World Wide Web Conference (2009) Google Scholar
  8. 8.
    Cristani, M., Perina, A., Castellani, U., Murino, V.: Geo-located image analysis using latent representations. In: Proceedings of IEEE CVPR (2008) Google Scholar
  9. 9.
    Divvala, S., Hoiem, D., Hays, J., Efros, A., Hebert, M.: An empirical study of context in object detection. In: Proceedings of IEEE CVPR (2009) Google Scholar
  10. 10.
    Dubinko, M., Kumar, R., Magnani, Novak J., Raghavan, P., Tomkins, A.: Visualizing tags over time. In: Proceedings of World Wide Web Conference (2006) Google Scholar
  11. 11.
    Hays, J., Efros, A.: IM2GPS: Estimating geographic information from a single image. In: Proceedings of IEEE CVPR (2008) Google Scholar
  12. 12.
    Jacobs, N., Satkin, S., Roman, N., Speyer, R., Pless, R.: Geolocating static cameras. In: Proceedings of IEEE International Conference on Computer Vision (2007) Google Scholar
  13. 13.
    Jain, V., Singhal, A., Luo, J.: Selective hidden random fields: Exploiting domain specific saliency for event classification. In: Proceedings of IEEE CVPR (2008) Google Scholar
  14. 14.
    Joshi, D., Luo, J.: Inferring generic activities and events from image content and bags of geo-tags. In: Proceedings of ACM CIVR (2008) Google Scholar
  15. 15.
    Joshi, D., Gallagher, A., Yu, J., Luo, J.: Inferring photographic location using geotagged web images. Multimed. Tools Appl. J. (2010) Google Scholar
  16. 16.
    Joshi, D., Luo, J., Yu, J., Lei, P., Gallagher, A.: Rich location-driven tag cloud suggestions based on public, community, and personal sources. In: Proceedings of ACM Int. Workshop on Connected Media Mining (2010) Google Scholar
  17. 17.
    Kennedy, L., Naaman, M., Ahern, S., Nair, R., Rattenbury, T.: How Flickr helps us make sense of the world: Context and content in community-contributed media collections. In: Proceedings of ACM Multimedia (2007) Google Scholar
  18. 18.
    Kennedy, L., Slaney, M., Weinberger, K.: Reliable tags using image similarity: Mining specificity and expertise from large-scale multimedia databases. In: ACM Workshop on Web-Scale Multimedia Corpus (2009) Google Scholar
  19. 19.
    Kleban, J., Moxley, E., Xu, J., Manjunath, B.S.: Global annotation on georeferenced photographs. In: Proceedings of ACM CIVR (2009) Google Scholar
  20. 20.
    Kosecka, J., Zhang, W.: Video compass. In: Proceedings of European Conference on Computer Vision (ECCV) (2002) Google Scholar
  21. 21.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: Proceedings of IEEE CVPR (2006) Google Scholar
  22. 22.
    Liao, L., Fox, D., Kautz, H.: Extracting places and activities from GPS traces using hierarchical conditional random fields. Int. J. Robot. Res. (2007) Google Scholar
  23. 23.
    Li, L.-J., Fei-Fei, L.: What, where and who? Classifying event by scene and object recognition. In: Proceedings of IEEE ICCV (2007) Google Scholar
  24. 24.
    Luo, J., Boutell, M., Brown, C.: Pictures are not taken in a vacuum: An overview of exploiting context for semantic scene content understanding. IEEE Signal Process. Mag. 23(2), 101–114 (2006) CrossRefGoogle Scholar
  25. 25.
    Luo, J., Yu, J., Joshi, D., Hao, W.: Event recognition: viewing the world with a third eye. In: Proceedings of ACM Multimedia (2008) Google Scholar
  26. 26.
    Moxley, E., Kleban, J., Manjunath, B.S.: SpiritTagger: A geo-aware tag suggestion tool mined from Flickr. In: Proceedings of ACM Multimedia Information Retrieval (MIR) (2008) Google Scholar
  27. 27.
    O’Hare, N., Smeaton, A.: Context-aware person identification in personal photo collections. IEEE Trans. Multimed. (2009) Google Scholar
  28. 28.
    Quack, T., Leibe, B., Van Gool, L.: World-scale mining of objects and events from community photo collections. In: Proceedings of CIVR (2008) Google Scholar
  29. 29.
    Torralba, A., Fergus, R., Freeman, W.T.: Tiny images. Technical Report MIT-CSAIL-TR-2007-024 (2007) Google Scholar
  30. 30.
    Toyama, K., Logan, R., Roseway, A.: Geographic location tags on digital images. In: Proceedings of ACM Multimedia (2003) Google Scholar
  31. 31.
    Tsai, C.-M., Qamra, A., Chang, E.: Extent: Inferring image metadata from context and content. In: Proceedings of IEEE ICME (2005) Google Scholar
  32. 32.
    Wei, X.-Y., Jiang, Y.-G., Ngo, C.-W.: Exploring inter-concept relationship with context space for semantic video indexing. In: Proceedings of ACM CIVR (2009) Google Scholar
  33. 33.
    Wolf, L., Bileschi, S.: A critical view of context. Int. J. Comput. Vis. 68(1), 43–52 (2006) CrossRefGoogle Scholar
  34. 34.
    Yu, J., Luo, J.: Leveraging probabilistic season and location context models for scene understanding. In: Proceedings of ACM CIVR (2008) Google Scholar
  35. 35.
    Yuan, J., Luo, J., Kautz, H., Wu, Y.: Mining GPS traces and visual words for event classification. In: Proceedings of ACM Multimedia Information Retrieval (MIR) (2008) Google Scholar
  36. 36.
    Zheng, V.W., Zheng, Y., Xie, X., Yang, Q.: Collaborative location and activity recommendations with GPS history data. In: Proceedings of World Wide Web Conference (2010) Google Scholar
  37. 37.
    Zheng, Y.-T., Zhao, M., Song, Y., Adam, H., Buddemeier, U., Bissacco, A., Brucher, F., Chua, T.-S., Neven, H.: Tour the world: Building a webscale landmark recognition engine. In: Proceedings of IEEE CVPR (2009) Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Dhiraj Joshi
    • 1
    Email author
  • Jiebo Luo
    • 1
  • Jie Yu
    • 1
  • Phoury Lei
    • 1
  • Andrew Gallagher
    • 1
  1. 1.Corporate Research and EngineeringEastman Kodak CompanyRochesterUSA

Personalised recommendations