Personalized Annotation for Photos with Visual Instance Search

  • Bao TruongEmail author
  • Thuyen V. Phan
  • Vinh-Tiep Nguyen
  • Minh-Triet Tran
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9733)


Emotional and memorable moments are usually kept and shared on different online services such as Facebook, Flickr, Instagram, and Google Photos. As a result, one of users’ practical needs is to have their photos annotated automatically, especially with personalized tags. This motivates the authors to propose a system that can suggest personalized annotations for a photo uploaded to online services. Our system provides 2 major features. First, the system automatically recommends personalized annotations for newly uploaded photos based on visually similar photos uploaded in the past. Second, our system propagates manual annotations of users to other similar photos existed in their albums. To evaluate the performance of our system, we use the Oxford 5K Building Dataset and our own dataset consisting of personal photos collected from Facebook. Our systems achieves the mean Average Precision of 0.844 and 0.749 respectively on these two datasets. This demonstrates that our proposed solution can be potentially integrated as a useful utility or extension for online photo sharing services.


Image Retrieval Visual Word Personal Photo Inverse Document Frequency Text Retrieval 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Belongie, S., Carson, C., Greenspan, H., Malik, J.: Color- and texture-based image segmentation using EM and its application to content-based image retrieval. In: 1998 Sixth International Conference on Computer Vision, pp. 675–682, January 1998Google Scholar
  2. 2.
    Brunelli, R.: Template Matching Techniques in Computer Vision: Theory and Practice. Wiley, New York (2009)CrossRefGoogle Scholar
  3. 3.
    Chen, M., Zheng, A., Weinberger, K.Q.: Fast image tagging. In: Dasgupta, S., Mcallester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning (ICML 2013), vol. 28, pp. 1274–1282. JMLR Workshop and Conference Proceedings, May 2013.
  4. 4.
    Gharavi-Alkhansari, M.: A fast globally optimal algorithm for template matching using low-resolution pruning. IEEE Trans. Image Process. 10(4), 526–533 (2001)CrossRefzbMATHGoogle Scholar
  5. 5.
    Lan, T., Mori, G.: A max-margin riffled independence model for image tag ranking. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013Google Scholar
  6. 6.
    Manning, C.D., Raghavan, P., Schtze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)CrossRefzbMATHGoogle Scholar
  7. 7.
    Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. In: Rosin, P.L., Marshall, A.D. (eds.) BMVC. British Machine Vision Association (2002)Google Scholar
  8. 8.
    Mikolajczyk, K., Schmid, C.: Scale & affine invariant interest point detectors. Int. J. Comput. Vis. 60(1), 63–86 (2004)CrossRefGoogle Scholar
  9. 9.
    Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: Proceedings of CVPR (2006)Google Scholar
  10. 10.
    Ouyang, W., Tombari, F., Mattoccia, S., Di Stefano, L., Cham, W.K.: Performance evaluation of full search equivalent pattern matching algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 34(1), 127–143 (2012)CrossRefGoogle Scholar
  11. 11.
  12. 12.
    Peleg, S., Werman, M., Rom, H.: A unified approach to the change of resolution: space and gray-level. IEEE Trans. Pattern Anal. Mach. Intell. 11(7), 739–742 (1989)CrossRefGoogle Scholar
  13. 13.
    Perd’och, M., Chum, O., Matas, J.: Efficient representation of local geometry for large scale object retrieval. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 9–16, June 2009Google Scholar
  14. 14.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of CVPR (2007)Google Scholar
  15. 15.
    Philbin, J., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: improving particular object retrieval in large scale image databases. In: Proceedings of CVPR (2008)Google Scholar
  16. 16.
    Rosenfeld, A., Vanderburg, G.J.: Coarse-fine template matching. IEEE Trans. Syst. Man Cybern. 7(2), 104–107 (1977)CrossRefGoogle Scholar
  17. 17.
    Rubner, Y., Tomasi, C., Guibas, L.: The earth mover’s distance as a metric for image retrieval. Int. J. Comput. Vis. 40(2), 99–121 (2000)CrossRefzbMATHGoogle Scholar
  18. 18.
    Schindler, G., Brown, M., Szeliski, R.: City-scale location recognition. In: Proceedings of CVPR (2007)Google Scholar
  19. 19.
    Shen, H.C., Wong, A.K.: Generalized texture representation and metric. Comput. Vis. Graph. Image Process. 23(2), 187–206 (1983). MathSciNetCrossRefGoogle Scholar
  20. 20.
    Sivic, J., Zisserman, A.: Video google: a text retrieval approach to object matching in videos. In: Proceedings of ICCV (2003)Google Scholar
  21. 21.
    Tuytelaars, T., Gool, L.V.: Content-based image retrieval based on local affinely invariant regions. In: International Conference on Visual Information Systems, pp. 493–500 (1999)Google Scholar
  22. 22.
    Tuytelaars, T., Gool, L.V.: Wide baseline stereo matching based on local, affinely invariant regions. In: Proceedings of BMVC, pp. 412–425 (2000)Google Scholar
  23. 23.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: 2001 Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, pp. I-511-I-518 (2001)Google Scholar
  24. 24.
    Werman, M., Peleg, S., Rosenfeld, A.: A distance metric for multidimensional histograms. Comput. Visi. Graph. Image Process. 32(3), 328–336 (1985). CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Bao Truong
    • 1
    Email author
  • Thuyen V. Phan
    • 1
  • Vinh-Tiep Nguyen
    • 1
  • Minh-Triet Tran
    • 1
  1. 1.Faculty of Information TechnologyUniversity of Science, VNU - HCMHo Chi Minh CityVietnam

Personalised recommendations