LOH and Behold: Web-Scale Visual Search, Recommendation and Clustering Using Locally Optimized Hashing

  • Yannis Kalantidis
  • Lyndon Kennedy
  • Huy Nguyen
  • Clayton Mellina
  • David A. Shamma
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9913)


We propose a novel hashing-based matching scheme, called Locally Optimized Hashing (LOH), based on a state-of-the-art quantization algorithm that can be used for efficient, large-scale search, recommendation, clustering, and deduplication. We show that matching with LOH only requires set intersections and summations to compute and so is easily implemented in generic distributed computing systems. We further show application of LOH to: (a) large-scale search tasks where performance is on par with other state-of-the-art hashing approaches; (b) large-scale recommendation where queries consisting of thousands of images can be used to generate accurate recommendations from collections of hundreds of millions of images; and (c) efficient clustering with a graph-based algorithm that can be scaled to massive collections in a distributed environment or can be used for deduplication for small collections, like search results, performing better than traditional hashing approaches while only requiring a few milliseconds to run. In this paper we experiment on datasets of up to 100 million images, but in practice our system can scale to larger collections and can be used for other types of data that have a vector representation in a Euclidean space.


Visual Search Semantic Concept Residual Vector Restricted Boltzmann Machine Fine Code 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Arandjelovic, R., Zisserman, A.: Multiple queries for large scale specific object retrieval. In: BMVC (2012)Google Scholar
  2. 2.
    Avrithis, Y., Kalantidis, Y., Anagnostopoulos, E., Emiris, I.Z.: Web-scale image clustering revisited. In: ICCV (2015)Google Scholar
  3. 3.
    Avrithis, Y., Kalantidis, Y., Tolias, G., Spyrou, E.: Retrieving landmark and non-landmark images from community photo collections. In: ACM Multimedia (2010)Google Scholar
  4. 4.
    Babenko, A., Lempitsky, V.: The inverted multi-index. In: CVPR (2012)Google Scholar
  5. 5.
    Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 584–599. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10590-1_38 Google Scholar
  6. 6.
    Datar, M., Immorlica, N., Indyk, P., Mirrokni, V.: Locality-sensitive hashing scheme based on p-stable distributions. In: Symposium on Computational Geometry (2004)Google Scholar
  7. 7.
    Dollár, P., Appel, R., Belongie, S., Perona, P.: Fast feature pyramids for object detection. PAMI 36(8), 1532–1545 (2014)CrossRefGoogle Scholar
  8. 8.
    Douze, M., Jégou, H., Perronnin, F.: Polysemous codes. In: ECCV (2016)Google Scholar
  9. 9.
    Fernando, B., Tuytelaars, T.: Mining multiple queries for image retrieval: on-the-fly learning of an object-specific mid-level representation. In: ICCV (2013)Google Scholar
  10. 10.
    Ge, T., He, K., Ke, Q., Sun, J.: Optimized product quantization. Technical report 4 (2014)Google Scholar
  11. 11.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)Google Scholar
  12. 12.
    Gong, Y., Lazebnik, S.: Iterative quantization: a procrustean approach to learning binary codes. In: CVPR (2011)Google Scholar
  13. 13.
    Gong, Y., Pawlowski, M., Yang, F., Brandy, L., Bourdev, L., Fergus, R.: Web scale photo hash clustering on a single machine. In: CVPR (2015)Google Scholar
  14. 14.
    Gordo, A., Perronnin, F., Gong, Y., Lazebnik, S.: Asymmetric distances for binary embeddings. PAMI 36(1), 33–47 (2014)CrossRefGoogle Scholar
  15. 15.
    Hsiao, K., Calder, J., Hero, A.O.: Pareto-depth for multiple-query image retrieval. arXiv preprint arXiv:1402.5176 (2014)
  16. 16.
    Jégou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. PAMI 33(1) (2011)Google Scholar
  17. 17.
    Jégou, H., Tavenard, R., Douze, M., Amsaleg, L.: Searching in one billion vectors: re-rank with source coding. In: ICASSP (2011)Google Scholar
  18. 18.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
  19. 19.
    Jin, Z., Hu, Y., Lin, Y., Zhang, D., Lin, S., Cai, D., Li, X.: Complementary projection hashing. In: ICCV (2013)Google Scholar
  20. 20.
    Kalantidis, Y., Tolias, G., Avrithis, Y., Phinikettos, M., Spyrou, E., Mylonas, P., Kollias, S.: Viral: visual image retrieval and localization. MTAP (2011)Google Scholar
  21. 21.
    Kalantidis, Y., Avrithis, Y.: Locally optimized product quantization for approximate nearest neighbor search. In: CVPR (2014)Google Scholar
  22. 22.
    Kennedy, L., Naaman, M., Ahern, S., Nair, R., Rattenbury, T.: How flickr helps us makesense of the world: context and content in community-contributed media collections. In: ACM Multimedia, vol. 3, pp. 631–640 (2007)Google Scholar
  23. 23.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  24. 24.
    Norouzi, M., Fleet, D.: Cartesian \(k\)-means. In: CVPR (2013)Google Scholar
  25. 25.
    Norouzi, M., Punjani, A., Fleet, D.J.: Fast search in hamming space with multi-index hashing. In: CVPR (2012)Google Scholar
  26. 26.
    Paulevé, L., Jégou, H., Amsaleg, L.: Locality sensitive hashing: a comparison of hash function types and querying mechanisms. Pattern Recogn. Lett. 31(11), 1348–1358 (2010)CrossRefGoogle Scholar
  27. 27.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575 (2014)
  28. 28.
    Shen, F., Shen, C., Liu, W., Shen, H.T.: Supervised discrete hashing. In: CVPR (2015)Google Scholar
  29. 29.
    Thomee, B., Elizalde, B., Shamma, D.A., Ni, K., Friedland, G., Poland, D., Borth, D., Li, L.J.: Yfcc100m: the new data in multimedia research. Commun. ACM 59(2), 64–73 (2016)CrossRefGoogle Scholar
  30. 30.
    Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., Li, L.J.: The new data and new challenges in multimedia research (2015). arXiv preprint arXiv:1503.01817
  31. 31.
    Tolias, G., Avrithis, Y., Jégou, H.: Image search with selective match kernels: aggregation across single and multiple images. Int. J. Comput. Vision, 1–15 (2015)Google Scholar
  32. 32.
    Torralba, A., Fergus, R., Weiss, Y.: Small codes and large image databases for recognition. In: CVPR (2008)Google Scholar
  33. 33.
    Wang, J., Shen, H.T., Yan, S., Yu, N., Li, S., Wang, J.: Optimized distances for binary code ranking. In: Proceedings of the ACM International Conference on Multimedia, pp. 517–526. ACM (2014)Google Scholar
  34. 34.
    Wang, J., Kumar, S., Chang, S.F.: Sequential projection learning for hashing with compact codes. In: ICML (2010)Google Scholar
  35. 35.
    Weiss, Y., Torralba, A., Fergus, R.: Spectral hashing. In: NIPS (2008)Google Scholar
  36. 36.
    Xu, B., Bu, J., Chen, C., Cai, D., He, X., Liu, W., Luo, J.: Efficient manifold ranking for image retrieval. In: SIGIR (2011)Google Scholar
  37. 37.
    Zhang, L., Zhang, Y., Tang, J., Lu, K., Tian, Q.: Binary code ranking with weighted hamming distance. In: CVPR (2013)Google Scholar
  38. 38.
    Zheng, Y., Zhao, M., Song, Y., Adam, H., Buddemeier, U., Bissacco, A., Brucher, F., Chua, T.S., Neven, H.: Tour the world: building a web-scale landmark recognition engine. In: CVPR (2009)Google Scholar
  39. 39.
    Zhu, C.Z., Huang, Y.H., Satoh, S.: Multi-image aggregation for better visual object retrieval. In: ICASSP (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Yannis Kalantidis
    • 1
  • Lyndon Kennedy
    • 2
  • Huy Nguyen
    • 1
  • Clayton Mellina
    • 1
  • David A. Shamma
    • 3
  1. 1.Computer Vision and Machine Learning GroupFlickr, YahooSan FranciscoUSA
  2. 2.Futurewei Technologies Inc.Santa ClaraUSA
  3. 3.CWI: Centrum Wiskunde & InformaticaAmsterdamThe Netherlands

Personalised recommendations