Spatial Verification via Compact Words for Mobile Instance Search

  • Bo Wang
  • Jie ShaoEmail author
  • Chengkun He
  • Gang Hu
  • Xing Xu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10133)


Instance search is a retrieval task that searches video segments or images relevant to a certain specific instance (object, person, or location). Selecting more representative visual words is a significant challenge for the problem of instance search, since spatial relations between features are leveraged in many state-of-the-art methods. However, with the popularity of mobile devices it is now feasible to adopt multiple similar photos from mobile devices as a query to extract representative visual words. This paper proposes a novel approach for mobile instance search, by spatial analysis with a few representative visual words extracted from multi-photos. We develop a scheme that applies three criteria, including BM25 with exponential IDF (EBM25), significance in multi-photos and separability to rank visual words. Then, a spatial verification method about position relations is applied to a few visual words to obtain the weight of each photo selected. In consideration of the limited bandwidth and instability of wireless channel, our approach only transmits a few visual words from mobile client to server and the number of visual words varies with bandwidth. We evaluate our approach on Oxford building dataset, and the experimental results demonstrate a notable improvement on average precision over several state-of-the-art methods including spatial coding, query expansion and multiple photos.


Mobile instance search Multiple photos Spatial verification 



This work is supported by the National Nature Science Foundation of China (grants No. 61672133, No. 61602089, and No. 61632007), and the Fundamental Research Funds for the Central Universities (grants No. ZYGX2015J058 and No. ZYGX2014Z007).


  1. 1.
    Arandjelovic, R., Zisserman, A.: Multiple queries for large scale specific object retrieval. In: British Machine Vision Conference, BMVC 2012, Surrey, UK, 3–7 September 2012, pp. 1–11 (2012)Google Scholar
  2. 2.
    Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. 30(1–7), 107–117 (1998)Google Scholar
  3. 3.
    Chum, O., Mikulík, A., Perdoch, M., Matas, J.: Total recall II: query expansion revisited. In: The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011, pp. 889–896 (2011)Google Scholar
  4. 4.
    Chum, O., Philbin, J., Sivic, J., Isard, M., Zisserman, A.: Total recall: automatic query expansion with a generative feature model for object retrieval. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, 14–20 October 2007, pp. 1–8. IEEE Computer Society (2007)Google Scholar
  5. 5.
    Jegou, H., Douze, M., Schmid, C.: Hamming embedding and weak geometric consistency for large scale image search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 304–317. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88682-2_24 CrossRefGoogle Scholar
  6. 6.
    Jegou, H., Douze, M., Schmid, C., Pérez, P.: Aggregating local descriptors into a compact image representation. In: The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13–18 June 2010, pp. 3304–3311 (2010)Google Scholar
  7. 7.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  8. 8.
    Murata, M., Nagano, H., Mukai, R., Kashino, K., Satoh, S.: BM25 with exponential IDF for instance search. IEEE Trans. Multimed. 16(6), 1690–1699 (2014)CrossRefGoogle Scholar
  9. 9.
    Perronnin, F., Dance, C.R.: Fisher kernels on visual vocabularies for image categorization. In: 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), Minneapolis, Minnesota, USA, 18–23 June 2007 (2007)Google Scholar
  10. 10.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), Minneapolis, Minnesota, USA, 18–23 June 2007 (2007)Google Scholar
  11. 11.
    Robertson, S.E., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4), 333–389 (2009)CrossRefGoogle Scholar
  12. 12.
    Sivic, J., Zisserman, A.: Video Google: a text retrieval approach to object matching in videos. In: 9th IEEE International Conference on Computer Vision (ICCV 2003), Nice, France, 14–17 October 2003, pp. 1470–1477 (2003)Google Scholar
  13. 13.
    Tao, R., Gavves, E., Snoek, C.G.M., Smeulders, A.W.M.: Locality in generic instance search from one example. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, 23–28 June 2014, pp. 2099–2106 (2014)Google Scholar
  14. 14.
    Yang, X., Qian, X., Xue, Y.: Scalable mobile image retrieval by exploring contextual saliency. IEEE Trans. Image Process. 24(6), 1709–1721 (2015)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Zhang, W., Ngo, C.: Searching visual instances with topology checking and context modeling. In: International Conference on Multimedia Retrieval, ICMR 2013, Dallas, TX, USA, 16–19 April 2013, pp. 57–64 (2013)Google Scholar
  16. 16.
    Zhang, W., Ngo, C.: Topological spatial verification for instance search. IEEE Trans. Multimed. 17(8), 1236–1247 (2015)CrossRefGoogle Scholar
  17. 17.
    Zhang, Z., Albatal, R., Gurrin, C., Smeaton, A.F.: Instance search with weak geometric correlation consistency. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9516, pp. 226–237. Springer, Heidelberg (2016). doi: 10.1007/978-3-319-27671-7_19 CrossRefGoogle Scholar
  18. 18.
    Zhou, W., Lu, Y., Li, H., Song, Y., Tian, Q.: Spatial coding for large scale partial-duplicate web image search. In: Proceedings of the 18th International Conference on Multimedia 2010, Firenze, Italy, 25–29 October 2010, pp. 511–520 (2010)Google Scholar
  19. 19.
    Zhu, C., Satoh, S.: Large vocabulary quantization for searching instances from videos. In: International Conference on Multimedia Retrieval, ICMR 2012, Hong Kong, China, 5–8 June 2012, p. 52 (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Bo Wang
    • 1
  • Jie Shao
    • 1
    Email author
  • Chengkun He
    • 1
  • Gang Hu
    • 1
  • Xing Xu
    • 1
  1. 1.School of Computer Science and EngineeringUniversity of Electronic Science and Technology of ChinaChengduChina

Personalised recommendations