Instance search retrospective with focus on TRECVID

  • George Awad
  • Wessel Kraaij
  • Paul Over
  • Shin’ichi Satoh
Trends and Surveys

Abstract

This paper presents an overview of the Video instance search benchmark which was run over a period of 6 years (2010–2015) as part of the TREC Video Retrieval workshop series. The main contributions of the paper include (i) an examination of the evolving design of the evaluation framework and its components (system tasks, data, measures); (ii) an analysis of the influence of topic characteristics (such as rigid/non-rigid, planar/non-planar, stationary/mobile on performance; (iii) a high-level overview of results and best-performing approaches. The instance search benchmark worked with a variety of large collections of data including Sound & Vision, Flickr, BBC Rushes for the first three pilot years and with the small world of the BBC EastEnders series for the last 3 years.

Keywords

Instance search Multimedia Evaluation TRECVID 

References

  1. 1.
    Arandjelovic O, Zisserman A (2005) Automatic face recognition for film character retrieval in feature-length films. In: IEEE computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, vol 1, pp 860–867. doi:10.1109/CVPR.2005.81
  2. 2.
    Arandjelovic R, Zisserman A (2012) Multiple queries for large scale specific object retrieval. In: Bowden R, Collomosse JP, Mikolajczyk K (eds) British machine vision conference, BMVC 2012, Surrey, UK, 3–7 Sept 2012. BMVA Press, pp 1–11. doi:10.5244/C.26.92
  3. 3.
    Arandjelovic R, Zisserman A (2012) Three things everyone should know to improve object retrieval. In: 2012 IEEE conference on computer vision and pattern recognition, Providence, RI, USA, 16–21 June 2012. IEEE Computer Society, pp 2911–2918. doi:10.1109/CVPR.2012.6248018
  4. 4.
    Babenko A, Lempitsky VS (2015) Aggregating local deep features for image retrieval. In: 2015 IEEE international conference on computer vision, ICCV 2015, Santiago, Chile, 7–13 Dec 2015. IEEE Computer Society, pp 1269–1277. doi:10.1109/ICCV.2015.150
  5. 5.
    Babenko A, Slesarev A, Chigorin A, Lempitsky VS (2014) Neural codes for image retrieval. In: Fleet DJ, Pajdla T, Schiele B, Tuytelaars T (eds) 13th European conference computer vision—ECCV 2014, Zurich, Switzerland, 6–12 Sept 2014, Proceedings, Part I. Lecture notes in computer science, vol 8689. Springer, pp 584–599. doi:10.1007/978-3-319-10590-1_38
  6. 6.
    Bay H, Ess A, Tuytelaars T, Gool LJV (2008) Speeded-up robust features (SURF). Comput Vis Image Underst 110(3):346–359. doi:10.1016/j.cviu.2007.09.014 CrossRefGoogle Scholar
  7. 7.
    Blog GM (2010) Open your eyes: Google Goggles now available on iPhone in Google Mobile App. http://googlemobile.blogspot.com/2010/10/open-your-eyes-google-goggles-now.html
  8. 8.
    Broder A (1997) On the resemblance and containment of documents. In: Proceedings of compression and complexity of sequences 1997. pp 21–29. doi:10.1109/SEQUEN.1997.666900
  9. 9.
    Chandrasekhar V, Chen DM, Tsai SS, Cheung N, Chen H, Takacs G, Reznik YA, Vedantham R, Grzeszczuk R, Bach J, Girod B (2011) The stanford mobile visual search data set. In Begen AC, Mayer-Patel K (eds) Proceedings of the second annual ACM SIGMM conference on multimedia systems, MMSys 2011, Santa Clara, CA, USA 23–25 Feb 2011. ACM, pp 117–122. doi:10.1145/1943552.1943568
  10. 10.
    Chum O, Perdoch M, Matas J (2009) Geometric min-hashing: finding a (thick) needle in a haystack. In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA. IEEE Computer Society, pp 17–24. doi:10.1109/CVPRW.2009.5206531
  11. 11.
    Chum O, Philbin J, Sivic J, Isard M, Zisserman A (2007) Total recall: automatic query expansion with a generative feature model for object retrieval. In: IEEE 11th international conference on computer vision, ICCV 2007, Rio de Janeiro, Brazil, 14–20 Oct 2007. IEEE, pp 1–8. doi:10.1109/ICCV.2007.4408891
  12. 12.
    Chum O, Philbin J, Zisserman A (2008) Near duplicate image detection: min-hash and tf-idf weighting. In Everingham M, Needham CJ, Fraile R (eds) Proceedings of the British machine vision conference 2008, Leeds, Sept 2008. British Machine Vision Association, pp 1–10. doi:10.5244/C.22.50
  13. 13.
    Csurka G, Dance C, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. Workshop on statistical learning in computer vision, ECCV, vol 1, no 1–22, pp 1–2Google Scholar
  14. 14.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 2005), 20–26 June 2005, San Diego, CA, USA. IEEE Computer Society, pp 886–893. doi:10.1109/CVPR.2005.177
  15. 15.
    Davis WR, Kosicki BB, Boroson DM, Kostishack D (1996) Micro air vehicles for optical surveillance. Linc Lab J 9(2):197–214Google Scholar
  16. 16.
    Deng J, Dong W, Socher R, Li L, Li K, Li F (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA. IEEE Computer Society, pp 248–255. doi:10.1109/CVPRW.2009.5206848
  17. 17.
    Everingham M, Eslami SMA, Gool LV, Williams CKI, Winn JM, Zisserman A (2015) The pascal visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136. doi:10.1007/s11263-014-0733-5 CrossRefGoogle Scholar
  18. 18.
    Everingham M, Sivic J, Zisserman A (2006) “Hello! My name is... Buffy”—automatic naming of characters in TV video. In: Proceedings of the British machine vision conferenceGoogle Scholar
  19. 19.
    Felzenszwalb PF, Girshick RB, McAllester DA, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645. doi:10.1109/TPAMI.2009.167 CrossRefGoogle Scholar
  20. 20.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395. doi:10.1145/358669.358692 MathSciNetCrossRefGoogle Scholar
  21. 21.
    Girshick RB, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE conference on computer vision and pattern recognition, CVPR 2014, Columbus, OH, USA, 23–28 June 2014. IEEE Computer Society, pp 580–587. doi:10.1109/CVPR.2014.81
  22. 22.
    Gross R, Matthews I, Cohn JF, Kanade T, Baker S (2010) Multi-pie. Image Vis Comput 28(5):807–813. doi:10.1016/j.imavis.2009.08.002 CrossRefGoogle Scholar
  23. 23.
    Harris C, Stephens M (1988) A combined corner and edge detector. In: Taylor CJ (ed) Proceedings of the Alvey vision conference, AVC 1988, Manchester, UK, Sept 1988. Alvey Vision Club, pp. 1–6. doi:10.5244/C.2.23
  24. 24.
    Hauptmann A, Smith M (1995) Text, speech, and vision for video segmentation: the informedia tm project. In: Proceeding of AAAI fall symposium computational models for integrating language and vision, Boston. CiteseerGoogle Scholar
  25. 25.
    Huang GB, Jain V, Learned-Miller E (2007) Unsupervised joint alignment of complex images. In: Computer Vision 2007. ICCV 2007. 11th International conference on IEEE, pp 1–8. IEEEGoogle Scholar
  26. 26.
    Huang J, Kumar R, Mitra M, Zhu W, Zabih R (1997) Image indexing using color correlograms. In: 1997 conference on computer vision and pattern recognition (CVPR ’97), 17–19 June 1997, San Juan, Puerto Rico. IEEE Computer Society, pp 762–768. doi:10.1109/CVPR.1997.609412
  27. 27.
    Jain M, van Gemert JC, Jégou H, Bouthemy P, Snoek CGM (2014) Action localization with tubelets from motion. In: 2014 IEEE conference on computer vision and pattern recognition, CVPR 2014, Columbus, OH, USA, 23–28 June 2014. IEEE Computer Society, pp 740–747. doi:10.1109/CVPR.2014.100
  28. 28.
    Jegou H, Douze M, Schmid C (2008) Hamming embedding and weak geometric consistency for large scale image search. In: Forsyth DA, Torr PHS, Zisserman A (eds) 10th European conference on computer vision—ECCV 2008, Marseille, France, 12–18 October 2008, Proceedings, Part I. Lecture notes in computer science, vol 5302. Springer, pp 304–317. doi:10.1007/978-3-540-88682-2_24
  29. 29.
    Jégou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128. doi:10.1109/TPAMI.2010.57 CrossRefGoogle Scholar
  30. 30.
    Jégou H, Perronnin F, Douze M, Sánchez J, Pérez P, Schmid C (2012) Aggregating local image descriptors into compact codes. IEEE Trans Pattern Anal Mach Intell 34(9):1704–1716. doi:10.1109/TPAMI.2011.235 CrossRefGoogle Scholar
  31. 31.
    Joly A, Buisson O (2009) Logo retrieval with a contrario visual query expansion. In: Proceedings of the 17th ACM international conference on multimedia, MM ’09. ACM, New York, NY, USA, pp 581–584. doi:10.1145/1631272.1631361
  32. 32.
    Kristan M, Matas J, Leonardis A, Vojíř T, Pflugfelder R, Fernandez G, Nebehay G, Porikli F, Čehovin L (2016) A novel performance evaluation methodology for single-target trackers. IEEE Trans Pattern Anal Mach Intell 38(11):2137–2155Google Scholar
  33. 33.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Bartlett PL, Pereira FCN, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25: 26th annual conference on neural information processing systems 2012. Proceedings of a meeting held, 3–6 December 2012, Lake Tahoe, Nevada, USA, pp 1106–1114. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks
  34. 34.
    Le QV, Ranzato M, Monga R, Devin M, Corrado G, Chen K, Dean J, Ng AY (2012) Building high-level features using large scale unsupervised learning. In: Proceedings of the 29th international conference on machine learning, ICML 2012, Edinburgh, Scotland, UK, 26 June–1 July 2012. icml.cc/Omnipress. http://icml.cc/discuss/2012/73.html
  35. 35.
    Lebeda K, Matas J, Chum O (2012) Fixing the locally optimized RANSAC. In: Bowden R, Collomosse JP, Mikolajczyk K (eds) British machine vision conference, BMVC 2012, Surrey, UK, 3–7 September 2012. BMVA Press, pp 1–11. doi:10.5244/C.26.95
  36. 36.
    Letessier P, Buisson O, Joly A (2012) Scalable mining of small visual objects. In: Babaguchi N, Aizawa K, Smith JR, Satoh S, Plagemann T, Hua X, Yan R (eds) Proceedings of the 20th ACM multimedia conference, MM ’12, Nara, Japan, 29 October–02 November 2012. ACM, pp 599–608. doi:10.1145/2393347.2393431
  37. 37.
    Liu Z, Li H, Zhou W, Tian Q (2012) Embedding spatial context information into inverted file for large-scale image retrieval. In: Babaguchi N, Aizawa K, Smith JR, Satoh S, Plagemann T, Hua X, Yan R (eds) Proceedings of the 20th ACM multimedia conference, MM ’12, Nara, Japan, 29 October–02 November 2012. ACM, pp 199–208. doi:10.1145/2393347.2393380
  38. 38.
    Lloyd SP (1982) Least squares quantization in PCM. IEEE Trans Inf Theory 28(2):129–136. doi:10.1109/TIT.1982.1056489 MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110. doi:10.1023/B:VISI.0000029664.99615.94 CrossRefGoogle Scholar
  40. 40.
    Matas J, Chum O, Urban M, Pajdla T (2004) Robust wide-baseline stereo from maximally stable extremal regions. Image Vis Comput 22(10):761–767. doi:10.1016/j.imavis.2004.02.006 CrossRefGoogle Scholar
  41. 41.
    Mikolajczyk K, Schmid C (2004) Scale & affine invariant interest point detectors. Int J Comput Vis 60(1):63–86. doi:10.1023/B:VISI.0000027790.02288.f2 CrossRefGoogle Scholar
  42. 42.
    Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10):1615–1630. doi:10.1109/TPAMI.2005.188 CrossRefGoogle Scholar
  43. 43.
    Mikolajczyk K, Tuytelaars T, Schmid C, Zisserman A, Matas J, Schaffalitzky F, Kadir T, Gool LJV (2005) A comparison of affine region detectors. Int J Comput Vis 65(1–2):43–72. doi:10.1007/s11263-005-3848-x CrossRefGoogle Scholar
  44. 44.
    Muja M, Lowe DG (2014) Scalable nearest neighbor algorithms for high dimensional data. IEEE Trans Pattern Anal Mach Intell 36(11):2227–2240. doi:10.1109/TPAMI.2014.2321376 CrossRefGoogle Scholar
  45. 45.
    Murase H, Nayar SK (1995) Visual learning and recognition of 3-d objects from appearance. Int J Comput Vis 14(1):5–24. doi:10.1007/BF01421486 CrossRefGoogle Scholar
  46. 46.
    Murata M, Nagano H, Mukai R, Kashino K, Satoh S (2014) BM25 with exponential IDF for instance search. IEEE Trans Multimed 16(6):1690–1699. doi:10.1109/TMM.2014.2323945 CrossRefGoogle Scholar
  47. 47.
    Niblack CW, Barber R, Equitz W, Flickner MD, Glasman EH, Petkovic D, Yanker P, Faloutsos C, Taubin G (1993) QBIC project: querying images by content, using color, texture, and shape. In: IS&T/SPIE’s symposium on electronic imaging: science and technology. International Society for Optics and Photonics, pp 173–187Google Scholar
  48. 48.
    Nistér D, Stewénius H (2006) Scalable recognition with a vocabulary tree. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 2006), 17–22 June 2006, New York, NY, USA. IEEE Computer Society, pp 2161–2168. doi:10.1109/CVPR.2006.264
  49. 49.
    Nowak E, Jurie F, Triggs B (2006) Sampling strategies for bag-of-features image classification. In: Leonardis A, Bischof H, Pinz A (eds) 9th European conference on computer vision—ECCV 2006, Graz, Austria, 7–13 May 2006, Proceedings, Part IV. Lecture notes in computer science, vol 3954. Springer, pp 490–503. doi:10.1007/11744085_38
  50. 50.
    Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recognit 29(1):51–59. doi:10.1016/0031-3203(95)00067-4 CrossRefGoogle Scholar
  51. 51.
    Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis 42(3):145–175. doi:10.1023/A:1011139631724 CrossRefMATHGoogle Scholar
  52. 52.
    Perronnin F, Sánchez J, Mensink T (2010) Improving the fisher kernel for large-scale image classification. In: Daniilidis K, Maragos P, Paragios N (eds) 11th European conference on computer vision—ECCV 2010, Heraklion, Crete, Greece, 5–11 September 2010, Proceedings, Part IV. Lecture notes in computer science, vol 6314. Springer, pp 143–156. doi:10.1007/978-3-642-15561-1_11
  53. 53.
    Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR 2007), 18–23 June 2007, Minneapolis, MN, USA. IEEE Computer Society. doi:10.1109/CVPR.2007.383172
  54. 54.
    Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2008) Lost in quantization: improving particular object retrieval in large scale image databases. In: 2008 IEEE computer society conference on computer vision and pattern recognition (CVPR 2008), 24-26 June 2008, Anchorage, Alaska, USA. IEEE Computer Society. doi:10.1109/CVPR.2008.4587635
  55. 55.
    Phillips PJ, Moon H, Rizvi SA, Rauss PJ (2000) The FERET evaluation methodology for face-recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104. doi:10.1109/34.879790 CrossRefGoogle Scholar
  56. 56.
    Rabinovich A, Vedaldi A, Galleguillos C, Wiewiora E, Belongie S (2007) Objects in context. In: IEEE 11th international conference on computer vision, ICCV 2007, Rio de Janeiro, Brazil, 14–20 October 2007. IEEE, pp 1–8. doi:10.1109/ICCV.2007.4408986
  57. 57.
    Razavian AS, Sullivan J, Carlsson S, Maki A (2016) Visual instance retrieval with deep convolutional networks. ITE Trans Media Technol Appl 4(3):251–258CrossRefGoogle Scholar
  58. 58.
    Romberg S, Lienhart R (2013) Bundle min-hashing. IJMIR 2(4):243–259. doi:10.1007/s13735-013-0040-x Google Scholar
  59. 59.
    Romberg S, Pueyo LG, Lienhart R, van Zwol R (2011) Scalable logo recognition in real-world images. In: Natale FGBD, Bimbo AD, Hanjalic A, Manjunath BS, Satoh S (eds.) Proceedings of the 1st international conference on multimedia retrieval, ICMR 2011, Trento, Italy, 18–20 April 2011. ACM, p 25. doi:10.1145/1991996.1992021
  60. 60.
    Schmid C, Mohr R, Bauckhage C (2000) Evaluation of interest point detectors. Int J Comput Vis 37(2):151–172. doi:10.1023/A:1008199403446 CrossRefMATHGoogle Scholar
  61. 61.
    Sivic J, Everingham M, Zisserman A (2005) Person spotting: video shot retrieval for face sets. In: Proceedings of the ACM international conference on image and video retrieval. http://www.robots.ox.ac.uk/~vgg
  62. 62.
    Sivic J, Zisserman A (2003) Video Google: a text retrieval approach to object matching in videos. In: Proceedings of ninth IEEE international conference on computer vision, 2003, vol 2, pp. 1470–1477. doi:10.1109/ICCV.2003.1238663
  63. 63.
    Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: 9th IEEE international conference on computer vision (ICCV 2003), 14–17 Oct 2003, Nice, France. IEEE Computer Society, pp 1470–1477. doi:10.1109/ICCV.2003.1238663
  64. 64.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380. doi:10.1109/34.895972 CrossRefGoogle Scholar
  65. 65.
    Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1):11–32. doi:10.1007/BF00130487 CrossRefGoogle Scholar
  66. 66.
    Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the 2014 IEEE conference on computer vision and pattern recognition, CVPR ’14. IEEE Computer Society, Washington, DC, USA, pp 1701–1708. doi:10.1109/CVPR.2014.220
  67. 67.
    Tolias G, Sicre R, Jégou H (2016) Particular object retrieval with integral max-pooling of cnn activations. In: Proceedings of international conference on learning representationsGoogle Scholar
  68. 68.
  69. 69.
    TRECVID (2001) Trec video retrieval evaluation notebook papers and slides. http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html
  70. 70.
    Uijlings JRR, Smeulders AWM., Scha RJH (2009) What is the spatial extent of an object? In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA. IEEE Computer Society, pp 770–777. doi:10.1109/CVPRW.2009.5206663
  71. 71.
    van Gemert J, Geusebroek J, Veenman CJ, Smeulders AWM (2008) Kernel codebooks for scene categorization. In: Forsyth DA, Torr PHS, Zisserman A (eds) 10th European conference on computer vision—ECCV 2008, Marseille, France, 12–18 Oct 2008. Proceedings, Part III. Lecture notes in computer science, vol 5304. Springer, pp 696–709. doi:10.1007/978-3-540-88690-7_52
  72. 72.
    van de Sande KEA, Gevers T, Snoek CGM (2010) Evaluating color descriptors for object and scene recognition. IEEE Trans Pattern Anal Mach Intell 32(9):1582–1596. doi:10.1109/TPAMI.2009.154 CrossRefGoogle Scholar
  73. 73.
    Wu Z, Ke Q, Isard M, Sun J (2009) Bundling features for large scale partial-duplicate web image search. In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA. IEEE Computer Society, pp 25–32. doi:10.1109/CVPRW.2009.5206566
  74. 74.
    Yang J, Yu K, Gong Y, Huang TS (2009) Linear spatial pyramid matching using sparse coding for image classification. In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA. IEEE Computer Society, pp 1794–1801. doi:10.1109/CVPRW.2009.5206757
  75. 75.
    Zhang J, Marszalek M, Lazebnik S, Schmid C (2007) Local features and kernels for classification of texture and object categories: a comprehensive study. Int J Comput Vis 73(2):213–238. doi:10.1007/s11263-006-9794-4 CrossRefGoogle Scholar
  76. 76.
    Zhang W, Ngo C (2015) Topological spatial verification for instance search. IEEE Trans Multimed 17(8):1236–1247. doi:10.1109/TMM.2015.2440997 CrossRefGoogle Scholar
  77. 77.
    Zhang Y, Jia Z, Chen T (2011) Image retrieval with geometry-preserving visual phrases. In: The 24th IEEE conference on computer vision and pattern recognition, CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011. IEEE Computer Society, pp 809–816. doi:10.1109/CVPR.2011.5995528
  78. 78.
    Zhao WL, Ngo CW, Wang H (2016) Fast covariant vlad for image search. IEEE Trans Multimed 18(9):1843–1854. doi:10.1109/TMM.2016.2585023
  79. 79.
    Zhu C, Huang Y, Satoh S (2014) Multi-image aggregation for better visual object retrieval. In: IEEE international conference on acoustics, speech and signal processing, ICASSP 2014, Florence, Italy, May 4–9, 2014. IEEE, pp 4304–4308. doi:10.1109/ICASSP.2014.6854414
  80. 80.
    Zhu C, Satoh S (2012) Large vocabulary quantization for searching instances from videos. In: Ip HH, Rui Y (eds) International conference on multimedia retrieval, ICMR ’12, Hong Kong, China, 5–8 June 2012. ACM, p 52. doi:10.1145/2324796.2324856
  81. 81.
    Zhu C, Satoh S (2013) Evaluation of visual object retrieval datasets. In: IEEE international conference on image processing, ICIP 2013, Melbourne, Australia, 15–18 Sept 2013. IEEE, pp 3954–3958. doi:10.1109/ICIP.2013.6738814
  82. 82.
    Zhu CZ, Jégou H, Ichi Satoh S (2013) Query-adaptive asymmetrical dissimilarities for visual object retrieval. In: Proceedings of the IEEE international conference on computer vision, pp 1705–1712Google Scholar
  83. 83.
    Zhu CZ, Jégou H, Satoh S (2013) Query-adaptive asymmetrical dissimilarities for visual object retrieval. In: The IEEE international conference on computer vision (ICCV)Google Scholar
  84. 84.
    Zobel J, Moffat A (2006) Inverted files for text search engines. ACM Comput Surv. doi:10.1145/1132956.1132959 Google Scholar

Copyright information

© Springer-Verlag London (outside the USA) 2017

Authors and Affiliations

  1. 1.Dakota Consulting, Inc.Silver SpringUSA
  2. 2.National Institute of Standards and TechnologyGaithersburgUSA
  3. 3.TNOThe HagueThe Netherlands
  4. 4.Leiden UniversityLeidenThe Netherlands
  5. 5.National Institute of InformaticsTokyoJapan

Personalised recommendations