Skip to main content
SpringerLink
Log in

Efficient interactive search for geo-tagged multimedia data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Due to the advances in mobile computing and multimedia techniques, there are vast amount of multimedia data with geographical information collected in multifarious applications. In this paper, we propose a novel type of image search namedinteractive geo-tagged image search which aims to find out a set of images based on geographical proximity and similarity of visual content, as well as the preference of users. Existing approaches for spatial keyword query and geo-image query cannot address this problem effectively since they do not consider these three type of information together for query. In order to solve this challenge efficiently, we propose the definition of interactive top-k geo-tagged image query and then present a framework including candidate search stage , interaction stage and termination stage. To enhance the searching efficiency in a large-scale database, we propose the candidate search algorithm named GI-SUPER Search based on a new notion called superior relationship and GIR-Tree, a novel index structure. Furthermore, two candidate selection methods are proposed for learning the preferences of the user during the interaction. At last, the termination procedure and estimation procedure are introduced in brief. Experimental evaluation on real multimedia dataset demonstrates that our solution has a really high performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://facebook.com/

  2. http://www.twitter.com/

  3. http://www.flickr.com/

  4. http://instagram.com/

  5. https://weixin.qq.com/

References

  1. Andersen R, Chellapilla K (2009) Finding dense subgraphs with size bounds. In: Algorithms and models for the web-graph, 6th international workshop, WAW 2009. Barcelona, Spain, February 12-13, 2009. Proceedings, pp 25–37

  2. Goldberg AV (1984) Finding a maximum density subgraph. University of California, Berkeley

    Google Scholar 

  3. Beckmann N, Kriegel H, Schneider R, Seeger B (1990) The r*-tree: an efficient and robust access method for points and rectangles. In: Proceedings of the 1990 ACM SIGMOD international conference on management of data. Atlantic City, NJ, May 23-25, 1990., pp 322–331

  4. Bianchi-Berthouze N (2003) K-DIME: an affective image filtering system. IEEE MultiMedia 10(3): 103–106

    Article  Google Scholar 

  5. Bȯrzsȯnyi S, Kossmann D, Stocker K (2001) The skyline operator. In: Proceedings of the 17th international conference on data engineering, April 2-6, 2001, Heidelberg, Germany, pp 421–430

  6. Cao X, Cong G, Jensen CS, Ooi BC (2011) Collective spatial keyword querying. In: Proceedings of the ACM SIGMOD international conference on management of data, SIGMOD 2011, Athens, Greece, June 12-16, 2011, pp 373–384

  7. Chen L, Cong G, Cao X, Tan K (2015) Temporal spatial-keyword top-k publish/subscribe. In: 31st IEEE International conference on data engineering, ICDE 2015, Seoul, South Korea, April 13-17, 2015, pp 255–266

  8. Chen J, Wang Y, Luo L, Yu J, Ma J (2016) Image retrieval based on image-to-class similarity. Pattern Recogn Lett 83:379–387

    Article  Google Scholar 

  9. Chum O, Philbin J, Zisserman A (2008) Near duplicate image detection: min-hash and tf-idf weighting. In: Proceedings of the British machine vision conference 2008, Leeds, September 2008, pp 1–10

  10. Cong G, Jensen CS, Wu D (2009) Efficient retrieval of the top-k most relevant spatial web objects. PVLDB 2(1):337–348

    Google Scholar 

  11. Deng J, Berg AC, Li F (2011) Hierarchical semantic indexing for large scale image retrieval. In: The 24th IEEE Conference on computer vision and pattern recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp 785–792

  12. Deng K, Li X, Lu J, Zhou X (2015) Best keyword cover search. IEEE Trans Knowl Data Eng 27(1):61–73

    Article  Google Scholar 

  13. Douze M, Ramisa A, Schmid C (2011) Combining attributes and fisher vectors for efficient image retrieval. In: The 24th IEEE Conference on computer vision and pattern recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp 745–752

  14. Felipe ID, Hristidis V, Rishe N (2008) Keyword search on spatial databases. In: Proceedings of the 24th international conference on data engineering, ICDE 2008, April 7-12, 2008, Cancu̇n, Mėxico, pp 656–665

  15. Gallo G, Grigoriadis MD, Tarjan RE (1989) A fast parametric maximum flow algorithm and applications. SIAM J Comput 18(1):30–55

    Article  MathSciNet  Google Scholar 

  16. Gosselin PH, Cord M (2008) Active learning methods for interactive image retrieval. IEEE Trans Image Process 17(7):1200–1211

    Article  MathSciNet  Google Scholar 

  17. Guo T, Cao X, Cong G (2015) Efficient algorithms for answering the m-closest keywords query. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, Melbourne, Victoria, Australia, May 31 - June 4, 2015, pp 405–418

  18. Guttman A (1984) R-trees: a dynamic index structure for spatial searching. In: SIGMOD’84, Proceedings of annual meeting, Boston, Massachusetts, June 18-21, 1984, pp 47–57

  19. Huang Y, Liu Q, Zhang S, Metaxas DN (2010) Image retrieval via probabilistic hypergraph ranking. In: The Twenty-Third IEEE conference on computer vision and pattern recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp 3376–3383

  20. Huang M, Liu A, Xiong N, Wang T, Vasilakos AV (2018) A low-latency communication scheme for mobile wireless sensor control systems. IEEE Trans Syst Man Cybern-Syst

  21. Ilyas IF, Beskales G, Soliman MA (2008) A survey of top-k query processing techniques in relational database systems. ACM Comput Surv 40(4):11:1–11:58

    Article  Google Scholar 

  22. Kamahara J, Nagamatsu T, Tanaka N (2012) Conjunctive ranking function using geographic distance and image distance for geotagged image retrieval. In: Proceedings of the ACM multimedia 2012 workshop on geotagging and its applications in multimedia, GeoMM@ACM Multimedia 2012. Nara, Japan, October 29, 2012, pp 9–14

  23. Kim G, Sigal L, Xing EP (2014) Joint summarization of large-scale collections of web images and videos for storyline reconstruction. In: 2014 IEEE Conference on computer vision and pattern recognition, CVPR 2014. Columbus, OH, USA, June 23-28, 2014, pp 4225–4232

  24. Kitanovski I, Strezoski G, Dimitrovski I, Madjarov G, Loskovska S (2017) Multimodal medical image retrieval system. Multimed Tools Appl 76(2):2955–2978

    Article  Google Scholar 

  25. Li Y, Zhang Y, Tao C, Zhu H (2016) Content-based high-resolution remote sensing image retrieval via unsupervised feature learning and collaborative affinity metric fusion. Remote Sens 8(9):709

    Article  Google Scholar 

  26. Li Y, Bie R, Zhang C, Miao Z, Wang Y, Wang J, Wu H (2017) Optimized learning instance-based image retrieval. Multimed Tools Appl 76 (15):16749–16766

    Article  Google Scholar 

  27. Liu J, Huang Z, Chen L, Shen HT, Yan Z (2012) Discovering areas of interest with geo-tagged images and check-ins. In: Proceedings of the 20th ACM multimedia conference, MM ’12, Nara, Japan, October 29 - November 02, 2012. pp 589–598. https://doi.org/10.1145/2393347.2393429

  28. Liu X, Liu Y, Liu A, Yang LT (2018) Defending on-off attacks using light probing messages in smart sensors for industrial communication systems. IEEE Trans Indus Inf

  29. Long C, Wong RC, Wang K, Fu AW (2013) Collective spatial keyword queries: a distance owner-driven approach. In: Proceedings of the ACM SIGMOD international conference on management of data, SIGMOD 2013. New York, NY, USA, June 22-27, 2013, pp 689–700

  30. Lowe DG (1999) Object recognition from local scale-invariant features. In: ICCV, pp 1150–1157

  31. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2): 91–110

    Article  Google Scholar 

  32. Memon MH, Li J, Memon I, Arain QA (2017) GEO matching regions: multiple regions of interests using content based image retrieval based on relative locations. Multimed Tools Appl 76(14):15377–15411. https://doi.org/10.1007/s11042-016-3834-z

    Article  Google Scholar 

  33. Rasiwasia N, Pereira JC, Coviello E, Doyle G, Lanckriet GRG, Levy R, Vasconcelos N (2010) A new approach to cross-modal multimedia retrieval. In: Proceedings of the 18th international conference on multimedia 2010. Firenze, Italy, October 25-29, 2010, pp 251–260. https://doi.org/10.1145/1873951.1873987

  34. Rocha-Junior JB, Gkorgkas O, Jonassen S, Nørvåg K (2011) Efficient processing of top-k spatial keyword queries. In: Advances in spatial and temporal databases - 12th international symposium, SSTD 2011. Minneapolis, MN, USA, August 24-26, 2011, Proceedings, pp 205–222

  35. Rocha-Junior JB, Nørvåg K (2012) Top-k spatial keyword queries on road networks. In: 15th International conference on extending database technology, EDBT ’12. Berlin, Germany, March 27-30, 2012, Proceedings, pp 168–179. https://doi.org/10.1145/2247596.2247617

  36. Singh S, Kumar P (2017) User specific context construction for personalized multimedia retrieval. In: Multimedia Tools Appl., vol 9, pp 1–28

  37. Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: 9th IEEE International conference on computer vision (ICCV 2003), 14-17 October 2003, Nice, France, pp 1470–1477

  38. Thomee B, Lew MS (2012) Interactive search in image retrieval: a survey. IJMIR 1(2):71–86

    Google Scholar 

  39. Wang Y, Wu L (2018) Beyond low-rank representations: orthogonal clustering basis reconstruction with optimized graph structure for multi-view spectral clustering. Neural Netw 103:1–8

    Article  Google Scholar 

  40. Wang Y, Cheema MA, Lin X, Zhang Q (2013) Multi-manifold ranking: using multiple features for better image retrieval. In: Advances in knowledge discovery and data mining, 17th Pacific-Asia conference, PAKDD 2013. Gold Coast, Australia, April 14-17, 2013, Proceedings, Part II, pp 449–460

    Chapter  Google Scholar 

  41. Wang Y, Lin X, Zhang Q (2013) Towards metric fusion on multi-view data: a cross-view based graph random walk approach. In: 22nd ACM International conference on information and knowledge management, CIKM’13. San Francisco, CA, USA, October 27 - November 1, 2013, pp 805–810

  42. Wang Y, Lin X, Zhang Q, Wu L (2014) Shifting hypergraphs by probabilistic voting. In: Advances in knowledge discovery and data mining - 18th Pacific-Asia Conference, PAKDD 2014. Tainan, Taiwan, May 13-16, 2014. Proceedings, Part II, pp 234–246

    Chapter  Google Scholar 

  43. Wang Y, Lin X, Wu L, Zhang W, Zhang Q (2014) Exploiting correlation consensus: towards subspace clustering for multi-modal data. In: Proceedings of the ACM international conference on multimedia, MM ’14. Orlando, FL, USA, November 03 - 07, 2014, pp 981–984

  44. Wang Y, Lin X, Wu L, Zhang W, Zhang Q (2015) LBMCH: learning bridging mapping for cross-modal hashing. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. Santiago, Chile, August 9-13, 2015, pp 999–1002

  45. Wang Y, Lin X, Wu L, Zhang W, Zhang Q, Huang X (2015) Robust subspace clustering for multi-view data by exploiting correlation consensus. IEEE Trans Image Process 24(11):3939–3949

    Article  MathSciNet  Google Scholar 

  46. Wang Y, Lin X, Wu L, Zhang W (2015) Effective multi-query expansions: robust landmark retrieval. In: Proceedings of the 23rd Annual ACM conference on multimedia conference, MM ’15. Brisbane, Australia, October 26 - 30, 2015, pp 79–88

  47. Wang Y, Zhang W, Wu L, Lin X, Fang M, Pan S (2016) Iterative views agreement: an iterative low-rank based structured optimization method to multi-view spectral clustering. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, IJCAI 2016. New York, NY, USA, 9-15 July 2016, pp 2153–2159

  48. Wang Y, Lin X, Wu L, Zhang W (2017) Effective multi-query expansions: collaborative deep networks for robust landmark retrieval. IEEE Trans Image Process 26(3):1393–1404

    Article  MathSciNet  Google Scholar 

  49. Wang Y, Zhang W, Wu L, Lin X, Zhao X (2017) Unsupervised metric fusion over multiview data by graph random walk-based cross-view diffusion. IEEE Trans Neural Netw Learn Syst 28(1):57–70

    Article  Google Scholar 

  50. Wang Y, Wu L, Lin X, Gao J (2018) Multiview spectral clustering via structured low-rank matrix factorization. IEEE Trans Neural Netw Learn Syst

  51. Wu L, Wang Y (2017) Robust hashing for multi-view data: jointly learning low-rank kernelized similarity consensus and hash functions. Image Vision Comput 57:58–66

    Article  Google Scholar 

  52. Wu L, Wang Y, Shepherd J (2013) Efficient image and tag co-ranking: a bregman divergence optimization method. In: ACM multimedia conference, MM ’13, Barcelona, Spain, October 21–25, 2013, pp 593–596

  53. Wu L, Huang X, Zhang C, Shepherd J, Wang Y (2015) An efficient framework of bregman divergence optimization for co-ranking images and tags in a heterogeneous network. Multimed Tools Appl 74(15):5635–5660

    Article  Google Scholar 

  54. Wu L, Wang Y, Gao J, Li X (2018) Deep adaptive feature embedding with local sample distributions for person re-identification. Pattern Recogn 73:275–288

    Article  Google Scholar 

  55. Wu L, Wang Y, Ge Z, Hu Q, Li X (2018) Structured deep hashing with convolutional neural networks for fast person re-identification. Comput Vis Image Underst 167:63–73

    Article  Google Scholar 

  56. Wu L, Wang Y, Li X, Gao J (2018) Deep attention-based spatially recursive networks for fine-grained visual recognition. IEEE Trans Cybern

  57. Wu L, Wang Y, Li X, Gao J (2018) What-and-where to match: deep spatially multiplicative integration networks for person re-identification. Pattern Recogn 76:727–738

    Article  Google Scholar 

  58. Xie Y, Yu H, Hu R (2014) Multimodal information joint learning for geotagged image search. In: 2013 IEEE International conference on multimedia and expo workshops. Chengdu, China, July 14–18, 2014, pp 1–6

  59. Yaegashi K, Yanai K (2010) Geotagged image recognition by combining three different kinds of geolocation features. In: Computer Vision - ACCV 2010 - 10th Asian conference on computer vision. Queenstown, New Zealand, November 8-12, 2010, Revised Selected Papers, Part II, pp 360–373

    Chapter  Google Scholar 

  60. Yang Y, Nie F, Xu D, Luo J, Zhuang Y, Pan Y (2012) A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Trans Pattern Anal Mach Intell 34(4):723–742

    Article  Google Scholar 

  61. Zhang D, Chee YM, Mondal A, Tung AKH, Kitsuregawa M (2009) Keyword search in spatial databases: towards searching by document. In: Proceedings of the 25th international conference on data engineering, ICDE 2009, March 29 2009 - April 2 2009, Shanghai, China, pp 688–699

  62. Zhang D, Tan K, Tung AKH (2013) Scalable top-k spatial keyword search. In: Joint 2013 EDBT/ICDT conferences, EDBT ’13 Proceedings. Genoa, Italy, March 18-22, 2013, pp 359–370

  63. Zhang C, Zhang Y, Zhang W, Lin X (2013) Inverted linear quadtree: efficient top k spatial keyword search. In: 29th IEEE International conference on data engineering, ICDE 2013. Brisbane, Australia, April 8-12, 2013, pp 901–912

  64. Zhang D, Chan C, Tan K (2014) Processing spatial keyword query as a top-k aggregation query. In: The 37th International ACM SIGIR conference on research and development in information retrieval, SIGIR ’14. Gold Coast, QLD, Australia - July 06 - 11, 2014, pp 355–364

  65. Zhang C, Zhang Y, Zhang W, Lin X (2016) Inverted linear quadtree: efficient top K spatial keyword search. IEEE Trans Knowl Data Eng 28(7):1706–1721

    Article  Google Scholar 

  66. Zhao S, Yao H, Yang Y, Zhang Y (2014) Affective image retrieval via multi-graph learning. In: Proceedings of the ACM international conference on multimedia, MM ’14. Orlando, FL, USA, November 03 - 07, 2014, pp 1025–1028

  67. Zhao P, Kuang X, Sheng VS, Xu J, Wu J, Cui Z (2015) Scalable top-k spatial image search on road networks. In: Database Systems for advanced applications - 20th international conference, DASFAA 2015. Hanoi, Vietnam, April 20-23, 2015, Proceedings, Part II, pp 379–396

    Chapter  Google Scholar 

  68. Zhu L, Shen J, Jin H, Zheng R, Xie L (2015) Content-based visual landmark search via multimodal hypergraph learning. IEEE Trans Cybern 45(12):2756–2769

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61702560, 61472450), the Key Research Program of Hunan Province(2016JC2018), project (2018JJ3691) of Science and Technology Plan of Hunan Province, and the Research and Innovation Project of Central South University Graduate Students (2018zzts177).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Central South University, Changsha, People’s Republic of China

    Jun Long, Lei Zhu, Chengyuan Zhang, Zhan Yang, Yunwu Lin & Ruipeng Chen

  2. Big Data and Knowledge Engineering Institute, Central South University, Changsha, People’s Republic of China

    Jun Long, Lei Zhu, Chengyuan Zhang, Zhan Yang, Yunwu Lin & Ruipeng Chen

Authors
  1. Jun Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengyuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yunwu Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ruipeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chengyuan Zhang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Long, J., Zhu, L., Zhang, C. et al. Efficient interactive search for geo-tagged multimedia data. Multimed Tools Appl 78, 30677–30706 (2019). https://doi.org/10.1007/s11042-018-6393-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6393-7

Keywords

Search

Navigation