Multimedia Tools and Applications

, Volume 77, Issue 11, pp 13853–13889 | Cite as

Extracting semantic knowledge from web context for multimedia IR: a taxonomy, survey and challenges

  • Teresa Bracamonte
  • Benjamin Bustos
  • Barbara Poblete
  • Tobias Schreck


Since its invention, the Web has evolved into the largest multimedia repository that has ever existed. This evolution is a direct result of the explosion of user-generated content, explained by the wide adoption of social network platforms. The vast amount of multimedia content requires effective management and retrieval techniques. Nevertheless, Web multimedia retrieval is a complex task because users commonly express their information needs in semantic terms, but expect multimedia content in return. This dissociation between semantics and content of multimedia is known as the semantic gap. To solve this, researchers are looking beyond content-based or text-based approaches, integrating novel data sources. New data sources can consist of any type of data extracted from the context of multimedia documents, defined as the data that is not part of the raw content of a multimedia file. The Web is an extraordinary source of context data, which can be found in explicit or implicit relation to multimedia objects, such as surrounding text, tags, hyperlinks, and even in relevance-feedback. Recent advances in Web multimedia retrieval have shown that context data has great potential to bridge the semantic gap. In this article, we present the first comprehensive survey of context-based approaches for multimedia information retrieval on the Web. We introduce a data-driven taxonomy, which we then use in our literature review of the most emblematic and important approaches that use context-based data. In addition, we identify important challenges and opportunities, which had not been previously addressed in this area.


Semantic knowledge extraction Multimedia retrieval Web retrieval Context data Big data 



This work was partially supported by the Millennium Nucleus Center for Semantic Web Research, Grant No. NC120004. In addition, B. Poblete was also partially supported by Project Enlace-Fondecyt ENL011/16 and Project Fondef ID16—10222. T. Bracamonte was also supported by PhD Scholarship Program of Conicyt, Chile (CONICYT-PCHA/Doctorado Nacional/2013-63130260).


  1. 1.
    Blanken HM, de Vries AP, Blok HE, Feng L (eds) (2007) Multimedia Retrieval. Springer, BerlinGoogle Scholar
  2. 2.
    Blei DM, Jordan MI (2003) Modeling annotated data. ACM, New YorkCrossRefGoogle Scholar
  3. 3.
    Bota H, Zhou K, Jose JM, Lalmas M (2014) Composite retrieval of heterogeneous web search. ACM, New YorkCrossRefGoogle Scholar
  4. 4.
    Brin S, Page L (2012) Reprint of: the anatomy of a large-scale hypertextual web search engine. Comput Netw 56(18):3825–3833. doi: 10.1016/j.comnet.2012.10.007 CrossRefGoogle Scholar
  5. 5.
    Cascia ML, Sethi S, Sclaroff S (1998) Combining textual and visual cues for content-based image retrieval on the World Wide Web. In: Proceedings of the IEEE workshop on content-based access of image and video libraries, CBAIVL ’98. IEEE, Washington, p 24CrossRefGoogle Scholar
  6. 6.
    Chen DL, Dolan WB (2011) Collecting highly parallel data for paraphrase evaluation Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, HLT ’11, vol 1. Association for Computational Linguistics, Stroudsburg, pp 190–200Google Scholar
  7. 7.
    Chen Z, Wenyin L, Zhang F, Li M, Zhang H (2001) Web mining for web image retrieval. J Am Soc Inf Sci Tec 52(10):831–839CrossRefGoogle Scholar
  8. 8.
    Chen Y, Yu N, Luo B, Chen X (2010) iLike: integrating visual and textual features for vertical search Proceedings of the 18th international conference on multimedia, MM ’10. ACM, New York, pp 221–230CrossRefGoogle Scholar
  9. 9.
    Chen C, Zhu Q, Lin L, Shyu ML (2013) Web media semantic concept retrieval via tag removal and model fusion. ACM Trans Intell Syst Technol 4:61:1–61:22Google Scholar
  10. 10.
    Choi J, Thomee B, Friedland G, Cao L, Ni K, Borth D, Elizalde B, Gottlieb L, Carrano C, Pearce R, Poland D (2014) The placing task: a large-scale geo-estimation challenge for social-media videos and images Proceedings of the 3rd ACM multimedia workshop on geotagging and its applications in multimedia, geoMM ’14. ACM, New York, pp 27–31. doi: 10.1145/2661118.2661125 Google Scholar
  11. 11.
    Craswell N, Szummer M (2007) Random walks on the click graph Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR ’07. ACM, New York, pp 239–246CrossRefGoogle Scholar
  12. 12.
    Datta R, Joshi D, Li J, Wang J (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60CrossRefGoogle Scholar
  13. 13.
    Duda R, Hart P, Stork D (2001) Pattern classification. 2nd edn. WileyGoogle Scholar
  14. 14.
    Dupplaw DP, Matthews M, Johansson R, Boato G, Costanzo A, Fontani M, Minack E, Demidova E, Blanco R, Griffiths T, Lewis P, Hare J, Moschitti A (2014) Information extraction from multimedia web documents: an open-source platform and testbed. Int J Multimed Inf Retr 3(2):97–111. doi: 10.1007/s13735-014-0051-2 CrossRefGoogle Scholar
  15. 15.
    Egenhofer MJ (1997) Query processing in spatial-query-by-sketch. J Vis Lang Comput 8(4):403–424. doi: 10.1006/jvlc.1997.0054 CrossRefGoogle Scholar
  16. 16.
    Eickhoff C, Li W, de Vries A (2013) Exploiting user comments for audio-visual content indexing and retrieval Proceedings of the 35th european conference on advances in information retrieval, ECIR’13. Springer, Berlin, pp 38–49Google Scholar
  17. 17.
    Feng W, Wang J (2012) Incorporating heterogeneous information for personalized tag recommendation in social tagging systems Proceedings of the 18th international conference on knowledge discovery and data mining, KDD ’12. ACM, New York, pp 1276–1284Google Scholar
  18. 18.
    Fu Z, Lu G, Ting KM, Zhang D (2011) A survey of audio-based music classification and annotation. IEEE Trans Multimedia 13(2):303–319. doi: 10.1109/TMM.2010.2098858 CrossRefGoogle Scholar
  19. 19.
    Gao B, Liu TY, Qin T, Zheng X, Cheng QS, Ma WY (2005) Web image clustering by consistent utilization of visual features and surrounding texts Proceedings 13th annual ACM international conference on multimedia, MM ’05. ACM, New York, pp 112–121Google Scholar
  20. 20.
    Gao Y, Wang M, Zha ZJ, Shen J, Li X, Wu X (2013) Visual-textual joint relevance learning for tag-based social image search. IEEE Trans Image Process 22(1):363–376. doi: 10.1109/TIP.2012.2202676 MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Ghias A, Logan J, Chamberlin D, Smith BC (1995) Query by humming: musical information retrieval in an audio database Proceedings of the 3rd international conference on multimedia, MULTIMEDIA ’95. ACM, New York, pp 231–236. doi: 10.1145/217279.215273 CrossRefGoogle Scholar
  22. 22.
    Gilbert A, Piras L, Wang J, Yan F, Dellandrea E, Gaizauskas R, Villegas M, Mikolajczyk K (2015) Overview of the imageclef 2015 scalable image annotation, localization and sentence generation task CLEF (Online working notes/labs/workshop)Google Scholar
  23. 23.
    Gui C, Liu J, Xu C, Lu H (2009) Web image retrieval via learning semantics of query image Proceedings of the IEEE international conference on multimedia and expo, ICME ’09. IEEE, pp 1476–1479Google Scholar
  24. 24.
    Hanjalic A, Kofler C, Larson M (2012) Intent and its discontents: The user at the wheel of the online video search engine Proceedings of the 20th ACM international conference on multimedia, MM ’12. doi: 10.1145/2393347.2396424. ACM, New York, pp 1239–1248CrossRefGoogle Scholar
  25. 25.
    Haslhofer B, Sanderson R, Simon R, van de Sompel H (2014) Open annotations on multimedia web resources. Multimed Tool Appl 70(2):847–867. doi: 10.1007/s11042-012-1098-9
  26. 26.
    Hauff C, Houben GJ (2012) Placing images on the world map: a microblog-based enrichment approach Proceedings of the 35th international conference on research and development in information retrieval, SIGIR ’12. ACM, New York, pp 691–700Google Scholar
  27. 27.
    He R, Jin H, Tao W, Sun A (2006) Unifying keywords and visual features within one-step search for web image retrieval Advances in multimedia information processing, PCM ’06. Springer, pp 527– 536Google Scholar
  28. 28.
    He X, Kan MY, Xie P, Chen X (2014) Comment-based multi-view clustering of web 2.0 items Proceedings of the 23rd international conference on World Wide Web, WWW ’14. ACM, New York, pp 771–782Google Scholar
  29. 29.
    Hu W, Xie N, Li L, Zeng X, Maybank S (2011) A survey on visual content-based video indexing and retrieval. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):797–819CrossRefGoogle Scholar
  30. 30.
    Ionescu B, Popescu A, Lupu M, Gınsca AL, Müller H (2014) Retrieving diverse social images at mediaeval 2014: challenge, dataset and evaluation Mediaeval 2014 workshopGoogle Scholar
  31. 31.
    Ionescu B, Popescu A, Radu AL, Müller H (2016) Result diversification in social image retrieval: a benchmarking framework. Multimed Tool Appl 75(2):1301–1331. doi: 10.1007/s11042-014-2369-4 CrossRefGoogle Scholar
  32. 32.
    Jain V, Varma M (2011) Learning to re-rank: query-dependent image re-ranking using click data Proceedings of the 20th international conference on World Wide Web, WWW ’11. ACM, New York, pp 277–286Google Scholar
  33. 33.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding Proceedings of the 22nd ACM international conference on multimedia, MM ’14. ACM, New York, pp 675–678. doi: 10.1145/2647868.2654889 Google Scholar
  34. 34.
    Jiang L, Yu SI, Meng D, Mitamura T, Hauptmann AG (2015) Bridging the ultimate semantic gap: a semantic search engine for internet videos Proceedings of the 5th ACM on international conference on multimedia retrieval, ICMR ’15. ACM, New York, pp 27–34. doi: 10.1145/2671188.2749399 CrossRefGoogle Scholar
  35. 35.
    Kamath KY, Caverlee J (2012) Content-based crowd retrieval on the real-time web Proceedings of the 21st international conference on information and knowledge management, CIKM ’12. ACM, New York, pp 195–204Google Scholar
  36. 36.
    Kaminskas M, Ricci F, Schedl M (2013) Location-aware music recommendation using auto-tagging and hybrid matching Proceedings of the 7th ACM conference on recommender systems, recsys ’13. doi: 10.1145/2507157.2507180. ACM, New York, pp 17–24CrossRefGoogle Scholar
  37. 37.
    Kannan A, Baker S, Ramnath K, Fiss J, Lin D, Vanderwende L, Ansary R, Kapoor A, Ke Q, Uyttendaele M, Wang XJ, Zhang L (2014) Mining text snippets for images on the web Proceedings of the 20th international conference on knowledge discovery and data mining, KDD ’14. ACM, New York, pp 1534–1543Google Scholar
  38. 38.
    Kherfi ML, Ziou D, Bernardi A (2004) Image retrieval from the World Wide Web: issues, techniques, and systems. ACM Comput Surv 36(1):35–67. doi: 10.1145/1013208.1013210 CrossRefGoogle Scholar
  39. 39.
    Kim YA, Ahmad MA (2013) Trust, distrust and lack of confidence of users in online social media-sharing communities. Knowl-Based Syst 37:438–450. doi: 10.1016/j.knosys.2012.09.002 CrossRefGoogle Scholar
  40. 40.
    Knees P, Schedl M (2013) A survey of music similarity and recommendation from music context data. ACM Trans Multimedia Comput Commun Appl 10(1):2:1–2:21. doi: 10.1145/2542205.2542206 CrossRefGoogle Scholar
  41. 41.
    Kofler C, Larson M, Hanjalic A (2016) User intent in multimedia search: A survey of the state of the art and future challenges. ACM Comput Surv 49(2):36:1–36:37. doi: 10.1145/2954930 CrossRefGoogle Scholar
  42. 42.
    van Leuken RH, Garcia L, Olivares X, van Zwol R (2009) Visual diversification of image search results Proceedings of the 18th international conference on World Wide Web, WWW ’09. ACM, New York, pp 341–350Google Scholar
  43. 43.
    Leung CHC, Chan AWS, Milani A, Liu J, Li Y (2012) Intelligent social media indexing and sharing using an adaptive indexing search engine. ACM Trans Intell Syst Technol 3(3):47:1–47:27CrossRefGoogle Scholar
  44. 44.
    Lew MS, Seve N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: State of the art and challenges. ACM Comput Surv 2(1):1–19Google Scholar
  45. 45.
    Li X, Snoek CGM, Worring M, Smeulders AWM (2012) Harvesting social images for bi-concept search. IEEE Trans Multimedia 14(4):1091–1104CrossRefGoogle Scholar
  46. 46.
    Li X, Uricchio T, Ballan L, Bertini M, Snoek CGM, Bimbo AD (2016) Socializing the semantic gap: A comparative survey on image tag assignment, refinement, and retrieval. ACM Comput Surv 49(1):14:1–14:39. doi: 10.1145/2906152 CrossRefGoogle Scholar
  47. 47.
    Liu X, Hue B (2013) Heterogeneous features and model selection for event-based media classification Proceedings of the 3rd ACM conference on international conference on multimedia retrieval, ICMR ’13. ACM, New York, pp 151–158CrossRefGoogle Scholar
  48. 48.
    Low Y, Agarwal D, Smola AJ (2011) Multiple domain user personalization Proceedings of the 17th international conference on knowledge discovery and data mining, KDD ’11. ACM, New York, pp 123–131Google Scholar
  49. 49.
    Mallik A, Ghosh H, Chaudhury S, Harit G (2013) Mowl: An ontology representation language for web-based multimedia applications. ACM Trans Multimedia Comput Commun Appl 10(1):8:1–8:21. doi: 10.1145/2542205.2542210 CrossRefGoogle Scholar
  50. 50.
    Mei T, Rui Y, Li S, Tian Q (2014) Multimedia search reranking: A literature survey. ACM Comput Surv 46(3):38:1–38:38. doi: 10.1145/2536798 CrossRefGoogle Scholar
  51. 51.
    Morrison D, Tsikrika T, Hollink V, Vries AP, Bruno É, Marchand-Maillet S (2013) Topic modelling of clickthrough data in image search. Multimed Tool Appl 66(3):493–515. doi: 10.1007/s11042-012-1038-8 CrossRefGoogle Scholar
  52. 52.
    Naaman M (2012) Social multimedia: highlighting opportunities for search and mining of multimedia data in social media applications. Multimed Tool Appl 56(1):9–34. doi: 10.1007/s11042-010-0538-7 CrossRefGoogle Scholar
  53. 53.
    Nie L, Yan S, Wang M, Hong R, Chua TS (2012) Harvesting visual concepts for image search with complex queries Proceedings of the 20th ACM international conference on multimedia, MM ’12. doi: 10.1145/2393347.2393363. ACM, New York, pp 59–68CrossRefGoogle Scholar
  54. 54.
    Perelman D, Bortnikov E, Lempel R, Sandler R (2012) Lightweight automatic face annotation in media pages Proceedings of the 21st international conference on World Wide Web, WWW ’12. ACM, New York, pp 939–948CrossRefGoogle Scholar
  55. 55.
    Petkos G, Papadopoulos S, Mezaris V, Kompatsiaris Y (2014) Social event detection at mediaeval 2014: challenges, datasets, and evaluation Mediaeval 2014 workshopGoogle Scholar
  56. 56.
    Poblete B, Bustos B, Mendoza M, Barrios JM (2010) Visual-semantic graphs: using queries to reduce the semantic gap in web image retrieval Proceedings 19th ACM international conference on information and knowledge management (CIKM’10). ACM, New York, pp 1553–1556. doi: 10.1145/1871437.1871670 Google Scholar
  57. 57.
    Popescu A, Grefenstette G (2011) Social media driven image retrieval Proceedings of the 1st ACM international conference on multimedia retrieval, ICMR ’11. ACM, New York, pp 33:1–33:8Google Scholar
  58. 58.
    Popescu A, Spyromitros-Xioufis E, Papadopoulos S, Le Borgne H, Kompatsiaris I (2015) Toward an automatic evaluation of retrieval performance with large scale image collections Proceedings of the 2015 workshop on community-organized multimodal mining: Opportunities for novel solutions, MMCommons ’15. ACM, New York, pp 7–12. doi: 10.1145/2814815.2814819 CrossRefGoogle Scholar
  59. 59.
    Schedl M, Orio N, Liem CCS, Peeters G (2013) A professionally annotated and enriched multimodal data set on popular music Proceedings of the 4th multimedia systems conference, MMSys ’13. doi: 10.1145/2483977.2483985. ACM, New York, pp 78–83CrossRefGoogle Scholar
  60. 60.
    Schmiedeke S, Xu P, Ferrané I, Eskevich M, Kofler C, Larson MA, Estève Y, Lamel L, Jones GJF, Sikora T (2013) Blip10000: a social video dataset containing spug content for tagging and retrieval Proceedings of the 4th ACM multimedia systems conference, MMSys ’13. ACM, New York, pp 96–101. doi: 10.1145/2483977.2483988 CrossRefGoogle Scholar
  61. 61.
    Shen HT, Ooi BC, Tan KL (2000) Giving meanings to WWW images Proceedings of the 8th international conference on multimedia, MM ’00. ACM, New York, pp 39–47Google Scholar
  62. 62.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  63. 63.
    Song Y, Vallmitjana J, Stent A, Jaimes A (2015) Tvsum: summarizing web videos using titles IEEE Conference on computer vision and pattern recognition, CVPR ’15. IEEE, pp 5179–5187. doi: 10.1109/CVPR.2015.7299154
  64. 64.
    Tan HK, Ngo CW (2011) Fusing heterogeneous modalities for video and image re-ranking Proceedings of the 1st international conference on multimedia retrieval, ICMR ’11. ACM, New York, pp 15:1–15:8Google Scholar
  65. 65.
    Tan S, Ngo CW, Tan HK, Pang L (2011) Cross media hyperlinking for search topic browsing Proceedings of the 19th international conference on multimedia, MM ’11. ACM, New York, pp 243– 252CrossRefGoogle Scholar
  66. 66.
    Tsikrika T, Diou C, de Vries A, Delopoulos A (2011) Reliability and effectiveness of clickthrough data for automatic image annotation. Multimed Tool Appl 55(1):27–52. doi: 10.1007/s11042-010-0584-1
  67. 67.
    Typke R, Wiering F, Veltkamp RC (2005) A survey of music information retrieval systems Proceedings of the 6th international conference on music information retrieval, ISMIR 2005, pp 153– 160Google Scholar
  68. 68.
    Villegas M, Paredes R (2012) Overview of the imageclef 2012 scalable web image annotation task CLEF (Online working notes/labs/workshop)Google Scholar
  69. 69.
    Wang J, Hua XS (2011) Interactive image search by color map. ACM Trans Intell Syst Technol 3(1):12:1–12:23CrossRefGoogle Scholar
  70. 70.
    Wang XJ, Ma WY, Li X (2004) Data-driven approach for bridging the cognitive gap in image retrieval IEEE International conference on multimedia and expo, ICME ’04, vol 3, pp 2231–2234Google Scholar
  71. 71.
    Wang D, Hoi S, Wu P, Zhu J, He Y, Miao C (2013) Learning to name faces: a multimodal learning scheme for search-based face annotation Proceedings of the 36th international conference on research and development in information retrieval, SIGIR ’13. ACM, New York, pp 443–452Google Scholar
  72. 72.
    Westerveld T (2000) Image retrieval: Content versus context. In: content-based multimedia information access, RIAO ’00, pp 276–284Google Scholar
  73. 73.
    White RW, Roth RA (2009) Exploratory search: beyond the query-response paradigm, vol 1. Morgan & Claypool Publishers, San RafaelGoogle Scholar
  74. 74.
    Wu L, Hoi S, Yu N (2009) Semantics-preserving bag-of-words models for efficient image annotation Proceedings 1st ACM workshop on large-scale multimedia retrieval and mining, LS-MMRM ’09. ACM, New York, pp 19–26Google Scholar
  75. 75.
    Xu S, Jiang H, Lau FCM (2011) Retrieving and ranking unannotated images through collaboratively mining online search results Proceedings of the 20th international conference on information and knowledge management, CIKM ’11. ACM, New York, pp 485–494Google Scholar
  76. 76.
    Yang CC, Chan KY (2005) Retrieving multimedia web objects based on pagerank algorithm Special interest tracks and posters of the 14th international conference on World Wide Web, WWW ’05. ACM, New York, pp 906–907CrossRefGoogle Scholar
  77. 77.
    Yatskar M, Vanderwende L, Zettlemoyer L (2014) See no evil, say no evil: description generation from densely labeled images. Lexical Comput Semant (*SEM 2014):110Google Scholar
  78. 78.
    Yu J, Tao D, Wang M, Rui Y (2015) Learning to rank using user clicks and visual features for image retrieval. IEEE Trans Cybern 4(45):767–779CrossRefGoogle Scholar
  79. 79.
    Zhao R, Grosky WI (2002) Narrowing the semantic gap—improved text-based web document retrieval using visual features. IEEE Trans Multimed 4(2):189–200CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of ChileSantiagoChile
  2. 2.Computer Graphics and Knowledge VisualizationGraz University of TechnologyGrazAustria

Personalised recommendations