Multimedia Tools and Applications

, Volume 77, Issue 20, pp 27447–27469 | Cite as

Multimedia and geographic data integration for cultural heritage information retrieval

  • Erasmo Purificato
  • Antonio M. RinaldiEmail author


In this paper a system providing an efficient integration between Content-Based Image Retrieval (CBIR) and Geographic Information Retrieval (GIR) is presented. Over the years, many CBIR systems have been proposed to give a solution for an efficient use of multimedia/visual contents and other issues as performance, quality of retrieval, data heterogeneity, and multimodal information integration. The aim of the proposed approach is to prove that the use of geographic data can improve the results obtained by an image matching system based only on visual data. Our framework is composed of three parts, each of them described in detail in this paper: the first part is dedicated to CBIR, with an experimental comparison of a large number of different multimedia features to choose the one to use in the system implementation; in the second part the methodology to integrate geographic and multimedia data is showed; in the last part is presented a GIR system implementation using a “points of interest” search. An Android application has been developed for the client-side using Apache Solr as server side provider for the information retrieval functionalities. An experimental evaluation is carried out to demonstrate the effective improvement given by the combination of geographic and multimedia data. Our results have been obtained using a real dataset composed of artworks located in Naples’s museums.


Content-based image retrieval Geographic information retrieval Multimodal query Digital cultural heritage 


  1. 1.
    Adams B, McKenzie G, Gahegan M (2015) Frankenplace: interactive thematic mapping for ad hoc exploratory search. In: Proceedings of the 24th international conference on world wide web. ACM, pp 12–22Google Scholar
  2. 2.
    Agnello F, Corsale R, Franco V, Lo Brutto M, Midulla P, Orlando P, Villa B (2003) Cultural heritage and information systems, an investigation into a dedicated hypertext. Int Archives Photogrammetry Remote Sens Spatial Inform Sci 34(5/W12):7–12Google Scholar
  3. 3.
    Ai L-F, Jun-Qing Y, He Y-F, Guan T (2013) High-dimensional indexing technologies for large scale content-based image retrieval: a review. J Zhejiang University Sci C 14(7):505–520CrossRefGoogle Scholar
  4. 4.
    Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B (2011) Content-based image retrieval in radiology: current status and future directions. J Digit Imag 24(2):208–222CrossRefGoogle Scholar
  5. 5.
    Basanth Kumar HB (2015) An overview on content-based image retrievalGoogle Scholar
  6. 6.
    Bhatti A, Butt SM, Butt MM (2014) Visual feature extraction for content-based image retrieval. Sci Int 26(1):1–5Google Scholar
  7. 7.
    Bosch A, Zisserman A, Munoz X (2007) Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM international conference on image and video retrieval. ACM, pp 401–408Google Scholar
  8. 8.
    Carson C, Belongie S, Greenspan H, Malik J (2002) Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans Pattern Anal Mach Intell 24(8):1026–1038CrossRefGoogle Scholar
  9. 9.
    Chang S-F, Sikora T, Purl A (2001) Overview of the mpeg-7 standard. IEEE Trans Circuits Syst Video Technol 11(6):688–695CrossRefGoogle Scholar
  10. 10.
    Chatzichristofis SA, Boutalis YS (2008) Cedd: color and edge directivity descriptor: a compact descriptor for image indexing and retrieval. In: International conference on computer vision systems. Springer, pp 312–322Google Scholar
  11. 11.
    Chatzichristofis SA, Boutalis YS (2010) Content based radiology image retrieval using a fuzzy rule based scalable composite descriptor. Multimed Tool Appl 46(2-3):493–519CrossRefGoogle Scholar
  12. 12.
    Chatzichristofis SA, Fcth YSB (2008) Fuzzy color and texture histogram-a low level feature for accurate image retrieval. In: Ninth international workshop on Image analysis for multimedia interactive services, 2008. WIAMIS’08. IEEE, pp 191–196Google Scholar
  13. 13.
    Chatzichristofis S, Boutalis Y, Lux M (2009) Selection of the proper compact composite descriptor for improving content based image retrieval. In: Proceedings of the 6th IASTED international conference, vol 134643, p 064Google Scholar
  14. 14.
    Chen C-C, Wactlar HD, Wang JZ, Kiernan K (2005) Digital imagery for significant cultural and historical materials. Int J Digit Libr 5(4):275–286CrossRefGoogle Scholar
  15. 15.
    Datta R, Li J, Wang JZ (2005) Content-based image retrieval: approaches and trends of the new age. In: Proceedings of the 7th ACM SIGMM international workshop on multimedia information retrieval. ACM, pp 253–262Google Scholar
  16. 16.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Computing Surveys (Csur) 40(2):5CrossRefGoogle Scholar
  17. 17.
    de Vries AP, Westerveld T (2004) A comparison of continuous vs. discrete image models for probabilistic image and video retrieval. In: 2004 international conference on Image processing, 2004. ICIP’04, vol 4. IEEE, pp 2387–2390Google Scholar
  18. 18.
    Deselaers T, Keysers D, Ney H (2004) Fire–flexible image retrieval engine: imageclef 2004 evaluation. In: Workshop of the cross-language evaluation forum for european languages. Springer, pp 688–698Google Scholar
  19. 19.
    Deselaers T, Keysers D, Ney H (2008) Features for image retrieval: an experimental comparison. Inform Retrieval 11(2):77–107CrossRefGoogle Scholar
  20. 20.
    Frederix G, Caenen G, Pauwels EJ (2000) Panoramic, adaptive and reconfigurable interface for similarity search. In: 2000 international conference on Image processing, 2000. Proceedings, vol 3. IEEE, pp 222–225Google Scholar
  21. 21.
    Giuca A-M, Seitz KA, Furst J, Raicu D (2012) Expanding diagnostically labeled datasets using content-based image retrieval. In: 2012 19th IEEE international conference on Image processing (ICIP). IEEE, pp 2397–2400Google Scholar
  22. 22.
    Golubovic N, Krintz C, Wolski R, Lafia S, Hervey T, Kuhn W (2016) Extracting spatial information from social media in support of agricultural management decisions. In: Proceedings of the 10th workshop on geographic information retrieval. ACM, p 4Google Scholar
  23. 23.
    Hariharan R, Hore B, Li C, Mehrotra S (2007) Processing spatial-keyword (sk) queries in geographic information retrieval (gir) systems. In: 19th international conference on Scientific and statistical database management, 2007. SSBDM’07. IEEE, pp 16–16Google Scholar
  24. 24.
    Huang J, Ravi Kumar S, Mitra M, Zhu W-J, Zabih R (1997) Image indexing using color correlograms. In: 1997 Proceedings of the IEEE computer society conference on Computer vision and pattern recognition, 1997. IEEE, pp 762–768Google Scholar
  25. 25.
    Iqbal Q, Aggarwal JK (2002) Cires: a system for content-based retrieval in digital image libraries. In: 7th international conference on Control, automation, robotics and vision, 2002. ICARCV 2002, vol 1. IEEE, pp 205–210Google Scholar
  26. 26.
    Irtaza A, Jaffar AM, Aleisa E, Choi T-S (2014) Embedding neural networks for semantic association in content based image retrieval. Multimed Tool Appl 72(2):1911–1931CrossRefGoogle Scholar
  27. 27.
    Irtaza A, Jaffar MA, Muhammad MS (2015) Content based image retrieval in a web 3.0 environment. Multimed Tool Appl 74(14):5055–5072CrossRefGoogle Scholar
  28. 28.
    Ivanova K, Dobreva M, Stanchev P, Totkov G (2012) Access to digital cultural heritage: innovative applications of automated metadata generation plovdiv university publishing house paisii hilendarskiGoogle Scholar
  29. 29.
    Ji Wx, Wang D, Hoi SCH, Pengcheng W, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval A comprehensive study. In: Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 157–166Google Scholar
  30. 30.
    Karamti H, Tmar M, Visani M, Urruty T, Gargouri F (2017) Vector space model adaptation and pseudo relevance feedback for content-based image retrieval. Multimed Tool Appl:1–27Google Scholar
  31. 31.
    Kim J, Vasardani M, Winter S (2017) Similarity matching for integrating spatial information extracted from place descriptions. Int J Geogr Inf Sci 31(1):56–80CrossRefGoogle Scholar
  32. 32.
    Koolen M, Kamps J, de Keijzer V (2009) Information retrieval in cultural heritage. Interdisc Sci Rev 34(2-3):268–284CrossRefGoogle Scholar
  33. 33.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  34. 34.
    Kumar A, Kim J, Cai W, Fulham M, Feng D (2013) Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data. J Digital Imag 26(6):1025–1039CrossRefGoogle Scholar
  35. 35.
    Larson RR (1996) Geographic information retrieval and spatial browsing. Geographic information systems and libraries: patrons, maps, and spatial information [papers presented at the 1995 Clinic on Library Applications of Data Processing, April 10-12, 1995]Google Scholar
  36. 36.
    Lehmann TM, Güld MO, Deselaers T, Keysers D, Schubert H, Spitzer K, Ney H, Wein BB (2005) Automatic categorization of medical images for content-based retrieval and data mining. Comput Med Imaging Graph 29(2):143–155CrossRefGoogle Scholar
  37. 37.
    Liew CL (2005) Online cultural heritage exhibitions: a survey of information retrieval features. Program 39(1):4–24MathSciNetCrossRefGoogle Scholar
  38. 38.
    Longley P (2005) Geographic information systems and science. Wiley, New YorkGoogle Scholar
  39. 39.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRefGoogle Scholar
  40. 40.
    Maliene V, Grigonis V, Paleviċius V, Griffiths S (2011) Geographic information system: old principles with new capabilities. Urban Des Int 16(1):1–6CrossRefGoogle Scholar
  41. 41.
    Mark DM (2003) Geographic information science: defining the field. Found Geographic Inform Sci 1:3–18CrossRefGoogle Scholar
  42. 42.
    Müller H, Müller W, Squire DM, Marchand-Maillet S, Pun T (2001) Performance evaluation in content-based image retrieval: overview and proposals. Pattern Recogn Lett 22(5):593–601zbMATHCrossRefGoogle Scholar
  43. 43.
    Ng JY-H, Yang F, Davis LS (2015) Exploiting local features from deep networks for image retrieval. arXiv:1504.05133
  44. 44.
    Ohm J-R (2001) The mpeg-7 visual description framework—concepts, accuracy, and applications. In: Computer analysis of images and patterns. Springer, pp 2–10Google Scholar
  45. 45.
    Rasuli B, Asadi S (2017) Web geographic information retrieval: Literature review and conceptual modelGoogle Scholar
  46. 46.
    Rinaldi AM (2011) Automatic web pages hierarchical classification using dynamic domain ontologies. Int J Knowl Web Intell 2(4):231–256CrossRefGoogle Scholar
  47. 47.
    Rinaldi AM (2014) A multimedia ontology model based on linguistic properties and audio-visual features. Inf Sci 277:234–246CrossRefGoogle Scholar
  48. 48.
    Ruigang F, Li B, Gao Y, Wang P (2016) Content-based image retrieval based on cnn and svm. In: 2016 2nd IEEE International Conference on Computer and Communications (ICCC). IEEE, pp 638–642Google Scholar
  49. 49.
    Shete DS, Chavan MS, Kolhapur K (2012) Content based image retrieval. Int J Emerging Technol Advan Eng 2(9):85–90Google Scholar
  50. 50.
    Shyu C-R, Klaric M, Scott GJ, Barb AS, Davis CH, Geoiris KP (2007) Geospatial information retrieval and indexing system—content mining, semantics modeling, and complex queries. IEEE Trans Geoscience Remote Sens 45(4):839–852CrossRefGoogle Scholar
  51. 51.
    Siggelkow S, Schael Mx, Burkhardt H (2001) Simba—search images by appearance. In: Joint pattern recognition symposium. Springer, pp 9–16Google Scholar
  52. 52.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  53. 53.
    Squire D, Muller W, Muller H, Raki J (1998) Content-based query of image databases, inspirations from text retrieval: inverted files frequency-based weights and relevance feedbackGoogle Scholar
  54. 54.
    Tangelder JWH, Veltkamp RC (2008) A survey of content based 3d shape retrieval methods. Multimed Tool Appl 39(3):441CrossRefGoogle Scholar
  55. 55.
    Tsai C-F (2007) A review of image retrieval methods for digital cultural heritage resources. Online Inf Rev 31(2):185–198CrossRefGoogle Scholar
  56. 56.
    Wang JZ, Li J, Wiederhold G (2001) Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans Pattern Anal Mach Intell 23(9):947–963CrossRefGoogle Scholar
  57. 57.
    Zaila YL (2017) GEIR: a full-fledged geographically enhanced information retrieval solution. PhD thesis almaGoogle Scholar
  58. 58.
    Zheng Y-T, Zha Z-J, Chua T-S (2011) Research and applications on georeferenced multimedia: a survey. Multimed Tool Appl 51(1):77–98CrossRefGoogle Scholar
  59. 59.
    Zheng L, Yi Y, Tian Q (2017) Sift meets cnn: a decade survey of instance retrieval. IEEE Trans Pattern Anal Mach IntellGoogle Scholar
  60. 60.
    Zhou XS, Huang TS (2003) Relevance feedback in image retrieval: A comprehensive review. Multimed Syst 8(6):536–544CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electrical Engineering and Information TechnologiesUniversity of Naples Federico IINapoliItaly

Personalised recommendations