Skip to main content
SpringerLink
Log in

Semantic analysis and retrieval in personal and social photo collections

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Semantic understanding of images has been an important topic in the research community for a long time as it is an important prerequisite to build meaningful retrieval systems which are accessible by both users and automatic reasoning algorithms. Recently, especially with the growing trend to share photos online, the social aspect of image retrieval becomes more and more prevalent and image retrieval more and more focusses specifically on photos and their special characteristics, especially on information outside the image itself. Researchers are starting to explore how and why photos are shot, shared and used and try to incorporate this additional knowledge to aid image analysis and retrieval. Several survey papers have been written in the past reviewing works in the general field of image analysis and retrieval. However, the social aspect of image retrieval and the focus on digital photos has not sufficiently been addressed in these works. In this article we give an overview over the current research field of semantic photo understanding, annotation and retrieval. We review over 160 contributions in the field and identify trending topics and implications for future directions of research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. http://blog.facebook.com/blog.php?post=2406207130

  2. http://www.alipr.com

  3. http://www.facebook.com/press/info.php?statistics

  4. http://brisc.sourceforge.net/

  5. http://www.gnu.org/software/gift/

  6. http://www.imgseek.net/

  7. http://www.apple.com/de/ilife/iphoto/

  8. http://picasa.google.com

  9. http://www.smilebooks.com

References

  1. 3rd special workshop on multimedia semantics (2005). Pisa, Italy

  2. ACM (2010) Multimedia Grand Challenge. http://comminfo.rutgers.edu/conferences/mmchallenge/

  3. Adobe (2005) Xmp specification. Tech. rep., Adobe Systems Inc. http://www.adobe.com/products/xmp/

  4. Aigrain P, Zhang H, Petkovic D (1996) Content-based representation and retrieval of visual media: a state-of-the-art review. In: Representation and retrieval of visual media in multimedia systems, pp 3–26

  5. Ames M, Eckles D, Naaman M, Spasojevic M, Van House N (2009) Requirements for mobile photoware. Personal and ubiquitous computing. doi:10.1007/s00779-009-0237-4

  6. Anguera X, Xu J, Oliver N (2008) Multimodal photo annotation and retrieval on a mobile phone. In: MIR ’08: proceeding of the 1st ACM international conference on multimedia information retrieval. ACM, New York, NY, USA, pp 188–194. doi:10.1145/1460096.1460127

    Chapter  Google Scholar 

  7. Ardizzone E, La Cascia M, Vella F (2008) Mean shift clustering for personal photo album organization. In: Proc. of ICIP 2008. 15th IEEE international conference on image processing, pp 85–88. doi:10.1109/ICIP.2008.4711697

  8. Arni T, Tang J, Sanderson M, Clough P (2008) Creating a test collection to evaluate diversity in image retrieval. Beyond binary relevance: preferences, diversity and set-level judgments

  9. Aslandogan Y, Yu C (2000) Diogenes: a web search agent for content based indexing of personal images. In: Proceedings of ACM SIGIR, pp 481–482

  10. Avcıbaş İ, Sankur B, Sayood K (2002) Statistical evaluation of image quality measures. J Electron Imaging 11:206

    Article  Google Scholar 

  11. Banerjee S, Evans B (2004) Unsupervised automation of photographic composition rules in digital still cameras. In: Proceedings of SPIE, vol 5301, pp 364–373

  12. Becker H, Naaman M, Gravano L (2010) Learning similarity metrics for event identification in social media. In: WSDM ’10: proceedings of the third ACM international conference on web search and data mining. ACM, New York, NY, USA, pp 291–300. doi:10.1145/1718487.1718524

    Chapter  Google Scholar 

  13. Bentley F, Metcalf C, Harboe G (2006) Personal vs. commercial content: the similarities between consumer use of photos and music. In: CHI ’06: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, pp 667–676. doi:10.1145/1124772.1124871

    Chapter  Google Scholar 

  14. Berners-Lee T, Hendler J, Lassila O et al (2001) The semantic web. Sci Am 284(5):28–37

    Article  Google Scholar 

  15. Bloehdorn S, Petridis K, Saathoff C, Simou N, Tzouvaras V, Avrithis Y, Handschuh S, Kompatsiaris Y, Staab S, Strintzis M (2005) Semantic annotation of images and videos for multimedia analysis. In: The semantic web: research and applications, pp 592–607

  16. Boutell M, Luo J (2004) Bayesian fusion of camera metadata cues in semantic scene classification. In: CVPR

  17. Boutell M, Luo J (2004) Photo classification by integrating image content and camera metadata. In: ICPR ’04: proceedings of the pattern recognition, 17th international conference on (ICPR’04), vol 4. IEEE Computer Society, Washington, DC, USA, pp 901–904. doi:10.1109/ICPR.2004.693

    Chapter  Google Scholar 

  18. Boutell MR, Luo J (2005) Beyond pixels: exploiting camera metadata for photo classification. Pattern Recogn 38(6):935–946

    Article  Google Scholar 

  19. Boutell M, Brown C, Luo J (2006) Exploiting context for semantic scene classification. Univ. Rochester, Rochester, NY, Tech. Rep 894

  20. Bovik A, Sheikh H (2005) No-reference quality assessment using natural scene statistics: JPEG2000. IEEE Trans Image Process 14(2005):1918–1927

    Google Scholar 

  21. Boyd D, Ellison NB (2007) Social network sites: definition, history, and scholarship. J Comput-Mediat Commun 13(1):210–230

    Article  Google Scholar 

  22. Bryant R (2007) Data-intensive supercomputing: the case for DISC. School of Computer Science, Carnegie Mellon University, Tech. Rep. Technical Report CMU-CS-07-128

  23. Burnett IS, Pereira F, Walle RVd, Koenen R (2006) The MPEG-21 book. Wiley, New York

    Book  Google Scholar 

  24. Cao L, Luo J, Huang TS (2008) Annotating photo collections by label propagation according to multiple similarity cues. In: MM ’08: proceeding of the 16th ACM international conference on multimedia. ACM, New York, NY, USA, pp 121–130. doi:10.1145/1459359.1459376

    Chapter  Google Scholar 

  25. Chalfen R (1987) Snapshot versions of life. Bowling Green University Popular Press

  26. Chandramouli K, Izquierdo E (2010) Semantic structuring and retrieval of event chapters in social photo collections. In: MIR ’10: proceedings of the international conference on multimedia information retrieval. ACM, New York, NY, USA, pp 507–516. doi:10.1145/1743384.1743472

    Chapter  Google Scholar 

  27. Chang NS, Fu KS (1980) Query-by-pictorial-example. IEEE Trans Softw Eng 6(6):519–524. doi:10.1109/TSE.1980.230801

    Article  Google Scholar 

  28. Chen C, Oakes M, Tait J (2006) Browsing personal images using episodic memory (time + location). Adv in IR 3936:362–372. doi:10.1007/11735106_32

    Google Scholar 

  29. Chianese A, Picariello A, Sansone L, Sapino ML (2004) Managing uncertainties in image databases: a fuzzy approach. Multimed Tools Appl 23(3):237–252. doi:10.1023/B:MTAP.0000031759.22145.5d

    Article  Google Scholar 

  30. Chu C, Kim S, Lin Y, Yu Y, Bradski G, Ng A, Olukotun K (2007) Map-reduce for machine learning on multicore. In: Advances in neural information processing systems 19: proceedings of the 2006 conference. MIT Press, Cambridge, p 281

    Google Scholar 

  31. Comité International des Télécommunications de Presse (1999) IPTC—NAA information interchange model version 4. Tech. rep. http://www.iptc.org/IIM/

  32. Cooper M, Foote J, Girgensohn A, Wilcox L (2005) Temporal event clustering for digital photo collections. ACM Trans Multimed Comput Commun Appl 1(3):269–288. doi:10.1145/1083314.1083317

    Article  Google Scholar 

  33. Crabtree A, Rodden T, Mariani J (2004) Collaborating around collections: informing the continued development of photoware. In: CSCW ’04: proceedings of the 2004 ACM conference on computer supported cooperative work. ACM, New York, NY, USA, pp 396–405. doi:10.1145/1031607.1031673

    Chapter  Google Scholar 

  34. Cusano C, Ciocca G, Schettini R (2003) Image annotation using SVM. In: Santini S, Schettini R (eds) Society of photo-optical instrumentation engineers (SPIE) conference series, pp 330–338

  35. Damera-Venkata N, Kite TD, Geisler WS, Evans BL, Bovik AC (2004) Image quality assessment based on a degradation model. IEEE Trans Image Process 9(4):636–650

    Article  Google Scholar 

  36. Datta R, Joshi D, Li J, Wang JZ (2006) Studying aesthetics in photographic images using a computational approach. In: Leonardis A, Bischof H, Pinz A (eds) ECCV (3). Lecture notes in computer science, vol 3953. Springer, pp 288–301. http://infolab.stanford.edu/~wangz/project/imsearch/Aesthetics/ECCV06/

  37. Datta R, Ge W, Liu J, Wang JZ (2007) Image retrieval: ideas, influences, and trends of the new age. In: ACM computing surveys, 2007. http://wang.ist.psu.edu/survey/analysis/

  38. Datta R, Li J, Wang JZ (2007) Learning the consensus on visual quality for next-generation image management. In: Proceedings of the ACM multimedia conference. http://infolab.stanford.edu/~wangz/project/imsearch/ALIP/ACMM M07A/

  39. Datta R, Li J, Wang J (2008) Algorithmic inferencing of aesthetics and emotion in natural images: an exposition. In: 15th IEEE international conference on image processing, pp 105–108

  40. Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113. doi:10.1145/1327452.1327492

    Article  Google Scholar 

  41. Digital Imaging Group (2001) DIG35 specification—metadata for digital images—version 1.1. Tech. rep. http://www.digitalimaging.org

  42. Duygulu P, Barnard K, De Freitas J, Forsyth D (2006) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Computer vision—ECCV 2002, pp 349–354

  43. Eisenthal Y, Dror G, Ruppin E (2006) Facial attractiveness: beauty and the machine. Neural Comput 18(1):119–142. doi:10.1162/089976606774841602

    Article  Google Scholar 

  44. Eskicioglu A (2000) Quality measurement for monochrome compressed images in the past 25 years. In: IEEE international conference on acoustics speech and signal processing, vol 4

  45. Eskicioglu A, Fisher P (1995) Image quality measures and their performance. IEEE Trans Commun 43(12):2959–2965. doi:10.1109/26.477498

    Article  Google Scholar 

  46. Fan X, Xie X, Li Z, Li M, Ma WY (2005) Photo-to-search: using multimodal queries to search the web from mobile devices. In: MIR ’05: proceedings of the 7th ACM SIGMM international workshop on multimedia information retrieval. ACM, New York, NY, USA, pp 143–150. doi:10.1145/1101826.1101851

    Chapter  Google Scholar 

  47. Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P (1995) Query by image and video content: the qbic system. Computer 28(9):23–32. doi:10.1109/2.410146

    Article  Google Scholar 

  48. Frankel C, Swain MJ, Athitsos V (1996) WebSeer: an image search engine for the world wide web. Tech. rep., University of Chicago, Chicago, IL, USA

  49. Frohlich D, Kuchinsky A, Pering C, Don A, Ariss S (2002) Requirements for photoware. In: CSCW ’02: proceedings of the 2002 ACM conference on computer supported cooperative work. ACM, New York, NY, USA, pp 166–175. doi:10.1145/587078.587102. http://portal.acm.org/citation.cfm?id=587102

    Chapter  Google Scholar 

  50. Frohlich D, Wall S, Kiddle G (2008) Collaborative photowork: challenging the boundaries between photowork and phototalk. In: Proc. of CHI workshop on collocated social practices surrounding photos

  51. Gargi U, Deng Y, Tretter DR (2002) Managing and searching personal photo collections. Tech. rep., HP Laboratories, Palo Alto

  52. Girgensohn A, Adcock J, Cooper MD, Foote J, Wilcox L (2003) Simplifying the management of large photo collections. In: INTERACT

  53. Girgensohn A, Adcock J, Wilcox L (2004) Leveraging face recognition technology to find and organize photos. In: MIR ’04: proceedings of the 6th ACM SIGMM international workshop on multimedia information retrieval. ACM, New York, NY, USA, pp 99–106. doi:10.1145/1026711.1026728

    Chapter  Google Scholar 

  54. Gong Z, U LH, Cheang CW (2005) Web image semantic clustering. In: Proceedings of the 4th international conference on ontologies, database and applications of semantics (ODBASE 2005). Lecture notes in computer science. Springer, Agia, Napa, Cyprus

  55. Graham A, Garcia-Molina H, Paepcke A, Winograd T (2002) Time as essence for photo browsing through personal digital libraries. In: JCDL ’02: proceedings of the 2nd ACM/IEEE-CS joint conference on digital libraries. ACM, New York, NY, USA, pp 326–335. doi:10.1145/544220.544301

    Chapter  Google Scholar 

  56. Gurrin C, Jones GJF, Lee H, O’Hare N, Smeaton AF, Murphy N (2005) Mobile access to personal digital photograph archives. In: MobileHCI ’05: proceedings of the 7th international conference on human computer interaction with mobile devices & services. ACM, New York, NY, USA, pp 311–314. doi:10.1145/1085777.1085842

    Chapter  Google Scholar 

  57. Halaschek-wiener C, Schain A, Golbeck J, Parsia B, Hendler J (2005) A flexible approach for managing digital images on the semantic web. In: th international workshop on knowledge markup and semantic annotation

  58. Hardman L, Nack F, Obrenovic Z, Kerherve B, Piersol K (2005) Canonical processes of media production. In: ACM workshop on multimedia for human communication—from capture to convey. http://homepages.cwi.nl/~media/projects/canonical/papers/model.pdf

  59. Hare J, Sinclair P, Lewis P, Martinez K, Enser P, Sandom C (2006) Bridging the semantic gap in multimedia information retrieval: top-down and bottom-up approaches. In: 3rd European semantic web conference

  60. Hare JS, Lewis PH, Enser PGB, Sandom CJ (2007) Semantic facets: an in-depth analysis of a semantic image retrieval system. In: ACM CIVR 2007: the 6th international conference on image and video retrieval, ACM. http://eprints.ecs.soton.ac.uk/14322/

  61. Hare JS, Samangooei S, Lewis PH, Nixon MS (2008) Semantic spaces revisited: investigating the performance of auto-annotation and semantic retrieval using semantic spaces. In: CIVR ’08: proceedings of the 2008 international conference on content-based image and video retrieval. ACM, New York, NY, USA, pp 359–368. doi:10.1145/1386352.1386399

    Chapter  Google Scholar 

  62. He X, Caia D, Wen JR, Ma WY, Zhang HJ (2007) Clustering and searching www images using link and page layout analysis. ACM Trans Multimed Comput Commun Appl (TOMCAP) 3:10

    Article  Google Scholar 

  63. Hollink L, Schreiber A, Wielemaker J, Wielinga B (2003) Semantic annotation of image collections. In: Knowledge capture, pp 41–48

  64. Hunter J (2001) Adding multimedia to the semantic web building an Mpeg-7 ontology. In: International semantic web working symposium (SWWS)

  65. Inoue M (2009) Image retrieval: research and use in the information explosion. Prog Inf 6:3–14

    Article  Google Scholar 

  66. Jaffe A, Naaman M, Tassa T, Davis M (2006) Generating summaries for large collections of geo-referenced photographs, pp 853–854. doi:10.1145/1135777.1135911

  67. Jeon J, Lavrenko V, Manmatha R (2003) Automatic image annotation and retrieval using cross-media relevance models. In: SIGIR ’03: proceedings of the 26th annual international ACM SIGIR conference on research and development in informaion retrieval. ACM, New York, NY, USA, pp 119–126. doi:10.1145/860435.860459

    Chapter  Google Scholar 

  68. Jia J, Yu N, Hua XS (2008) Annotating personal albums via web mining. In: MM ’08: proceeding of the 16th ACM international conference on multimedia. ACM, New York, NY, USA, pp 459–468. doi:10.1145/1459359.1459421

    Chapter  Google Scholar 

  69. Ke Y, Tang X, Jing F (2006) The design of high-level features for photo quality assessment. In: CVPR ’06: proceedings of the 2006 IEEE computer society conference on computer vision and pattern recognition. IEEE Computer Society, Washington, DC, USA, pp 419–426. doi:10.1109/CVPR.2006.303

    Google Scholar 

  70. Kennedy LS, Naaman M (2008) Generating diverse and representative image search results for landmarks. In: WWW ’08: proceeding of the 17th international conference on world wide web. ACM, New York, NY, USA, pp 297–306. doi:10.1145/1367497.1367539

    Chapter  Google Scholar 

  71. Kennedy L, Naaman M, Ahern S, Nair R, Rattenbury T (2007) How flickr helps us make sense of the world: context and content in community-contributed media collections. In: MULTIMEDIA ’07: proceedings of the 15th international conference on multimedia. ACM, New York, NY, USA, pp 631–640. doi:10.1145/1291233.1291384

    Chapter  Google Scholar 

  72. Kherfi ML, Ziou D, Bernardi A (2004) Image retrieval from the world wide web: issues, techniques, and systems. ACM Comput Surv 36(1):35–67. doi:10.1145/1013208.1013210

    Article  Google Scholar 

  73. Kirk D, Sellen A, Rother C, Wood K (2006) Understanding photowork. In: CHI ’06: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, pp761–770. doi:10.1145/1124772.1124885

    Chapter  Google Scholar 

  74. Ku W, Kankanhalli M, Lim J (2007) Using camera settings templates to classify photos. In: International workshop on advanced image technology

  75. Lacerda Y, de Figueiredo H, de Souza Baptista C, Sampaio M (2008) Photogeo: a self-organizing system for personal photo collections, pp 258–265. doi:10.1109/ISM.2008.81

  76. Lafon Y, Bos B (2002) Describing and retrieving photos using rdf and http. http://www.w3.org/TR/photo-rdf/

  77. Lavrenko V, Manmatha R, Jeon J (2004) A model for learning the semantics of pictures. In: Advances in neural information processing systems, vol 16

  78. Lew MS (2000) Next-generation web searches for visual content. IEEE Comput 33(11):46–53. doi:10.1109/2.881694

    Google Scholar 

  79. Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: state of the art and challenges. ACM Trans Multimed Comput Commun Appl 2(1):1–19. doi:10.1145/1126004.1126005.

    Article  Google Scholar 

  80. Li J, Wang JZ (2008) Real-time computerized annotation of pictures. IEEE Trans Pattern Anal Mach Intell 30(6):985–1002

    Article  Google Scholar 

  81. Li X et al (2002) Blind image quality assessment. In: Proc. IEEE int. conf. image proc, vol 1, pp 449–452

  82. Lindley SE, Durrant AC, Kirk DS, Taylor AS (2008) Collocated social practices surrounding photos. In: CHI ’08: CHI ’08 extended abstracts on human factors in computing systems. ACM, New York, NY, USA, pp 3921–3924. doi:10.1145/1358628.1358957

    Google Scholar 

  83. Liu Y, Zhang D, Lu G, Ma WY (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40(1):262–282. doi:10.1016/j.patcog.2006.04.045

    Article  MATH  Google Scholar 

  84. Loui AC (2000) Automatic image event segmentation and quality screening for albuming applications. In: ICME, pp 1125–1128

  85. Loui AC, Savakis AE (2003) Automated event clustering and quality screening of consumer pictures for digital albuming. IEEE Trans Multimedia 5(3):390–402

    Article  Google Scholar 

  86. Lu G, Williams B (1999) An integrated WWW image retrieval system. In: Proceedings of the AusWeb99, Lismore, Australia

  87. Luo J, Boutell M, Brown C (2006) Pictures are not taken in a vacuum. IEEE Signal Process Mag 23(2):101– 114

    Article  Google Scholar 

  88. Lux M (2009) Caliph & Emir: Mpeg-7 photo annotation and retrieval. In: MM ’09: proceedings of the seventeen ACM international conference on multimedia. ACM, New York, NY, USA, pp 925–926. doi:10.1145/1631272.1631456

    Chapter  Google Scholar 

  89. Lux M, Chatzichristofis SA (2008) Lire: lucene image retrieval: an extensible java cbir library. In: MM ’08: proceeding of the 16th ACM international conference on multimedia. ACM, New York, NY, USA, pp 1085–1088. doi:10.1145/1459359.1459577

    Chapter  Google Scholar 

  90. Maekawa T, Hara T, Nishio S (2006) Image classification for mobile web browsing. In: WWW ’06: proc.of the 15th intl. conf. on world wide web. ACM, New York, NY, USA, pp 43–52. doi:10.1145/1135777.1135789

    Chapter  Google Scholar 

  91. Manjunath BS, Salembier P, Sikora T (2002) Introduction to MPEG-7: multimedia content description interface. Wiley, New York

    Google Scholar 

  92. Mansoor AB, Haider M, Mian AS, Khan SA (2009) A hybrid image quality measure for automatic image quality assessment. In: SCIA ’09: proceedings of the 16th scandinavian conference on image analysis. Springer, Berlin, Heidelberg, pp 91–98. doi:10.1007/978-3-642-02230-2_10

    Google Scholar 

  93. Marchand-Maillet S, Beretta G (2005) The benchathlon network. http://www.benchathlon.net

  94. Marchand-Maillet S, Worring M (2006) Benchmarking image and video retrieval: an overview. In: MIR ’06: proceedings of the 8th ACM international workshop on multimedia information retrieval. ACM, New York, NY, USA, pp 297–300. doi:10.1145/1178677.1178718

    Chapter  Google Scholar 

  95. Mei T, Wang B, Hua X, Zhou H, Li S (2006) Probabilistic multimodality fusion for event based home photo clustering. In: IEEE international conference on multimedia and expo, pp 1757–1760

  96. Metadata Working Group (2009) Guidelines for handling image metadata—version 1.0.1. Tech. rep., Metadata Working Group

  97. Miller G (1995) WordNet: a lexical database for english. Commun ACM 38(11):41

    Article  Google Scholar 

  98. Miller AD, Edwards WK (2007) Give and take: a study of consumer photo-sharing culture and practice. In: CHI ’07: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, pp 347–356. doi:10.1145/1240624.1240682

    Chapter  Google Scholar 

  99. Monaghan F, O’Sullivan D (2006) Automating photo annotation using services and ontologies. In: Proc IEEE international conference on mobile data management, p 79. doi:10.1109/MDM.2006.39

  100. Monaghan F, O’Sullivan D (2007) Leveraging ontologies, context and social networks to automate photo annotation. In: SAMT, pp 252–255

  101. Mori Y, Takahashi H, Oka R (1999) Image-to-word transformation based on dividing and vector quantizing images with words. In: Proceedings of the first international workshop on multimedia intelligent storage and retrieval management (MISRM’99)

  102. Mulhem P, Lim JH (2003) Home photo retrieval: time matters. In: Proc. of the second international conference on image and video retrieval (CIVR). LNCS. Springer, Urbana-Champaign, IL, USA, pp 321–330

  103. Müller H, Marchand-Maillet S, Pun T (2002) The truth about corel—evaluation in image retrieval. In: CIVR ’02: proceedings of the international conference on image and video retrieval. Springer, London, UK, pp 38–49

    Google Scholar 

  104. Müller H, Müller W, Marchand-Maillet S, Pun T, Squire D (2003) A framework for benchmarking in CBIR. Multimed Tools Appl 21(1):55–73

    Article  Google Scholar 

  105. Müller H, Müller W, Squire DM, Marchand-Maillet S, Pun T (2001) Performance evaluation in content-based image retrieval: overview and proposals. Pattern Recogn Lett 22(5):593–601. doi:10.1016/S0167-8655(00)00118-5

    Article  MATH  Google Scholar 

  106. Munson EV, Tsymbalenko Y (2001) To search for images on the web, look at the text, then look at the images. In: Proceedings of the first international workshop on web document analysis (WDA2001)

  107. Naaman M, Harada S, Wang Q, Paepcke A (2004) Adventures in space and time: browsing personal collections of geo-referenced digital photographs. Technical Report 2004-26, Stanford InfoLab. http://ilpubs.stanford.edu:8090/647/

  108. Naaman M, Song YJ, Paepcke A, Garcia-Molina H (2004) Automatic organization for digital photographs with geographic coordinates. In: JCDL ’04: proceedings of the 4th ACM/IEEE-CS joint conference on digital libraries. ACM, New York, NY, USA, pp 53–62. doi:10.1145/996350.996366

    Google Scholar 

  109. Nack F, Hardman L (2002) Towards a syntax for multimedia semantics. Tech. rep., Centrum voor Wiskunde en Informatica (CWI), Amsterdam

  110. Nair R, Reid N, Davis M (2005) Photo loi: browsing multi-user photo collections. In: MULTIMEDIA ’05: proceedings of the 13th annual ACM international conference on multimedia. ACM, New York, NY, USA, pp 223–224. doi:10.1145/1101149.1101187

    Chapter  Google Scholar 

  111. Negoescu RA, Gatica-Perez D (2008) Analyzing flickr groups. In: CIVR ’08: proceedings of the 2008 international conference on content-based image and video retrieval. ACM, New York, NY, USA, pp 417–426. doi:10.1145/1386352.1386406

    Chapter  Google Scholar 

  112. Nov O, Naaman M, Ye C (2009) Motivational, structural and tenure factors that impact online community photo sharing. In: Third international conference on weblogs and social media (ICWSM), San Jose, California

  113. Obrador P, Anguera X, de Oliveira R, Oliver N (2009) The role of tags and image aesthetics in social image search. In: WSM ’09: proceedings of the first SIGMM workshop on Social media. ACM, New York, NY, USA, pp 65–72. doi:10.1145/1631144.1631158

    Chapter  Google Scholar 

  114. O’Hare N, Gurrin C, Lee H, Murphy N, Smeaton AF, Jones GJ (2005) My digital photos: where and when? In: Proceedings of the 13th annual ACM international conference on multimedia (MM). ACM, New York, NY, USA, pp 261–262. doi:10.1145/1101149.1101196. http://portal.acm.org/citation.cfm?id=1101196#

    Chapter  Google Scholar 

  115. Oliva A, Torralba A (2002) Scene-centered description from spatial envelope properties. In: Biologically motivated computer vision, vol 2525/2010, pp 263–272

  116. On AV an IT Storage Systems, TSC, Equipment (2002) Exchangeable image file format for digital still cameras: exif version 2.2. Tech. rep., Japan Electronics and Information Technology Industries Association

  117. Ortega-Binderberger M, Mehrotra S, Chakrabarti K, Porkaew K (2000) WebMARS: a multimedia search engine

  118. Peng Wu DT (2009) Close & closer: social cluster and closeness from photo collections. In: Proceedings of MM’09, p 709

  119. Penta A, Picariello A, Tanca L (2007) Towards a definition of an image ontology. In: International workshop on database and expert systems applications, pp 74–78. doi:10.1109/DEXA.2007.83

  120. Petersen T (1994) Art and architecture thesaurus. Oxford, New York

    Google Scholar 

  121. Pigeau A (2010) Myownlife: incremental and hierarchical classification of a personal image collection on mobile devices. Multimed Tools Appl 46(2–3):289–306. doi:10.1007/s11042-009-0373-x

    Article  Google Scholar 

  122. Pigeau A, Gelgon M (2005) Building and tracking hierarchical geographical & temporal partitions for image collection management on mobile devices. In: MULTIMEDIA ’05: proceedings of the 13th annual ACM international conference on multimedia. ACM, New York, NY, USA, pp 141–150. doi:10.1145/1101149.1101170

    Chapter  Google Scholar 

  123. Platt J (2000) Autoalbum: clustering digital photographs using probabalistic model merging. citeseer.ist.psu.edu/platt00autoalbum.html

  124. Platt JC, Czerwinski M, Field BA (2002) Phototoc: automatic clustering for browsing personal photographs. Tech. Rep. MSR-TR-2002-17, Microsoft Research. http://research.microsoft.com/~jplatt/abstracts/phototoc.html

  125. Ren K, Calic J (2009) FreeEye—interactive intuitive interface for large-scale image browsing. In: ACM multimedia, p 757

  126. Rodden K, Wood KR (2003) How do people manage their digital photographs? In: Gilbert Cockton PK (ed) Proceedings of the conference on human factors and computing systems, ACM, pp 409–416

  127. Sandhaus P, Boll S (2009) From usage to annotation: analysis of personal photo albums for semantic photo understanding. In: Proceedings of the first SIGMM workshop on social media co-located with acm multimedia conference, Bejing, China

  128. Sandhaus P, Boll S, Fageth R (2008) Employing a photo’s life cycle for multimedia retrieval. In: MS ’08: proceeding of the 2nd ACM workshop on multimedia semantics. ACM, New York, NY, USA, pp 56–59. doi:10.1145/1460676.1460688

    Chapter  Google Scholar 

  129. Sandhaus P, Thieme S, Boll S (2008) Processes of photo book production. Multimedia Syst 14(6):351–357

    Article  Google Scholar 

  130. Santini S, Gupta A, Jain R (2001) Emergent semantics through interaction in image databases. IEEE Trans Knowl Data Eng 13(3):337–351. doi:10.1109/69.929893

    Article  Google Scholar 

  131. Scherp A, Jain R (2007) Towards an ecosystem for semantics. In: MS ’07: workshop on multimedia information retrieval on the many faces of multimedia semantics. ACM, New York, NY, USA, pp 3–12. doi:10.1145/1290067.1290069

    Chapter  Google Scholar 

  132. Scherp A, Boll S, Cremer H (2006) Emergent semantics in personalized multimedia content. In: Fourth special workshop on multimedia semantics (WMS), Chania, Greece

  133. Sclaroff S, La Cascia M, Sethi S: Unifying textual and visual cues for content-based image retrieval on the world wide web. Comput Vis Image Underst 75(1–2):86–98 (1999). doi:10.1006/cviu.1999.0765

    Article  Google Scholar 

  134. Shadbolt N, Lee TB, Hall W (2006) The semantic web revisited. In: IEEE intelligent systems, vol 21, pp 96–101. doi:10.1109/MIS.2006.62

  135. Shirahatti NV, Barnard K (2005) Evaluating image retrieval. In: CVPR ’05: proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 1. IEEE Computer Society, Washington, DC, USA, pp 955–961. doi:10.1109/CVPR.2005.147

    Google Scholar 

  136. Sinha P, Jain R (2008) Classification and annotation of digital photos using optical context data. In: CIVR. ACM, New York, NY, USA, pp 309–318. doi:10.1145/1386352.1386394

    Chapter  Google Scholar 

  137. Sinha P, Jain R (2008) Semantics in digital photos: a contenxtual analysis. In: ICSC ’08: proceedings of the 2008 IEEE international conference on semantic computing. IEEE Computer Society, Washington, DC, USA, pp 58–65. doi:10.1109/ICSC.2008.87

    Chapter  Google Scholar 

  138. Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380. doi:10.1109/34.895972

    Article  Google Scholar 

  139. Sun X, Yao H, Ji R, Liu S (2009) Photo assessment based on computational visual attention model. In: Proceedings of MM’09, p 541

  140. Tan T, Chen J, Mulhem P, Kankanhalli M (2002) SmartAlbum: a multi-modal photo annotation system. In: MULTIMEDIA ’02: proceedings of the tenth ACM international conference on multimedia. ACM, New York, NY, USA, pp 87–88. doi:10.1145/641007.641025

    Chapter  Google Scholar 

  141. Tong H, Li M, Zhang H, He J, Zhang C (2004) Classification of digital photos taken by photographers or home users. In: Aizawa K, Nakamura Y, Satoh S (eds) PCM (1). Lecture notes in computer science, vol 3331. Springer, pp 198–205

  142. Troncy R, van Ossenbruggen J, Pan JZ, Stamou G (2007) Image annotation on the semantic web. W3c note, W3C. http://www.w3.org/2005/Incubator/mmsem/XGR-image-annotation/

  143. Trust G (2000) Union list of artists names online

  144. Tsymbalenko Y, Munson EV (2001) Using HTML metadata to find relevant images on the world wide web. In: Proceedings of internet computing 2001, vol 2. CSREA Press, Las Vegas, pp 842–848

    Google Scholar 

  145. Tuffield MM, Gibbins N, O’Hara K, Brewster C, Dupplaw DP, Wilks Y, Sleeman D, Shadbolt NR (2006) Image annotation with photocopain. In: International workshop on semantic web annotations for multimedia

  146. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86. doi:10.1162/jocn.1991.3.1.71

    Article  Google Scholar 

  147. Vailaya A, Figueiredo M, Jain AK, Zhang J (1999) Content-based hierarchical classification of vacation images. Tech. Rep. MSU-CPS-99-9, Department of Computer Science, Michigan State University, East Lansing, Michigan

  148. Van House N, Davis M, Ames M, Finn M, Viswanathan V (2005) The uses of personal networked digital imaging: an empirical study of cameraphone photos and sharing. In: CHI ’05: CHI ’05 extended abstracts on human factors in computing systems. ACM, New York, NY, USA, pp 1853–1856. doi:10.1145/1056808.1057039

    Chapter  Google Scholar 

  149. Van House N, Davis M, Takhteyev Y, Ames M, Finn M (2004) The social uses of personal photography: methods for projecting future imaging applications. University of California, Berkeley, Working Papers 3, 2005

  150. Viana W, Filho JB, Gensel J, Oliver MV, Martin H (2007) Photomap—automatic spatiotemporal annotation for mobile photos. In: W2GIS’07: proceedings of the 7th international conference on web and wireless geographical information systems. Springer, Berlin, Heidelberg, pp 187–201

    Chapter  Google Scholar 

  151. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. citeseer.ist.psu.edu/article/viola01rapid.html

  152. Waal H, Couprie L, Tholen E, Vellekoop G (1985) Iconclass: an iconographic classification system. North-Holland, Amsterdam, The Netherlands

  153. Wagenaar WA (1986) My memory: a study of autobiographical memory over six years. Cogn Psychol 18(2):225–252. doi:10.1016/0010-0285(86)90013-7. http://www.sciencedirect.com/science/article/B6WCR-4D6YVW4-1M/2/0b3ea4936d6016c24e6e896cc48c15e3

    Article  Google Scholar 

  154. Wang YM, Zhang H (2004) Detecting image orientation based on low-level visual content. Comput Vis Image Underst (CVIU) 3(3):328–346

    Article  Google Scholar 

  155. Wang Z, Bovik A, Lu L (2002) Why is image quality assessment so difficult? In: IEEE international conference on acoustics speech and signal processing, vol 4. IEEE, pp 3313–3316

  156. Wenyin L, Sun Y, Zhang H (2000) MiAlbum—a system for home photo managemet using the semi-automatic image annotation approach. In: MULTIMEDIA ’00: proceedings of the eighth ACM international conference on multimedia. ACM, New York, NY, USA, pp 479–480. doi:10.1145/354384.379011

    Chapter  Google Scholar 

  157. Yan R, Fleury MO, Merler M, Natsev A, Smith JR (2009) Large-scale multimedia semantic concept modeling using robust subspace bagging and mapreduce. In: LS-MMRM ’09: proceedings of the first ACM workshop on large-scale multimedia retrieval and mining. ACM, New York, NY, USA, pp 35–42. doi:10.1145/1631058.1631067

    Chapter  Google Scholar 

  158. Yang MH, Kriegman DJ, Ahuja N (2002) Detecting faces in images: a survey. IEEE Trans Pattern Anal Mach Intell 24(1):34–58. doi:10.1109/34.982883

    Article  Google Scholar 

  159. Yavlinsky A, Schofield E, Rüger S (2005) Automated image annotation using global features and robust nonparametric density estimation. In: Image and video retrieval, pp 507–517

  160. You J, Perkis A, Hannuksela MM, Gabbouj M (2009) Perceptual quality assessment based on visual attention analysis. In: Proceedings of MM’09, p 561

  161. Zagoris K, Chatzichristofis SA, Papamarkos N, Boutalis YS (2009) img(anaktisi): a web content based image retrieval system. In: SISAP ’09: proceedings of the 2009 second international workshop on similarity search and applications. IEEE Computer Society, Washington, DC, USA, pp 154–155. doi:10.1109/SISAP.2009.15

    Chapter  Google Scholar 

  162. Zhang T, Chao H, Willis C, Tretter D (2010) Consumer image retrieval by estimating relation tree from family photo collections. Tech. rep., HP Laboratories

  163. Zhao W, Chellappa R, Phillips PJ, Rosenfeld A (2003) Face recognition: a literature survey. ACM Comput Surv 35(4):399–458. doi:10.1145/954339.954342

    Article  Google Scholar 

  164. Zhao M, Teo Y, Liu S, Chua TS, Jain R (2006) Automatic person annotation of family photo album. In: CIVR, pp 163–172

  165. Zhao J, Klyne G, Shotton D (2008) Building a semantic web image repository for biological research images. In: The semantic web: research and applications, pp 154–169

  166. Zhu C, Li K, Lv Q, Shang L, Dick RP (2009) Iscope: personalized multi-modality image search for mobile devices. In: MobiSys ’09: proceedings of the 7th international conference on mobile systems, applications, and services. ACM, New York, NY, USA, pp 277–290. doi:10.1145/1555816.1555845

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Health, OFFIS – Institute for Information Technology, Escherweg 2, Oldenburg, Germany

    Philipp Sandhaus

  2. Department of Computing Science - Multimedia and Internet Technologies, University of Oldenburg, Escherweg 2, Oldenburg, Germany

    Susanne Boll

Authors
  1. Philipp Sandhaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susanne Boll
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Sandhaus.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sandhaus, P., Boll, S. Semantic analysis and retrieval in personal and social photo collections. Multimed Tools Appl 51, 5–33 (2011). https://doi.org/10.1007/s11042-010-0673-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0673-1

Keywords

Search

Navigation