Multimedia Tools and Applications

, Volume 76, Issue 5, pp 7141–7173 | Cite as

Time and space for segmenting personal photo sets

  • Nuno DatiaEmail author
  • João Moura Pires
  • Nuno Correia


A personal collection of photos shows large variability in the depicted items, making difficult a fully automated solution to cope with sensory and semantic gaps. Emotions and non-visual contextual information can be very important to address those problems. Manual annotations are key, but their time-consuming nature alienate users from doing them. One solution is to lower the annotation effort, building solutions on top of algorithms that prepare a context separation, making possible the reuse of annotations. In this paper we present a segmentation algorithm that uses spatio-temporal information to segment personal photo collections. The algorithm is assessed in a user study, using the participants own photos. The results show users make none or few changes to the proposed segmentations, indicating an acceptance of the algorithm outcome.


Empirical user study Segmentation algorithm Formalisation Personal photo collections 


  1. 1.
    Allen J (1983) Maintaining knowledge about temporal intervals. Commun ACM 26(11):832–843CrossRefzbMATHGoogle Scholar
  2. 2.
    Breunig M M, Kriegel H P, Ng R T, Sander J (2000) Lof: identifying density-based local outliers. SIGMOD Rec 29(2):93–104. doi: 10.1145/335191.335388 CrossRefGoogle Scholar
  3. 3.
    Bruneau P, Pigeau A, Gelgon M, Picarougne F (2010) Geo-temporal structuring of a personal image database with two-level variational-bayes mixture estimation. In: Detyniecki M, Leiner U, Nrnberger A (eds) adaptive multimedia retrieval. Identifying, summarizing, and recommending image and music, lecture notes in computer science, vol 5811. Springer, Berlin Heidelberg, pp 127–139CrossRefGoogle Scholar
  4. 4.
    Cao L, Luo J, Kautz HS, Huang TS (2008) Annotating collections of photos using hierarchical event and scene models. In: CVPR, IEEE Computer Society. doi: 10.1109/CVPR.2008.4587382
  5. 5.
    Cobley P, Haeffner N (2009) Digital cameras and domestic photography: communication, agency and structure. Vis Commun 8(2):123–146CrossRefGoogle Scholar
  6. 6.
    Cohen J (1988) Statistical power analysis for the behavioral sciences. Psychology PressGoogle Scholar
  7. 7.
    Cohen J (1992) A power primer. Psychol Bull 112(1):155CrossRefGoogle Scholar
  8. 8.
    Comaniciu D, Meer P (2002) Mean shift: A robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619CrossRefGoogle Scholar
  9. 9.
    Cooper M, Foote J, Girgensohn A, Wilcox L (2005) Temporal event clustering for digital photo collections. ACM Transactions on Multimedia Computing. Communicat Appl (TOMCCAP) 1(3):269– 288Google Scholar
  10. 10.
    Cooper ML (2011) Clustering geo-tagged photo collections using dynamic programming. In: Proceedings of the 19th ACM International Conference on Multimedia, MM ’11. ACM, New York, pp 1025–1028. doi: 10.1145/2072298.2071929
  11. 11.
    Datia N, Moura-Pires J, Correia N (2014) Summarised presentation of personal photo sets. In: Gurrin C, Hopfgartner F, Hurst W, Johansen H, Lee H, OConnor N (eds) MultiMedia modeling, lecture notes in computer science, vol 8325. Springer International Publishing, pp 195–206Google Scholar
  12. 12.
    Datta R, Joshi D, Li J, Wang J Z (2008) Image retrieval: Ideas, influences, and trends of the new age. ACM Comput Surv 40 (2):1–60. doi: 10.1145/1348246.1348248 CrossRefGoogle Scholar
  13. 13.
    Do TMT, Blom J, Gatica-Perez D (2011) Smartphone usage in the wild: a large-scale analysis of applications and context. In: Proceedings of the 13th international conference on multimodal interfaces, ICMI ’11. ACM, New York, pp 353–360. doi: 10.1145/2070481.2070550
  14. 14.
    Foote J (2000) Automatic audio segmentation using a measure of audio novelty. In: International Conference on Multimedia and Expo, ICME 2000, vol. 1, pp 452–455Google Scholar
  15. 15.
    Friedman W (2004) Time in autobiographical memory. Social Cognition 22(Special issue):591–605. doi: 10.1521/soco.22.5.591.50766
  16. 16.
    Gargi U (2003) Consumer media capture: Time-based analysis and event clustering. Tech. rep., Technical Report HPL-2003-165, HP LaboratoriesGoogle Scholar
  17. 17.
    Georgescul M, Clark A, Armstrong S (2006) An analysis of quantitative aspects in the evaluation of thematic segmentation algorithms. In: Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, Association for Computational Linguistics, pp 144–151Google Scholar
  18. 18.
    Gozali J, Kan M, Sundaram H (2012) Hidden markov model for event photo stream segmentation. In: 2012 IEEE international conference on Multimedia and Expo Workshops (ICMEW). IEEE, pp 25–30Google Scholar
  19. 19.
    Graham A, Garcia-Molina H, Paepcke A, Winograd T (2002) Time as essence for photo browsing through personal digital libraries. In: Proceedings of the second ACM/IEEE-CS joint conference on Digital libraries, pp 326–335Google Scholar
  20. 20.
    Gye L (2007) Picture this: the impact of mobile camera phones on personal photographic practices. Continuum 21(2):279–288CrossRefGoogle Scholar
  21. 21.
    House N A V (2009) Collocated photo sharing, story-telling, and the performance of self. International Journal of Human-Computer Studies 67(12):1073–1086. doi: 10.1016/j.ijhcs.09.003 CrossRefGoogle Scholar
  22. 22.
    Janssen S, Chessa A, Murre J (2006) Memory for time: how people date events. Mem Cogn 34(1):138CrossRefGoogle Scholar
  23. 23.
    Johnson J, et al. (2010) Designing with the mind in mind: simple guide to understanding user interface design rules. Morgan KaufmannGoogle Scholar
  24. 24.
    Kang H, Bederson B B, Suh B (2007) Capture, annotate, browse, find, share: Novel interfaces for personal photo management. Int J Hum Comput Interact 23 (3):315–337. doi: 10.1080/10447310701702618 CrossRefGoogle Scholar
  25. 25.
    Kellerman A (1989) Time, space, and society: geographical societal perspectives. Kluwer Academic PubGoogle Scholar
  26. 26.
    Kirk D S, Sellen A (2010) On human remains: Values and practice in the home archiving of cherished objects. ACM Transactions on Computer-Human Interaction (TOCHI) 17(3):10. doi: 10.1145/1806923.1806924 CrossRefGoogle Scholar
  27. 27.
    Kwok S C, Shallice T (2012) Macaluso, E. Functional anatomy of temporal organisation and domain-specificity of episodic memory retrieval. Neuropsychologia 50 (12):2943–2955. doi: 10.1016/j.neuropsychologia2012.07.025 Google Scholar
  28. 28.
    Latif K, Mustofa K, Tjoa A (2006) An approach for a personal information management system for photos of a lifetime by exploiting semantics. In: Bressan S, Kng J, Wagner R (eds) Database and expert systems applications, lecture notes in computer science, vol 4080. Springer Berlin Heidelberg, pp 467–477. doi: 10.1007/11827405_46
  29. 29.
    Lietz P (2010) Research into questionnaire design. Int J Mark Res 52(2):249–272CrossRefGoogle Scholar
  30. 30.
    Loui A, Savakis A (2003) Automated event clustering and quality screening of consumer pictures for digital albuming. IEEE Trans Multimedia 5:390–402CrossRefGoogle Scholar
  31. 31.
    Lux M, Kogler M, del Fabro M (2010) Why did you take this photo: a study on user intentions in digital photo productions.. In: Proceedings of the 2010 ACM workshop on Social, adaptive and personalized multimedia interaction and access, SAPMIA ’10. ACM, New York, pp 41–44. doi: 10.1145/1878061.1878075
  32. 32.
    McGill R, Tukey J W, Larsen W A (1978) Variations of box plots. Am Stat 32(1):12–16Google Scholar
  33. 33.
    Naaman M, Song Y J, Paepcke A, Garcia-Molina H (2004) Automatic organization for digital photographs with geographic coordinates. In: Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, JCDL ’04. ACM Press, New York, pp 53–62Google Scholar
  34. 34.
    Nielsen J (1994) Usability inspection methods. In: Conference companion on Human factors in computing systems. ACM, pp 413–414Google Scholar
  35. 35.
    Pevzner L, Hearst M A (2002) A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics 28(1):19–36CrossRefGoogle Scholar
  36. 36.
    Platt J, Czerwinski M, Field B (2003) Phototoc: automatic clustering for browsing personal photographs. Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia Proceedings of the 2003 Joint Conference of the Fourth International Conference on 1 1:6–10CrossRefGoogle Scholar
  37. 37.
    Reeves L M, Lai J, Larson J A, Oviatt S, Balaji T S, Buisine S, Collings P, Cohen P, Kraal B, Martin J C, McTear M, Raman T, Stanney K M, Su H, Wang Q Y (2004) Guidelines for multimodal user interface design. Commun ACM 47(1):57–59CrossRefGoogle Scholar
  38. 38.
    Seltman HJ (2012) Experimental design and analysis. Online at:
  39. 39.
    St Jacques P, Rubin D, LaBar K, Cabeza R (2008) The short and long of it: Neural correlates of temporal-order memory for autobiographical events. J Cognitive Neurosci 20(7):1327–1341, cited By (since 1996)29Google Scholar
  40. 40.
    Sun F, Li H, Wang X (2013) Photo 4w: mobile photo management on what, where, who and when. Neurocomputing intelligent Processing Techniques for Semantic-based Image and Video Retrieval 119:59–64. doi: 10.1016/j.neucom.2012.03.038
  41. 41.
    Sun Y, Zhang H, Zhang L, Li M (2002) Myphotos: a system for home photo management and processing. In: Proceedings of the 10th ACM international conference on Multimedia ’02. ACM Press, New York, pp 81–82. doi: 10.1145/641007.641022
  42. 42.
    Tulving E (2002) Episodic memory: from mind to brain. Annual Review of Psychology 53(1):1–25CrossRefGoogle Scholar
  43. 43.
    Viana W, Bringel Filho J, Gensel J, Villanova-Oliver M, Martin H (2008) PhotoMap: from location and time to context-aware photo annotations. Journal of Location Based Services 2(3):211–235CrossRefGoogle Scholar
  44. 44.
    von Watzdorf S, Michahelles F (2010) Accuracy of positioning data on smartphones. In: Proceedings of the 3rd International Workshop on Location and the Web, ACM, p 2Google Scholar
  45. 45.
    Whittaker S, Bergman O, Clough P (2010) Easy on that trigger dad: a study of long term family photo retrieval. Pers Ubiquit Comput 14(1):31–43CrossRefGoogle Scholar
  46. 46.
    Zerubavel E (1985) Hidden rhythms: schedules and calendars in social life. University of California PressGoogle Scholar
  47. 47.
    Zerubavel E (1996) Social memories: steps to a sociology of the past. Qual Sociol 19(3):283–299CrossRefGoogle Scholar
  48. 48.
    Zhao M, Teo Y, Liu S, Chua TS, Jain R (2006) Automatic person annotation of family photo album. In: Sundaram H, Naphade M, Smith J, Rui Y (eds) Image and video retrieval, lecture notes in computer science, vol 4071. Springer, Berlin Heidelberg, pp 163–172. doi: 10.1007/11788034_17 CrossRefGoogle Scholar
  49. 49.
    Zuzanek J, Smale J (1993) Life-cycle variations in across-the-week allocation of time to selected daily activities. SOCIETY AND LEISURE-MONTREAL- 15:559–559Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.ISELInstituto Politécnico de LisboaLisboaPortugal
  2. 2.FCTUniversidade Nova de LisboaCaparicaPortugal

Personalised recommendations