Multimedia Tools and Applications

, Volume 75, Issue 7, pp 3787–3811 | Cite as

Personalized multimedia content delivery on an interactive table by passive observation of museum visitors

  • Svebor KaramanEmail author
  • Andrew D. Bagdanov
  • Lea Landucci
  • Gianpaolo D’Amico
  • Andrea Ferracani
  • Daniele Pezzatini
  • Alberto Del Bimbo


The amount of multimedia data collected in museum databases is growing fast, while the capacity of museums to display information to visitors is acutely limited by physical space. Museums must seek the perfect balance of information given on individual pieces in order to provide sufficient information to aid visitor understanding while maintaining sparse usage of the walls and guaranteeing high appreciation of the exhibit. Moreover, museums often target the interests of average visitors instead of the entire spectrum of different interests each individual visitor might have. Finally, visiting a museum should not be an experience contained in the physical space of the museum but a door opened onto a broader context of related artworks, authors, artistic trends, etc. In this paper we describe the MNEMOSYNE system that attempts to address these issues through a new multimedia museum experience. Based on passive observation, the system builds a profile of the artworks of interest for each visitor. These profiles of interest are then used to drive an interactive table that personalizes multimedia content delivery. The natural user interface on the interactive table uses the visitor’s profile, an ontology of museum content and a recommendation system to personalize exploration of multimedia content. At the end of their visit, the visitor can take home a personalized summary of their visit on a custom mobile application. In this article we describe in detail each component of our approach as well as the first field trials of our prototype system built and deployed at our permanent exhibition space at LeMurate ( in Florence together with the first results of the evaluation process during the official installation in the National Museum of Bargello (


Computer vision Video surveillance Cultural heritage Multimedia museum Personalization Natural interaction Passive profiling 



This work was partially supported by Thales Italia and the MNEMOSYNE project (POR-FSE 2007-2013, A.IV-OB.2). Andrew D. Bagdanov acknowledges the support of Ramon y Cajal Fellowship RYC-2012-11776.


  1. 1.
    Anderson P, Blackwood A (2004) Mobile and pda technologies and their future use in education. In: Proceedings of JISC Technology and Standards Watch, vol. 4, pp 3–33Google Scholar
  2. 2.
    Avraham T, Gurvich I, Lindenbaum M, Markovitch S (2012) Learning implicit transfer for person re-identification. In: Proceedings of ECCV - Workshops and Demonstrations, pp 381–390Google Scholar
  3. 3.
    Baber C, Bristow H, Cheng SL, Hedley A, Kuriyama Y, Lien M, Pollard J, Sorrell P (2001) Augmenting museums and art galleries. In: 13rd Conference on Human-Computer Interaction Interact 2001, pp 439–447Google Scholar
  4. 4.
    Bagdanov AD, Del Bimbo A, Di Fina D, Karaman S, Lisanti G, Masi I (2013) Multi-target data association using sparse reconstruction. In: Proceedings of International Conference on Image Analysis and Processing (ICIAP), pp 239–248Google Scholar
  5. 5.
    Bagdanov AD, Del Bimbo A, Landucci L, Pernici F (2011) Mnemosyne: Enhancing the museum experience through interactive media and visual profiling, In: Multimedia for Cultural Heritage, pp 39–50. SpringerGoogle Scholar
  6. 6.
    Ballagas R, Rohs M, Sheridan JG, Borchers J (2004) Byod: Bring your own device. In: UbiComp 2004 Workshop on Ubiquitous Display EnvironmentsGoogle Scholar
  7. 7.
    Bangor A, Kortum PT, Miller JT (2008) An empirical evaluation of the system usability scale. Intl J Human– Comput Interact 24(6):574–594CrossRefGoogle Scholar
  8. 8.
    Bartoli F, Lisanti G, Karaman S, Bagdanov AD, Bimbo AD (2014) Unsupervised scene adaptation for faster multi-scale pedestrian detection. In: Proceedings of the 22nd International Conference on Pattern Recognition (ICPR 2014)Google Scholar
  9. 9.
    Bay H, Fasel B, Van Gool L (2006) Interactive museum guide: Fast and robust recognition of museum objects. In: Proceedings of the first international workshop on mobile visionGoogle Scholar
  10. 10.
    Bazzani L, Cristani M, Perina A, Murino V (2012) Multiple-shot person re-identification by chromatic and epitomic analyses. Pattern Recogn Lett 33(7):898–903CrossRefGoogle Scholar
  11. 11.
    Billsus D, Pazzani MJ (2000) User modeling for adaptive news access. User Model User-Adap Inter 10(2-3):147–180CrossRefGoogle Scholar
  12. 12.
    Bimbo AD, Lisanti G, Masi I, Pernici F (2010) Person detection using temporal and geometric context with a pan tilt zoom camera. In: 20th international conference on pattern recognition (ICPR). IEEE, pp 3886–3889Google Scholar
  13. 13.
    Bowen JP (2010) A brief history of early museums online. The Rutherford JournalGoogle Scholar
  14. 14.
    Bowen JP, Filippini-Fantoni S (2004) Personalization and the web from a museum perspective, In: Proceedings of the museums and the web conference (MW2004)Google Scholar
  15. 15.
    Breitenstein MD, Reichlin F, Leibe B, Koller-Meier E, Van Gool L (2009) Robust tracking-by-detection using a detector confidence particle filter, In: 12th international conference on computer vision, pp 1515–1522Google Scholar
  16. 16.
    Breitenstein MD, Reichlin F, Leibe B, Koller-Meier E, Van Gool L (2011) Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Trans Patt Anal Machine Intell 33(9):1820–1833CrossRefGoogle Scholar
  17. 17.
    Bruns E, Brombach B, Zeidler T, Bimber O (2007) Enabling mobile phones to support large-scale museum guidance. MultiMe IEEE 14(2):16–25CrossRefGoogle Scholar
  18. 18.
    Brusilovsky P, Maybury MT (2002) From adaptive hypermedia to the adaptive web. Commun ACM 45(5):30–33CrossRefGoogle Scholar
  19. 19.
    Cheng DS, Cristani M, Stoppa M, Bazzani L, Murino V (2011) Custom pictorial structures for re-identification. In: Proceedings of BMVC, vol. 2, p 6Google Scholar
  20. 20.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, vol. 1, pp 886– 893Google Scholar
  21. 21.
    Delgado J, Ishii N (1999) On-line learning of user preferences in recommender systems. In: Proceedings of international joint conference on artificial intelligence (IJCAI-99), Workshop on machine learning for information filtering, Stockholm, SwedenGoogle Scholar
  22. 22.
    Farenzena M, Bazzani L, Perina A, Murino V, Cristani M (2010) Person re-identification by symmetry-driven accumulation of local features. In: Proceedings of CVPR: 2360–2367Google Scholar
  23. 23.
    Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Patt Anal Mach Intell 32(9):1627–1645CrossRefGoogle Scholar
  24. 24.
    Hartley RI, Zisserman A (2004) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, ISBN: 0521540518Google Scholar
  25. 25.
    Hatala M, Wakkary R (2005) User modeling and semantic technologies in support of a tangible interface. JUser Model User Adap Interact 15(3-4):339–380CrossRefGoogle Scholar
  26. 26.
    Huang Z, Chen H, Zeng D (2004) Applying associative retrieval techniques to alleviate the sparsity problem in collaborative filtering. ACM Trans Inf Syst (TOIS) 22(1):116–142CrossRefGoogle Scholar
  27. 27.
    Hyvönen E (2009) Semantic portals for cultural heritage, In: Handbook on ontologies, pp 757–778. SpringerGoogle Scholar
  28. 28.
    Jin C, Takahashi S, Tanaka J (2006) Interaction between small size device and large screen in public space. In: Knowledge-based intelligent information and engineering systems, pp 197–204, SpringerGoogle Scholar
  29. 29.
    Karaman S, Bagdanov AD (2012) Identity inference: generalizing person re-identification scenarios. In: Computer Vision–ECCV 2012. Workshops and demonstrations, pp 443–452, SpringerGoogle Scholar
  30. 30.
    Karaman S, Bagdanov AD, D’Amico G, Landucci L, Ferracani A, Pezzatini D, Del Bimbo A (2013) Passive profiling and natural interaction metaphors for personalized multimedia museum experiences, In: New trends in image analysis and processing–ICIAP 2013, pp 247–256. Springer Berlin HeidelbergGoogle Scholar
  31. 31.
    Karaman S, Lisanti G, Bagdanov AD, Bimbo AD Leveraging local neighborhood topology for large scale person re-identification. Pattern Recognition (2014). doi: 10.1016/j.patcog.2014.06.003. In press
  32. 32.
    Karaman S, Lisanti G, Bagdanov AD, Del Bimbo A (2013) From re-identification to identity inference: labelling consistency by local similarity constraints, In: Person re-identification, advances in computer vision and pattern recognition, pp 287–307, SpringerGoogle Scholar
  33. 33.
    Kuflik T, Stock O, Zancanaro M, Gorfinkel A, Jbara S, Kats S, Sheidin J, Kashtan N (2011) A visitor’s guide in an active museum: Presentations, communications, and reflection. J Comput Cultural Herit (JOCCH) 3(3):11Google Scholar
  34. 34.
    Kumar R, Raghavan P, Rajagopalan S, Tomkins A (1998) Recommendation systems: a probabilistic analysis. In: Proceedings of the 39th annual symposium on foundations of computer science 1998, pp 664–673Google Scholar
  35. 35.
    Leibe B, Schindler K, Cornelis N, Van Gool L (2008) Coupled object detection and tracking from static cameras and moving vehicles. Patt Anal Mach Intell IEEE Trans 30(10):1683–1698CrossRefGoogle Scholar
  36. 36.
    Leibe B, Seemann E, Schiele B (2005) Pedestrian detection in crowded scenes. In: IEEE computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, vol. 1, pp 878–885Google Scholar
  37. 36.
    Mikolajczyk K, Schmid C, Zisserman A (2004) Human detection based on a probabilistic assembly of robust part detectors. In: Computer Vision-ECCV 2004, pp 69–82Google Scholar
  38. 38.
    Negroponte N (1996) Being digital. Random House LLCGoogle Scholar
  39. 39.
    Norman DA (1999) The invisible computer: why good products can fail, the personal computer is so complex, and information appliances are the solution. MIT pressGoogle Scholar
  40. 40.
    Papageorgiou C, Poggio T (1999) Trainable pedestrian detection. In: Proceedings of the 1999 international conference on image processing, 1999. ICIP 99, vol. 4, pp 35–39Google Scholar
  41. 41.
    Pechenizkiy M, Calders T (2007) A framework for guiding the museum tours personalization. In: Proceedings of the workshop on personalized access to Cultural Heritage (PATCH07), pp 11– 28Google Scholar
  42. 42.
    Popescul A, Pennock DM, Lawrence S (2001) Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments. In: Proceedings of the seventeenth conference on uncertainty in artificial intelligence, pp 437–444Google Scholar
  43. 43.
    Prosser B, Zheng WS, Gong S, Xiang T (2010) Person re-identification by support vector ranking. In: Proceedings of BMVCGoogle Scholar
  44. 44.
    Rashid AM, Albert I, Cosley D, Lam SK, McNee SM, Konstan JA, Riedl J (2002) Getting to know you: learning new user preferences in recommender systems. In: Proceedings of the 7th international conference on Intelligent user interfaces, pp 127–134Google Scholar
  45. 45.
    Rukzio E, Schmidt A, Hussmann H (2004) An analysis of the usage of mobile phones for personalized interactions with ubiquitous public displays. In: Workshop ubiquitous display environments in conjunction with UbiCompGoogle Scholar
  46. 46.
    Sakamura K (2003) Personalized digital museum assistant. Digital Museum: 2000Google Scholar
  47. 47.
    Sampaio I, Ramalho G, Corruble V, Prudêncio R (2006) Acquiring the preferences of new users in recommender systems-the role of item controversy. In: Proceedings of the ECAI 2006 workshop on recommender systems, pp 107–110Google Scholar
  48. 48.
    Savia E, Puolamäki K, Sinkkonen J, Kaski S (2005) Two-way latent grouping model for user preference prediction. In: Proceedings of the UAI’05, CiteseerGoogle Scholar
  49. 49.
    Sparacino F (2002) The museum wearable: real-time sensor-driven understanding of visitors’ interests for personalized visually-augmented museum experiences, In: Proceedings of Museums and the Web (MW2002)Google Scholar
  50. 50.
    Sparacino F (2004) Scenographies of the past and museums of the future: from the wunderkammer to body-driven interactive narrative spaces. In: Proceedings of the 12th annual ACM international conference on multimedia, pp 72–79. ACMGoogle Scholar
  51. 51.
    Starner T, Mann S, Rhodes B, Levine J, Healey J, Kirsch D, Picard RW, Pentland A (1997) Augmented reality through wearable computing. Presence: Teleoperators Virtual Environ 6(4):386–398CrossRefGoogle Scholar
  52. 52.
    Stock O, Not E, Zancanaro M (2005) Intelligent interactive information presentation for cultural tourism. In: Multimodal intelligent information presentation, pp 95–111, SpringerGoogle Scholar
  53. 53.
    Ungar LH, Foster DP (1998) Clustering methods for collaborative filtering. In: AAAI Workshop on recommendation systems, vol 1Google Scholar
  54. 54.
    Wang X, Han TX, Yan S (2009) An hog-lbp human detector with partial occlusion handling. In: IEEE 12th International Conference on Computer Vision, pp 32–39Google Scholar
  55. 55.
    Wang Y, Van Sambeek R, Schuurmans Y, Aroyo L, Stash N, Rutledge L, Gorgels P (2008) Be your own curator with the chip tour wizard. In: Proceedings of the museums and the web conference (MW2008)Google Scholar
  56. 56.
    Wang Y, Stash N, Sambeek R, Schuurmans Y, Aroyo L, Schreiber G, Gorgels P (2009) Cultivating personalized museum tours online and on-site. Interdisc Sci Rev 34(2-3):139–153CrossRefGoogle Scholar
  57. 57.
    Artabus Accessed: 2014-03-18
  58. 58. Accessed: 2014-03-18
  59. 59.
    Peach home page Accessed: 2014-04-07
  60. 60.
    Polo museale fiorentino :: Sito ufficiale Accessed: 2014-03-18
  61. 61.
    The san francisco museum of modern art (sfmoma) Accessed: 2014-03-18
  62. 62.
    Site officiel du musée du louvre Accessed: 2014-03-18
  63. 63.
    Tate Accessed: 2014-03-18
  64. 64.
    Virtual museum exhibit home page Accessed: 2014-03-18
  65. 65.
    Web gallery of art Accessed: 2014-03-18
  66. 66.
    Wu B, Nevatia R (2005) Detection of multiple, partially occluded humans in a single image by bayesian combination of edgelet part detectors, In: 10th IEEE international conference on computer vision, vol 1, pp 90–97Google Scholar
  67. 67.
    Zancanaro M, Stock O, Alfaro I (2003) Using cinematic techniques in a multimedia museum guide, In: Proceedings of museums and the web 2003Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Svebor Karaman
    • 1
    Email author
  • Andrew D. Bagdanov
    • 2
  • Lea Landucci
    • 1
  • Gianpaolo D’Amico
    • 1
  • Andrea Ferracani
    • 1
  • Daniele Pezzatini
    • 1
  • Alberto Del Bimbo
    • 1
  1. 1.Media Integration and Communication Center (MICC)University of FlorenceFirenzeItaly
  2. 2.Computer Vision CenterBarcelonaSpain

Personalised recommendations