Dynamic Narratives for Heritage Tour

  • Anurag Ghosh
  • Yash Patel
  • Mohak Sukhwani
  • C. V. Jawahar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9913)


We present a dynamic story generation approach for the egocentric videos from the heritage sites. Given a short video clip of a ‘heritage-tour’ our method selects a series of short descriptions from the collection of pre-curated text and create a larger narrative. Unlike in the past, these narratives are not merely monotonic static versions from simple retrievals. We propose a method to generate on the fly dynamic narratives of the tour. The series of the text messages selected are optimised over length, relevance, cohesion and information simultaneously. This results in ‘tour guide’ like narratives which are seasoned and adapted to the participants selection of the tour path. We simultaneously use visual and gps cues for precision localization on the heritage site which is conceptually formulated as a graph. The efficacy of the approach is demonstrated on a heritage site, Golconda Fort, situated in Hyderabad, India. We validate our approach on two hours of data collected over multiple runs across the site for our experiments.


Storytelling Digital heritage Egocentric perception 


  1. 1.
    Ikeuchi, K., Oishi, T., Takamatsu, J., Sagawa, R., Nakazawa, A., Kurazume, R., Nishino, K., Kamakura, M., Okamoto, Y.: The great buddha project: digitally archiving, restoring, and analyzing cultural heritage objects. In: IJCV (2007)Google Scholar
  2. 2.
    Adabala, N., Datha, N., Joy, J., Kulkarni, C., Manchepalli, A., Sankar, A., Walton, R.: An interactive multimedia framework for digital heritage narratives. In: ACMMM (2010)Google Scholar
  3. 3.
    Torii, A., Sivic, J., Pajdla, T.: Visual localization by linear combination of image descriptors. In: ICCV Workshop (2011)Google Scholar
  4. 4.
    Van Aart, C., Wielinga, B., Van Hage, W.R.: Mobile cultural heritage guide: location-aware semantic search. In: Knowledge Engineering and Management by the Masses (2010)Google Scholar
  5. 5.
    Panda, J., Brown, M.S., Jawahar, C.V.: Offline mobile instance retrieval with a small memory footprint. In: ICCV (2013)Google Scholar
  6. 6.
    Lu, Z., Grauman, K.: Story-driven summarization for egocentric video. In: CVPR (2013)Google Scholar
  7. 7.
    Chen, X., Yuille, A.L.: A time-efficient cascade for real-time object detection: with applications for the visually impaired. In: CVPR (2005)Google Scholar
  8. 8.
    Ezaki, N., Bulacu, M., Schomaker, L.: Text detection from natural scene images: towards a system for visually impaired persons. In: ICPR (2004)Google Scholar
  9. 9.
    Schwarze, T., Lauer, M., Schwaab, M., Romanovas, M., Bohm, S., Jurgensohn, T.: An intuitive mobility aid for visually impaired people based on stereo vision. In: ICCV Workshops (2015)Google Scholar
  10. 10.
    Rodríguez, A., Yebes, J.J., Alcantarilla, P.F., Bergasa, L.M., Almazán, J., Cela, A.: Assisting the visually impaired: obstacle detection and warning system by acoustic feedback. In: Sensors (2012)Google Scholar
  11. 11.
    Pradeep, V., Medioni, G., Weiland, J.: Robot vision for the visually impaired. In: CVPR (2010)Google Scholar
  12. 12.
    Lin, T.Y., Belongie, S., Hays, J.: Cross-view image geolocalization. In: CVPR (2013)Google Scholar
  13. 13.
    Martin, E., Vinyals, O., Friedland, G., Bajcsy, R.: Precise indoor localization using smart phones. In: ACMMM (2010)Google Scholar
  14. 14.
    Bay, H., Fasel, B., Gool, L.V.: Interactive museum guide. In: UBICOMP Workshop (2005)Google Scholar
  15. 15.
    Schroth, G., Huitl, R., Chen, D., Abu-Alqumsan, M., Al-Nuaimi, A., Steinbach, E.: Mobile visual location recognition. Signal Process. Mag. 28(4), 77–89 (2011)CrossRefGoogle Scholar
  16. 16.
    Kim, G., Xing, E.P.: Reconstructing storyline graphs for image recommendation from web community photos. In: CVPR (2014)Google Scholar
  17. 17.
    Wang, D., Li, T., Ogihara, M.: Generating pictorial storylines via minimum-weight connected dominating set approximation in multi-view graphs. In: AAAI (2012)Google Scholar
  18. 18.
    Riedl, M.O., Young, R.M.: From linear story generation to branching story graphs. IEEE Comput. Graph. Appl. 26(3), 23–31 (2006)CrossRefGoogle Scholar
  19. 19.
    Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., Courville, A.: Describing videos by exploiting temporal structure. In: ICCV (2015)Google Scholar
  20. 20.
    Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., Schiele, B.: Translating video content to natural language descriptions. In: ICCV (2013)Google Scholar
  21. 21.
    Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., Saenko, K.: Sequence to sequence-video to text. In: ICCV (2015)Google Scholar
  22. 22.
    Arandjelović, R., Zisserman, A.: Three things everyone should know to improve object retrieval. In: CVPR (2012)Google Scholar
  23. 23.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. In: IJCV (2004)Google Scholar
  24. 24.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)Google Scholar
  25. 25.
    Hu, C., Chen, W., Chen, Y., Liu, D.: Adaptive Kalman filtering for vehicle navigation. Positioning 1(04) (2009)Google Scholar
  26. 26.
    Tolman, B.W.: GPS precise absolute positioning via Kalman filtering. Ionosphere 2(L1), L2 (2008)MathSciNetGoogle Scholar
  27. 27.
    Marchal, F., Hackney, J., Axhausen, K.: Efficient map-matching of large GPS data sets-tests on a speed monitoring experiment in Zurich. Arbeitsbericht Verkehrs-und Raumplanung (2004)Google Scholar
  28. 28.
    Mitchell, S., OSullivan, M., Dunning, I.: PuLP: a linear programming toolkit for python. The University of Auckland, New Zealand (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Anurag Ghosh
    • 1
  • Yash Patel
    • 1
  • Mohak Sukhwani
    • 1
  • C. V. Jawahar
    • 1
  1. 1.CVIT, IIIT HyderabadHyderabadIndia

Personalised recommendations