Querying Multiple Simultaneous Video Streams with 3D Interest Maps

  • Axel CarlierEmail author
  • Lilian Calvet
  • Pierre Gurdjos
  • Vincent Charvillat
  • Wei Tsang Ooi
Part of the Multimedia Systems and Applications book series (MMSA)


With proliferation of mobile devices equipped with cameras and video recording applications, it is now common to observe multiple mobile cameras filming the same scene at an event from a diverse set of view angles. These recorded videos provide a rich set of data for someone to re-experience the event at a later time. Not all the videos recorded, however, show a desirable view. Navigating through a large collection of videos to find a video with a better viewing angle can be time consuming. We propose a query-response interface in which users can intuitively switch to another video with an alternate, better, view, by selecting a 2D region within a video as a query. The system would then response with another video that has a better view of the selected region, maximizing the viewpoint entropy. The key to our system is a lightweight 3D scene structure, also termed 3D interest map. A 3D interest map is naturally an extension of saliency maps in the 3D space since most users film what they find interesting from their respective viewpoints. A user study with more than 35 users shows that our video query system achieves a suitable compromise between accuracy and run-time.


  1. 1.
    Calvet, L., Gurdjos, P., Charvillat, V.: Camera tracking using concentric circle markers: paradigms and algorithms. In: ICIP, pp. 1361–1364 (2012)Google Scholar
  2. 2.
    Carlier, A., Calvet, L., Nguyen, D.T.D., Ooi, W.T., Gurdjos, P., Charvillat, V.: 3D interest maps from simultaneous video recordings. In: Proceedings of the 22nd ACM International Conference on Multimedia, MM ’14, pp. 577–586 (2014)Google Scholar
  3. 3.
    Chandra, S., Chiu, P., Back, M.: Towards portable, high definition multi-camera video capture using smartphone for tele-immersion. In: IEEE International Symposium on Multimedia (2013)Google Scholar
  4. 4.
    Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)CrossRefGoogle Scholar
  5. 5.
    Dezfuli, N., Huber, J., Olberding, S., Mühlhäuser, M.: CoStream: in-situ co-construction of shared experiences through mobile video sharing during live events. In: CHI Extended Abstracts, pp. 2477–2482 (2012)Google Scholar
  6. 6.
    Ercan, A.O., Yang, D.B., Gamal, A.E., Guibas, L.J.: Optimal placement and selection of camera network nodes for target localization. In: IEEE DCOSS, pp. 389–404 (2006)Google Scholar
  7. 7.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010)CrossRefGoogle Scholar
  8. 8.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004). ISBN: 0521540518CrossRefzbMATHGoogle Scholar
  9. 9.
    Kansal, A., Zhao, F.: Location and mobility in a sensor network of mobile phones. In: ACM Network and Operating Systems Support for Digital Audio and Video (2007)Google Scholar
  10. 10.
    Ke, Y., Hoiem, D., Sukthankar, R.: Computer vision for music identification. In: CVPR (1), pp. 597–604 (2005)Google Scholar
  11. 11.
    Kim, J.S., Gurdjos, P., Kweon, I.S.: Geometric and algebraic constraints of projected concentric circles and their applications to camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 27(4), 637–642 (2005)CrossRefGoogle Scholar
  12. 12.
    Lee, H., Tessens, L., Morbée, M., Aghajan, H.K., Philips, W.: Sub-optimal camera selection in practical vision networks through shape approximation. In: ACIVS, pp. 266–277 (2008)Google Scholar
  13. 13.
    Oliensis, J., Hartley, R.: Iterative extensions of the sturm/triggs algorithm: convergence and nonconvergence. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2217–2233 (2007)CrossRefGoogle Scholar
  14. 14.
    Philip Kelly, C.O.C., Kim, C., O’Connor, N.E.: Automatic camera selection for activity monitoring in a multi-camera system for tennis. In: ACM/IEEE International Conference on Distributed Smart Cameras (2009)Google Scholar
  15. 15.
    Pollefeys, M., Gool, L.J.V., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Koch, R.: Visual modeling with a hand-held camera. Int. J. Comput. Vis. 59(3), 207–232 (2004)CrossRefGoogle Scholar
  16. 16.
    Saini, M.K., Gadde, R., Yan, S., Ooi, W.T.: MoViMash: online mobile video mashup. In: ACM Multimedia, pp. 139–148 (2012)Google Scholar
  17. 17.
    Saini, M., Venkatagiri, S.P., Ooi, W.T., Chan, M.C.: The Jiku mobile video dataset. In: Proceedings of the Fourth Annual ACM SIGMM Conference on Multimedia Systems, MMSys, Oslo (2013)CrossRefGoogle Scholar
  18. 18.
    Shrestha, P., Barbieri, M., Weda, H., Sekulovski, D.: Synchronization of multiple camera videos using audio-visual features. IEEE Trans. Multimedia 12(1), 79–92 (2010)CrossRefGoogle Scholar
  19. 19.
    Shrestha, P., de With, P.H.N., Weda, H., Barbieri, M., Aarts, E.H.L.: Automatic mashup generation from multiple-camera concert recordings. In: ACM Multimedia, pp. 541–550 (2010)Google Scholar
  20. 20.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. 25(3), 835–846 (2006)CrossRefGoogle Scholar
  21. 21.
    Sturm, P.F.: Algorithms for plane-based pose estimation. In: CVPR, pp. 1706–1711 (2000)Google Scholar
  22. 22.
    Tanskanen, P., Kolev, K., Meier, L., Camposeco, F., Saurer, O., Pollefeys, M.: Live metric 3D reconstruction on mobile phones. In: ICCV (2013)CrossRefGoogle Scholar
  23. 23.
    Tatler, B.W.: The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. J. Vis. 7(14), 1–17 (2007)CrossRefGoogle Scholar
  24. 24.
    Tessens, L., Morbée, M., Lee, H., Philips, W., Aghajan, H.K.: Principal view determination for camera selection in distributed smart camera networks. In: IEEE ICDSC, pp. 1–10 (2008)Google Scholar
  25. 25.
    Vázquez, P.P., Feixas, M., Sbert, M., Heidrich, W.: Viewpoint selection using viewpoint entropy. In: Proceedings of VMV 2001, pp. 273–280 (2001)Google Scholar
  26. 26.
    Wang, C., Shen, H.W.: Information theory in scientific visualization. Entropy 13(1), 254–273 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Axel Carlier
    • 1
    Email author
  • Lilian Calvet
    • 2
  • Pierre Gurdjos
    • 1
  • Vincent Charvillat
    • 1
  • Wei Tsang Ooi
    • 3
  1. 1.IRIT, UMR 5505Université ToulouseToulouseFrance
  2. 2.Simula Research LaboratoryFornebuNorway
  3. 3.School of ComputingNational University of Singapore119077Singapore

Personalised recommendations