The Video Browser Showdown: a live evaluation of interactive video search tools

  • Klaus Schoeffmann
  • David Ahlström
  • Werner Bailer
  • Claudiu Cobârzan
  • Frank Hopfgartner
  • Kevin McGuinness
  • Cathal Gurrin
  • Christian Frisson
  • Duy-Dinh Le
  • Manfred Del Fabro
  • Hongliang Bai
  • Wolfgang Weiss
Regular Paper


The Video Browser Showdown evaluates the performance of exploratory video search tools on a common data set in a common environment and in presence of the audience. The main goal of this competition is to enable researchers in the field of interactive video search to directly compare their tools at work. In this paper, we present results from the second Video Browser Showdown (VBS2013) and describe and evaluate the tools of all participating teams in detail. The evaluation results give insights on how exploratory video search tools are used and how they perform in direct comparison. Moreover, we compare the achieved performance to results from another user study where 16 participants employed a standard video player to complete the same tasks as performed in VBS2013. This comparison shows that the sophisticated tools enable better performance in general, but for some tasks common video players provide similar performance and could even outperform the expert tools. Our results highlight the need for further improvement of professional tools for interactive search in videos.


Video browsing Video search Video retrieval Exploratory search 



This work was funded by the Austrian Federal Ministry for Transport, Innovation and Technology (bmvit) and Austrian Science Fund (FWF): TRP 273-N15 and the European Regional Development Fund and the Carinthian Economic Promotion Fund (KWF), supported by Lakeside Labs GmbH, Klagenfurt, Austria. The research leading to these results has received funding from the European Union’s Seventh Framework Programme. (FP7/2007-2013) under grant agreement no. 287532 entitled “TOSCA-MP”. Christian Frisson has been funded by the numediart research program in digital arts technologies, grant from the Walloon Region of Belgium, from 2010 to 2013. He would like to acknowledge his colleagues part of the VideoCycle team and doctoral jury, Thierry Dutoit (advisor) and Stéphane Dupont, that lead to the participation of the team to VBS 2013. The content used in the experiments has been made available for the EBU MIM/SCAIE project by the Flemish public broadcasting organisation VRT (, Belgium.


  1. 1.
    Adams B, Greenhill S, Venkatesh S (2012) Towards a video browser for the digital native. In: Proceedings of IEEE International Conference on Multimedia and Expo Workshops, pp 127–132. doi: 10.1109/ICMEW.2012.29
  2. 2.
    Ahlström D, Hudelist MA, Schoeffmann K, Schaefer G (2012) A user study on image browsing on touchscreens. In: Proceedings of the 20th ACM International Conference on Multimedia, MM ’12, pp 925–928. ACM, New York. doi: 10.1145/2393347.2396348
  3. 3.
    Bailer W, Schoeffmann K, Ahlström D, Weiss W, del Fabro M (2013) Interactive evaluation of video browsing tools. In: Proceedings of the Multimedia Modeling Conference, pp 81–91Google Scholar
  4. 4.
    Bailer W, Weiss W, Kienast G, Thallinger G, Haas W (2010) A video browsing tool for content management in post-production. Int J Digit Multimed Broadcasting 2010:1–17. doi: 10.1155/2010/856761 CrossRefGoogle Scholar
  5. 5.
    Barnes C, Goldman DB, Shechtman E, Finkelstein A (2010) Video tapestries with continuous temporal zoom. In: ACM SIGGRAPH 2010 Papers, SIGGRAPH ’10, pp 89:1–89:9. ACM, New York. doi: 10.1145/1833349.1778826
  6. 6.
    Borgo R, Chen M, Daubney B, Grundy E, Heidemann G, H-ferlin B, H-ferlin M, Leitte H, Weiskopf D, Xie X (2012) State of the art report on video-based graphics and video visualization. Comput Graph Forum 31(8):2450–2477. doi: 10.1111/j.1467-8659.2012.03158.x CrossRefGoogle Scholar
  7. 7.
    Christel M, Stevens S, Kanade T, Mauldin M, Reddy R, Wactlar H (1996) Multimedia tools and applications. In: Furht B (ed) Techniques for the creation and exploration of digital video libraries., The Kluwer International series in engineering and computer scienceSpringer, US, pp 283–327. doi: 10.1007/978-1-4613-1387-8_8 Google Scholar
  8. 8.
    Christel MG, Smith MA, Taylor CR, Winkler DB (1998) Evolving video skims into useful multimedia abstractions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’98, pp 171–178. ACM Press/Addison-Wesley Publishing Co., New York. doi: 10.1145/274644.274670
  9. 9.
    del Fabro M, Münzer B, Böszörmenyi L (2013) AAU video browser with augmented navigation bars. In: Proceedings of the Multimedia Modeling Conference, pp 544–546Google Scholar
  10. 10.
    del Fabro M, Schoeffmann K, Böszörmenyi L (2010) Instant video browsing: a tool for fast non-sequential hierarchical video browsing. In: Proceedings of the 6th Symposium of the Workgroup Human-Computer Interaction and Usability, Engineering, pp 443–446. doi: 10.1007/978-3-642-16607-5_30
  11. 11.
    Frisson C, Dupont S, Moinet A, Picard-Limpens C, Ravet T, Siebert X, Dutoit T(2013) Videocycle: user-friendly navigation by similarity in video databases. In: Proceedings of the Multimedia Modeling Conference, pp 550–553Google Scholar
  12. 12.
    Girgensohn A, Shipman F, Wilcox L (2011) Adaptive clustering and interactive visualizations to support the selection of video clips. In: Proceedings of the 1st ACM International Conference on Multimedia Retrieval, pp 34:1–34:8. doi: 10.1145/1991996.1992030
  13. 13.
    Huber J, Steimle J, Lissermann R, Olberding S, Mühlhäuser M (2010) Wipe‘n’watch: spatial interaction techniques for interrelated video collections on mobile devices. In: Proceedings of the 24th BCS Interaction Specialist Group Conference, pp 423–427Google Scholar
  14. 14.
    Hürst W, Götz G, Welte M (2007) Interactive video browsing on mobile devices. In: Proceedings of the 15th international conference on Multimedia, pp 247–256. doi: 10.1145/1291233.1291284
  15. 15.
    Jansen M, Heeren W, van Dijk B (2008) Videotrees: improving video surrogate presentation using hierarchy. In: Content-based multimedia indexing. International Workshop on CBMI 2008, pp 560–567. doi: 10.1109/CBMI.2008.4564997
  16. 16.
    Jégou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33:117–128CrossRefGoogle Scholar
  17. 17.
    Jégou H, Douze M, Schmid C, Perez P (2010) Aggregating local descriptors into a compact image representation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3304–3311Google Scholar
  18. 18.
    Jiang J, Zhang XP (2011) A smart video player with content-based fast-forward playback. In: Proceedings of the 19th ACM international conference on Multimedia, pp 1061–1064. doi: 10.1145/2072298.2071938
  19. 19.
    Le DD, Lam V, Ngo TD, Tran VQ, Nguyen VH, Duong DA, Satoh S (2013) NII-UIT-VBS: A video browsing tool for known item search. In: Proceedings of the Multimedia Modeling Conference, pp 547–549Google Scholar
  20. 20.
    Lowe D (1999) Object recognition from local scale-invariant features. In: IEEE International Conference on Computer Vision, pp 1150–1157Google Scholar
  21. 21.
    Marchand-Maillet S, Morrison D, Szekely E, Bruno E (2010) Interactive representations of multimodal databases. In: Thiran J-P, Marques F, Bourlard H (eds) Multimodal signal processing, theory and applications for human-computer interaction. Elsevier, Amsterdam, ISBN: 9780123748256Google Scholar
  22. 22.
    Meixner B, Köstler J, Kosch H (2011) A mobile player for interactive non-linear video. In: Proceedings of the 19th ACM international conference on Multimedia, pp 779–780. doi: 10.1145/2072298.2072453
  23. 23.
    Mueller C, Smole M, Schöffmann K (2012) A demonstration of a hierarchical multi-layout 3D video browser. In: Proceedings of the IEEE International Conference on Multimedia and Expo Workshops. doi: 10.1109/ICMEW.2012.121
  24. 24.
    Over P, Awad G, Michel M, Fiscus J, Sanders G, Shaw B, Kraaij W, Smeaton AF, Quénot G (2012) Trecvid 2012: an overview of the goals, tasks, data, evaluation mechanisms and metrics. In: Proceedings of TRECVID 2012Google Scholar
  25. 25.
    de Rooij O, Snoek CG, Worring M (2008) Balancing thread based navigation for targeted video search. In: Proceedings of the 2008 international conference on content-based image and video retrieval, pp 485–494. doi: 10.1145/1386352.1386414
  26. 26.
    Schoeffmann K, Boeszoermenyi L (2011) Image and video browsing with a cylindrical 3D storyboard. In: Proceedings of the 1st ACM International Conference on Multimedia Retrieval, ICMR ’11, pp 63:1–63:2. ACM, New York. doi: 10.1145/1991996.1992059
  27. 27.
    Schoeffmann K, Cobarzan C (2013) An evaluation of interactive search with modern video players. In: IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013, pp 1–4. doi: 10.1109/ICMEW.2013.6618282
  28. 28.
    Schoeffmann K, Fabro MD (2011) Hierarchical video browsing with a 3D carousel. In: Proceedings of the 19th ACM international conference on Multimedia, pp 827–828. doi: 10.1145/2072298.2072479
  29. 29.
    Schoeffmann K, Hopfgartner F, Marques O, Böszörmenyi L, Jose JM (2010) Video browsing interfaces and applications: a review. SPIE Rev 1(1):018004. doi: 10.1117/6.0000005 Google Scholar
  30. 30.
    Schoeffmann K, Taschwer M, Böszörmenyi L (2010) The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In: Proceedings of the first annual ACM SIGMM conference on Multimedia systems, pp 247–258. doi: 10.1145/1730836.1730867
  31. 31.
    Scott D, Guo J, Gurrin C, Hopfgartner F, McGuinness K, O’Connor N, Smeaton A, Yang Y, Zhang Z (2013) DCU at MMM 2013 Video Browser Showdown. In: Proceedings of the Multimedia Modeling Conference, pp 541–543Google Scholar
  32. 32.
    Scott D, Hopfgartner F, Guo J, Gurrin C (2013) Evaluating novice and expert users on handheld video retrieval systems. In: Proceedings of MMM, pp 69–78Google Scholar
  33. 33.
    Smeaton AF, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp 321–330.
  34. 34.
    Snoek C, Worring M, de Rooij O, van de Sande K, Yan R, Hauptmann A (2008) Videolympics: real-time evaluation of multimedia retrieval systems. IEEE Multimed 15(1):86–91. doi: 10.1109/MMUL.2008.21 CrossRefGoogle Scholar
  35. 35.
    Tao K, Dong Y, Bian Y, Chang X, Bai H (2012) The France Telecom Orange Labs (Beijing) video semantic indexing systems. In: TRECVID 2012Google Scholar
  36. 36.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 511–518 (2001)Google Scholar
  37. 37.
    Wilkins P, Byrne D, Jones GJF, Lee H, Keenan G, McGuinness K, O’Connor NE, O’Hare N, Smeaton AF, Adamek T, Troncy R, Amin A, Benmokhtar R, Dumont E, Huet B, Mérialdo B, Tolias G, Spyrou E, Avrithis YS, Papadopoulos G, Mezaris V, Kompatsiaris I, Mörzinger R, Schallauer P, Bailer W, Chandramouli K, Izquierdo E, Goldmann L, Haller M, Samour A, Cobet A, Sikora T, Praks P, Hannah D, Halvey M, Hopfgartner F, Villa R, Punitha P, Goyal A, Jose JM (2008) K-space at trecvid 2008. In: TRECVIDGoogle Scholar
  38. 38.
    Worring M, Sajda P, Santini S, Shamma DA, Smeaton AF, Yang Q (2012) Where is the user in multimedia retrieval? IEEE MultiMed 19(4):6–10CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  • Klaus Schoeffmann
    • 1
  • David Ahlström
    • 1
  • Werner Bailer
    • 2
  • Claudiu Cobârzan
    • 1
  • Frank Hopfgartner
    • 3
  • Kevin McGuinness
    • 4
  • Cathal Gurrin
    • 4
  • Christian Frisson
    • 5
  • Duy-Dinh Le
    • 6
  • Manfred Del Fabro
    • 1
  • Hongliang Bai
    • 7
  • Wolfgang Weiss
    • 2
  1. 1.Alpen-Adria-Universität KlagenfurtKlagenfurtAustria
  2. 2.Joanneum ResearchGrazAustria
  3. 3.Technische Universität BerlinBerlinGermany
  4. 4.Dublin City UniversityDublinIreland
  5. 5.Université de MonsMonsBelgium
  6. 6.National Institute of InformaticsTokyoJapan
  7. 7.Orange Labs International CentersBeijingChina

Personalised recommendations