Advertisement

Evaluation of Visual Content Descriptors for Supporting Ad-Hoc Video Search Tasks at the Video Browser Showdown

  • Sabrina KletzEmail author
  • Andreas Leibetseder
  • Klaus Schoeffmann
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10704)

Abstract

Since 2017 the Video Browser Showdown (VBS) collaborates with TRECVID and interactively evaluates Ad-Hoc Video Search (AVS) tasks, in addition to Known-Item Search (KIS) tasks. In this video search competition the participants have to find relevant target scenes to a given textual query within a specific time limit, in a large dataset consisting of 600 h of video content. Since usually the number of relevant scenes for such an AVS query is rather high, the teams at the VBS 2017 could find only a small portion of them. One way to support them at the interactive search would be to automatically retrieve other similar instances of an already found target scene. However, it is unclear which content descriptors should be used for such an automatic video content search, using a query-by-example approach. Therefore, in this paper we investigate several different visual content descriptors (CNN Features, CEDD, COMO, HOG, Feature Signatures and HOF) for the purpose of similarity search in the TRECVID IACC.3 dataset, used for the VBS. Our evaluation shows that there is no single descriptor that works best for every AVS query, however, when considering the total performance over all 30 AVS tasks of TRECVID 2016, CNN features provide the best performance.

Keywords

Video Browser Showdown Ad-Hoc Video Search TRECVID Video retrieval Similarity search Content descriptors 

Notes

Acknowledgement

This work is supported by the Alpen-Adria University Klagenfurt and Lakeside Labs GmbH, Klagenfurt, Austria and funding from the European Regional Development Fund and the Carinthian Economic Promotion Fund (KWF) under grant KWF 20214 u. 3520/26336/38165.

References

  1. 1.
    Awad, G., Fiscus, J., Michel, M., Joy, D., Kraaij, W., Smeaton, A.F., Quénot, G., Eskevich, M., Aly, R., Ordelman, R.: TRECVID 2016: evaluating video search, video event detection, localization, and hyperlinking. In: Proceedings of TRECVID (2016)Google Scholar
  2. 2.
    Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 584–599. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_38 Google Scholar
  3. 3.
    Beecks, C., Kirchhoff, S., Seidl, T.: Signature matching distance for content-based image retrieval. In: Proceedings of 3rd International ACM Conference on Multimedia Retrieval (2013)Google Scholar
  4. 4.
    Blaz̆ek, A., Lokoc̆, J., Kubon̆, D.: Video hunter at VBS 2017. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 493–498. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_47 CrossRefGoogle Scholar
  5. 5.
    Chatzichristofis, S.A., Boutalis, Y.S.: CEDD: color and edge directivity descriptor: a compact descriptor for image indexing and retrieval. In: Gasteratos, A., Vincze, M., Tsotsos, J.K. (eds.) ICVS 2008. LNCS, vol. 5008, pp. 312–322. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-79547-6_30 CrossRefGoogle Scholar
  6. 6.
    Cisco: The Zettabyte Era: Trends and Analysis. Technical report, Cisco (2017). http://tinyurl.com/cisco-trends-2017
  7. 7.
    Cobârzan, C., Schoeffmann, K., Bailer, W., Hürst, W., Blažek, A., Lokoč, J., Vrochidis, S., Barthel, K.U., Rossetto, L.: Interactive video search tools: a detailed analysis of the video browser showdown 2015. Multimedia Tools Appl. 76(4), 5539–5571 (2017)CrossRefGoogle Scholar
  8. 8.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. I, pp. 886–893. IEEE (2005)Google Scholar
  9. 9.
    Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 428–441. Springer, Heidelberg (2006).  https://doi.org/10.1007/11744047_33 CrossRefGoogle Scholar
  10. 10.
    Hürst, W., Ching, A.I.V., Schoeffmann, K., Primus, M.J.: Storyboard-based video browsing using color and concept indices. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 480–485. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_45 CrossRefGoogle Scholar
  11. 11.
    Joe Yue-Hei, N., Fan, Y., Davis, L.S.: Exploiting local features from deep networks for image retrieval. In: Proceedings of IEEE Conference Workshop on Computer Vision and Pattern Recognition, pp. 53–61 (2015)Google Scholar
  12. 12.
    Kletz, S., Schoeffmann, K., Münzer, B., Primus, J.M., Husslein, H.: Surgical action retrieval for assisting video review of laparoscopic skills. In: Proceedings of ACMMM Conference Workshop on Educational and Knowledge Technologies (2017)Google Scholar
  13. 13.
    Laptev, I., Marszałek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: Proceedings of 26th IEEE Conference on Computer Vision and Pattern Recognition (2008)Google Scholar
  14. 14.
    Lu, Y.-J., Nguyen, P.A., Zhang, H., Ngo, C.-W.: Concept-based interactive search system. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 463–468. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_42 CrossRefGoogle Scholar
  15. 15.
    Moumtzidou, A., et al.: VERGE in VBS 2017. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 486–492. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_46 CrossRefGoogle Scholar
  16. 16.
    Nguyen, V.-T., Ngo, T.D., Le, D.-D., Tran, M.-T., Duong, D.A., Satoh, S.: Semantic extraction and object proposal for video search. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 475–479. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_44 CrossRefGoogle Scholar
  17. 17.
    Rossetto, L., Giangreco, I., Tănase, C., Schuldt, H., Dupont, S., Seddati, O.: Enhanced retrieval and browsing in the IMOTION system. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 469–474. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_43 CrossRefGoogle Scholar
  18. 18.
    Sanou, B.: World in 2016: ICT Facts and Figures. Technical report, International Telecommunication Union (ITU) (2017). http://tinyurl.com/itu-facts-2016
  19. 19.
    Schoeffmann, K., Hudelist, M.A., Huber, J.: Video interaction tools: a survey of recent work. ACM Comput. Surv. 48(1), 14:1–14:34 (2015)Google Scholar
  20. 20.
    Schoeffmann, K., Primus, M.J., Muenzer, B., Petscharnig, S., Karisch, C., Xu, Q., Huerst, W.: Collaborative feature maps for interactive video search. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133, pp. 457–462. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51814-5_41 CrossRefGoogle Scholar
  21. 21.
    Schoeffmann, K.: A user-centric media retrieval competition: the video browser showdown 2012–2014. IEEE MultiMedia 21(4), 8–13 (2014)CrossRefGoogle Scholar
  22. 22.
    Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: Proceedings of 8th ACM International Workshop on Multimedia Information Retrieval, p. 321. ACM Press (2006)Google Scholar
  23. 23.
    Vassou, S.A., Amanatiadis, A., Christodoulou, K., Chatzichristoos, S.A.: CoMo: a compact composite moment-based descriptor for image retrieval. In: Proceedings of 15th International Workshop on Content-Based Multimedia Indexing (2017)Google Scholar
  24. 24.
    Yilmaz, E., Kanoulas, E., Aslam, J.A.: A simple and efficient sampling method for estimating AP and NDCG. In: Proceedings of 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 603. ACM Press (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Sabrina Kletz
    • 1
    Email author
  • Andreas Leibetseder
    • 1
  • Klaus Schoeffmann
    • 1
  1. 1.Institute of Information TechnologyAlpen-Adria University (AAU)KlagenfurtAustria

Personalised recommendations