Advertisement

Combining Boolean and Multimedia Retrieval in vitrivr for Large-Scale Video Search

  • Loris SauterEmail author
  • Mahnaz Amiri Parian
  • Ralph Gasser
  • Silvan Heller
  • Luca Rossetto
  • Heiko Schuldt
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11962)

Abstract

This paper presents the most recent additions to the vitrivr multimedia retrieval stack made in preparation for the participation to the 9\(^{th}\) Video Browser Showdown (VBS) in 2020. In addition to refining existing functionality and adding support for classical Boolean queries and metadata filters, we also completely replaced our storage engine \(\textsf {ADAM}_{pro}\) by a new database called Cottontail DB. Furthermore, we have added support for scoring based on the temporal ordering of multiple video segments with respect to a query formulated by the user. Finally, we have also added a new object detection module based on Faster-RCNN and use the generated features for object instance search.

Keywords

Video Browser Showdown Interactive video retrieval 

Notes

Acknowledgements

This work was partly supported by the Hasler Foundation in the context of the project City-Stories (contract no. 17055).

References

  1. 1.
    Berns, F., Rossetto, L., Schoeffmann, K., Beecks, C., Awad, G.: V3C1 dataset: an evaluation of content characteristics. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 334–338. ACM (2019)Google Scholar
  2. 2.
    Chen, B.C., Davis, L.S., Lim, S.N.: An analysis of object embeddings for image retrieval. arXiv preprint arXiv:1905.11903 (2019)
  3. 3.
    Gasser, R., Rossetto, L., Schuldt, H.: Towards an all-purpose content-based multimedia information retrieval system. arXiv preprint arXiv:1902.03878 (2019)
  4. 4.
    Giangreco, I.: Database support for large-scale multimedia retrieval. Ph.D. thesis, University of Basel (2018)Google Scholar
  5. 5.
    Giangreco, I., Schuldt, H.: ADAM\(_{pro}\): database support for big multimedia retrieval. Datenbank-Spektrum 16(1), 17–26 (2016)CrossRefGoogle Scholar
  6. 6.
    Gurrin, C., et al.: A test collection for interactive lifelog retrieval. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11295, pp. 312–324. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-05710-7_26CrossRefGoogle Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
  8. 8.
    Krasin, I., et al.: Openimages: a public dataset for large-scale multi-label and multi-class image classification (2017). Dataset available from https://storage.googleapis.com/openimages/web/index.html
  9. 9.
    Lokoč, J., et al.: Interactive search or sequential browsing? A detailed analysis of the video browser showdown 2018. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 15(1), 29 (2019)Google Scholar
  10. 10.
    Lokoč, J., Kovalčík, G., Souček, T., Moravec, J., Čech, P.: VIRET: a video retrieval tool for interactive known-item search. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 177–181. ACM (2019)Google Scholar
  11. 11.
    Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. CoRR abs/1506.01497 (2015). http://arxiv.org/abs/1506.01497
  12. 12.
    Rossetto, L., Berns, F., Schoeffman, K., Awad, G., Beeks, C.: The V3C1 dataset: advancing the state of the art in video retrieval. ACM SIGMultimedia Rec. 11(2) (2019) Google Scholar
  13. 13.
    Rossetto, L., Gasser, R., Heller, S., Amiri Parian, M., Schuldt, H.: Retrieval of structured and unstructured data with vitrivr. In: Proceedings of the ACM Workshop on Lifelog Search Challenge, pp. 27–31. ACM (2019)Google Scholar
  14. 14.
    Rossetto, L., Gasser, R., Schuldt, H.: Query by semantic sketch. CoRR abs/1909.12526 (2019). https://arxiv.org/abs/1909.12526
  15. 15.
    Rossetto, L., Giangreco, I., Heller, S., Tănase, C., Schuldt, H.: Searching in video collections using sketches and sample images – the Cineast system. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9517, pp. 336–341. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-27674-8_30CrossRefGoogle Scholar
  16. 16.
    Rossetto, L., et al.: IMOTION – searching for video sequences using multi-shot sketch queries. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9517, pp. 377–382. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-27674-8_36CrossRefGoogle Scholar
  17. 17.
    Rossetto, L., et al.: IMOTION—a content-based video retrieval engine. In: He, X., Luo, S., Tao, D., Xu, C., Yang, J., Hasan, M.A. (eds.) MMM 2015. LNCS, vol. 8936, pp. 255–260. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-14442-9_24CrossRefGoogle Scholar
  18. 18.
    Rossetto, L., Giangreco, I., Tanase, C., Schuldt, H.: Vitrivr: a flexible retrieval stack supporting multiple query modes for searching in multimedia collections. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 1183–1186. ACM (2016)Google Scholar
  19. 19.
    Rossetto, L., Amiri Parian, M., Gasser, R., Giangreco, I., Heller, S., Schuldt, H.: Deep learning-based concept detection in vitrivr. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11296, pp. 616–621. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-05716-9_55CrossRefGoogle Scholar
  20. 20.
    Rossetto, L., Parian, M.A., Gasser, R., Giangreco, I., Heller, S., Schuldt, H.: Deep learning-based concept detection in vitrivr at the video browser showdown 2019-final notes. arXiv preprint arXiv:1902.10647 (2019)
  21. 21.
    Rossetto, L., Schuldt, H., Awad, G., Butt, A.A.: V3C – a research video collection. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11295, pp. 349–360. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-05710-7_29CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Mathematics and Computer ScienceUniversity of BaselBaselSwitzerland
  2. 2.Department of InformaticsUniversity of ZurichZurichSwitzerland
  3. 3.Numediart InstituteUniversity of MonsMonsBelgium

Personalised recommendations