Skip to main content
SpringerLink
Log in

Competitive Interactive Video Retrieval in Virtual Reality with vitrivr-VR

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12573))

Included in the following conference series:

Abstract

Virtual Reality (VR) has emerged and developed as a new modality to interact with multimedia data. In this paper, we present vitrivr-VR, a prototype of an interactive multimedia retrieval system in VR based on the open source full-stack multimedia retrieval system vitrivr. We have implemented query formulation tailored to VR: Users can use speech-to-text to search collections via text for concepts, OCR and ASR data as well as entire scene descriptions through a video-text co-embedding feature that embeds sentences and video sequences into the same feature space. Result presentation and relevance feedback in vitrivr-VR leverages the capabilities of virtual spaces.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://vitrivr.org.

  2. 2.

    https://unity.com.

  3. 3.

    https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@0.9.

  4. 4.

    https://docs.microsoft.com/en-us/windows/mixed-reality/voice-input-in-unity.

References

  1. Duane, A., Þór Jónsson, B., Gurrin, C.: VRLE: lifelog interaction prototype in virtual reality: lifelog search challenge at ACM ICMR 2020. In: Proceedings of the Third Annual Workshop on Lifelog Search Challenge (2020)

    Google Scholar 

  2. Gasser, R., Rossetto, L., Heller, S., Schuldt, H.: Cottontail DB: an open source database system for multimedia retrieval and analysis. In: Proceedings of the 28th ACM International Conference on Multimedia (2020)

    Google Scholar 

  3. Giunchi, D., James, S., Steed, A.: 3D sketching for interactive model retrieval in virtual reality. In: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering (2018)

    Google Scholar 

  4. Heller, S., et al.: Towards explainable interactive multi-modal video retrieval with vitrivr. In: International Conference on Multimedia Modeling MMM (2021)

    Google Scholar 

  5. Heller, S., Parian, M.A., Gasser, R., Sauter, L., Schuldt, H.: Interactive lifelog retrieval with vitrivr. In: Proceedings of the Third ACM Workshop on Lifelog Search Challenge (2020)

    Google Scholar 

  6. Heller, S., Parian, M., Pasquinelli, M., Schuldt, H.: Vitrivr-explore: guided multimedia collection exploration for ad-hoc video search. In: Satoh, S., et al. (eds.) SISAP 2020. LNCS, vol. 12440, pp. 379–386. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60936-8_30

    Chapter  Google Scholar 

  7. Heller, S., Sauter, L., Schuldt, H., Rossetto, L.: Multi-stage queries and temporal scoring in vitrivr. In: IEEE International Conference on Multimedia & Expo Workshops (2020)

    Google Scholar 

  8. Jónsson, B.Þ., Khan, O.S., Koelma, D.C., Rudinac, S., Worring, M., Zahálka, J.: Exquisitor at the video browser showdown 2020. In: Ro, Y., et al. (eds.) MMM 2020. LNCS, vol. 11962, pp. 796–802. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2_72

    Chapter  Google Scholar 

  9. Khanwalkar, S., Balakrishna, S., Jain, R.: Exploration of large image corpuses in virtual reality. In: Proceedings of the 24th ACM International Conference on Multimedia (2016)

    Google Scholar 

  10. Kratochvíl, M., Veselý, P., Mejzlík, F., Lokoč, J.: SOM-hunter: video browsing with relevance-to-SOM feedback loop. In: Ro, Y., et al. (eds.) MMM 2020. LNCS, vol. 11962, pp. 790–795. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2_71

    Chapter  Google Scholar 

  11. Li, X., Xu, C., Yang, G., Chen, Z., Dong, J.: W2VV++ fully deep learning for ad-hoc video search. In: Proceedings of the 27th ACM International Conference on Multimedia (2019)

    Google Scholar 

  12. Lokoč, J., et al.: A W2VV++ case study with automated and interactive text-to-video retrieval. In: ACM Multimedia (2020)

    Google Scholar 

  13. Nakazato, M., Huang, T.S.: 3D MARS: immersive virtual reality for content-based image retrieval. In: IEEE International Conference on Multimedia and Expo (2001)

    Google Scholar 

  14. Rossetto, L., et al.: Interactive video retrieval in the age of deep learning-detailed evaluation of VBS 2019. IEEE Trans. Multimed. 23, 243–256 (2020)

    Article  Google Scholar 

  15. Rossetto, L., Giangreco, I., Heller, S., Tănase, C., Schuldt, H.: Searching in video collections using sketches and sample images – the Cineast system. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9517, pp. 336–341. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27674-8_30

    Chapter  Google Scholar 

  16. Rossetto, L., Giangreco, I., Tanase, C., Schuldt, H.: Vitrivr: a flexible retrieval stack supporting multiple query modes for searching in multimedia collections. In: Proceedings of the 24th ACM International Conference on Multimedia (2016)

    Google Scholar 

  17. Rossetto, L., Parian, M.A., Gasser, R., Giangreco, I., Heller, S., Schuldt, H.: Deep learning-based concept detection in vitrivr. In: International Conference on Multimedia Modeling (2019)

    Google Scholar 

  18. Sauter, L., Parian, M.A., Gasser, R., Heller, S., Rossetto, L., Schuldt, H.: Combining Boolean and multimedia retrieval in vitrivr for large-scale video search. In: International Conference on Multimedia Modeling (2020)

    Google Scholar 

Download references

Acknowledgements

This work was partly supported by the Hasler Foundation in the context of the project City-Stories (contract no. 17055).

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, University of Basel, Basel, Switzerland

    Florian Spiess, Ralph Gasser, Silvan Heller, Loris Sauter & Heiko Schuldt

  2. Department of Informatics, University of Zurich, Zurich, Switzerland

    Luca Rossetto

Authors
  1. Florian Spiess
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ralph Gasser
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Silvan Heller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luca Rossetto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Loris Sauter
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Heiko Schuldt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Florian Spiess .

Editor information

Editors and Affiliations

  1. Charles University, Prague, Czech Republic

    Jakub Lokoč

  2. Charles University, Prague, Czech Republic

    Tomáš Skopal

  3. Klagenfurt University, Klagenfurt, Austria

    Klaus Schoeffmann

  4. CERTH-ITI, Thessaloniki, Greece

    Vasileios Mezaris

  5. Renmin University of China, Beijing, China

    Xirong Li

  6. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  7. Queen Mary University of London, London, UK

    Ioannis Patras

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Spiess, F., Gasser, R., Heller, S., Rossetto, L., Sauter, L., Schuldt, H. (2021). Competitive Interactive Video Retrieval in Virtual Reality with vitrivr-VR. In: Lokoč, J., et al. MultiMedia Modeling. MMM 2021. Lecture Notes in Computer Science(), vol 12573. Springer, Cham. https://doi.org/10.1007/978-3-030-67835-7_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67835-7_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67834-0

  • Online ISBN: 978-3-030-67835-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation