Skip to main content

Competitive Interactive Video Retrieval in Virtual Reality with vitrivr-VR

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12573)

Abstract

Virtual Reality (VR) has emerged and developed as a new modality to interact with multimedia data. In this paper, we present vitrivr-VR, a prototype of an interactive multimedia retrieval system in VR based on the open source full-stack multimedia retrieval system vitrivr. We have implemented query formulation tailored to VR: Users can use speech-to-text to search collections via text for concepts, OCR and ASR data as well as entire scene descriptions through a video-text co-embedding feature that embeds sentences and video sequences into the same feature space. Result presentation and relevance feedback in vitrivr-VR leverages the capabilities of virtual spaces.

Keywords

  • Video Browser Showdown
  • Virtual Reality
  • Interactive video retrieval

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-67835-7_42
  • Chapter length: 7 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-67835-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 1.

Notes

  1. 1.

    https://vitrivr.org.

  2. 2.

    https://unity.com.

  3. 3.

    https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@0.9.

  4. 4.

    https://docs.microsoft.com/en-us/windows/mixed-reality/voice-input-in-unity.

References

  1. Duane, A., Þór Jónsson, B., Gurrin, C.: VRLE: lifelog interaction prototype in virtual reality: lifelog search challenge at ACM ICMR 2020. In: Proceedings of the Third Annual Workshop on Lifelog Search Challenge (2020)

    Google Scholar 

  2. Gasser, R., Rossetto, L., Heller, S., Schuldt, H.: Cottontail DB: an open source database system for multimedia retrieval and analysis. In: Proceedings of the 28th ACM International Conference on Multimedia (2020)

    Google Scholar 

  3. Giunchi, D., James, S., Steed, A.: 3D sketching for interactive model retrieval in virtual reality. In: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering (2018)

    Google Scholar 

  4. Heller, S., et al.: Towards explainable interactive multi-modal video retrieval with vitrivr. In: International Conference on Multimedia Modeling MMM (2021)

    Google Scholar 

  5. Heller, S., Parian, M.A., Gasser, R., Sauter, L., Schuldt, H.: Interactive lifelog retrieval with vitrivr. In: Proceedings of the Third ACM Workshop on Lifelog Search Challenge (2020)

    Google Scholar 

  6. Heller, S., Parian, M., Pasquinelli, M., Schuldt, H.: Vitrivr-explore: guided multimedia collection exploration for ad-hoc video search. In: Satoh, S., et al. (eds.) SISAP 2020. LNCS, vol. 12440, pp. 379–386. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60936-8_30

    CrossRef  Google Scholar 

  7. Heller, S., Sauter, L., Schuldt, H., Rossetto, L.: Multi-stage queries and temporal scoring in vitrivr. In: IEEE International Conference on Multimedia & Expo Workshops (2020)

    Google Scholar 

  8. Jónsson, B.Þ., Khan, O.S., Koelma, D.C., Rudinac, S., Worring, M., Zahálka, J.: Exquisitor at the video browser showdown 2020. In: Ro, Y., et al. (eds.) MMM 2020. LNCS, vol. 11962, pp. 796–802. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2_72

    CrossRef  Google Scholar 

  9. Khanwalkar, S., Balakrishna, S., Jain, R.: Exploration of large image corpuses in virtual reality. In: Proceedings of the 24th ACM International Conference on Multimedia (2016)

    Google Scholar 

  10. Kratochvíl, M., Veselý, P., Mejzlík, F., Lokoč, J.: SOM-hunter: video browsing with relevance-to-SOM feedback loop. In: Ro, Y., et al. (eds.) MMM 2020. LNCS, vol. 11962, pp. 790–795. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2_71

    CrossRef  Google Scholar 

  11. Li, X., Xu, C., Yang, G., Chen, Z., Dong, J.: W2VV++ fully deep learning for ad-hoc video search. In: Proceedings of the 27th ACM International Conference on Multimedia (2019)

    Google Scholar 

  12. Lokoč, J., et al.: A W2VV++ case study with automated and interactive text-to-video retrieval. In: ACM Multimedia (2020)

    Google Scholar 

  13. Nakazato, M., Huang, T.S.: 3D MARS: immersive virtual reality for content-based image retrieval. In: IEEE International Conference on Multimedia and Expo (2001)

    Google Scholar 

  14. Rossetto, L., et al.: Interactive video retrieval in the age of deep learning-detailed evaluation of VBS 2019. IEEE Trans. Multimed. 23, 243–256 (2020)

    CrossRef  Google Scholar 

  15. Rossetto, L., Giangreco, I., Heller, S., Tănase, C., Schuldt, H.: Searching in video collections using sketches and sample images – the Cineast system. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9517, pp. 336–341. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27674-8_30

    CrossRef  Google Scholar 

  16. Rossetto, L., Giangreco, I., Tanase, C., Schuldt, H.: Vitrivr: a flexible retrieval stack supporting multiple query modes for searching in multimedia collections. In: Proceedings of the 24th ACM International Conference on Multimedia (2016)

    Google Scholar 

  17. Rossetto, L., Parian, M.A., Gasser, R., Giangreco, I., Heller, S., Schuldt, H.: Deep learning-based concept detection in vitrivr. In: International Conference on Multimedia Modeling (2019)

    Google Scholar 

  18. Sauter, L., Parian, M.A., Gasser, R., Heller, S., Rossetto, L., Schuldt, H.: Combining Boolean and multimedia retrieval in vitrivr for large-scale video search. In: International Conference on Multimedia Modeling (2020)

    Google Scholar 

Download references

Acknowledgements

This work was partly supported by the Hasler Foundation in the context of the project City-Stories (contract no. 17055).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Florian Spiess .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Spiess, F., Gasser, R., Heller, S., Rossetto, L., Sauter, L., Schuldt, H. (2021). Competitive Interactive Video Retrieval in Virtual Reality with vitrivr-VR. In: , et al. MultiMedia Modeling. MMM 2021. Lecture Notes in Computer Science(), vol 12573. Springer, Cham. https://doi.org/10.1007/978-3-030-67835-7_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67835-7_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67834-0

  • Online ISBN: 978-3-030-67835-7

  • eBook Packages: Computer ScienceComputer Science (R0)