Skip to main content

Measuring quality of experience for 360-degree videos in virtual reality

Abstract

In recent years, we witness dramatic growing attention in immersive media technologies like 360-degree videos and virtual reality (VR). However, measuring the quality-of-experience (QoE) for 360-degree VR videos is not a trivial task. Streaming such videos to head mounted displays (HMDs) is extremely bandwidth-demanding when compared to traditional 2D videos. In HTTP adaptive streaming, QoE tends to deteriorate significantly during fluctuating network conditions, which results in various bitrate changes and causes multiple stalling events during playback. Thus, understanding how the human visual system perceives 360-degree video with the effect of stalling and different bitrate levels becomes inevitable. In this paper, we investigate the impact of stalling on users QoE under different bitrate levels and the interaction between stalling event and bitrate level for 360-degree videos in VR. To aim this, we first build a 360-degree videos database by encoding videos in three different bitrate levels (1, 5, and 15 Mbps) with 4K resolutions (3840 × 1920 pixels). We then simulate various stalling events in the videos and conduct a subjective experiment in a virtual reality environment to investigate the human responses. Finally, we use a Bayesian method to estimate and predict the QoE while measuring the quality drop owing to various stalling events and bitrate changes. Proposed solution and prediction results show a strong dependency between playback stalling and bitrate of 360-degree video in VR. Stalling always impacts the QoE of 360-degree videos, but the strength of this negative impact depends on the video bitrate level. The adverse effect of stalling events is more profound when bitrate level approaches to the high and low end, which is in close agreement with subjective opinion.

This is a preview of subscription content, access via your institution.

References

  1. Le Callet P, Möller S, Perkis A. Qualinet white paper on definitions of quality of experience (2012). European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). Ver.1.2. 2013

  2. Liu Y T, Yun S, Mao Y N, et al. A study on quality of experience for adaptive streaming service. In: Proceedings of IEEE International Conference on Communications Workshops (ICC), 2013. 682–686

  3. Lewcio B, Belmudez B, Enghardt T, et al. On the way to high-quality video calls in future mobile networks. In: Proceedings of the 3rd International Workshop on Quality of Multimedia Experience (QoMEX), 2011. 43–48

  4. Zare A, Aminlou A, Hannuksela M M, et al. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In: Proceedings of the 24th ACM International Conference on Multimedia, 2016. 601–605

  5. Qian F, Ji L, Han B, et al. Optimizing 360 video delivery over cellular networks. In: Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, 2016. 1–6

  6. Anwar M S, Wang J, Ullah A, et al. User profile analysis for enhancing QoE of 360 panoramic video in virtual reality environment. In: Proceedings of 2018 International Conference on Virtual Reality and Visualization (ICVRV), 2018. 106–111

  7. Zhang B, Zhao J, Yang S, et al. Subjective and objective quality assessment of panoramic videos in virtual reality environments. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2017. 163–168

  8. Li C, Xu M, Du X Z, et al. Bridge the gap between VQA and human behavior on omnidirectional video: a large-scale dataset and a deep learning model. In: Proceedings of the 26th ACM International Conference on Multimedia, 2018. 932–940

  9. Schatz R, Sackl A, Timmerer C, et al. Towards subjective quality of experience assessment for omnidirectional video streaming. In: Proceedings of 2017 9th International Conference on Quality of Multimedia Experience (QoMEX), 2017. 1–6

  10. Zou W, Yang F. Measuring quality of experience of novel 360-degree streaming video during stalling. In: Proceedings of International Conference on Communications and Networking in China, 2017. 417–424

  11. Zhang W, Zou W, Yang F. The impact of stalling on the perceptual quality of http-based omnidirectional video streaming. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019. 4060–4064

  12. Chen C, Choi L K, de Veciana G, et al. Modeling the time-varying subjective quality of HTTP video streams with rate adaptations. IEEE Trans Image Process, 2014, 23: 2206–2221

    MathSciNet  Article  Google Scholar 

  13. Singla A, Fremerey S, Robitza W, et al. Measuring and comparing qoe and simulator sickness of omnidirectional videos in different head mounted displays. In: Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX), 2017. 1–6

  14. Huyen T, Ngoc N P, Pham C T, et al. A subjective study on QoE of 360 video for VR communication. In: Proceedings of 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), 2017. 1–6

  15. Tasaka S. Bayesian hierarchical regression models for QoE estimation and prediction in audiovisual communications. IEEE Trans Multimedia, 2017, 19: 1195–1208

    Article  Google Scholar 

  16. Ahmad S, Li K, Amin A, et al. A multilayer prediction approach for the student cognitive skills measurement. IEEE Access, 2018, 6: 57470–57484

    Article  Google Scholar 

  17. Zhang Y, Wang Y, Liu F, et al. Subjective panoramic video quality assessment database for coding applications. IEEE Trans Broadcast, 2018, 64: 461–473

    Article  Google Scholar 

  18. Corbillon X, de Simone F, Simon G. 360-degree video head movement dataset. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. 199–204

  19. Lo W-C, Fan C-L, Lee J, et al. 360 video viewing dataset in head-mounted virtual reality. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. 211–216

  20. Wu C, Tan Z, Wang Z, et al. A dataset for exploring user behaviors in VR spherical video streaming. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. 193–198

  21. Staelens N, Moens S, van den Broeck W, et al. Assessing quality of experience of IPTV and video on demand services in real-life environments. IEEE Trans Broadcast, 2010, 56: 458–466

    Article  Google Scholar 

  22. International Telecommunication Union. ITU-T Rec. P.910. Subjective video quality assessment methods for multimedia applications. 2008

  23. International Telecommunication Union. ITU-R Rec. BT.500-10. Methodology for the subjective assessment of the quality of television pictures. 1993

  24. Yao S-H, Fan C-L, Hsu C-H. Towards quality-of-experience models for watching 360 videos in head-mounted virtual reality. In: Proceedings of the 11th International Conference on Quality of Multimedia Experience (QoMEX), 2019. 1–3

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jing Wang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Anwar, M.S., Wang, J., Ullah, A. et al. Measuring quality of experience for 360-degree videos in virtual reality. Sci. China Inf. Sci. 63, 202301 (2020). https://doi.org/10.1007/s11432-019-2734-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-019-2734-y

Keywords

  • quality of experience
  • stalling
  • virtual reality
  • 360-degree video