Advertisement

Two-Layer FoV Prediction Model for Viewport Dependent Streaming of 360-Degree Videos

  • Yunqiao Li
  • Yiling XuEmail author
  • Shaowei Xie
  • Liangji Ma
  • Jun Sun
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 262)

Abstract

As the representative and most widely used content form of Virtual Reality (VR) application, omnidirectional videos provide immersive experience for users with 360-degree scenes rendered. Since only part of the omnidirectional video can be viewed at a time due to human’s eye characteristics, field of view (FoV) based transmission has been proposed by ensuring high quality in the FoV while reducing the quality out of that to lower the amount of transmission data. In this case, transient content quality reduction will occur when the user’s FoV changes, which can be improved by predicting the FoV beforehand. In this paper, we propose a two-layer model for FoV prediction. The first layer detects the heat maps of content in offline process, while the second layer predicts the FoV of a specific user online during his/her viewing period. We utilize a LSTM model to calculate the viewing probability of each region given the results from the first layer, the user’s previous orientations and the navigation speed. In addition, we set up a correction model to check and correct the unreasonable results. The performance evaluation shows that our model obtains higher accuracy and less undulation compared with widely used approaches.

Keywords

Omnidirectional video Field of view prediction FoV-based transmission 

Notes

Acknowledgements

This paper is supported in part by National Natural Science Foundation of China (61650101), Scientific Research Plan of the Science and Technology Commission of Shanghai Municipality (16511104203), in part by the 111 Program (B07022).

References

  1. 1.
    Virtual and augmented reality: Understanding the race for the next computing platform. Technical report. The Goldman Sachs Group Inc. (2016)Google Scholar
  2. 2.
    Bao, Y., Wu, H., Zhang, T., Ramli, A.A., Liu, X.: Shooting a moving target: motion-prediction-based transmission for 360-degree videos. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 1161–1170. IEEE (2016)Google Scholar
  3. 3.
    Corbillon, X., Simon, G., Devlic, A., Chakareski, J.: Viewport-adaptive navigable 360-degree video delivery. In: 2017 IEEE International Conference on Communications (ICC), pp. 1–7. IEEE (2017)Google Scholar
  4. 4.
    De Abreu, A., Ozcinar, C., Smolic, A.: Look around you: Saliency maps for omnidirectional images in vr applications. In: 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6. IEEE (2017)Google Scholar
  5. 5.
    Fan, C.L., Lee, J., Lo, W.C., Huang, C.Y., Chen, K.T., Hsu, C.H.: Fixation prediction for 360 video streaming in head-mounted virtual reality. In: Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 67–72. ACM (2017)Google Scholar
  6. 6.
    Graf, M., Timmerer, C., Mueller, C.: Towards bandwidth efficient adaptive streaming of omnidirectional video over http: Design, implementation, and evaluation. In: Proceedings of the 8th ACM on Multimedia Systems Conference, pp. 261-271. ACM (2017)Google Scholar
  7. 7.
    Hochreiter, Sepp, Schmidhuber, Jürgen: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  8. 8.
    Mohammad H., Swaminathan, V.: Adaptive 360 vr video streaming: Divide and conquer. In: 2016 IEEE International Symposium on Multimedia (ISM), pp. 107–110. IEEE (2016)Google Scholar
  9. 9.
    Hu, Y., Xie, S., Xu, Y., Sun, J.: Dynamic VR live streaming over MMT. In: 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pp. 1–4. IEEE (2017)Google Scholar
  10. 10.
    Lipton, Z.C., Berkowitz, J., Elkan, C.: A critical review of recurrent neural networks for sequence learning (2015). arXiv:1506.00019
  11. 11.
    Lo, W.C., Fan, C.L., Lee, J., Huang, C.Y., Chen, K.T., Hsu, C.H.: 360 video viewing dataset in head-mounted virtual reality. In: Proceedings of the 8th ACM on Multimedia Systems Conference, pp. 211–216 (2017)Google Scholar
  12. 12.
    Qian, F., Ji, L., Han, B., Gopalakrishnan, V.: Optimizing 360 video delivery over cellular networks. In: Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, pp. 1–6 (2016)Google Scholar
  13. 13.
    Sitzmann, Vincent, Serrano, Ana, Pavel, Amy, Agrawala, Maneesh, Gutierrez, Diego, Masia, Belen, Wetzstein, Gordon: Saliency in vr: How do people explore virtual environments? IEEE Trans. Vis. Comput. Graph. 24(4), 1633–1642 (2018)CrossRefGoogle Scholar
  14. 14.
    Wu, Z., Su, L., Huang, Q., Wu, B., Li, J., Li, G.: Video saliency prediction with optimized optical flow and gravity center bias. In: 2016 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2016)Google Scholar
  15. 15.
    Xie, S., Xu, Y., Qian, Q., Shen, Q. Ma, Z., Zhang, W.: Modeling the perceptual impact of viewport adaptation for immersive video. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5. IEEE (2018)Google Scholar
  16. 16.
    Xu, M., Song, Y., Wang, J., Qiao, M., Huo, L., Wang, Z.: Modeling Attention in Panoramic Video: A Deep Reinforcement Learning Approach (2017)Google Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2019

Authors and Affiliations

  • Yunqiao Li
    • 1
  • Yiling Xu
    • 1
    Email author
  • Shaowei Xie
    • 1
  • Liangji Ma
    • 1
  • Jun Sun
    • 1
  1. 1.Shanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations