Skip to main content
SpringerLink
Log in

Visual attention-aware quality estimation framework for omnidirectional video using spherical Voronoi diagram

  • Research Article
  • Published:
Quality and User Experience Aims and scope Submit manuscript

Abstract

Omnidirectional video (ODV) enables viewers to look at every direction from a fixed point and provides a much more immersive experience than traditional 2D video. Assessing the video quality is important for delivering ODV to the end-user with the best possible quality. For this goal, two aspects of ODV should be considered. The first is the spherical nature of ODV and the related projection distortions when the ODV is stored in a planar format. The second is the interactive look-around consumption nature of ODV. Related to this aspect, visual attention, that identifies the regions that attract the viewer’s attention, is important for ODV quality assessment. Considering these aspects, in this paper, we study in particular objective full-reference quality assessment for ODV. To this end, we propose a quality assessment framework based on the spherical Voronoi diagram and visual attention. In this framework, a given ODV is subdivided into multiple planar patches with low projection distortions using the spherical Voronoi diagram. Afterwards, each planar patch is analyzed separately by a quality metric for traditional 2D video, obtaining a quality score for each patch. Then, the patch scores are combined based on visual attention into a final quality score. To validate the proposed framework, we create a dataset of ODVs with scaling and compression distortions, and conduct subjective experiments in order to gather the subjective quality scores and the visual attention data for our ODV dataset. The evaluation of the proposed framework based on our dataset shows that both the use of the spherical Voronoi diagram and visual attention are crucial for achieving state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://v-sense.scss.tcd.ie/research/voronoi-based-objective-metrics/.

References

  1. Knorr S, Ozcinar C, Fearghail CO, Smolic A (2018) Director’s cut—a combined dataset for visual attention analysis in cinematic VR content. In: The 15th ACM SIGGRAPH European conference on visual media production. https://doi.org/10.1145/3278471.3278472

  2. Rana A, Ozcinar C, Smolic A (2019) Towards generating ambisonics using audio-visual cue for virtual reality. In: 44th international conference on acoustics, speech, and signal processing (ICASSP)

  3. Ozcinar C, De Abreu A, Knorr S, Smolic A (2017) Estimation of optimal encoding ladders for tiled \(360^{\circ }\) VR video in adaptive streaming systems. In: The 19th IEEE international symposium on multimedia (ISM 2017). Taichung, Taiwan

  4. Warburton DE, Bredin SS, Horita LT, Zbogar D, Scott JM, Esch BT, Rhodes RE (2007) The health benefits of interactive video game exercise. Appl Physiol Nutrit Metabol 32(4):655–663

    Article  Google Scholar 

  5. Freina L, Ott M (2015) A literature review on immersive virtual reality in education: state of the art and perspectives. In: The international scientific conference elearning and software for education, vol 1. “ Carol I” National Defence University, p 133

  6. Ozcinar C, De Abreu A, Smolic A (2017) Viewport-aware adaptive 360\(^{\circ }\) video streaming using tiles for virtual reality. In: 2017 international conference on image processing (ICIP). Beijing, China

  7. Möller S, Raake A (2014) Quality of experience: advanced concepts, applications and methods. Springer, Berlin

    Book  Google Scholar 

  8. Sun Y, Lu A, Yu L (2017) Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE Signal Process Lett 24(9):1408–1412. https://doi.org/10.1109/LSP.2017.2720693

    Article  Google Scholar 

  9. Zakharchenko V, Choi KP, Park JH (2016) Quality metric for spherical panoramic video. Proc SPIE 9970:9970. https://doi.org/10.1117/12.2235885

    Article  Google Scholar 

  10. Yu M, Lakshman H, Girod B (2015) A framework to evaluate omnidirectional video coding schemes. In: 2015 IEEE international symposium on mixed and augmented reality, pp 31–36. https://doi.org/10.1109/ISMAR.2015.12

  11. Li C, Xu M, Du X, Wang Z (2018) Bridge the gap between VQA and human behavior on omnidirectional video: A large-scale dataset and a deep learning model. CoRR abs/1807.10990

  12. Upenik E, Ebrahimi T (2019) Saliency driven perceptual quality metric for omnidirectional visual content. In: 2019 IEEE international conference on image processing (ICIP), pp 4335–4339. https://doi.org/10.1109/ICIP.2019.8803637

  13. Ozcinar C, Cabrera J, Smolic A (2019) Visual attention-aware omnidirectional video streaming using optimal tiles for virtual reality. IEEE J Emerg Sel Topics Circuits Syst 9(1):217–230. https://doi.org/10.1109/JETCAS.2019.2895096

    Article  Google Scholar 

  14. Ye Y, Alshina E, Boyce J (2017) Algorithm descriptions of projection format conversion and video quality metrics in 360lib. Technical Report JVET-F1003, ISO/IEC JTC1/SC29/WG11/N16888, Hobart, AU

  15. Sun W, Gu K, Ma S, Zhu W, Liu N, Zhai G (2018) A Large-Scale compressed 360-degree spherical image database: From subjective quality evaluation to objective model comparison. In: 2018 IEEE 20th international workshop on multimedia signal processing (MMSP), pp 1–6. https://doi.org/10.1109/MMSP.2018.8547102

  16. Ozcinar C, Smolic A (2018) Visual attention in omnidirectional video for virtual reality applications. In: 10th international conference on quality of multimedia experience (QoMEX 2018). Sardinia, Italy

  17. Singla A, Fremerey S, Raake A, List P, Feiten B (2017) AhG8: Measurement of user exploration behavior for omnidirectinal (\(360^{\circ }\)) videos with a head mounted display. Technical report Macau, China

  18. Gutiérrez J, David E, Rai Y, Le Callet P (2018) Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360 still images. Signal Process Image Commun 69:35–42

    Article  Google Scholar 

  19. David EJ, Gutiérrez J, Coutrot A, Da Silva MP, Callet PL (2018) A dataset of head and eye movements for \(360^{\circ }\) videos. In: Proceedings of the 9th ACM multimedia systems conference. ACM, pp 432–437

  20. De Abreu A, Ozcinar C, Smolic A (2017) Look around you: saliency maps for omnidirectional images in VR applications. In: 2017 ninth international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–6

  21. Rai Y, Le Callet P, Guillotel P (2017) Which saliency weighting for omni directional image quality assessment? In: 2017 ninth international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–6

  22. John B, Raiturkar P, Le Meur O, Jain E (2018) A Benchmark of Four Methods for Generating \(360^{\circ }\) Saliency Maps from Eye Tracking Data. In: Proceedings of the first IEEE international conference on artificial intelligence and virtual reality. Taichung, Taiwan

  23. Duan H, Zhai G, Min X, Zhu Y, Fang Y, Yang X (2018) Perceptual quality assessment of omnidirectional images. In: 2018 IEEE international symposium on circuits and systems (ISCAS), pp 1–5. https://doi.org/10.1109/ISCAS.2018.8351786

  24. Luz G, Ascenso J, Brites C, Pereira F (2017) Saliency-driven omnidirectional imaging adaptive coding: Modeling and assessment. In: 2017 IEEE 19th international workshop on multimedia signal processing (MMSP), pp 1–6. https://doi.org/10.1109/MMSP.2017.8122228

  25. Kim HG, Lim H, Ro YM (2019) Deep virtual reality image quality assessment with human perception guider for omnidirectional image. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2019.2898732

    Article  Google Scholar 

  26. Aurenhammer F (1991) Voronoi diagrams—a survey of a fundamental data structure. ACM Comput Surv 23(3):345–405. https://doi.org/10.1145/116873.116880

    Article  Google Scholar 

  27. Croci S, Knorr S, Goldmann L, Smolic A (2017) A framework for quality control in cinematic VR based on voronoi patches and saliency. In: International conference on 3D immersion. Brussels, Belgium

  28. Croci S, Ozcinar C, Zerman E, Cabrera J, Smolic A (2019) Voronoi-based objective quality metrics for omnidirectional video. In: 11th international conference on quality of multimedia experience (QoMEX 2019)

  29. Li C, Xu M, Zhang S, Callet PL (2019) State-of-the-art in \(360^{\circ }\) video/image processing: perception, assessment and compression. CoRR abs/1905.00161. http://arxiv.org/abs/1905.00161

  30. Zhang Y, Wang Y, Liu F, Liu Z, Li Y, Yang D, Chen Z (2018) Subjective panoramic video quality assessment database for coding applications. IEEE Trans Broadcast 64(2):461–473. https://doi.org/10.1109/TBC.2018.2811627

    Article  Google Scholar 

  31. Singla A Goring S, Raake A, Meixner B, Koenen R, Buchholz T (2019) Subjective quality evaluation of tile-based streaming for omnidirectional videos. In: 10th ACM multimedia systems conference (MMSys 2019)

  32. Schatz R, Sackl A, Timmerer C, Gardlo B (2017) Towards subjective quality of experience assessment for omnidirectional video streaming. In: Proceedings of 9th international conference on quality multimedia expo. (QoMEX), pp 1–6

  33. Ohm JR, Sullivan G (2011) Vision, applications and requirements for high efficiency video coding (HEVC). Technical Report MPEG2011/N11891, ISO/IEC JTC1/SC29/WG11, Geneva, Switzerland

  34. Upenik E, Rerabek M, Ebrahimi T (2017) On the performance of objective metrics for omnidirectional visual content. In: 2017 ninth international conference on quality of multimedia experience (QoMEX)

  35. Tran HTT, Ngoc NP, Bui CM, Pham MH, Thang TC (2017) An evaluation of quality metrics for 360 videos. In: 2017 ninth international conference on ubiquitous and future networks (ICUFN), pp 7–11. https://doi.org/10.1109/ICUFN.2017.7993736

  36. Orduna M, Díaz C, Muñoz L, Pérez P, Benito I, García N (2019) Video multimethod assessment fusion (VMAF) on 360vr contents. CoRR abs/1901.06279

  37. Upenik E, Reřábek M, Ebrahimi T (2016) A testbed for subjective evaluation of omnidirectional visual content. In: Proceedings of the picture coding symposium (PCS)

  38. Chen S, Zhang Y, Li Y, Chen Z, Wang Z (2018) Spherical structural similarity index for objective omnidirectional video quality assessment. In: 2018 IEEE international conference on multimedia and expo (ICME), pp 1–6. https://doi.org/10.1109/ICME.2018.8486584

  39. Li Z, Aaron A, Katsavounidis I, Moorthy A, Manohara M (2019) Toward a practical perceptual video quality metric. https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

  40. Barman N, Schmidt S, Zadtootaghaj S, Martini MG, Möller S (2018) An evaluation of video quality assessment metrics for passive gaming video streaming. In: Proceedings of the 23rd Packet Video Workshop. ACM, pp 7–12. https://doi.org/10.1145/3210424.3210434

  41. Rassool R (2017) VMAF reproducibility: Validating a perceptual practical video quality metric. In: 2017 IEEE international symposium on broadband multimedia systems and broadcasting (BMSB), pp 1–2. https://doi.org/10.1109/BMSB.2017.7986143

  42. Bampis CG, Li Z, Bovik AC (2018) Spatiotemporal feature integration and model fusion for full reference video quality assessment. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2018.2868262

    Article  Google Scholar 

  43. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  44. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The thrity-seventh asilomar conference on signals, systems computers, 2003, vol 2, pp 1398–1402 . https://doi.org/10.1109/ACSSC.2003.1292216

  45. Abbas A, Adsumilli B (2016) AhG8: New GoPro test sequences for virtual reality video coding. Technical Report JVET-D0026, JTC1/SC29/WG11, ISO/IEC, Chengdu, China

  46. Asbun E, He H, Y, H, Ye Y (2016) AhG8: InterDigital test sequences for virtual reality video coding. Technical Report JVET-D0039, JTC1/SC29/WG11, ISO/IEC, Chengdu, China

  47. Bang G, Lafruit G, Tanimoto M (2016) Description of 360 3D video application exploration experiments on divergent multiview video Technical Report MPEG2015/ M16129, JTC1/SC29/WG11, ISO/IEC, Chengdu, China

  48. x265 HEVC Encoder / H.265 Video Codec. http://x265.org/ (2018)

  49. FFmpeg. https://ffmpeg.org. Accessed 15 Jan 2019

  50. HLS Authoring Specification for Apple Devices. https://developer.apple.com (2018)

  51. Xu M, Li C, Chen Z, Wang Z, Guan Z (2018) Assessing visual quality of omnidirectional videos. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2018.2886277

    Article  Google Scholar 

  52. https://github.com/Archer-Tatsu/Evaluation_VR-onebar-vive. Accessed 15 Jan 2019

  53. Singla A, Fremerey S, Robitza W, Lebreton P, Raake A (2017) Comparison of subjective quality evaluation for HEVC encoded omnidirectional videos at different bit-rates for UHD and FHD resolution. In: Proceedings of the on thematic workshops of ACM multimedia 2017, Thematic workshops ’17. ACM, New York, NY, USA, pp 511–519. https://doi.org/10.1145/3126686.3126768

  54. ITU-R: Methodology for the subjective assessment of the quality of television pictures. ITU-R Recommendation BT.500-13 (2012)

  55. ITU-T: Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P.910 (2008)

  56. Seshadrinathan K, Soundararajan R, Bovik AC, Cormack LK (2010) Study of subjective and objective quality assessment of video. IEEE Trans Image Process 19(6):1427–1441. https://doi.org/10.1109/TIP.2010.2042111

    Article  MathSciNet  MATH  Google Scholar 

  57. VQEG: Final report from the video quality experts group on the validation of objective models of video quality assessment. Technical report, ITU, COM 9-80-E, Geneva, Switzerland (2000)

  58. Video multi-method assessment fusion (VMAF). https://github.com/Netflix/vmaf. Accessed 15 Jan 2019

  59. Video quality measurement tool (VQMT). https://mmspg.epfl.ch/vqmt. Accessed 15 Jan 2019

  60. 360lib. https://jvet.hhi.fraunhofer.de/svn/svn_360Lib/trunk. Accessed 15 Jan 2019

  61. ITU-T: Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models. ITU-T Recommendation P.1401 (2012)

  62. Zhang Z, Xu Y, Yu J, Gao S (2018) Saliency detection in 360\(^\circ \) videos: 15th European conference, Munich, Germany, September 8–14, 2018, Proceedings, Part VII, pp 504–520. https://doi.org/10.1007/978-3-030-01234-2_30

  63. Knorr S, Croci S, Smolic A (2017) A modular scheme for artifact detection in stereoscopic omni-directional images. In: Irish machine vision and image processing conference. Maynooth, Ireland

  64. de Albuquerque Azevedo RG, Birkbeck N, Simone FD, Janatra I, Adsumilli B, Frossard P (2019) Visual distortions in 360-degree videos. CoRR abs/1901.01848

Download references

Acknowledgements

This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under the Grant Number 15/RP/2776. This work has also been partially supported by the Ministerio de Economía, Industria y Competitividad (AEI/FEDER) of the Spanish Government under project TEC2016-75981 (IVME)

Author information

Authors and Affiliations

  1. V-SENSE Project, School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland

    Simone Croci, Cagri Ozcinar, Emin Zerman & Aljosa Smolic

  2. Communication Systems Group, Technical University of Berlin, Berlin, Germany

    Sebastian Knorr

  3. Grupo de Tratamiento de Imágenes, Information Processing and Telecommunications Center and ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain

    Julián Cabrera

Authors
  1. Simone Croci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cagri Ozcinar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emin Zerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sebastian Knorr
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Julián Cabrera
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Aljosa Smolic
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simone Croci.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Croci, S., Ozcinar, C., Zerman, E. et al. Visual attention-aware quality estimation framework for omnidirectional video using spherical Voronoi diagram. Qual User Exp 5, 4 (2020). https://doi.org/10.1007/s41233-020-00032-3

Download citation

  • Received:

  • Published:

  • DOI: https://doi.org/10.1007/s41233-020-00032-3

Keywords

Search

Navigation