Skip to main content

3D video subjective quality: a new database and grade comparison study


This paper presents a research study on the subjective assessment of 3D video quality using a newly constructed 3D video database (3DVCL@FER). This database consists of 8 original 3D video sequences, each degraded with 22 different degradation types, including degradations specific to stereoscopic systems. The subjective assessment was done with the support of a purpose-built easily customizable grade collection platform and conducted in two research laboratories, in Croatia and Portugal. Subjective scores for quality, depth and comfort were collected and DMOS (Difference Mean Opinion Score) values were calculated. Different objective measures (for image, 3D image, 2D video and 3D video) were separately compared with DMOS values for quality, depth and comfort. The 3D video grade-annotated database described is publicly accessible and can be used in research-related activities like assessment of existing objective measures, using the entire database or parts of it, and construction of new objective measures specific to 3D video degradations. The system presented can also be used to collect and compare subjective quality grades originating from different sites to study the effect of different observation conditions and observer/graders populations on the DMOS quality values for 3D video depth and comfort.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. 1.

    Aflaki P, Hannuksela MM, Hakala J, Häkkinenb J, Gabbouj M (2011) Estimation of subjective quality for mixed-resoultion stereoscopic video. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON) 1–4

  2. 2.

    Banitalebi-Dehkordi A, Pourazad MT, Nasiopoulos P (2015) The effect of frame rate on 3D video quality and bitrate. 3D Res 6(1):1–13. doi:10.1007/s13319-014-0034-3

    Article  Google Scholar 

  3. 3.

    Battisti F, Bosc E, Carli M, Le Callet P, Perugia S (2015) Objective image quality assessment of 3D synthesized views. Signal Process Image Commun 30:78–88. doi:10.1016/j.image.2014.10.005

    Article  Google Scholar 

  4. 4.

    Chen M-J, Su C-C, Kwon D-K, Cormack LK, Bovik AC (2013) Full-reference quality assessment of stereopairs accounting for rivalry. Signal Process Image Commun 28(10):1143–1155. doi:10.1016/j.image.2013.05.006

    Article  Google Scholar 

  5. 5.

    Cheng E, Burton P, Burton J, Joseski A, Burnett I (2012) RMIT3DV: pre-announcement of a creative commons uncompressed HD 3D video database. Qual Multimed Exp (QoMEX 2012) 212–217

  6. 6. Accesed 21 July 2015

  7. 7. Accesed 21 July 2015

  8. 8.

    De Silva V, Arachchi HK, Ekmekcioglu E, Kondoz A (2013) Towards an impairment metric for stereoscopic video: a full-reference video quality metric to assess compressed stereoscopic video. IEEE Trans Image Process 22(9):3392–3404. doi:10.1109/TIP.2013.2268422

    MathSciNet  Article  MATH  Google Scholar 

  9. 9.

    Dumic E, Grgic S, Jiménez Bernejo D, Silva Cruz LA (2014) Benchmark on state of the art objective measures for 3D stereoscopic video quality assesment on the Nantes database. Proc. ELMAR-2014 119–123

  10. 10.

    Dumic E, Grgic S, Grgic M (2010) Comparison of HDTV formats using objective video quality measures. Multimed Tools Appl 49(3):409–424. doi:10.1007/s11042-009-0441-2

    Article  Google Scholar 

  11. 11.

    Dumic E, Grgic S, Grgic M (2014) IQM2 - New image quality measure based on steerable pyramid wavelet transform and structural similarity index. SIViP 8(6):1159–1168. doi:10.1007/s11760-014-0654-3

    Article  Google Scholar 

  12. 12. Accesed 21 July 2015

  13. 13.

    Goldmann L, De Simone F, Ebrahimi T (2010) A comprehensive database and subjective evaluation methodology for quality of experience in stereoscopic video. Electronic Imaging (EI), 3D Image Processing (3DIP) and Applications, SPIE Proceedings 7526

  14. 14. Accesed 21 July 2015

  15. 15. Accesed 21 July 2015

  16. 16. Accesed 21 July 2015

  17. 17. Accesed 21 July 2015

  18. 18. Accesed 21 July 2015

  19. 19. Accesed 21 July 2015

  20. 20. Accesed 21 July 2015

  21. 21. Accesed 21 July 2015

  22. 22. Accesed 21 July 2015

  23. 23. Accesed 21 July 2015

  24. 24.

    ITU-R BT.2021 (2012) Subjective methods for the assessment of stereoscopic 3DTV systems. International Telecommunication Union/ITU radiocommunication sector

  25. 25.

    ITU-R BT.500-13 (2012) Methodology for the subjective assessment of the quality of television pictures. International Telecommunication Union/ITU radiocommunication sector

  26. 26.

    ITU-T Recommendation P.910 (2008) Subjective video quality assessment methods for multimedia applications

  27. 27.

    Joveluro P, Malekmohamadi H, Fernando WAC, Kondoz AM (2010) Perceptual video quality metric for 3D video quality assessment. 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON) 1–4

  28. 28.

    Lin Y, Wu J (2014) Quality assessment of stereoscopic 3D image compression by binocular integration behaviors. IEEE Trans Image Process 23(4):1527–1542. doi:10.1109/TIP.2010.2092435

    MathSciNet  Article  MATH  Google Scholar 

  29. 29.

    Lopez JP, Rodrigo JA, Jimenez D, Menendez JM (2013) Stereoscopic 3D video quality assessment based on depth maps and video motion. EURASIP J Image Video Process 62:1–14. doi:10.1186/1687-5281-2013-62

    Google Scholar 

  30. 30.

    Ma R, Cheung N-M, Au OC, Tian D (2013) Novel distortion metric for depth coding of 3D video. 2013 International Conference on Image Processing (ICIP) 1714–1718

  31. 31.

    Moorthy AK, Su C-C, Mittal A, Bovik AC (2013) Subjective evaluation of stereoscopic image quality. Signal Process Image Commun 28(9):870–883. doi:10.1016/j.image.2012.08.004

    Article  Google Scholar 

  32. 32.

    Mysirlidis C et al. (2015) STESCAL3D: subjective evaluation of HD stereo video streaming using H.264 SVC in diverse laboratory environments. Qual Multim Exp (QoMEX 2015) 1–6

  33. 33.

    Pinson MH, Wolf S (2004) A new standardized method for objectively measuring video quality. IEEE Trans Broadcast 50(3):312–322

    Article  Google Scholar 

  34. 34.

    QUALINET Multimedia Databases v5.5, Accesed 21 July 2015

  35. 35.

    Sheikh HR (2004) Image quality assessment using natural scene statistics. Ph.D. dissertation, University of Texas at Austin

  36. 36.

    Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430–444. doi:10.1109/TIP.2005.859378

    Article  Google Scholar 

  37. 37.

    Urvoy M, Barkowsky M, Cousseau R, Koudota Y, Ricorde V, Le Callet P, Gutierrez J, Garcia N (2012) NAMA3DS1-COSPAD1: Subjective video quality assessment database on coding conditions introducing freely available high quality 3D stereoscopic sequences. Quality of Multimedia Experience (QoMEX 2012) 109–114

  38. 38.

    VQEG 3DTV Group (2012) Test plan for establishing a ground truth for quality of experience in 3D for assessment methodologies in 3D video quality assessment, GroTruQoE3D1, Draft Version 1.0

  39. 39.

    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc 13(4):600–612. doi:10.1109/TIP.2003.819861

    Article  Google Scholar 

  40. 40.

    Wang Z, Li Q (2011) Information content weighting for perceptual image quality assessment. IEEE Trans Image Process 20(5):1185–1198. doi:10.1109/TIP.2010.2092435

    MathSciNet  Article  MATH  Google Scholar 

  41. 41.

    Xing L, You J, Ebrahimi T, Perkis A (2013) Stereoscopic quality datasest under various test conditions. Qual Multimed Exp (QoMEX 2013) 136–141

  42. 42.

    Zota V, Janssen J-K (2010) Tiefenbehandlung, 2D-Videos in 3D abspielen, c’t Magazin für Computer Technik, Heise Verlag 6:116

Download references

Author information



Corresponding author

Correspondence to Emil Dumić.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dumić, E., Grgić, S., Šakić, K. et al. 3D video subjective quality: a new database and grade comparison study. Multimed Tools Appl 76, 2087–2109 (2017).

Download citation


  • 3D video quality
  • Subjective assessment
  • Objective quality measures
  • 3DVCL@FER video database