Multimedia Tools and Applications

, Volume 75, Issue 12, pp 6849–6870 | Cite as

Subjective and objective quality assessment of videos in error-prone network environments

  • Soo-Jin Kim
  • Chan-Byoung Chae
  • Jong-Seok Lee


Compression and transmission are two fundamental stages involved in wireless video communications, each of which may cause degradation of the quality of experience (QoE) of end users by producing compression artifacts and packet loss artifacts, respectively. They have their own unique perceptual influences. To provide insight for designing QoE-aware content delivery applications, this paper studies subjective and objective quality of videos containing both types of artifacts. First, subjective quality assessment is conducted, from which interaction between the two types of artifacts during quality perception is investigated. Second, using the subjective data, the performance of the state-of-the-art objective quality metrics is evaluated, with the aim of examining suitability of the existing metrics for their use in error-prone video communication applications. Finally, the developed data set is made publicly available for the community.


Subjective quality assessment Objective quality assessment Packet loss Paired comparison Quality of experience 



This work was supported in part by the Students’ Association of the Graduate School of Yonsei University funded by the Graduate School of Yonsei University, in part by the MSIP(Ministry of Science, ICT & Future Planning), Korea in the ICT R&D Program 2013 (KCA-2012-911-01-106) and in part by the IT Consilience Creative Program (NIPA-2014-H0201-14-1002) supervised by the NIPA (National IT Industry Promotion Agency).


  1. 1.
    Bauer DF (1972) Constructing confidence sets using rank statistics. J Amer Stat Assoc 67(339):687–690CrossRefzbMATHGoogle Scholar
  2. 2.
    Boulos F, Parrein B, Le Callet P, Hands David S (2009) Perceptual effects of packet loss on H.264/AVC encoded videos. In: Proceedings of the fourth international workshop on video processing and quality metrics for consumer electronics, pp 1–6Google Scholar
  3. 3.
    Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39:324–345MathSciNetzbMATHGoogle Scholar
  4. 4.
    Chandler DM, Hemami SS (2007) VSNR: A aavelet-based visual signal-to-noise ratio for natural images. IEEE Trans Image Proc 16(9):2284–2298MathSciNetCrossRefGoogle Scholar
  5. 5.
    Chikkerur S, Sundaram V, Reisslein M, Karam L (2011) Objective video quality assessment methods: a classification, review, and performance comparison. IEEE Trans Broadcast 57(2):165–182CrossRefGoogle Scholar
  6. 6.
    De Simone F, Tagliasacchi M, Naccari M, Tubaro S, Ebrahimi T (2010) A H.264/AVC video database for the evaluation of quality metrics. In: Proceedings of the IEEE international conference on acoustics speech and signal processing (ICASSP). IEEE, pp 2430–2433Google Scholar
  7. 7.
    ETSI, TS 101 154 V1.9.1: Digital Video Broadcasting (DVB); specification for the use of video and audio coding in broadcasting applications based on the MPEG-2 transport stream (2010)Google Scholar
  8. 8.
    Gilbert EN (1960) Capacity of a burst-noise channel. Bell Syst Tech J 39(1960):1253–1265MathSciNetCrossRefGoogle Scholar
  9. 9.
    Glickman ME (1999) Parameter estimation in large dynamic paired comparison experiments. J Roy Stat Soc Ser C (Appl Statist) 48(3):377–394CrossRefzbMATHGoogle Scholar
  10. 10.
    Hong D, Horowitz M, Eleftheriadis A, Wiegand T (2010) H.264 hierarchical P coding in the context of ultra-low delay, low complexity applications. In: Proceedings of the picture coding symposium (PCS), pp 146–149Google Scholar
  11. 11.
    Hore A, Ziou F (2010) Image quality metrics: PSNR vs. SSIM. In: Proc. IEEE int. conf. on pattern recognition, vol 10, pp 2366–2369Google Scholar
  12. 12.
    [ITU], Available: NTIA General Video Quality Metric (VQM) Software (2013)
  13. 13.
    Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG: H.264/AVC JM Reference Software. Available: (2012)
  14. 14.
    Justel A, Pena D, Zamar R (1997) A multivariate Kolmogorov-Smirnov test of goodness of fit. Stat Prob Lett 35(3):251–259MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Kim SJ, Chae CB, Lee JS (2012) Quality perception of coding artifacts and packet loss in networked video communications. In: Proc. IEEE glob. telecom. conf., pp 1357–1361Google Scholar
  16. 16.
    Lee JS, Goldmann L, Ebrahimi T (2013) Paired comparison-based subjective quality assessment of stereoscopic images. Multimed Tools Appl 67(1):31–48CrossRefGoogle Scholar
  17. 17.
    Lee JS, Simone FD, Ebrahimi T (2011) Subjective quality assessment of paried comparison: application to scalable video coding. IEEE Trans Multimed 13(5):882–893CrossRefGoogle Scholar
  18. 18.
    Liu T, Boyce J, Yang H, Wu Z (2009) A novel video quality metric for low bit-late video considering both coding and packet-loss artifacts. IEEE J Select Topics Sig Proc 3(2):280–293CrossRefGoogle Scholar
  19. 19.
    Mitsa T, Varkur K (1993) Evaluation of contrast sensitivity functions for formulation of quality measures incorporated in halftoning algorithms. In: Proc. IEEE int. conf. acoust., speech and sig. proc., vol 5, pp 301–304Google Scholar
  20. 20.
    The SVT high definition multi format test Set. Available: (2000)
  21. 21.
    Ou YF, Ma Z, Wang Y (2009) A novel quality metric for compressed video considering both frame rate and quantization artifacts. In: Proceedings of the international workshop video processing and quality metrics for consumer (VPQM), pp 1–5Google Scholar
  22. 22.
    Park J, Seshadrinathan K, Lee S, Bovik AC (2013) Video quality pooling adaptive to perceptual distortion severity. IEEE Trans Image Proc 22(2):610–620MathSciNetCrossRefGoogle Scholar
  23. 23.
    Peli E (1990) Contrast in complex images. J. Opt. Soc. Amer. A 7(10):2032–2039CrossRefGoogle Scholar
  24. 24.
    Pitrey Y, Engelke U, Barkowsky M, Pépion R, Le Callet P et al (2011) Aligning subjective tests using a low cost common set. In: QoE for multimedia content sharingGoogle Scholar
  25. 25.
    Recommendation ITU com 9-80-E: final report from the video quality experts group on the validation of objective models of video quality assessment (2000)Google Scholar
  26. 26.
    Recommendation ITU-R BT.500-13: methodology for the subjective assessment of the quality of television pictures (2012)Google Scholar
  27. 27.
    Recommendation ITU-T J.144: objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference (2004)Google Scholar
  28. 28.
    Recommendation ITU-T P.910: subjective video quality assessment methods for multimedia applications (1999)Google Scholar
  29. 29.
    Seshadrinathan K, Bovik AC (2010) Motion tuned spatio-temporal quality assessment of natural videos. IEEE Trans Image Proc 19(2):335–350MathSciNetCrossRefGoogle Scholar
  30. 30.
    Seshadrinathan K, Soundararajan R, Bovik AC, Cormack LK (2010) Study of subjective and objective quality assessment of video. IEEE Trans Image Proc 19(6):1427–1441MathSciNetCrossRefGoogle Scholar
  31. 31.
    Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Proc 15(2):430–444CrossRefGoogle Scholar
  32. 32.
    Simone FD, Naccari M, Tagliasacchi M, Dufaux F, Tubaro S, Ebrahimi T (2011) Subjective quality assessment of H.264/AVC video streaming with packet losses. EURASIP J Image Video Proc 2011:1–12CrossRefGoogle Scholar
  33. 33.
    Spearman C (1904) The proof and measurement of association between two things, vol 15Google Scholar
  34. 34.
    Tan W, Patti A (2010) Improving error resilience of scalable H.264 (SVC) via drift control. In: Proc. IEEE int. conf. acoust., speech and sig. proc., pp 2302–2305Google Scholar
  35. 35.
    Venkata ND, Kite T, Geisler W, Evans B, Bovik A (2000) Image quality assessment based on a degradation model. IEEE Trans Image Proc 9(4):636–650CrossRefGoogle Scholar
  36. 36.
    Wang Z, Bovik AC (2002) A universal image quality index. IEEE Sig Proc Lett 9(2):81–84CrossRefGoogle Scholar
  37. 37.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc 16(9):600–612CrossRefGoogle Scholar
  38. 38.
    Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: Proc. of asilomar conf. on sign., syst. and computers, vol 2, pp 1398–1402Google Scholar
  39. 39.
    Wickelmaier F, Schmid C (2004) A MATLAB function to estimate choice model parameters from paired-comparison data. Behav Res Methods Instruments Comp 36:29–40CrossRefGoogle Scholar
  40. 40.
    Winkler S (2012) Analysis of public image and video databases for quality assessment. IEEE J Select Topics Sig Proc 6(6):1–10CrossRefGoogle Scholar
  41. 41.
    Wu HR, Rao KR (2005) Digital video image quality and perceptual coding. CRC PressGoogle Scholar
  42. 42.
    You J, Korhonen J, Perkis A (2010) Spatial and temporal pooling of image quality metrics for perceptual video quality assessment on packet loss streams. In: Proc. IEEE int. conf. acoust., speech and sig. proc., vol 7491, pp 1002–1005Google Scholar
  43. 43.
    Zhang F, Li S, Ma L, Wong YC, Ngan KN (2011) IVP subjective quality video database.

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.School of Integrated TechnologyYonsei UniversityYeonsu-guKorea

Personalised recommendations