Multimedia Tools and Applications

, Volume 61, Issue 3, pp 787–817 | Cite as

Framework for the integrated video quality assessment

  • Mu Mu
  • Piotr RomaniakEmail author
  • Andreas Mauthe
  • Mikołaj Leszczuk
  • Lucjan Janowski
  • Eduardo Cerqueira


Through years of development Content Networks (CN) have become more sophisticated and more technically diverse. Modern CN are designed to be more adaptive to communication environment, devices and user requirements. However, one open issue is the still fluctuating quality of service provision. As a result user experience can be negatively affected. In order to maintain a satisfactory level of user experience it is crucial to develop a feasible solution to measure the extent to which video services meet users’ expectation. Assessing video quality with respect to users’ subjective opinions is a complex task. In this paper we address challenges of this task and design an integrated framework using a number of comprehensive functional modules. Our framework integrates objective quality assessment models of Artifacts Measurement (AM) and Quality of Delivery (QoD) approaches. Only the fittest models are activated by the framework considering requirements of individual evaluation tasks. We also introduce our recent work of realising key functional modules of the framework. Joint subjective experiments between two institutes have also been carried out for the purpose of model implementation and evaluation. Results from experiments verify the concept of an integrated framework and show the effectiveness of its key modules in estimating the quality level of video services.


Video quality metrics Quality of experience Content distribution network Mean opinion score Perceptual quality 



Application of Piotr Romaniak’s, Mikolaj Leszczuk’s and Lucjan Janowski’s research leading to these results has received funding from the European Community’s Seventh Framework Program (FP7/2007-2013) under grant agreement n°218086 (INDECT). Mu Mu’s work is also supported by Agilent Laboratories UK, European Commission within the FP7 Project: P2P-Next and Framework for Innovation and Research in MediaCityUK (FIRM). Eduardo Cerqueira was supported by The National Council for Scientific and Technological Development (CNPq—Brazil) and Fundacao de Amparo a Pesquisa do Estado do Para (FAPESPA). The authors thank Piotr Borkowski from AGH University of Science and Technology, Roswitha Gostner from Lancaster University and Francisco Garcia from Agilent Laboratories for their input.


  1. 1.
    Addis B, Leyffer S (2006) A trust-region algorithm for global optimization. Comput Optim Appl 35(3):287–304. doi: 10.1007/s10589-006-8716-2 MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Botta A, Pescape A (2011) Ip packet interleaving: bridging the gap between theory and practice. In: Proc. IEEE symp. computers and communications (ISCC), pp 1022–1029. doi: 10.1109/ISCC.2011.5983976
  3. 3.
    Botta A, Pescapè A, Ventre G (2008) Quality of service statistics over heterogeneous networks: analysis and applications. Elesevier EJORGoogle Scholar
  4. 4.
    Brand ando T, Queluz M (2010) No-reference quality assessment of h.264/avc encoded video. IEEE Trans Circuits Syst Video Technol 20(11):1437–1447. doi: 10.1109/TCSVT.2010.2077474 CrossRefGoogle Scholar
  5. 5.
    Chan A, Zeng K, Mohapatra P, Lee SJ, Banerjee S (2010) Metrics for evaluating video streaming quality in lossy ieee 802.11 wireless networks. In: Proceedings of the 29th conference on information communications, INFOCOM’10. IEEE Press, Piscataway, NJ, pp 1613–1621. Google Scholar
  6. 6.
    Claypool M, Zhu Y (2003) Using interleaving to ameliorate the effects of packet loss in a video stream. In: Proc. 23rd int distributed computing systems workshops conf, pp 508–513. doi: 10.1109/ICDCSW.2003.1203604
  7. 7.
    CMP Media LLC (2009) Measuring IPTV QoS performance at the box.
  8. 8.
    Dosselmann R, Yang XD (2007) A prototype no-reference video quality system. In: Fourth Canadian conference on computer and robot vision, CRV ’07, vol 2007, pp 411–417Google Scholar
  9. 9.
    Eskicioglu AM (2000) Quality measurement for monochrome compressed images in the past 25 years. In: Proceedings of the international conference on acoustics, speech and signal processing (ICASSP). IEEE, Istanbul, Turkey, pp 1907–1910Google Scholar
  10. 10.
    Eskicioglu AM, Fisher PS (1995) Image quality measures and their performance. IEEE Trans Commun 43(12):2959–2965CrossRefGoogle Scholar
  11. 11.
    Farias MCQ, Mitra SK (2005) No-reference video quality metric based on artifact measurements. In: IEEE international conference on image processing, ICIP 2005, vol 3, pp III–141–4Google Scholar
  12. 12.
    Fukumoto K (2004) Taking bounded variables seriously: extended beta binomial, asymmetric logit, and time series. In: Research workshop in applied statisticsGoogle Scholar
  13. 13.
    Geerts D, De Moor K, Ketyko I, Jacobs A, Van den Bergh J, Joseph W, Martens L, De Marez L (2010) Linking an integrated framework with appropriate methods for measuring qoe. In: Second international workshop on quality of multimedia experience (QoMEX)Google Scholar
  14. 14.
    ITU-R (1998) ITU-R BT.1129 subjective assessment of standard definition digital television (SDTV) systems. ITU-RGoogle Scholar
  15. 15.
    ITU-R (1998) ITU-R BT.500 methodology for the subjective assessment of the quality of television pictures, Geneva, SwitzerlandGoogle Scholar
  16. 16.
    ITU-T (1996) ITU-T P.800 methods for subjective determination of transmission quality. ITU-T, Geneva, SwitzerlandGoogle Scholar
  17. 17.
    ITU-T (1999) ITU-T P.910 subjective video quality assessment methods for multimedia applications. ITU-TGoogle Scholar
  18. 18.
    Jain R (2004) Quality of experience. IEEE Multimed 11(1):96–95CrossRefGoogle Scholar
  19. 19.
    Kangasharju J, Mu M, Colussi GD (2008) Application-level fairness. In: Proc. international conference on information networking 2008 (ICOIN 2008), Busan, KoreaGoogle Scholar
  20. 20.
    Kwon JC, Jang SH, Chin Y, Oh SJ (2010) A novel video quality impairment monitoring scheme over an ipty service with packet loss. In: 2010 second international workshop on quality of multimedia experience (QoMEX), pp 224–229. doi: 10.1109/QOMEX.2010.5516127
  21. 21.
    Lee J, Hoppel K (1989) Noise modeling and estimation of remotely-sensed images. In: Proc. international geoscience and remote sensing, Vancouver, Canada, vol 2, pp 1005–1008Google Scholar
  22. 22.
    Liang YJ, Apostolopoulos JG, Girod B (2003) Analysis of packet loss for compressed video: does burst-length matter? In: Proc. IEEE int. conf. acoustics, speech, and signal processing (ICASSP ’03), vol 5. doi: 10.1109/ICASSP.2003.1200063
  23. 23.
    Liu T, Wang Y, Boyce JM, Yang H, Wu Z (2009) A novel video quality metric for low bit-rate video considering both coding and packet-loss artifacts. IEEE J Select Top Signal Process 3(2):280–293. doi: 10.1109/JSTSP.2009.2015069 CrossRefGoogle Scholar
  24. 24.
    MSU Graphics & Media Lab (Video Group) (2009) MSU video quality measurement tool.
  25. 25.
    Mu M, Mauthe A, Garcia F (2008) A utility-based qos model for emerging multimedia applications. In: Proc. first IEEE future multimedia networking (FMN 08) workshop, Cardiff, UKGoogle Scholar
  26. 26.
    Mu M, Gostner R, Mauthe A, Garcia F, Tyson G (2009) Visibility of individual packet loss on h.264 encoded video stream—a user study on the impact of packet loss on perceived video quality. In: Proc. sixteenth annual multimedia computing and networking (MMCN’09), San Jose, California, USAGoogle Scholar
  27. 27.
    Mu M, Mauthe A, Haley R, Garcia F (2011) Discrete quality assessment in IPTV content distribution networks. Signal Process Image Commun 26(7):339–357, Advances in IPTV Technologies, ElsevierCrossRefGoogle Scholar
  28. 28.
    Ohm J-R (1999) Bildsignalverarbeitung für Multimedia-Systeme, BerlinGoogle Scholar
  29. 29.
    OPTICOM (2007) Perceptual evaluation of video quality.
  30. 30.
    Pinson M, Wolf S, Gallagher M (2004) The impact of monitor resolution and type on subjective video quality testingGoogle Scholar
  31. 31.
    van den Branden Lambrecht CJ, Verscheure O (1996) Perceptual quality measure using a spatio-temporal model of the human visual system. In: Proc. SPIE, vol 2668, pp 450–461Google Scholar
  32. 32.
    VQEG (2004) Index VQEG test sequences.
  33. 33.
    VQEG (2008) Final report from the video quality experts group on the validation of objective models of multimedia quality assessment, phase I.
  34. 34.
    VQEG (2008) Final report from the video quality experts group on the validation of objective models of video quality assessment, phase II. VQEG final report of FR-TV phase II validation testGoogle Scholar
  35. 35.
    VQEG (2009) The video quality experts group.
  36. 36.
    Wang Y (2006) Survey of objective video quality measurements. EMC Corporation Hopkinton.
  37. 37.
    Wang Z (1998) Objective image/video quality measurement—a literature survey. In: EE 381k: multidimensional digital signal processingGoogle Scholar
  38. 38.
    Wang Z, Lu L, Bovik AC (2004) Video quality assessment based on structural distortion measurement. Signal Process Image Commun 19(2):121–213CrossRefGoogle Scholar
  39. 39.
    Winkler S (1999) A perceptual distortion metric for digital color video. In: Proc. SPIE, vol 3644, pp 175–184Google Scholar
  40. 40.
    Winkler S (2005) Digital video quality—vision models and metrics. John Wiley & Sons, LtdGoogle Scholar
  41. 41.
    Winkler S (2007) Video quality and beyond. In: Proc. European signal processing conference, Poznan, PolandGoogle Scholar
  42. 42.
    Wolf S, Pinson MH (1999) Spatial-temporal distortion metrics for in-service quality monitoring of any digital video system. In: Proc. SPIE, vol 3845, pp 266–277Google Scholar
  43. 43.
    Yang F, Wan S, Xie Q, Wu HR (2010) No-reference quality assessment for networked video via primary analysis of bit stream. IEEE Trans Circuits Syst Video Technol 20(11):1544–1554. doi: 10.1109/TCSVT.2010.2087433 CrossRefGoogle Scholar
  44. 44.
    Yim C, Bovik AC (2011) Evaluation of temporal variation of video quality in packet loss networks. Signal Process Image Commun 26(1):24–38. doi: 10.1016/j.image.2010.11.002. MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Mu Mu
    • 1
  • Piotr Romaniak
    • 2
    Email author
  • Andreas Mauthe
    • 1
  • Mikołaj Leszczuk
    • 2
  • Lucjan Janowski
    • 2
  • Eduardo Cerqueira
    • 3
  1. 1.The School of Computing and CommunicationsLancaster UniversityLancasterUK
  2. 2.The Department of TelecommunicationsAGH University of Science and TechnologyKrakowPoland
  3. 3.The Federal University of ParaBelemBrazil

Personalised recommendations