Skip to main content
SpringerLink
Log in

Visual quality assessment algorithms: what does the future hold?

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Creating algorithms capable of predicting the perceived quality of a visual stimulus defines the field of objective visual quality assessment (QA). The field of objective QA has received tremendous attention in the recent past, with many successful algorithms being proposed for this purpose. Our concern here is not with the past however; in this paper we discuss our vision for the future of visual quality assessment research. We first introduce the area of quality assessment and state its relevance. We describe current standards for gauging algorithmic performance and define terms that we will use through this paper. We then journey through 2D image and video quality assessment. We summarize recent approaches to these problems and discuss in detail our vision for future research on the problems of full-reference and no-reference 2D image and video quality assessment. From there, we move on to the currently popular area of 3D QA. We discuss recent databases, algorithms and 3D quality of experience. This yet-nascent technology provides for tremendous scope in terms of research activities and we summarize each of them. We then move on to more esoteric topics such as algorithmic assessment of aesthetics in natural images and in art. We discuss current research and hypothesize about possible paths to tread. Towards the end of this article, we discuss some other areas of interest including high-definition (HD) quality assessment, immersive environments and so on before summarizing interesting avenues for future work in multimedia (i.e., audio-visual) quality assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. For eg., the LIVE IQA database has five distortion categories [69], while the TID2008 database incorporates 17 distortion classes [56]!

  2. The best performing algorithm—MOVIE [63] has an SROCC of ∼0.79 with human perception.

  3. We define blurry images as those with poor quality, however blur can also be associated with positive aesthetics, such as an image of a softened, wrinkle-free face—future work needs to disambiguate such cases.

  4. Indeed, AT&T recently announced a cap on the bandwidth that users could utilize on their 3G-enabled phones, and other service providers may follow suit [17].

  5. Recently Nintendo announced the Nintendo DS 3D gaming device, which utilizes an autostereoscopic display [53].

References

  1. Alexandre B, Le Callet P, Patrizio C, Romain C (2009) Quality assessment of stereoscopic images. In: EURASIP journal on image and video processing

  2. Amatriain X, Pujol JM, Oliver N (2009) I like it, i like it not. In: Proceedings int conf UMAP’09

  3. Barkowsky M, Eskofier B, Bialkowski J, Bitto R, Kaup A (2009) Temporal trajectory aware video quality measure. IEEE J Sel Top Sig Proc 3(2):266–279, Issue on Visual Media Quality Assessment

    Article  Google Scholar 

  4. Barland R, Saadane A (2006) A reference free quality metric for compressed images. In: Proc of 2nd int workshop on video processing and quality metrics for consumer electronics

  5. Beerends JG, De Caluwe FE (1999) The influence of video quality on perceived audio quality and vice versa. J Audio Eng Soc 47(5):355–362

    Google Scholar 

  6. Campisi P, Carli M, Giunta G, Neri A (2003) Blind quality assessment system for multimedia communications using tracing watermarking. IEEE Trans Signal Process 51(4):996–1002

    Article  MathSciNet  Google Scholar 

  7. Campisi P, Le Callet P, Marini E (2007) Stereoscopic images quality assessment. In: Proceedings of 15th European signal processing conference (EUSIPCO07)

  8. Cermak GW (2009) Consumer opinions about frequency of artifacts in digital video. IEEE J Sel Top Sig Proc 3(2):336–343

    Article  Google Scholar 

  9. Chandler DM, Hemami SS (2007) VSNR: a wavelet-based visual signal-to-noise ratio for natural images. IEEE Trans Image Process 16(9):2284–2298

    Article  MathSciNet  Google Scholar 

  10. Channappayya SS, Bovik AC, Caramanis C, Heath RW (2008) Design of linear equalizers optimized for the structural similarity index. IEEE Trans Image Process 17(6):857–872

    Article  MathSciNet  Google Scholar 

  11. Channappayya SS, Bovik AC, Heath RW (2006) A linear estimator optimized for the structural similarity index and its application to image denoising. In: IEEE international conference on image processing, pp 2637–2640

  12. Chen GH, Yang CL, Xie SL (2006) Gradient-based structural similarity for image quality assessment. In: IEEE international conference on image processing, pp 2929–2932

  13. Chen MJ, Bovik AC (2009) No reference image blur assessment using multiscale gradient. In: 1st international workshop on quality of multimedia experience (QoMEX)

  14. Chen MJ, Bovik AC (2010) Fast structural similarity index. In: IEEE international conferene on acoustics, speech and signal processing (ICASSP)

  15. Datta R, Joshi D, Li J, Wang JZ (2006) Studying aesthetics in photographic images using a computational approach. Lect Notes Comput Sci 3953:288

    Article  Google Scholar 

  16. Datta R, Li J, Wang JZ (2008) Algorithmic inferencing of aesthetics and emotion in natural images: an exposition. In: IEEE intl conf image proc, pp 105–108

  17. Dignan L (2010) Tiered mobile data plans accelerate: verizon wireless to follow AT&T’s lead. ZDnet.com

  18. Ebert R (2010) Why I hate 3-D (and you should too). Newsweek

  19. Farias MCQ, Moore MS, Foley JM, Mitra SK (2004) Perceptual contributions of blocky, blurry, and fuzzy impairments to overall annoyance. In: Proceedings of the IS&T/SPIE human vision and electronic imaging IX, vol 5292, pp 109–120

  20. Gabarda S, Cristobal G (2007) Blind image quality assessment through anisotropy. J Opt Soc Am A 24:B42–B51

    Article  Google Scholar 

  21. Girod B (1993) What’s wrong with mean-squared error? In: Watson AB (ed) Digital images and human vision, pp 207–220

  22. Goldmann L, De Simone F, Ebrahimi T (2010) A comprehensive database and subjective evaluation methodology for quality of experience in stereoscopic video. In: Electronic imaging (EI), 3D image processing (3DIP) and applications

  23. Goldmann L, De Simone F, Ebrahimi T (2010) Impact of acquisition distortions on the quality of stereoscopic images. In: Electronic imaging (EI), 3D image processing (3DIP) and applications

  24. Gorley P, Holliman N (2008) Stereoscopic image quality metrics and compression. In: Proc. SPIE stereoscopic displays and applications XIX, 6803

  25. Hekstra AP, Beerends JG, Ledermann D, De Caluwe FE, Kohler S, Koenen RH, Rihs S, Ehrsam M, Schlauss D (2002) PVQM–a perceptual video quality measure. Signal Process, Image Commun 17(10):781–798

    Article  Google Scholar 

  26. Kandadai S, Hardin J, Creusere CD (2008) Audio quality assessment using the mean structural similarity measure. In: Proceedings of international conference on acoustics, speech, and signal processing, pp 221–224

  27. Kanumuri S, Cosman PC, Reibman AR, Vaishampayan VA (2006) Modeling packet-loss visibility in MPEG-2 video. IEEE Trans Multimedia 8(2):341–355

    Article  Google Scholar 

  28. Kayargadde V, Martens JB (1996) Perceptual characterization of images degraded by blur and noise: model. J Opt Soc Am A 13:1178–1188

    Article  Google Scholar 

  29. Ke Y, Tang X, Jing F (2006) The design of high-level features for photo quality assessment. In: IEEE conf. comp. vis. pat. recog. 1

  30. Keimel C, Oelbaum T, Diepold K (2009) No-reference video quality evaluation for high-definition video. In: Proceedings of the international conference on image processing, San Diego, CA, USA

  31. Kortum P, Sullivan M (2010) The effect of content desirability on subjective video quality ratings. In: Human factors: the journal of the human factors and ergonomics society

  32. Laboratory for Image and Video Engineering (LIVE) (2010) Quality assessment research at LIVE. http://live.ece.utexas.edu/research/quality/

  33. Le Callet P, Perchard S, Tourancheau S, Ninassi A, Barba D (2007) Towards the next generation of video and image quality metrics: impact of display, resolution, content and visual attention in subjective assessment. Second International Workshop on Image media Quality and its Applications 83(73.05):10–51

    Google Scholar 

  34. Li C, Bovik AC (2009) Three-component weighted structural similarity index. Proc SPIE 7242:72420Q

    Article  Google Scholar 

  35. Li C, Chen T (2009) Aesthetic visual quality assessment of paintings. IEEE J Sel Top Sig Proc 3(2):236–252

    Article  Google Scholar 

  36. Li Q, Wang Z (2009) Reduced-Reference Image Quality Assessment Using Divisive Normalization-Based Image Representation. IEEE J Sel Top Sig Proc 3(2):202–211

    Article  Google Scholar 

  37. Luo Y, Tang X (2008) Photo and video quality evaluation: focusing on the subject. In: Eur. conf. comp. vis., pp 386–399

  38. Marziliano P, Dufaux F, Winkler S, Ebrahimi T (2004) Perceptual blur and ringing metrics: application to JPEG2000. Signal Process Image Commun 19(2):163–172

    Article  Google Scholar 

  39. Meesters LMJ, IJsselsteijn WA, Seuntiens PJH (2004) A survey of perceptual evaluations and requirements of three-dimensional TV. IEEE Trans Circuits Syst Video Technol 14(3):381–391

    Article  Google Scholar 

  40. Meesters LMJ, Martens JB (2002) A single-ended blockiness measure for JPEG-coded images. Signal Process 82(3):369–387

    Article  MATH  Google Scholar 

  41. Miyata K, Saito M, Tsumura N, Haneishi H, Miyake Y (1997) Eye movement analysis and its application to evaluation of image quality. In: Final program and proceedings of the fifth IS&TSID color imaging conference. Color science, systems and applications, pp 116–119

  42. Moorthy AK, Bovik AC (2009) A motion-compensated approach to video quality assessment. In: Proc. IEEE asilomar conference on signals, systems and computers

  43. Moorthy AK, Bovik AC (2009) Visual importance pooling for image quality assessment. IEEE J Sel Top Sig Proc 3(2):193–201, Issue on Visual Media Quality Assessment

    Article  Google Scholar 

  44. Moorthy AK, Bovik AC (2010) A two-step framework for constructing blind image quality indices. IEEE Signal Process Lett 17(2):587–599

    Google Scholar 

  45. Moorthy AK, Geisler WS, Bovik AC (2010) Evaluating the task dependence of eye movements for compressed videos. In: Fifth international workshop on video processing and quality metrics for consumer electronics (VPQM)

  46. Moorthy AK, Obrador P, Oliver N, Bovik AC (2010) Towards computational models of visual aesthetic appeal of consumer videos. In: European conference on computer vision (ECCV)

  47. Moorthy AK, Seshadrinathan K, Bovik AC (2009) Digital video quality assessment algorithms. Springer

  48. Moorthy AK, Seshadrinathan K, Soundararajan R, Bovik AC (2010) Wireless video quality assessment: a study of subjective scores and objective algorithms’. IEEE Trans Circuits Syst Video Technol 20(4):513–516

    Article  Google Scholar 

  49. Naccari M, Tagliasacchi M, Pereira F, Tubaro S (2008) No-reference modeling of the channel induced distortion at the decoder for H. 264/AVC video coding. In: Proceedings of the international conference on image processing, San Diego, CA, USA

  50. Narvekar ND, Karam LJ (2009) A no-refernece perceptual image sharpness metric based on probability of blur detection. In: 1st international workshop on quality of multimedia experience (QoMEX)

  51. Ninassi A, Le Meur O, Le Callet P, Barba D (2009) Considering temporal variations of spatial visual distortions in video quality assessment. IEEE J Sel Top Sig Proc 3(2):253–265, Issue on Visual Media Quality Assessment

    Article  Google Scholar 

  52. Ninassi A, Le Meur O, Le Callet P, Barba D, Tirel A (2006) Task impact on the visual attention in subjective image quality assessment. In: The 14th European signal processing conference

  53. Nintendo (2010) Nintendo introduces glasses-free 3d gaming. http://www.nintendo.com/whatsnew/detail/MvsPLa99GTbqTKaOvwh93lOJHhSHJ_x2. Accessed August 2010

  54. Ong EP, Lin W, Lu Z, Yao S, Yang X, Jiang L (2003) No-reference JPEG-2000 image quality metric. In: International conference on multimedia and expo, vol 1, pp 6–9

  55. Parvez Sazzad ZM, Kawayoke Y, Horita Y (2008) No reference image quality assessment for JPEG2000 based on spatial features. Signal Process, Image Commun 23(4):247–268

    Google Scholar 

  56. Ponomarenko N, Carli M, Lukin V, Egiazarian K, Astola J, Battisti F (2008) Tampere image database. http://www.ponomarenko.info/tid2008.htm

  57. Rouse DM, Hemami SS (2008) Understanding and simplifying the structural similarity metric. In: 15th IEEE international conference on image processing, ICIP 2008, pp 1188–1191

  58. Saad MA, Bovik AC (2009) Natural motion statistics for no-reference video quality assessment. In: International workshop on quality of multimedia experience, pp 163–167

  59. Saad MA, Bovik AC, Charrier C (2010) A DCT statistics-based blind image quality index. IEEE Signal Process Lett 17(6):583–586

    Article  Google Scholar 

  60. Sadaka NG, Karam LJ, Ferzli R, Abousleman GP (2008) A no-reference perceptual image sharpness metric based on saliency-weighted foveal pooling. In: 15th IEEE international conference on image processing, ICIP 2008, pp 369–372

  61. Savakis AE, Etz SP, Loui AC (2000) Evaluation of image appeal in consumer photography. In: SPIE Proc, human vis elec img, pp 111–121

  62. Sazzad ZMP, Kawayoke Y, Horita Y (2007) Spatial features based no reference image quality assessment for JPEG2000. IEEE Int’l Conf Image Proc 3:517–520

    Google Scholar 

  63. Seshadrinathan K (2008) Video quality assessment based on motion models. PhD thesis, The University of Texas at Austin

  64. Seshadrinathan K, Bovik AC (2009) Video quality assessment in the essential guide to video processing, chapter 14. Academic Press

  65. Seshadrinathan K, Safranek RJ, Chen J, Pappas TN, Sheikh HR, Simoncelli EP, Wang Z, Bovik AC (2009) Image quality assessment in the essential guide to image processing, chapter 20. Academic Press

  66. Seshadrinathan K, Soundararajan R, Bovik AC, Cormack LK (2010) Study of subjective and objective quality assessment of video. IEEE Trans Image Process 19(6):1427–1441

    Article  Google Scholar 

  67. Seuntiens P, Meesters LMJ, Ijsselsteijn W (2006) Perceived quality of compressed stereoscopic images: effects of symmetric and asymmetric JPEG coding and camera separation. ACM Tran App Percep (TAP) 3(2):109

    Google Scholar 

  68. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430–444

    Article  Google Scholar 

  69. Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451

    Article  Google Scholar 

  70. Sheikh HR, Bovik AC, Cormack L (2005) No-reference quality assessment using natural scene statistics: JPEG 2000. IEEE Trans Image Process 14(11):1918–1927

    Article  Google Scholar 

  71. Sheskin D (2004) Handbook of parametric and nonparametric statistical procedures. CRC Press

  72. Shnayderman A, Gusev A, Eskicioglu AM (2003) A multidimensional image quality measure using singular value decomposition. 5294:82–92

  73. Sugimoto O, Kawada R, Wada M, Matsumoto S (2000) Objective measurement scheme for perceived picture quality degradation caused by MPEG encoding without any reference pictures. Proc SPIE 4310:932

    Article  Google Scholar 

  74. Suthaharan S (2003) Perceptual quality metric for digital video coding. Electron Lett 39(5):431–433

    Article  Google Scholar 

  75. Toyama image database. http://mict.eng.u-toyama.ac.jp/mict/index2.html. Accessed August 2010

  76. Varghese G, Wang Z (2010) Video denoising based on a spatiotemporal Gaussian scale mixture model. IEEE Trans Circuits Syst Video Technol 20(7):1032–1040

    Article  Google Scholar 

  77. Video Quality Experts Group (VQEG) (2000) Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment phase I. http://www.its.bldrdoc.gov/vqeg/projects/frtv_phaseI

  78. Vu CT, Larson EC, Chandler DM (2008) Visual fixation patterns when judging image quality: effects of distortion type, amount, and subject experience. In: IEEE Southwest symposium on image analysis and interpretation, SSIAI 2008, pp 73–76

  79. Vuori T, Olkkonen M (2006) The effect of image sharpness on quantitative eye movement data and on image quality evaluation while viewing natural images. In: Proceedings of SPIE, vol 6059, p 605903

  80. Wang Z, Bovik AC (2006) Modern image quality assessment, vol 2. Morgan & Claypool Publishers

  81. Wang Z, Bovik AC (2009) Mean squared error: love it or leave it?—a new look at fidelity measures. IEEE Signal Process Mag 26(1):98-117

    Article  Google Scholar 

  82. Wang Z, Li Q (2007) Video quality assessment using a statistical model of human visual speed perception. J Opt Soc Am 24(12):B61–B69

    Article  Google Scholar 

  83. Wang Z, Shang X (2006) Spatial pooling strategies for perceptual image quality assessment. In: IEEE international conference on Image Processing

  84. Wang Z, Simoncelli EP (2008) Maximum differentiation (MAD) competition: a methodology for comparing computational models of perceptual quantities. J Vis 8(12):1–13

    Article  MATH  Google Scholar 

  85. Wang Z, Sheikh HR, Bovik AC (2002) No-reference perceptual quality assessment of JPEG compressed images. IEEE Int’l Conf. Image Proc. 1:477–480

    Google Scholar 

  86. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: From error measurement to structural similarity. IEEE Signal Process Lett 13(4):600–612

    Google Scholar 

  87. Wang Z, Lu L, Bovik AC (2004) Video quality assesssment based on structural distortion measurement. Signal Process, Image Commun 19(2):121–132

    Article  Google Scholar 

  88. Wang Z, Wu G, Sheikh HR, Simoncelli EP, Yang EH, Bovik AC (2006) Quality-aware images. IEEE Trans Image Process 15(6):1680–1689

    Article  Google Scholar 

  89. Winkler S, Dufaux F (2003) Video quality evaluation for mobile applications. In: Proc. of SPIE conference on visual communications and image processing, vol 5150. Lugano, Switzerland, pp 593–603

    Google Scholar 

  90. Winkler S, Sharma A, McNally D (2001) Perceptual video quality and blockiness metrics for multimedia streaming applications. In: Proceedings of the international symposium on wireless personal multimedia communications, pp 547–552

  91. Yang KC, Guest C-C, El-Maleh K, Das PK (2007) Perceptual temporal quality metric for compressed video. IEEE Trans Multimedia 9(7):1528–1535

    Article  Google Scholar 

  92. Yarbus AL (1967) Eye movements and vision. Plenum press

  93. Zhu X, Milanfar P (2009) A no-reference sharpness metric sensitive to blur and noise. In: 1st international workshop on quality of multimedia experience (QoMEX)

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, 78712, USA

    Anush Krishna Moorthy & Alan Conrad Bovik

Authors
  1. Anush Krishna Moorthy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alan Conrad Bovik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anush Krishna Moorthy.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Moorthy, A.K., Bovik, A.C. Visual quality assessment algorithms: what does the future hold?. Multimed Tools Appl 51, 675–696 (2011). https://doi.org/10.1007/s11042-010-0640-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0640-x

Keywords

Search

Navigation