Skip to main content
SpringerLink
Log in

Full-Reference Image Quality Metrics Performance Evaluation Over Image Quality Databases

  • Research Article - Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

A quantitative predictive performance evaluation of 18 well-known and commonly used full-reference image quality assessment metrics has been conducted in the present work. The process has been run over six publicly available and subjectively rated image quality databases for four degradation types namely JPEG and JPEG2000 compression, noise and Gaussian blur. Results show that the existing predictive performance evaluation tools of the different full-reference image quality metrics are significantly impacted by the choice of the image quality database. Three of them, namely Toyama, LIVE and TID, have been found to give different assessment results. The visual information fidelity (VIF) quality metric has been found to have superior predictive capabilities to its counterparts. MS-SSIM (multi-scale structural similarity index), MSSIM (modified SSIM) and VIFP (pixel-based VIF) have also closer performances in terms of their correlation to the subjective human ratings, accuracy and monotonicity to the VIF model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Reference

  1. VQEG.: Final VQEG report on the validation of objective quality metrics for video quality assessment. http://www.its.bldrdoc.gov/vqeg/projects/frtv_phaseI/ (2000)

  2. Horita, Y.; Kawayoke, Y.; Parvez Sazzad, Z.M.: Image quality evaluation database. http://mict.eng.u-toyama.ac.jp/mictdb.html

  3. Sheikh, H.R.; Wang, Z.; Cormack, L.; Bovik, A.C.: LIVE Image Quality Assessment Database Release 2. http://live.ece.utexas.edu/research/quality

  4. Sheikh, H.R.: Image quality assessment using natural scene statistics. Ph.D. dissertation, University of Texas at Austin, May 2004

  5. Le Callet, P.; Autrusseau, F.: Subjective quality assessment IRCCyN/IVC database. http://www.irccyn.ec-nantes.fr/ivcdb/ (2005)

  6. Chandler D.M, Hemami S.S: VSNR: a wavelet-based visual signal-to-noise ratio for natural images. IEEE Trans. Image Process. 16(9), 2284–2298 (2007)

    Article  MathSciNet  Google Scholar 

  7. Chandler, D.M.; Lim, K.H.S.; Hemami, S.S.: Effects of spatial correlations and global precedence on the visual fidelity of distorted images. In: Rogowitz, B.E.; Pappas, T.N.; Daly, S. (eds.) Proceedings of SPIE Human Vision and Electronic Imaging XI, San Jose, CA (2006)

  8. Ponomarenko, N.; Lukin, V.; Zelensky, A.; Egiazarian, K.; Carli, M.; Battisti, F.: TID2008: a database for evaluation of full-reference visual quality assessment metrics. Adv. Modern Radioelectron. 10, 30–45 (2009). http://www.ponomarenko.info/tid2008.htm

  9. Larson, E.C.; Chandler, D.M.: Most apparent distortion: full- reference image quality assessment and the role of strategy. J. Electron. Imaging 19(1), 011006. http://vision.okstate.edu/csiq/ (2010)

  10. Mitsa, T.; Varkur, K.L.: Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Minneapolis, MN, vol. V, pp. 301–304 (1993)

  11. Damera-Venkata N, Kite T, Geisler W, Evans B, Bovik A: Image quality assessment based on a degradation model. IEEE Trans. Image Process. 9(4), 636–650 (2000)

    Article  Google Scholar 

  12. Wang Z, Bovik A.C: A Universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)

    Article  Google Scholar 

  13. Wang Z, Bovik A.C, Sheikh H.R, Simoncelli E.P: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  14. Wang, Z.; Simoncelli, E.P.; Bovik, A.C.: Multi-scale structural similarity for image quality assessment. In: Proceedings of 37th IEEE Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, Nov. 09–12, 2003

  15. Sheikh H.R, Bovik A.C: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006)

    Article  Google Scholar 

  16. Sheikh H.R, Bovik A.C, de Veciana G: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14(12), 2117–2128 (2005)

    Article  Google Scholar 

  17. Shnayderman A, Gusev A, Eskicioglu A. M: An SVD-based grayscale image quality measure for local and global assessment. IEEE Trans. Image Process. 15(2), 422–429 (2006)

    Article  Google Scholar 

  18. Egiazarian, K.; Astola, J.; Ponomarenko, N.; Lukin, V.; Battisti, F.; Carli, M.: New full-reference quality metrics based on HVS. In: Proceedings of the Second International Workshop on Video Processing and Quality Metrics, Scottsdale, USA (2006)

  19. Ponomarenko, N.; Silvestri, F.; Egiazarian, K.; Carli, M.; Astola, J.; Lukin, V.: On between-coefficient contrast masking of DCT basis functions. In: Proceedings of the Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics, VPQM-07, Scottsdale, Arizona, USA, 25–26 January 2007

  20. Chandler D.M, Hemami S.S: VSNR: a wavelet-based visual signal-to-noise ratio for natural images. IEEE Trans. Image Process. 16(9), 2284–2298 (2007)

    Article  MathSciNet  Google Scholar 

  21. Wang, Z.; Bovik, A.C.; Lu, L.: Why is image quality assessment so difficult? IEEE International Conference on Acoustics, Speech, & Signal Processing, May 2002

  22. Mansouri A, Aznaveh A.M, Torkamani-Azar F, Jahanshahi J.A: Image quality assessment using the singular value decomposition theorem. Opt. Rev. 16(2), 49–53 (2009)

    Article  Google Scholar 

  23. Zhang, L.; Zhang, L.; Mou, X.: RFSIM: a feature based image quality assessment metric using Riesz transforms. In: Proceedings of ICIP 2010, Hong Kong, 26–29 Sept 2010

  24. Wang, Z.; Bovik, A.C.; Simoncelli, E.P.: Structural Approaches to image quality assessment. In: Al Bovik (ed.) Handbook of Image and Video Processing, 2nd edn. Academic Press June (2005)

  25. Miyahara M, Kotani K, Algazi V.R: Objective picture quality scale (PQS) for image coding. IEEE Trans. Commun. 46(9), 1215–1225 (1998)

    Article  Google Scholar 

  26. Wang, Z.; Simoncelli, E.P.: Translation insensitive image similarity in complex wavelet domain. In: IEEE International Conference on Acoustics, Speech and Signal Processing, vol. II, pp. 573–576, Philadelphia, PA, Mar 2005

  27. Gaubatz, M.: Metrix MUX Visual Quality Assessment Package. http://foulard.ece.cornell.edu/gaubatz/metrix_mux (2007)

  28. Péchard, S.: Qualité d’usage en télévision haute définition : évaluations subjectives et métriques objectives. Ph.D. dissertation, Ecole Polytechnique de l’Université de Nantes, October 2008

  29. VQEG.: Final report from the video quality experts group on the validation of objective models of video quality assessment—Phase II, Rapport technique, Video Quality Experts Group. http://www.vqeg.org (2003)

  30. VQEG.: Multimedia Test Plan 1.19. Video Quality Experts Group (2007)

  31. Sheikh H.R, Sabir M.F, Bovik A.C: A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans. Image Process. 15(11), 3440–3451 (2006)

    Article  Google Scholar 

  32. Nauge, M.; Larabi, M.-C.; Fernandez, C.: Benchmark de métriques de qualité sur bases de données d’images compressées. Conférence sur la Compression et Représentation des Signaux Audiovisuels, CORESA 2010, Lyon, France, 26–27 October 2010

  33. Okarma, K.: Combined full-reference image quality metric linearly correlated with subjective assessment. Lecture Notes in Computer Science, vol. 6113, pp. 539–546 (2010)

  34. Okarma K: Video quality assessment using the combined full- reference approach. Adv. Intell. Soft Comput. 84, 51–58 (2010)

    Article  Google Scholar 

  35. Streijl R.C, Winkler S, Hands D.S: Perceptual quality measurement: towards a more efficient process for validating objective models. IEEE Signal Process. Mag. 27(4), 136–140 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratoire L2TI, Institut Galilée, University Paris 13, 99, Avenue J-B Clément, 93430, Villetaneuse, France

    Atidel Lahoulou & Emmanuel Viennet

  2. Ecole Nationale Polytechnique, 10, Avenue Hassen Badi, BP 182 El-Harrach, Algiers, 16200, Algeria

    Atidel Lahoulou & Mourad Haddadi

  3. Department of Computer Science, King Saud University, Riyadh, Saudi Arabia

    Ahmed Bouridane

  4. School of Computing, Engineering and Information Sciences, Northumbria University, Newcastle, UK

    Ahmed Bouridane

Authors
  1. Atidel Lahoulou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed Bouridane
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emmanuel Viennet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mourad Haddadi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Atidel Lahoulou.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lahoulou, A., Bouridane, A., Viennet, E. et al. Full-Reference Image Quality Metrics Performance Evaluation Over Image Quality Databases. Arab J Sci Eng 38, 2327–2356 (2013). https://doi.org/10.1007/s13369-012-0509-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-012-0509-6

Keywords

Search

Navigation