Subjective Ratings and Image Quality Databases

  • Yong Ding


Since human visual system (HVS) is the ultimate receiver of visual signals, ideal image quality assessment (IQA) should be conducted by subjective experiments. Moreover, well-organized subjective study provides the golden standard for evaluating and training objective IQA models. A few IQA databases have been constructed following rigorous experimental setups and flows. Such databases provide subjective experimental results for collected distorted images that can well guide objective study. In this chapter, the general framework of subjective IQA study and the information about several representative IQA databases are introduced.


Subjective image quality assessment Single stimulus Double stimulus Mean opinion score Database 


  1. Autrusseau, F., & Bas, P. (2009). Subjective quality assessment of the broken arrows watermarking technique.
  2. Carosi, M., Pankajakshan, V., & Autrusseau, F. (2010). Toward a simplified perceptual quality metric for watermarking application. In Conference on Multimedia on Mobile Devices. San Jose.Google Scholar
  3. Chandler, D. M. (2013). Seven challenges in image quality assessment: Past, present, and future research. ISRN Signal Processing, 2013(8), 905685.Google Scholar
  4. Chandler, D. M., & Hemami, S. S. (2007). VSNR: A wavelet-based visual signal-to-noise ratio for natural image. IEEE Transactions on Image Processing, 16(9), 2284–2298.MathSciNetCrossRefGoogle Scholar
  5. Choe, J. H., Jeong, T. U., Choi, H. S., Lee, E. J., & Lee, S. W. (1999). Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P. 910, 12(2), 3665–3673.Google Scholar
  6. Deng, C., Ma, L., Lin, W., & Ngan, K. N. (2015). Visual quality assessment. Switzerland.Google Scholar
  7. Engelke, U., Maeder, A. J., & Zepernick, H. J. (2009). Visual attention for image quality database.
  8. Engelke, U., Zepernick, H. J., & Kusuma, M. (2010). Wireless imaging quality (WIQ) database.
  9. Horita, H., Shibata, K., & Kawayoka, Y. (2018). Toyama image quality evaluation database.
  10. ITU-R B.T. (2012). Methodology for the subjective assessment of the quality television pictures. Geneva, Switzerland: International Telecommunication Union. ITU-R Recommendation BT.500–13.Google Scholar
  11. Lambooij, M., IJsselsteijn, W., Bouwhuis, D. G., & Heynderickx, I. (2011). Evaluation of stereoscopic images: beyond 2d quality. IEEE Transactions on Broadcasting, 57(2), 432–444.Google Scholar
  12. Larson, E. C., & Chandler, D. M. (2014). Categorical image quality (CSIQ) database.
  13. Le Callet, P., & Autrusseau, F. (2015). Subjective quality assessment irccyn/ivc database.
  14. Liu, H., Klomp, N., & Heynderickx, I. (2013). TUD image quality database: Perceived ringing.
  15. Marini, E., Autrusseau, F., Le Callet, P., & Campisi, P. (2007). Evaluation of standard watermarking techniques. In Proceedings of the International Social for Optical and Photonics (p. 6505).Google Scholar
  16. Ninassi, A., Le Callet, P., & Autrusseau, F. (2006, January). Pseudo no reference image quality metric using perceptual data hiding. In Conference on Human Vision and Electronic Imaging XI. San Jose.Google Scholar
  17. Ponomarenko, N., Lukin, V., & Zelensky, A. (2009). TID2008-a database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics, 10, 30–45.Google Scholar
  18. Seuntiens, P., Meesters, L., & IJsselsteijn, W. (2006). Perceived quality of compressed stereoscopic images: Effects of symmetric and asymmetric jpeg coding and camera separation. ACM Transactions on Applied Perception, 3(2), 95–109.Google Scholar
  19. Shahid, M., Rossholm, A., Lövström, B., & Zepernick, H. J. (2014). No-reference image and video quality assessment: A classification and review of recent approaches. EURASIP Journal on Image and Video Processing, 2014(40), 1–32.Google Scholar
  20. Sheikh, H. R., Sabir, M. F., & Bovik, A. C. (2006). A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing, 15(11), 3440–3451.CrossRefGoogle Scholar
  21. Sheikh, H. R., Wang, Z., Cormack, L., & Bovik, A. C. (2014). LIVE image quality assessment database release 2.
  22. Vu, C., Phan, T., Singh, P., & Chandler, D. M. (2012). Digitally retouched image quality (DRIQ) database.
  23. Winkler, S. (2012). Analysis of public image and video databases for quality assessment. IEEE Journal of Selected Topics in Signal Processing, 6(6), 616–625.CrossRefGoogle Scholar
  24. Wu, H. R., & Rao, K. R. (2006). Digital video image quality and perceptual coding. Journal of Electronic Imaging, 16(6), 039901.Google Scholar
  25. Wu, C. C., Chen, K. T., Chang, Y. C., & Lei, C. L. (2013). Crowdsourcing multimedia QoE evaluation: A trusted framework. IEEE Transactions on Multimedia, 15(5), 1121–1137.CrossRefGoogle Scholar
  26. Xu, Q., Huang, Q., & Yao, Y. (2012). Online crowdsourcing subjective image quality assessment. ACM International Conference on Multimedia, 359–368.Google Scholar
  27. Zaric, A., Tatalovic, N., Brajkovic, N., Hlevnjak, H., Loncaric, M., Dumic, E., & Grgic, S. (2011, September). Vcl@fer image quality assessment database. Proceeding of the International Symposium ELMAR. Zadar, Croatia.Google Scholar

Copyright information

© Zhejiang University Press, Hangzhou and Springer-Verlag GmbH Germany 2018

Authors and Affiliations

  1. 1.Zhejiang UniversityHangzhouChina

Personalised recommendations