As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects’ limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Bercovitz J (1998) Image-side perspective and stereoscopy. In: Proc. SPIE, vol 3295
Bosc E, Pépion R, Callet PL, Köppel M, Ndjiki-Nya P, Pressigout M, Morin L (2011) Towards a new quality metric for 3-D synthesized view assessment. IEEE J Sel Topics Signal Process 5(7):1332–1343
Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39:324–345
Chen KT, Wu CC, Chang YC, Lei CL (2009) A crowdsourceable QoE evaluation framework for multimedia content. In: Proc. ACM multimedia, pp 491–500
Davidson RR (1970) On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. J Am Stat Assoc 65(329):317–328
Eichhorn A, Ni P, Eg R (2010) Randomized pair comparison- an economic and robust method for audiovisual quality assessment. In: Proc. int. workshop on network and operating systems support for digital audio and video. Amsterdam, The Netherlands, pp 63–68
Glickman ME (1999) Parameter estimation in large dynamic paired comparison experiments. J R Stat Soc, Ser C, Appl Stat 48(3):377–394
Goldmann L, Simone FD, Ebrahimi T (2010) Impact of acquisition distortions on the quality of stereoscopic images. In: Proc. int. workshop on video processing and quality metrics for consumer electronics. Scottsdale, Arizona, USA
Gotchev A, Akar GB, Capin T, Strohmeier D, Boev A (2011) Three-dimensional media for mobile devices. Proc IEEE 99(4):708–741
Huynh-Thu Q, Barkowsky M, Callet PL (2011) The importance of visual attention in improving the 3D-TV viewing experience: overview and new perspectives. IEEE Trans Broadcast 57(2):421–431
Huynh-Thu Q, Callet PL, Barkowsky M (2010) Video quality assessment: from 2D to 3D- challenges and future trends. In: Proc. int. conf. image processing. Hong Kong, China, pp 4025–4028
Ijsselsteijn WA, de Ridder H, Vliegen J (2000) Subjective evaluation of stereoscopic images: effects of camera parameters and display duration. IEEE Trans Circuits Syst Video Technol 10(2):225–233
Jumisko-Pyykkö S, Utriainen T (2011) A hybrid method for quality evaluation in the context of use for mobile (3D) television. Multimed Tools Appl 55(2):185–225
Kooi FL, Toet A (2004) Visual comfort of binocular and 3D displays. Displays 25(2–3):99–108
Lee JS, Simone FD, Ramzan N, Zhao Z, Kurutepe E, Sikora T, Ostermann J, Izquierdo E, Ebrahimi T (2010) Subjective evaluation of scalable video coding for content distribution. In: Proc. ACM multimedia. Firenze, Italy, pp 65–72
Liu X, Yang LT, Sohn K (2011) High-speed inter-view frame mode decision procedure for multi-view video coding. Future Gener Comput Syst. doi:10.1016/j.future.2011.05.013
Meesters LMJ, Ijsselsteijn WA, Seuntiëns PJH (2004) A survey of perceptual evaluations and requirements of three-dimensional TV. IEEE Trans Circuits Syst Video Technol 14(3):381–391
International Telecommunication Union (2002) Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT.500-11
Rao PV, Kupper LL (1967) Ties in paired-comparison experiments: a generalization of the Bradley-Terry model. J Am Stat Assoc 62(317):194–204
Seuntiiëns P, Meesters L, Ijsselsteijn W (2006) Perceived quality of compressed stereoscopic images: effects of symmetric and asymmetric JPEG coding and camera separation. ACM Trans Appl Percept 3(2):95–109
Smolic A, Kauff P, Knorr S, Hornung A, Kunter M, Müller M, Lang M (2011) Three-dimensional video postproduction and processing. Proc IEEE 99(4):607–625
International Telecommunication Union (2000) Subjective assessment of stereoscopic television pictures. Recommendation ITU-R BT.1438
International Telecommunication Union (1999) Subjective video quality assessment methods for multimedia applications. Recommendation ITU-R P.910
Zilly F, Kluger J, Kauff P (2011) Production rules for stereo acquisition. Proc IEEE 99(4):590–606
This work was supported in part by the Ministry of Knowledge Economy, Korea, under the IT Consilience Creative Program (NIPA-2010-C1515-1001-0001), in part by Yonsei University Research Fund of 2011, and in part by the COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (Qualinet).
About this article
Cite this article
Lee, JS., Goldmann, L. & Ebrahimi, T. Paired comparison-based subjective quality assessment of stereoscopic images. Multimed Tools Appl 67, 31–48 (2013). https://doi.org/10.1007/s11042-012-1011-6
- Stereoscopic image
- Subjective quality
- Paired comparison
- Quality of experience (QoE)