Multimedia Tools and Applications

, Volume 67, Issue 1, pp 31–48 | Cite as

Paired comparison-based subjective quality assessment of stereoscopic images

  • Jong-Seok LeeEmail author
  • Lutz Goldmann
  • Touradj Ebrahimi


As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects’ limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.


Stereoscopic image Subjective quality Paired comparison Quality of experience (QoE) 



This work was supported in part by the Ministry of Knowledge Economy, Korea, under the IT Consilience Creative Program (NIPA-2010-C1515-1001-0001), in part by Yonsei University Research Fund of 2011, and in part by the COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (Qualinet).


  1. 1.
    Bercovitz J (1998) Image-side perspective and stereoscopy. In: Proc. SPIE, vol 3295Google Scholar
  2. 2.
    Bosc E, Pépion R, Callet PL, Köppel M, Ndjiki-Nya P, Pressigout M, Morin L (2011) Towards a new quality metric for 3-D synthesized view assessment. IEEE J Sel Topics Signal Process 5(7):1332–1343CrossRefGoogle Scholar
  3. 3.
    Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39:324–345MathSciNetzbMATHGoogle Scholar
  4. 4.
    Chen KT, Wu CC, Chang YC, Lei CL (2009) A crowdsourceable QoE evaluation framework for multimedia content. In: Proc. ACM multimedia, pp 491–500Google Scholar
  5. 5.
    Davidson RR (1970) On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. J Am Stat Assoc 65(329):317–328CrossRefGoogle Scholar
  6. 6.
    Eichhorn A, Ni P, Eg R (2010) Randomized pair comparison- an economic and robust method for audiovisual quality assessment. In: Proc. int. workshop on network and operating systems support for digital audio and video. Amsterdam, The Netherlands, pp 63–68Google Scholar
  7. 7.
    Glickman ME (1999) Parameter estimation in large dynamic paired comparison experiments. J R Stat Soc, Ser C, Appl Stat 48(3):377–394zbMATHCrossRefGoogle Scholar
  8. 8.
    Goldmann L, Simone FD, Ebrahimi T (2010) Impact of acquisition distortions on the quality of stereoscopic images. In: Proc. int. workshop on video processing and quality metrics for consumer electronics. Scottsdale, Arizona, USAGoogle Scholar
  9. 9.
    Gotchev A, Akar GB, Capin T, Strohmeier D, Boev A (2011) Three-dimensional media for mobile devices. Proc IEEE 99(4):708–741CrossRefGoogle Scholar
  10. 10.
    Huynh-Thu Q, Barkowsky M, Callet PL (2011) The importance of visual attention in improving the 3D-TV viewing experience: overview and new perspectives. IEEE Trans Broadcast 57(2):421–431CrossRefGoogle Scholar
  11. 11.
    Huynh-Thu Q, Callet PL, Barkowsky M (2010) Video quality assessment: from 2D to 3D- challenges and future trends. In: Proc. int. conf. image processing. Hong Kong, China, pp 4025–4028Google Scholar
  12. 12.
    Ijsselsteijn WA, de Ridder H, Vliegen J (2000) Subjective evaluation of stereoscopic images: effects of camera parameters and display duration. IEEE Trans Circuits Syst Video Technol 10(2):225–233CrossRefGoogle Scholar
  13. 13.
    Jumisko-Pyykkö S, Utriainen T (2011) A hybrid method for quality evaluation in the context of use for mobile (3D) television. Multimed Tools Appl 55(2):185–225CrossRefGoogle Scholar
  14. 14.
    Kooi FL, Toet A (2004) Visual comfort of binocular and 3D displays. Displays 25(2–3):99–108CrossRefGoogle Scholar
  15. 15.
    Lee JS, Simone FD, Ramzan N, Zhao Z, Kurutepe E, Sikora T, Ostermann J, Izquierdo E, Ebrahimi T (2010) Subjective evaluation of scalable video coding for content distribution. In: Proc. ACM multimedia. Firenze, Italy, pp 65–72Google Scholar
  16. 16.
    Liu X, Yang LT, Sohn K (2011) High-speed inter-view frame mode decision procedure for multi-view video coding. Future Gener Comput Syst. doi: 10.1016/j.future.2011.05.013 Google Scholar
  17. 17.
    Meesters LMJ, Ijsselsteijn WA, Seuntiëns PJH (2004) A survey of perceptual evaluations and requirements of three-dimensional TV. IEEE Trans Circuits Syst Video Technol 14(3):381–391CrossRefGoogle Scholar
  18. 18.
    International Telecommunication Union (2002) Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT.500-11Google Scholar
  19. 19.
    Rao PV, Kupper LL (1967) Ties in paired-comparison experiments: a generalization of the Bradley-Terry model. J Am Stat Assoc 62(317):194–204MathSciNetCrossRefGoogle Scholar
  20. 20.
    Seuntiiëns P, Meesters L, Ijsselsteijn W (2006) Perceived quality of compressed stereoscopic images: effects of symmetric and asymmetric JPEG coding and camera separation. ACM Trans Appl Percept 3(2):95–109CrossRefGoogle Scholar
  21. 21.
    Smolic A, Kauff P, Knorr S, Hornung A, Kunter M, Müller M, Lang M (2011) Three-dimensional video postproduction and processing. Proc IEEE 99(4):607–625CrossRefGoogle Scholar
  22. 22.
    International Telecommunication Union (2000) Subjective assessment of stereoscopic television pictures. Recommendation ITU-R BT.1438Google Scholar
  23. 23.
    International Telecommunication Union (1999) Subjective video quality assessment methods for multimedia applications. Recommendation ITU-R P.910Google Scholar
  24. 24.
    Zilly F, Kluger J, Kauff P (2011) Production rules for stereo acquisition. Proc IEEE 99(4):590–606CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Jong-Seok Lee
    • 1
    Email author
  • Lutz Goldmann
    • 2
  • Touradj Ebrahimi
    • 2
  1. 1.School of Integrated TechnologyYonsei UniversityIncheonKorea
  2. 2.Multimedia Signal Processing Group (MMSPG)Swiss Federal Institute of Technology in Lausanne (EPFL)LausanneSwitzerland

Personalised recommendations