Skip to main content
SpringerLink
Log in

Paired comparison-based subjective quality assessment of stereoscopic images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects’ limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bercovitz J (1998) Image-side perspective and stereoscopy. In: Proc. SPIE, vol 3295

  2. Bosc E, Pépion R, Callet PL, Köppel M, Ndjiki-Nya P, Pressigout M, Morin L (2011) Towards a new quality metric for 3-D synthesized view assessment. IEEE J Sel Topics Signal Process 5(7):1332–1343

    Article  Google Scholar 

  3. Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39:324–345

    MathSciNet  MATH  Google Scholar 

  4. Chen KT, Wu CC, Chang YC, Lei CL (2009) A crowdsourceable QoE evaluation framework for multimedia content. In: Proc. ACM multimedia, pp 491–500

  5. Davidson RR (1970) On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. J Am Stat Assoc 65(329):317–328

    Article  Google Scholar 

  6. Eichhorn A, Ni P, Eg R (2010) Randomized pair comparison- an economic and robust method for audiovisual quality assessment. In: Proc. int. workshop on network and operating systems support for digital audio and video. Amsterdam, The Netherlands, pp 63–68

    Google Scholar 

  7. Glickman ME (1999) Parameter estimation in large dynamic paired comparison experiments. J R Stat Soc, Ser C, Appl Stat 48(3):377–394

    Article  MATH  Google Scholar 

  8. Goldmann L, Simone FD, Ebrahimi T (2010) Impact of acquisition distortions on the quality of stereoscopic images. In: Proc. int. workshop on video processing and quality metrics for consumer electronics. Scottsdale, Arizona, USA

  9. Gotchev A, Akar GB, Capin T, Strohmeier D, Boev A (2011) Three-dimensional media for mobile devices. Proc IEEE 99(4):708–741

    Article  Google Scholar 

  10. Huynh-Thu Q, Barkowsky M, Callet PL (2011) The importance of visual attention in improving the 3D-TV viewing experience: overview and new perspectives. IEEE Trans Broadcast 57(2):421–431

    Article  Google Scholar 

  11. Huynh-Thu Q, Callet PL, Barkowsky M (2010) Video quality assessment: from 2D to 3D- challenges and future trends. In: Proc. int. conf. image processing. Hong Kong, China, pp 4025–4028

  12. Ijsselsteijn WA, de Ridder H, Vliegen J (2000) Subjective evaluation of stereoscopic images: effects of camera parameters and display duration. IEEE Trans Circuits Syst Video Technol 10(2):225–233

    Article  Google Scholar 

  13. Jumisko-Pyykkö S, Utriainen T (2011) A hybrid method for quality evaluation in the context of use for mobile (3D) television. Multimed Tools Appl 55(2):185–225

    Article  Google Scholar 

  14. Kooi FL, Toet A (2004) Visual comfort of binocular and 3D displays. Displays 25(2–3):99–108

    Article  Google Scholar 

  15. Lee JS, Simone FD, Ramzan N, Zhao Z, Kurutepe E, Sikora T, Ostermann J, Izquierdo E, Ebrahimi T (2010) Subjective evaluation of scalable video coding for content distribution. In: Proc. ACM multimedia. Firenze, Italy, pp 65–72

  16. Liu X, Yang LT, Sohn K (2011) High-speed inter-view frame mode decision procedure for multi-view video coding. Future Gener Comput Syst. doi:10.1016/j.future.2011.05.013

    Google Scholar 

  17. Meesters LMJ, Ijsselsteijn WA, Seuntiëns PJH (2004) A survey of perceptual evaluations and requirements of three-dimensional TV. IEEE Trans Circuits Syst Video Technol 14(3):381–391

    Article  Google Scholar 

  18. International Telecommunication Union (2002) Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT.500-11

  19. Rao PV, Kupper LL (1967) Ties in paired-comparison experiments: a generalization of the Bradley-Terry model. J Am Stat Assoc 62(317):194–204

    Article  MathSciNet  Google Scholar 

  20. Seuntiiëns P, Meesters L, Ijsselsteijn W (2006) Perceived quality of compressed stereoscopic images: effects of symmetric and asymmetric JPEG coding and camera separation. ACM Trans Appl Percept 3(2):95–109

    Article  Google Scholar 

  21. Smolic A, Kauff P, Knorr S, Hornung A, Kunter M, Müller M, Lang M (2011) Three-dimensional video postproduction and processing. Proc IEEE 99(4):607–625

    Article  Google Scholar 

  22. International Telecommunication Union (2000) Subjective assessment of stereoscopic television pictures. Recommendation ITU-R BT.1438

  23. International Telecommunication Union (1999) Subjective video quality assessment methods for multimedia applications. Recommendation ITU-R P.910

  24. Zilly F, Kluger J, Kauff P (2011) Production rules for stereo acquisition. Proc IEEE 99(4):590–606

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Ministry of Knowledge Economy, Korea, under the IT Consilience Creative Program (NIPA-2010-C1515-1001-0001), in part by Yonsei University Research Fund of 2011, and in part by the COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (Qualinet).

Author information

Authors and Affiliations

  1. School of Integrated Technology, Yonsei University, 406-840, Incheon, Korea

    Jong-Seok Lee

  2. Multimedia Signal Processing Group (MMSPG), Swiss Federal Institute of Technology in Lausanne (EPFL), 1015, Lausanne, Switzerland

    Lutz Goldmann & Touradj Ebrahimi

Authors
  1. Jong-Seok Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lutz Goldmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Touradj Ebrahimi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jong-Seok Lee.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lee, JS., Goldmann, L. & Ebrahimi, T. Paired comparison-based subjective quality assessment of stereoscopic images. Multimed Tools Appl 67, 31–48 (2013). https://doi.org/10.1007/s11042-012-1011-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-012-1011-6

Keywords

Search

Navigation