Skip to main content
SpringerLink
Log in

Assessing the Quality of Experience of 3DTV and Beyond: Tackling the Multidimensional Sensation

  • Chapter
  • First Online:
3D Future Internet Media

Abstract

Quality of experience in 3D media requires new and innovative concepts for subjective assessment methodologies. Capturing the observer’s opinion may be achieved by providing multiple voting scales, such as 2D image quality, depth quantity, and visual comfort. Pooling these different scales to achieve a single quality percept may be performed differently by each human observer. The chapter dives into the complexity of this subject by explaining the QoE concept using 3DTV as an example. It explains the meaning of the different scales, the current approaches to assess each of them, and the individual influence factors related to the voting which affects reproducibility of the obtained results. Methodologies for assessing the overall preference of experience using pair comparisons with a reasonable number of stimuli are provided. The viewers may also create their own attributes for evaluation in the Open Profiling methodology which has been recently adapted for 3DTV. The drawback of all these assessment methods is that they are intrusive in the sense that the assessor needs to concentrate on the task at hand. Medical and psychophysical measurement methods, such as EEG, EOG, EMG, and fMRI, may eliminate this drawback and are introduced with respect to the different QoE influence factors. Their value at this early stage of development is mostly in supporting and partly predicting subjectively perceived and annotated QoE. The chapter closes with a brief review of the most important technical constraints that impact on the capture, transmission, and display of 3DTV signals.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Le Callet P, Möller S, Perkis A (2012) Qualinet white paper on definitions of quality of experience (2012), European network on quality of experience in multimedia systems and services (COST Action IC 1003), (Version 1.1) Published online by COST Action IC 2003, http://www.qualinet.eu/images/stories/QoE_whitepaper_v1.2.pdf

  2. Kaptein RG, Kuijsters A, Lambooij M et al (2008) Performance evaluation of 3D-TV systems. In: Image quality and system performance V. Presented at the society of photo-optical instrumentation engineers (SPIE) conference, vol 6808

    Google Scholar 

  3. Lambooij M, IJsselsteijn W, Bouwhuis DG, Heynderickx I (2011) Evaluation of stereoscopic images: beyond 2D quality. IEEE Trans Broadcast 57(2):432–444

    Article  Google Scholar 

  4. Seuntiens PJH, Meesters LMJ, IJsselsteijn WA (2005) Perceptual attributes of crosstalk in 3D images. Displays 26(4–5):177–183

    Article  Google Scholar 

  5. Chen W, Fournier J, Barkowsky M, Le Callet P (2012) Exploration of quality of experience of stereoscopic images: binocular depth. VPQM, Scottsdale, Arizona

    Google Scholar 

  6. Barkowsky M, Cousseau R, Le Callet P (2011) Is visual fatigue changing the perceived depth accuracy on an autostereoscopic display? SPIE electronic imaging: stereoscopic displays and applications, vol 7863

    Google Scholar 

  7. Chen W, Fournier J, Barkowsky M, Le Callet P (2011) New stereoscopic video shooting rule based on stereoscopic distortion parameters and comfortable viewing zone, SPIE electronic imaging: stereoscopic displays and applications, vol 7863

    Google Scholar 

  8. Cutting JE, Vishton PM (1995) Perceiving layout and knowing distances: the integration, relative potency, and contextual use of different information about depth. In: Epstein W, Rogers S (eds) Handbook of perception and cognition, vol 5, Perception of space and motion. Academic Press, San Diego, pp 69–117

    Google Scholar 

  9. Lambooij M, IJsselsteijn W, Fortuin M, Heynderickx I (2009) Visual discomfort and visual fatigue of stereoscopic displays: a review. J Imag Tech 53(3):1–14

    Google Scholar 

  10. Tam WJ, Filippo S, Carlos V, Ron R, Namho H (2012) Visual comfort: stereoscopic objects moving in the horizontal and mid-sagittal planes. In: Society of photo-optical instrumentation engineers (SPIE) conference series

    Google Scholar 

  11. Hoffman DM, Girshick AR, Akeley K, Banks MS (2008) Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. J Vis 8(3):1–30

    Article  Google Scholar 

  12. Kim J, Shibata T, Hoffman DM, Banks MS (2011) Assessing vergence-accommodation conflict as a source of discomfort in stereo displays. J Vis 11(11):324

    Article  Google Scholar 

  13. Yano S, Ide S, Mitsuhashi T, Thwaites H (2002) A study of visual fatigue and visual comfort for 3D HDTV/HDTV images. Displays 23(4):191–201

    Article  Google Scholar 

  14. Chen W, Fournier J, Barkowsky M, Le Callet P (2010) New requirements of subjective video quality assessment methodologies for 3DTV, fifth international workshop on video processing and quality metrics (VPQM), Scottsdale

    Google Scholar 

  15. Yano S, Emoto M, Mitsuhashi T (2004) Two factors in visual fatigue caused by stereoscopic HDTV images. Displays 25(4):141–150

    Article  Google Scholar 

  16. Kuze J, Ukai K (2008) Subjective evaluation of visual fatigue caused by motion images. Displays 29(2):159–166

    Article  Google Scholar 

  17. Speranza F, Tam W, Renaud R, Hur N (2006) Effect of disparity and motion on visual comfort of stereoscopic images. Proc SPIE 6055:94–103

    Google Scholar 

  18. Ide S, Yamanoue H, Okui M, Okano F, Bitou M, Terashima N (2002) Parallax distribution for ease of viewing in stereoscopic HDTV. In: SPIE proceedings, vol 4660, pp 38–45

    Google Scholar 

  19. Nojiri Y, Yamanoue H, Hanazato A, Okano F (2003) Measurement of parallax distribution, and its application to the analysis of visual comfort for stereoscopic HDTV. In: Proceedings of SPIE, vol 5006, pp 195–205

    Google Scholar 

  20. Nojiri Y, Yamanoue H, Ide S, Yano S, Okana F (2006) Parallax distribution and visual comfort on stereoscopic HDTV. In: Proceedings of IBC, pp 373–380

    Google Scholar 

  21. Kooi FL, Toet A (2004) Visual comfort of binocular and 3D displays. Displays 25(2–3):99–108

    Article  Google Scholar 

  22. Pastoor S (1995) Human factors of 3D imaging: results of recent research at Heinrich-Hertz-Institut Berlin. In: Proceedings of IDW, pp 69–72

    Google Scholar 

  23. Li J, Barkowsky M, Wang J, Le Callet P (2011) Study on visual discomfort induced by stimulus movement at fixed depth on stereoscopic displays using shutter glasses, digital signal processing (DSP), 17th international conference on, IEEE

    Google Scholar 

  24. Harris JM, Mckee SP, Watamaniuk S (1998) Visual search for motion-in-depth: stereomotion does not ‘pop out’ from disparity noise. Nat Neurosci 1(2):165–168

    Article  Google Scholar 

  25. Cho S-H, Kang H-B (2012) An assessment of visual discomfort caused by motion-in-depth in stereoscopic 3D video. In: Proceedings of the British machine vision conference, pp 1–10

    Google Scholar 

  26. Lee S-I, Jung YJ, Sohn H, Ro YM, Park HW (2011) Visual discomfort induced by fast salient object motion in stereoscopic video, SPIE stereoscopic displays and applications XXII, vol 7863

    Google Scholar 

  27. Li J, Barkowsky M, Le Callet P (2011) The influence of relative disparity and planar motion velocity on visual discomfort of stereoscopic videos. In: Quality of multimedia experience (QoMEX), 2011 third international workshop on, pp 155–160

    Google Scholar 

  28. Gutierrez J, Perez P, Jaureguizar F, Cabrera J, Garcia N (2011) Subjective assessment of the impact of transmission errors in 3DTV compared to HDTV. In: 3DTV conference: the true vision—capture, transmission and display of 3D video (3DTV-CON), pp 1–4

    Google Scholar 

  29. International Telecommunication Union—Radiocommunication Sector (2012) Recommendation ITU-R BT.2021: subjective methods for the assessment of stereoscopic 3DTV systems. ITU-R broadcasting service

    Google Scholar 

  30. Stelmach LB, Tam WJ (1998) Display duration and stereoscopic depth discrimination. Can J Exp Psychol 52(1):56–61

    Article  Google Scholar 

  31. Gutierrez J, Pérez P, Jaureguizar F, Cabrera J, Garcia N (2012) Validation of a novel approach to subjective quality evaluation of conventional and 3D broadcasted video services. In: Quality of multimedia experience (QoMEX), 2012 fourth international workshop on, pp 230–235

    Google Scholar 

  32. ITU-T Study Group 12 (1997) ITU-T P.910 Subjective video quality assessment methods for multimedia applications www.itu.ch

  33. Question ITU-R 211/11 (1974) ITU-R BT.500-13 Methodology for the subjective assessment of the quality of television pictures www.itu.ch

  34. Question ITU-R 81/6 (2003) SAMVIQ subjective assessment methodology for video quality www.itu.ch

  35. Barkowsky M, Li J, Han T, et al (2013) Towards standardized 3DTV QoE assessment: cross-lab study on display technology and viewing environment parameters, SPIE electronic imaging stereoscopic displays and applications, vol 8648

    Google Scholar 

  36. Zielinski S, Rumsey F, Bech S (2008) On some biases encountered in modern audio quality listening tests—a review. J AES 56(6):427–451

    Google Scholar 

  37. Lee J-S, Goldmann L, Ebrahimi T (2012) Paired comparison-based subjective quality assessment of stereoscopic images, multimedia tools and applications, pp 1–18 http://www.springer.com/computer/information+systems+and+applications/journal/11042

  38. Dykstra O (1960) Rank analysis of incomplete block designs: a method of paired comparisons employing unequal repetitions on pairs. Biometrics 16(2):176–188

    Article  MATH  Google Scholar 

  39. Li J, Barkowsky M, Le Callet P (2012) Analysis and improvement of a paired comparison method in the application of 3DTV subjective experiment. In: Image processing (ICIP), 19th IEEE international conference on, pp 629–632

    Google Scholar 

  40. Li J, Barkowsky M, Le Callet P (2013) Boosting paired comparison methodology in measuring visual discomfort of 3DTV: performances of three different designs. In: IS&T/SPIE electronic imaging, pp 86481V-86481V-12

    Google Scholar 

  41. Li J, Barkowsky M, Le Callet P (2013) Subjective assessment methodology for preference of experience in 3DTV. In: 11th IEEE IVMSP workshop: 3D image/video technologies and applications. http://www.ivmsp2013.org/

  42. Silverstein DA, Farrell JE (1998) Quantifying perceptual image quality. In: Proceedings of ST&T’s image processing, image quality, image capture, systems conference, pp 242–246

    Google Scholar 

  43. Strohmeier D, Jumisko-Pyykkö S, Kunze K, Bici MO (2011) The extended-OPQ method for user-centered quality of experience evaluation: a study for mobile 3D video broadcasting over DVB-H. EURASIP J Image Video Process 2011. http://jivp.eurasipjournals.com/content/2011/1/538294

  44. Kunze K, Strohmeier D, Jumisko-Pyykkö S (2011) Comparison of two mixed methods approaches for multimodal quality evaluations: open profiling of quality and conventional profiling. In: Quality of multimedia experience (QoMEX), third international workshop on, pp 137–142

    Google Scholar 

  45. Tashakkori A, Teddlie C (2008) Quality of inferences in mixed methods research: calling for an integrative framework. In: Advances in mixed methods research, pp 101–119

    Google Scholar 

  46. Strohmeier D, Jumisko-Pyykkö S, Kunze K (2010) Open profiling of quality: a mixed method approach to understanding multimodal quality perception. Adv Multimed 2010:1–28

    Article  Google Scholar 

  47. Niedermeyer E, Da Silva FL (2005) Electroencephalography: basic principles, clinical applications, and related fields. Lippincott Williams & Wilkins, Philadelphia

    Google Scholar 

  48. Teplan M (2002) Fundamentals of EEG measurement. Meas Sci Rev 2(2):1–11

    Google Scholar 

  49. Kim Y-J, Lee EC (2011) EEG based comparative measurement of visual fatigue caused by 2D and 3D displays. In: HCI international 2011–posters’ extended abstracts, Springer, pp 289–292

    Google Scholar 

  50. Reiter U, De Moor K (2012) Content categorization based on implicit and explicit user feedback: combining self-reports with EEG emotional state analysis. In: Quality of multimedia experience (QoMEX), 2012 fourth international workshop on, pp 266–271

    Google Scholar 

  51. Li M, Lu B-L (2009) Emotion classification based on gamma-band EEG. In: Engineering in medicine and biology society, annual international conference of the IEEE, pp 1223–1226

    Google Scholar 

  52. Kim D, Jung YJ, Kim E, Ro YM, Park H (2011) Human brain response to visual fatigue caused by stereoscopic depth perception. In: Digital signal processing (DSP), 17th international conference on, pp 1–5

    Google Scholar 

  53. Backus BT, Fleet DJ, Parker AJ, Heeger DJ (2001) Human cortical activity correlates with stereoscopic depth perception. J Neurophysiol 86(4):2054–2068

    Google Scholar 

  54. Tsao DY, Vanduffel W, Sasaki Y et al (2003) Stereopsis activates V3A and caudal intraparietal areas in macaques and humans. Neuron 39(3):555–568

    Article  Google Scholar 

  55. Georgieva S, Peeters R, Kolster H, Todd JT, Orban GA (2009) The processing of three-dimensional shape from disparity in the human brain. J Neurosci 29(3):727–742

    Article  Google Scholar 

  56. Tootell RB et al (1997) Functional analysis of V3A and related areas in human visual cortex. J Neurosci 17(18):7060–7078

    Google Scholar 

  57. Emoto M, Niida T, Okano F (2005) Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television. J Disp Technol 1(2):328–340

    Article  Google Scholar 

  58. Li H-C, Seo J, Kham K, Lee S (2008) Measurement of 3D visual fatigue using event-related potential (ERP): 3D oddball paradigm. In: IEEE 3DTV conference: the true vision-capture, transmission and display of 3D video, pp 213–216

    Google Scholar 

  59. Nahar NK, Sheedy J, Hayes J, Tai Y-C (2007) Objective measurements of lower-level visual stress. Optom Vis Sci 84(7):620–629

    Article  Google Scholar 

  60. Tsubota K, Nakamori K (1993) Dry eyes and video display terminals. New Engl J Med 328(8):584

    Article  Google Scholar 

  61. Lee EC, Heo H, Park KR (2010) The comparative measurements of eyestrain caused by 2D and 3D displays. IEEE Trans Consum Electron 56(3):1677–1683

    Article  Google Scholar 

  62. Yu J-H, Lee B-H, Kim D-H (2012) EOG based eye movement measure of visual fatigue caused by 2D and 3D displays, biomedical and health informatics (BHI), 2012 IEEE-EMBS international conference on, pp 305–308

    Google Scholar 

  63. Kim D, Choi S, Park S, Sohn K (2011) Stereoscopic visual fatigue measurement based on fusional response curve and eye-blinks. In: Digital signal processing (DSP), 17th international conference on, pp 1–6

    Google Scholar 

  64. Divjak M, Bischof H (2009) Eye blink based fatigue detection for prevention of computer vision syndrome. In: IAPR conference on machine vision applications, Tokyo

    Google Scholar 

  65. Li J, Barkowsky M, Le Callet P (2013) Visual discomfort is not always proportional to eye blinking rate: exploring some effects of planar and in-depth motion on 3DTV QoE. Proceedings of VPQM 2013

    Google Scholar 

  66. Wang K, Barkowsky M, Brunnstrom K, Sjostrom M, Cousseau R, Le Callet P (2012) Perceived 3D TV transmission quality assessment: multi-laboratory results using absolute category rating on quality of experience scale. IEEE Trans Broadcast 58(4):544–557

    Article  Google Scholar 

  67. Bosc E, Köppel M, Pépion R, Pressigout M, Morin L, Ndjiki-Nya P et al (2011) Can 3D synthesized views be reliably assessed through usual subjective and objective evaluation protocols? In: Proceedings of international conference on image processing, Brussels, Belgium, pp 2597–2600

    Google Scholar 

  68. Tam WJ, Speranza F, Vázquez CA, Zhang L (2009) Temporal sub-sampling of depth maps in depth image based rendering of stereoscopic image sequences, SPIE stereoscopic displays and applications XX

    Google Scholar 

  69. Tourancheau S, Wang K, Bulat J, Cousseau R, Janowski L, Brunnström K et al (2012) Reproducibility of crosstalk measurements on active glasses 3D LCD displays based on temporal characterization. SPIE electronic imaging stereoscopic displays and applications XXIII, 8288

    Google Scholar 

  70. Society for Information Display—International Committee for Display Metrology (2012) Information display measurements standard (IDMS). http://www.sid.org/ICDM.aspx

  71. Xing L, You J, Ebrahimi T, Perkis A (2012) Assessment of stereoscopic crosstalk perception. IEEE Trans Multimed 14(2):326–337

    Article  Google Scholar 

  72. Pinson M, Janowski L, Pepion R et al (2012) The influence of subjects and environment on audiovisual subjective tests: an international study. IEEE J Sel Top Signal Process 6(6):640–651

    Article  Google Scholar 

  73. Li J, Kaller O, De Simone F et al (2013) Cross-lab study on preference of experience in 3DTV: influence from display technology and test environment. In: fifth international workshop on quality of multimedia experience (QoMEX)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LUNAM Université, Université de Nantes, IRCCyN UMR CNRS 6597, Polytech Nantes, Rue Christian Pauc, 50609, 44306, Nantes, France

    Jing Li, Marcus Barkowsky & Patrick Le Callet

Authors
  1. Jing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcus Barkowsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patrick Le Callet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Li .

Editor information

Editors and Affiliations

  1. Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey Faculty Engineering & Physical Sciences, Guildford, United Kingdom

    Ahmet Kondoz

  2. Dept. of Computer Science, Hellenic Open University, Patras, Greece

    Tasos Dagiuklas

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this chapter

Cite this chapter

Li, J., Barkowsky, M., Le Callet, P. (2014). Assessing the Quality of Experience of 3DTV and Beyond: Tackling the Multidimensional Sensation. In: Kondoz, A., Dagiuklas, T. (eds) 3D Future Internet Media. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8373-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-8373-1_11

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-8372-4

  • Online ISBN: 978-1-4614-8373-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation