Skip to main content
SpringerLink
Log in

Modeling immersive media experiences by sensing impact on subjects

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As immersive technologies target to provide higher quality of multimedia experiences, it is important to understand the quality of experience (QoE) perceived by users from various multimedia rendering schemes, in order to design and optimize human-centric immersive multimedia systems. In this study, various QoE-related aspects, such as depth perception, sensation of reality, content preference, and perceived quality are investigated and compared for presentation of 2D and 3D contents. Since the advantages of implicit over explicit QoE assessment have become essential, the way these QoE-related aspects influence brain and periphery is also investigated. In particular, two classification schemes using electroencephalography (EEG) and peripheral signals (electrocardiography and respiration) are carried out, to explore if it is possible to automatically recognize the QoE-related aspects under investigation. In addition, a decision-fusion scheme is applied to EEG and peripheral features, to explore the advantage of integrating information from the two modalities. The results reveal that the highest monomodal average informedness is achieved in the high beta EEG band (0.14 % ± 0.09, p < 0.01), when recognizing sensation of reality. The highest and significantly non-random multimodal average informedness is achieved when high beta EEG band is fused with peripheral features (0.17 % ± 0.1, p < 0.01), for the case of sensation of reality. Finally, a temporal analysis is conducted to explore how the EEG correlates for the case of sensation of reality change over time. The results reveal that the right cortex is more involved when sensation of reality is low, and the left when sensation of reality is high, indicating that approach and withdrawal-related processes occur during sensation of reality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://mmspg.epfl.ch/mimesis

  2. http://www.nvp3d.com

  3. http://www.red.com/products/scarlet

  4. http://www.red.com/learn/red-101/redcode-file-format

  5. http://www.psiaudio.com/product/active-monitors/a14-m

  6. https://www.egi.com/clinical-division/clinical-division-clinical-products/ges-300

References

  1. Bilchick KC, Berger RD (2006) Heart rate variability. J Cardiovasc Electrophysiol 17(6):691–694

    Article  Google Scholar 

  2. Blankertz B, Lemm S, Treder M, Haufe S, Müller KR (2011) Single-trial analysis and classification of ERP components: a tutorial. NeuroImage 56(2):814–825

    Article  Google Scholar 

  3. Davidson R, Ekman P, Saron C, Senulis J, Friesen W (1990) Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I. J Pers Soc Psychol 58(2):330–341

    Article  Google Scholar 

  4. Dores AR, Barbosa F, Monteiro L, Leitão M, Reis M, Coelho CM, Ribeiro E, Carvalho IP, Sousa L, Castro-Caldas A (2014) Amygdala activation in response to 2D and 3D emotion-inducing stimuli. PsychNology Journal 12(1–2):29–43

    Google Scholar 

  5. Emoto M, Niida T, Okana F (2005) Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television. J Disp Technol 1 (2):328–340

    Article  Google Scholar 

  6. Fukunaga K (1990) Introduction to Statistical Pattern Recognition. Academic press

  7. Hagura H, Nakajima M, Owaki T, Takeda T (2006) Study of asthenopia caused by the viewing of stereoscopic images: measuring by MEG and other devices. In: Proceedings of SPIE, vol 6057, pp 192–202

  8. Hastie T, Tibshirani R, Friedman J (2009) The Elements of Statistical Learning, vol. 2. Springer

  9. Hayward V, Astley OR, Cruz-Hernandez M, Grant D (2004) Robles-De-La-Torre, G: Haptic interfaces and devices. Sens Rev 24(1):16–29

    Article  Google Scholar 

  10. Hofman D, Schutter DJ (2011) Asymmetrical frontal resting-state beta oscillations predict trait aggressive tendencies and behavioral inhibition. Soc Cogn Affect Neurosci:1–8

  11. ITU-R BT.2021 (2012) Subjective methods for the assessment of stereoscopic 3DTV systems International Telecommunication Union

  12. ITU-R BT.500-13 (2012) Methodology for the subjective assessment of the quality of television pictures International Telecommunication Union

  13. ITU-T P.910 (2008) Subjective video quality assessment methods for multimedia applications International Telecommunication Union

  14. Jensen O, Goel P, Kopell N, Pohja M, Hari R, Ermentrout B (2005) On the human sensorimotor-cortex beta rhythm: sources and modeling. Neuroimage 26(2):347–355

    Article  Google Scholar 

  15. Kim D, Jung YJ, Kim E, Ro YM, Park HW (2011) Human brain response to visual fatigue caused by stereoscopic depth perception. In: Proc. Int. Conf. Digital Signal Processing. Corfu, Greece , pp 1–5

  16. Kittler J, Hatef M, Duin RP, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239

    Article  Google Scholar 

  17. Kroupi E, Hanhart P, Lee JS, Rerabek M, Ebrahimi T (2014) EEG correlates during video quality perception. In: Proceedings of the 22nd European Signal Processing Conference (EUSIPCO), Lisbon, Portugal

  18. Kroupi E, Hanhart P, Lee JS, Rerabek M, Ebrahimi T (2014) Predicting subjective sensation of reality during multimedia consumption based on EEG and peripheral physiological signals. In: Proceedings of the IEEE International Conference on Multimedia and Expo

  19. Kroupi E, Hanhart P, Lee JS, Rerabek M, Ebrahimi T (2014) User-independent classification of 2D versus 3D multimedia experiences through EEG and physiological signals. In: Proceedings of the 8th International Workshop on Video Processing and Quality Metrics for Consumer Electronics-VPQM

  20. Kulkarni SD, Minor MA, Deaver MW, Pardyjak ER, Hollerbach JM (2012) Design, sensing, and control of a scaled wind tunnel for atmospheric display. IEEE/ASME Trans Mechatron 17(4):635– 645

    Article  Google Scholar 

  21. Lee JS, De Simone F, Ebrahimi T (2011) Subjective quality evaluation of foveated video coding using audio-visual focus of attention. IEEE J Sel Top Sign Proces 5(7):1322–1331

    Article  Google Scholar 

  22. Li HCO, Seo J, Kham K, Lee S (2008) Measurement of 3D visual fatigue using event-related potential (ERP): 3D oddball paradigm , Proc. 3DTV Conf. Istanbul, Turkey, pp 213–216

  23. Muller K, Anderson CW, Birch GE (2003) Linear and nonlinear methods for brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 11(2):165–169

    Article  Google Scholar 

  24. Nunez P, Srinivasan R (2006) Electric Fields of the Brain: The Neurophysics of EEG. Oxford University Press

  25. Pan J, Tompkins WJ (1985) A real-time QRS detection algorithm. IEEE Trans Biomed Eng 3:230– 236

    Article  Google Scholar 

  26. Picard R, Vyzas E, Healey J (2001) Toward machine emotional intelligence: Analysis of affective physiological state. Pattern Analysis and Machine Intelligence. IEEE Transactions on 23(10):1175– 1191

    Google Scholar 

  27. Powers DMW (2011) Evaluation: From precision, recall and f-measure to ROC, informedness, markedness & correlation. J Mach Learn Technol 2(1):37–63

    MathSciNet  Google Scholar 

  28. von der Pütten AM, Klatt J, Broeke ST, McCall R, Krämer NC, Wetzel R, Blum L, Oppermann L, Klatt J (2012) Subjective and behavioral presence measurement and interactivity in the collaborative augmented reality game TimeWarp. Interacting with Computers 24:317–325

    Article  Google Scholar 

  29. Sanchez-Vives MV, Slater M (2005) From presence to consciousness through virtual reality. Nat Rev Neurosci 6(4):332–339

    Article  Google Scholar 

  30. Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):32

    MathSciNet  Google Scholar 

  31. Scholler S, Bosse S, Treder MS, Blankertz B, Curio G, Müller KR, Wiegand T (2012) Toward a direct measure of video quality perception using EEG. IEEE Trans Image Process 21(5):2619– 2629

    Article  MathSciNet  Google Scholar 

  32. Schutter DJ, de Weijer AD, Meuwese JD, Morgan B, Van Honk J (2008) Interrelations between motivational stance, cortical excitability, and the frontal electroencephalogram asymmetry of emotion: a transcranial magnetic stimulation study. Hum Brain Mapp 29(5):574–580

    Article  Google Scholar 

  33. Slater M (2009) Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc, B 364(1535):3549–3557

    Article  Google Scholar 

  34. Slater M, Wilbur S (1997) A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators and Virtual Environments 6(6):603–616

    Article  Google Scholar 

  35. Stoakley R, Conway MJ, Pausch R (1995) Virtual reality on a WIM: interactive worlds in miniature. In: Proc. SIGCHI Conf. Human Factors in Computing Systems. CO, Denver, pp 265–272

  36. Theodoridis S, Koutroumbas K (1998) Pattern Recognition. Academic Press

Download references

Author information

Authors and Affiliations

  1. Multimedia Signal Processing Group, EPFL, Station 11, CH-1015, Lausanne, Switzerland

    Eleni Kroupi, Philippe Hanhart, Martin Rerabek & Touradj Ebrahimi

  2. School of Integrated Technology, Yonsei University, Seoul, South Korea

    Jong-Seok Lee

Authors
  1. Eleni Kroupi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philippe Hanhart
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jong-Seok Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martin Rerabek
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Touradj Ebrahimi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jong-Seok Lee.

Additional information

The research leading to these results has been performed in the framework of two Swiss National Foundation for Scientific Research (FN 200020-132673-1 and FN 200021-143696-1), FP7 EC EUROSTAR funded Project - Transcoders Of the Future TeleVision (TOFuTV), QoE-Net Initial Training Network (H2020-MSCA-ITN-2014), the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Science, ICT and Future Planning (MSIP), Korea (2013R1A1A1007822), and the IT Consilience Creative Program funded by MSIP, Korea (IITP-2015-R0346-15-1008).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kroupi, E., Hanhart, P., Lee, JS. et al. Modeling immersive media experiences by sensing impact on subjects. Multimed Tools Appl 75, 12409–12429 (2016). https://doi.org/10.1007/s11042-015-2980-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2980-z

Keywords

Search

Navigation