Multimedia Tools and Applications

, Volume 75, Issue 20, pp 12409–12429 | Cite as

Modeling immersive media experiences by sensing impact on subjects

  • Eleni Kroupi
  • Philippe Hanhart
  • Jong-Seok Lee
  • Martin Rerabek
  • Touradj Ebrahimi


As immersive technologies target to provide higher quality of multimedia experiences, it is important to understand the quality of experience (QoE) perceived by users from various multimedia rendering schemes, in order to design and optimize human-centric immersive multimedia systems. In this study, various QoE-related aspects, such as depth perception, sensation of reality, content preference, and perceived quality are investigated and compared for presentation of 2D and 3D contents. Since the advantages of implicit over explicit QoE assessment have become essential, the way these QoE-related aspects influence brain and periphery is also investigated. In particular, two classification schemes using electroencephalography (EEG) and peripheral signals (electrocardiography and respiration) are carried out, to explore if it is possible to automatically recognize the QoE-related aspects under investigation. In addition, a decision-fusion scheme is applied to EEG and peripheral features, to explore the advantage of integrating information from the two modalities. The results reveal that the highest monomodal average informedness is achieved in the high beta EEG band (0.14 % ± 0.09, p < 0.01), when recognizing sensation of reality. The highest and significantly non-random multimodal average informedness is achieved when high beta EEG band is fused with peripheral features (0.17 % ± 0.1, p < 0.01), for the case of sensation of reality. Finally, a temporal analysis is conducted to explore how the EEG correlates for the case of sensation of reality change over time. The results reveal that the right cortex is more involved when sensation of reality is low, and the left when sensation of reality is high, indicating that approach and withdrawal-related processes occur during sensation of reality.


EEG Heart rate Respiration Immersiveness Fusion Quality of experience 


  1. 1.
    Bilchick KC, Berger RD (2006) Heart rate variability. J Cardiovasc Electrophysiol 17(6):691–694CrossRefGoogle Scholar
  2. 2.
    Blankertz B, Lemm S, Treder M, Haufe S, Müller KR (2011) Single-trial analysis and classification of ERP components: a tutorial. NeuroImage 56(2):814–825CrossRefGoogle Scholar
  3. 3.
    Davidson R, Ekman P, Saron C, Senulis J, Friesen W (1990) Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I. J Pers Soc Psychol 58(2):330–341CrossRefGoogle Scholar
  4. 4.
    Dores AR, Barbosa F, Monteiro L, Leitão M, Reis M, Coelho CM, Ribeiro E, Carvalho IP, Sousa L, Castro-Caldas A (2014) Amygdala activation in response to 2D and 3D emotion-inducing stimuli. PsychNology Journal 12(1–2):29–43Google Scholar
  5. 5.
    Emoto M, Niida T, Okana F (2005) Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television. J Disp Technol 1 (2):328–340CrossRefGoogle Scholar
  6. 6.
    Fukunaga K (1990) Introduction to Statistical Pattern Recognition. Academic pressGoogle Scholar
  7. 7.
    Hagura H, Nakajima M, Owaki T, Takeda T (2006) Study of asthenopia caused by the viewing of stereoscopic images: measuring by MEG and other devices. In: Proceedings of SPIE, vol 6057, pp 192–202Google Scholar
  8. 8.
    Hastie T, Tibshirani R, Friedman J (2009) The Elements of Statistical Learning, vol. 2. SpringerGoogle Scholar
  9. 9.
    Hayward V, Astley OR, Cruz-Hernandez M, Grant D (2004) Robles-De-La-Torre, G: Haptic interfaces and devices. Sens Rev 24(1):16–29CrossRefGoogle Scholar
  10. 10.
    Hofman D, Schutter DJ (2011) Asymmetrical frontal resting-state beta oscillations predict trait aggressive tendencies and behavioral inhibition. Soc Cogn Affect Neurosci:1–8Google Scholar
  11. 11.
    ITU-R BT.2021 (2012) Subjective methods for the assessment of stereoscopic 3DTV systems International Telecommunication UnionGoogle Scholar
  12. 12.
    ITU-R BT.500-13 (2012) Methodology for the subjective assessment of the quality of television pictures International Telecommunication UnionGoogle Scholar
  13. 13.
    ITU-T P.910 (2008) Subjective video quality assessment methods for multimedia applications International Telecommunication UnionGoogle Scholar
  14. 14.
    Jensen O, Goel P, Kopell N, Pohja M, Hari R, Ermentrout B (2005) On the human sensorimotor-cortex beta rhythm: sources and modeling. Neuroimage 26(2):347–355CrossRefGoogle Scholar
  15. 15.
    Kim D, Jung YJ, Kim E, Ro YM, Park HW (2011) Human brain response to visual fatigue caused by stereoscopic depth perception. In: Proc. Int. Conf. Digital Signal Processing. Corfu, Greece , pp 1–5Google Scholar
  16. 16.
    Kittler J, Hatef M, Duin RP, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239CrossRefGoogle Scholar
  17. 17.
    Kroupi E, Hanhart P, Lee JS, Rerabek M, Ebrahimi T (2014) EEG correlates during video quality perception. In: Proceedings of the 22nd European Signal Processing Conference (EUSIPCO), Lisbon, PortugalGoogle Scholar
  18. 18.
    Kroupi E, Hanhart P, Lee JS, Rerabek M, Ebrahimi T (2014) Predicting subjective sensation of reality during multimedia consumption based on EEG and peripheral physiological signals. In: Proceedings of the IEEE International Conference on Multimedia and ExpoGoogle Scholar
  19. 19.
    Kroupi E, Hanhart P, Lee JS, Rerabek M, Ebrahimi T (2014) User-independent classification of 2D versus 3D multimedia experiences through EEG and physiological signals. In: Proceedings of the 8th International Workshop on Video Processing and Quality Metrics for Consumer Electronics-VPQMGoogle Scholar
  20. 20.
    Kulkarni SD, Minor MA, Deaver MW, Pardyjak ER, Hollerbach JM (2012) Design, sensing, and control of a scaled wind tunnel for atmospheric display. IEEE/ASME Trans Mechatron 17(4):635– 645CrossRefGoogle Scholar
  21. 21.
    Lee JS, De Simone F, Ebrahimi T (2011) Subjective quality evaluation of foveated video coding using audio-visual focus of attention. IEEE J Sel Top Sign Proces 5(7):1322–1331CrossRefGoogle Scholar
  22. 22.
    Li HCO, Seo J, Kham K, Lee S (2008) Measurement of 3D visual fatigue using event-related potential (ERP): 3D oddball paradigm , Proc. 3DTV Conf. Istanbul, Turkey, pp 213–216Google Scholar
  23. 23.
    Muller K, Anderson CW, Birch GE (2003) Linear and nonlinear methods for brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 11(2):165–169CrossRefGoogle Scholar
  24. 24.
    Nunez P, Srinivasan R (2006) Electric Fields of the Brain: The Neurophysics of EEG. Oxford University PressGoogle Scholar
  25. 25.
    Pan J, Tompkins WJ (1985) A real-time QRS detection algorithm. IEEE Trans Biomed Eng 3:230– 236CrossRefGoogle Scholar
  26. 26.
    Picard R, Vyzas E, Healey J (2001) Toward machine emotional intelligence: Analysis of affective physiological state. Pattern Analysis and Machine Intelligence. IEEE Transactions on 23(10):1175– 1191Google Scholar
  27. 27.
    Powers DMW (2011) Evaluation: From precision, recall and f-measure to ROC, informedness, markedness & correlation. J Mach Learn Technol 2(1):37–63MathSciNetGoogle Scholar
  28. 28.
    von der Pütten AM, Klatt J, Broeke ST, McCall R, Krämer NC, Wetzel R, Blum L, Oppermann L, Klatt J (2012) Subjective and behavioral presence measurement and interactivity in the collaborative augmented reality game TimeWarp. Interacting with Computers 24:317–325CrossRefGoogle Scholar
  29. 29.
    Sanchez-Vives MV, Slater M (2005) From presence to consciousness through virtual reality. Nat Rev Neurosci 6(4):332–339CrossRefGoogle Scholar
  30. 30.
    Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):32MathSciNetGoogle Scholar
  31. 31.
    Scholler S, Bosse S, Treder MS, Blankertz B, Curio G, Müller KR, Wiegand T (2012) Toward a direct measure of video quality perception using EEG. IEEE Trans Image Process 21(5):2619– 2629MathSciNetCrossRefGoogle Scholar
  32. 32.
    Schutter DJ, de Weijer AD, Meuwese JD, Morgan B, Van Honk J (2008) Interrelations between motivational stance, cortical excitability, and the frontal electroencephalogram asymmetry of emotion: a transcranial magnetic stimulation study. Hum Brain Mapp 29(5):574–580CrossRefGoogle Scholar
  33. 33.
    Slater M (2009) Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc, B 364(1535):3549–3557CrossRefGoogle Scholar
  34. 34.
    Slater M, Wilbur S (1997) A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators and Virtual Environments 6(6):603–616CrossRefGoogle Scholar
  35. 35.
    Stoakley R, Conway MJ, Pausch R (1995) Virtual reality on a WIM: interactive worlds in miniature. In: Proc. SIGCHI Conf. Human Factors in Computing Systems. CO, Denver, pp 265–272Google Scholar
  36. 36.
    Theodoridis S, Koutroumbas K (1998) Pattern Recognition. Academic PressGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Eleni Kroupi
    • 1
  • Philippe Hanhart
    • 1
  • Jong-Seok Lee
    • 2
  • Martin Rerabek
    • 1
  • Touradj Ebrahimi
    • 1
  1. 1.Multimedia Signal Processing GroupEPFLLausanneSwitzerland
  2. 2.School of Integrated TechnologyYonsei UniversitySeoulSouth Korea

Personalised recommendations