Advertisement

Exploiting sensing devices availability in AR/VR deployments to foster engagement

  • Nicholas Vretos
  • Petros Daras
  • Stylianos Asteriadis
  • Enrique Hortal
  • Esam Ghaleb
  • Evaggelos Spyrou
  • Helen C. Leligou
  • Panagiotis Karkazis
  • Panagiotis Trakadas
  • Kostantinos Assimakopoulos
S.I. : VR in Education
  • 167 Downloads

Abstract

Currently, in all augmented reality (AR) or virtual reality (VR) educational experiences, the evolution of the experience (game, exercise or other) and the assessment of the user’s performance are based on her/his (re)actions which are continuously traced/sensed. In this paper, we propose the exploitation of the sensors available in the AR/VR systems to enhance the current AR/VR experiences, taking into account the users’ affect state that changes in real time. Adapting the difficulty level of the experience to the users’ affect state fosters their engagement which is a crucial issue in educational environments and prevents boredom and anxiety. The users’ cues are processed enabling dynamic user profiling. The detection of the affect state based on different sensing inputs, since diverse sensing devices exist in different AR/VR systems, is investigated, and techniques that have been undergone validation using state-of-the-art sensors are presented.

Keywords

Affect state detection Engagement Interpretation of interaction Multimodal affect state detection 

Notes

Acknowledgements

The work presented in this document has been partially funded through H2020-MaTHiSiS Project. This project has received funding from the European Union’s Horizon 2020 Programme (H2020-ICT-2015) under Grant Agreement No. 687772.

References

  1. Anagnostopoulos CN, Iliou T, Giannoukos I (2015) Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artif Intell Rev 43(2):155–177CrossRefGoogle Scholar
  2. Baker RSJ, D’Mello SK, Rodrigo MT, Graesser AC (2010) Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. Int. J. Hum.-Comput. Stud. 68:223–241CrossRefGoogle Scholar
  3. Baltrušaitis T, Robinson P, Morency LP (2016) Openface: an open source facial behavior analysis toolkit. 2016 IEEE Winter conference on IEEE applications of computer vision (WACV)Google Scholar
  4. Burkhardt F et al (2005) A database of german emotional speech. In: Proceedings of interspeech, LissabonGoogle Scholar
  5. Chen C, Liu K, Kehtarnavaz N (2016) Real-time human action recognition based on depth motion maps. J Real-Time Image Proc 12(1):155–163CrossRefGoogle Scholar
  6. Costantini G et al (2014) Emovo corpus: an Italian emotional speech database. In: Chair NCC, Choukri K, Declerck T, Loftsson H, Maegaard B, Mariani J, Moreno A, Odijk J, Piperidis S (eds) Proceedings of the ninth international conference on language resources and evaluation (LREC’14). European Language Resources Association (ELRA), Reykjavik, IcelandGoogle Scholar
  7. Coutrix C et al (2012) Identifying emotions expressed by mobile users through 2D surface and 3D motion gestures. In: Proceedings of the 2012 ACM conference on ubiquitous computingGoogle Scholar
  8. Csikszentmihalyi M (1996) Flow and the psychology of discovery and invention. Harper Collins, New YorkGoogle Scholar
  9. Csíkszentmihályi M (2008) Flow: the psychology of optimal experience. Harper Perennial, New YorkGoogle Scholar
  10. Du Y, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. In: Proceedings of the IEEE international conference on computer vision and pattern recognition (CVPR)Google Scholar
  11. El Ayadi M, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recogn 44(3):572–587CrossRefzbMATHGoogle Scholar
  12. Ghaleb E, Popa M, Hortal E, Asteriadis S (2017). Multimodal fusion based on information gain for emotion recognition in the wild, intelligent systems conference 2017, 7–8 Sept 2017, London, UKGoogle Scholar
  13. Hansen DW, Ji Q (2010). In the eye of the beholder: a survey of models for eyes. IEEE Trans Pattern Anal Mach IntellGoogle Scholar
  14. Haq S, Jackson P (2009) Speaker-dependent audio-visual emotion recognition. In: Proceedings of the international conference on auditory-visual speech processing (AVSP’08), Norwich, UKGoogle Scholar
  15. Kanade T, Cohn JF, Lucey P, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn–Kanade Dataset (CK+): a complete expression dataset for action unit and emotion-specified expression, San Francisco, USAGoogle Scholar
  16. Kim M et al (2013) A touch based affective user interface for smartphone. In: IEEE international conference on consumer electronics (ICCE). IEEEGoogle Scholar
  17. Leifman G et al (2016) Learning gaze transitions from depth to improve video saliency estimation. arXiv preprint arXiv:1603.03669
  18. Li W, Zhang Z, Liu Z (2010) Action recognition based on a bag of 3d points. In: Proceedings of IEEE international conference in computer vision and pattern recognition workshop (CVPRW)Google Scholar
  19. Mora KAF, Monay F, Odobez J-M (2014) Eyediap: a database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. In: Proceedings of the symposium on eye tracking research and applications. ACMGoogle Scholar
  20. Nottingham Trent University (ed) (2017) Adaptation and Personalization principles based on MaTHiSiS findings. Deliverable for the MaTHiSiS project. http://www.mathisis-project.eu/en/content/d61-adaptation-and-personalization-principles-based-mathisis-findings. Accessed 3/4/2018
  21. Pedregosa F et al (2011) Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12:2825–2830MathSciNetzbMATHGoogle Scholar
  22. Rikert TD, Jones MJ (1998) Gaze estimation using morphable models. In: Proceedings of third IEEE international conference on automatic face and gesture recognition. IEEEGoogle Scholar
  23. Vemulapalli R, Arrate F, Chellappa R (2014) Human action recognition by representing 3d skeletons as points in a lie group. In: Proceedings of the IEEE international conference on computer vision and pattern recognition (CVPR)Google Scholar
  24. Wang L, Zhang J, Zhou L, Tang C, Li W (2015) Beyond covariance: feature representation with nonlinear kernel matrices. In: Proceedings of the IEEE international conference on computer vision (ICCV)Google Scholar
  25. Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. In: 2013 IEEE conference on computer vision and pattern recognition, Portland, pp 532–539Google Scholar
  26. Zhang X, Sugano Y, Fritz M, Bulling A (2015) Appearance-based gaze estimation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4511–4520Google Scholar
  27. Zhang X, Sugano Y, Fritz M, Bulling A (2017) MPIIGaze: real-world dataset and deep appearance-based gaze estimation. IEEE Trans Pattern Anal Mach IntellGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  • Nicholas Vretos
    • 1
  • Petros Daras
    • 1
  • Stylianos Asteriadis
    • 2
  • Enrique Hortal
    • 2
  • Esam Ghaleb
    • 2
  • Evaggelos Spyrou
    • 3
  • Helen C. Leligou
    • 4
  • Panagiotis Karkazis
    • 4
  • Panagiotis Trakadas
    • 4
  • Kostantinos Assimakopoulos
    • 4
  1. 1.Centre of Research and Technology HellasThermi, ThessalonikiGreece
  2. 2.University of MaastrichtMaastrichtThe Netherlands
  3. 3.National Centre for Scientific Research “Demokritos”Agia Paraskevi, AthensGreece
  4. 4.Technological Educational Institute of Sterea ElladaPsahna, HalkidaGreece

Personalised recommendations