Advertisement

Augmenting VR/AR Applications with EEG/EOG Monitoring and Oculo-Vestibular Recoupling

  • John K. ZaoEmail author
  • Tzyy-Ping Jung
  • Hung-Ming Chang
  • Tchin-Tze Gan
  • Yu-Te Wang
  • Yuan-Pin Lin
  • Wen-Hao Liu
  • Guang-Yu Zheng
  • Chin-Kuo Lin
  • Chia-Hung Lin
  • Yu-Yi Chien
  • Fang-Cheng Lin
  • Yi-Pai Huang
  • Sergio José Rodríguez Méndez
  • Felipe A. Medeiros
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9743)

Abstract

Head-mounted virtual reality and augmented reality displays (a.k.a. VR/AR goggles) created a revolutionary multimedia genre that is seeking ever-broadening applications and novel natural human interfaces. Adding neuromonitoring and neurofeedback to this genre is expected to introduce a new dimension to user interaction with the cyber-world. This study presents the development of a Neuromonitoring VR/AR Goggle armed with electroence-phalo-gram and electrooculogram sensors, programmable milli-Ampere current stimulators and wireless fog/cloud computing support. Beside of its potential use in mitigating cybersickness, this device may have potential applications in augmented cognition ranging from feedback-controlled perceptual training to on-line learning and virtual social interactions. A prototype of the device has been made from a Samsung Gear VR for S6. This study explains its technical design to ensure precision data sampling, synchronous event marking, real-time signal processing and big data cloud computing support. This study also demonstrates the effective-ness in measuring the event-related potentials during a visual oddball experiment.

Keywords

Virtual reality Mixed reality Augmented cognition Cybersickness Electroencephalography Electrooculography Oculo-vestibular recoupling 

References

  1. 1.
    Stein, J.: Inside the box. TIME Mag. 186(6), 30–39 (2015)Google Scholar
  2. 2.
    Virtual Reality: Grand Illusions. The Economist 416(8953) (2015)Google Scholar
  3. 3.
    Brown, S.: Interactive Neurological Methods for Characterizing Syntactic Elements of Virtual Reality. Personal communication (2015)Google Scholar
  4. 4.
    Cevette, M.J., Stepanek, J., Cocco, D., Galea, A.M., Pradhan, G.N., Wagner, L.S., Oakley, S.R., Smith, B.E., Zapala, D.A., Brookler, K.H.: Oculo-vestibular recoupling using galvanic vestibular stimulation to mitigate simulator sickness. Aviat. Space Environ. Med. 83(6), 549–555 (2012)CrossRefGoogle Scholar
  5. 5.
    Yeh, K.H., She, H.C.: On-line synchronous scientific argumentation learning: nurturing students’ argumentation ability and conceptual change in science context. Comput. Educ. 55(2), 586–602 (2010)CrossRefGoogle Scholar
  6. 6.
    Bulling, A., Gellersen, H.: Toward mobile eye-based human-computer interaction. IEEE Pervasive Comput. 9(4), 8–12 (2010)CrossRefGoogle Scholar
  7. 7.
    Chen, S.C., She, H.C., Chuang, M.H., Wu, J.Y., Tsai, J.L., Jung, T.P.: Eye movements predict students’ computer-based assessment performance of physics concepts in different presentation modalities. Comput. Educ. 74(1), 61–72 (2014)CrossRefGoogle Scholar
  8. 8.
    Schleicher, R., Galley, N., Briest, S., Galley, L.: Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51(7), 982–1010 (2008)CrossRefGoogle Scholar
  9. 9.
    Jung, T.P., Makeig, S., Stensmo, M., Sejnowski, T.J.: Estimating alertness from the EEG power spectrum. IEEE Trans. Biomed. Eng. 44(1), 60–69 (1997)CrossRefGoogle Scholar
  10. 10.
    Jap, B.T., Lal, S., Fischer, P., Bekiaris, E.: Using EEG spectral components to assess algorithms for detecting fatigue. Expert Syst. Appl. 36(2), 2352–2359 (2009)CrossRefGoogle Scholar
  11. 11.
    Wang, Y.T., Huang, K.C., Wei, C.S., Huang, T.Y., Ko, L.W., Lin, C.T., Cheng, C.K., Jung, T.P.: Developing an EEG-based on-line closed-loop lapse detection and mitigation system. Front. Neurosci. 8, 321 (2014)Google Scholar
  12. 12.
    Decety, J., Jackson, P.L.: The functional architecture of human empathy. Behav. Cogn. Neurosci. Rev. 3(2), 71–100 (2004)CrossRefGoogle Scholar
  13. 13.
    Lin, Y.P., Wang, C.H., Jung, T.P., Wu, T.L., Jeng, S.K., Duann, J.R., Chen, J.H.: EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)CrossRefGoogle Scholar
  14. 14.
    LaViola Jr., J.J.: A discussion of cybersickness in virtual environments. ACM SIGCHI Bull. 32(1), 47–56 (2000)CrossRefGoogle Scholar
  15. 15.
    Kim, D.J., Yogendrakumar, V., Chiang, J., Ty, E., Wang, Z.J., McKeown, M.J.: Noisy galvanic vestibular stimulation modulates the amplitude of EEG synchrony patterns. PLoS One 8(7), e69055 (2013)CrossRefGoogle Scholar
  16. 16.
    Oculus Mobile SDK. Oculus. https://developer.oculus.com/documentation/mobilesdk/latest/. Accessed Mar 2016
  17. 17.
    OpenGL ES. Khronos Group. https://www.khronos.org/opengles/. Accessed 2016
  18. 18.
    Zao, J.K., Gan, T.T., You, C.K., Chung, C.E., Wang, Y.T., Méndez, S.J.R., Mullen, T., Yu, C., Kothe, C., Hsiao, C.T., Chu, S.L.: Pervasive brain monitoring and data sharing based on multi-tier distributed computing and linked data technology. Front. Hum. Neurosci. 8, 370 (2014)CrossRefGoogle Scholar
  19. 19.
    Rodríguez Méndez, S., Zao, J.K.: https://w3id.org/BCI-ontology. Accessed Mar 2016
  20. 20.
    Semantic Sensor Network Ontology. W3C. https://www.w3.org/2005/Incubator/ssn/ssnx/ssn. Accessed 2009–2011
  21. 21.
    Linked Data. http://linkeddata.org. Accessed 2009
  22. 22.
    CerebraWeb.net. NCTU Pervasive Embedded Technology (PET) Laboratory. http://cerebraweb.net. Accessed 2015
  23. 23.
    Kim, D.J., Yogendrakumar, V., Chiang, J., Ty, E., Wang, Z.J., McKeown, M.J.: Noisy galvanic vestibular stimulation modulates the amplitude of EEG synchrony patterns. PLoS One 8(7), e69055 (2013)CrossRefGoogle Scholar
  24. 24.
    Yocto Project. Linux Foundation Collaborative Project. https://www.yoctoproject.org/
  25. 25.
    Message Queuing Telemetry Transport. http://mqtt.org/. Accessed 1999
  26. 26.
    Makeig, S., Debener, S., Onton, J., Delorme, A.: Mining event-related brain dynamics. Trends Cogn. Sci. 8(5), 204–210 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • John K. Zao
    • 1
    • 2
    • 4
    Email author
  • Tzyy-Ping Jung
    • 1
    • 2
    • 6
  • Hung-Ming Chang
    • 2
    • 4
  • Tchin-Tze Gan
    • 2
  • Yu-Te Wang
    • 1
    • 6
  • Yuan-Pin Lin
    • 6
  • Wen-Hao Liu
    • 4
  • Guang-Yu Zheng
    • 4
  • Chin-Kuo Lin
    • 3
  • Chia-Hung Lin
    • 3
  • Yu-Yi Chien
    • 5
  • Fang-Cheng Lin
    • 1
    • 2
    • 5
  • Yi-Pai Huang
    • 1
    • 2
    • 5
  • Sergio José Rodríguez Méndez
    • 4
  • Felipe A. Medeiros
    • 1
    • 7
  1. 1.NGoggle Inc.San DiegoUSA
  2. 2.Cerebra Technologies Co. Ltd.HsinchuTaiwan, ROC
  3. 3.Powerforce Tech. Co. Ltd.HsinchuTaiwan, ROC
  4. 4.Department of Computer ScienceNational Chiao Tung UniversityHsinchuTaiwan, ROC
  5. 5.Department of PhotonicsNational Chiao Tung UniversityHsinchuTaiwan, ROC
  6. 6.Swartz Center for Computational Neuroscience, Institute for Neural ComputationUniversity of California at San DiegoLa JollaUSA
  7. 7.Hamilton Glaucoma Center, Shiley Eye InstituteUniversity of California at San DiegoLa JollaUSA

Personalised recommendations