Abstract
Recently research into Brain-Computer Interfacing (BCI) applications for healthy users, such as games, has been initiated. But why would a healthy person use a still-unproven technology such as BCI for game interaction? BCI provides a combination of information and features that no other input modality can offer. But for general acceptance of this technology, usability and user experience will need to be taken into account when designing such systems. Therefore, this chapter gives an overview of the state of the art of BCI in games and discusses the consequences of applying knowledge from Human-Computer Interaction (HCI) to the design of BCI for games. The integration of HCI with BCI is illustrated by research examples and showcases, intended to take this promising technology out of the lab. Future research needs to move beyond feasibility tests, to prove that BCI is also applicable in realistic, real-world settings.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Due to different ITR measures used in BCI, a comparison between keyboard and BCI is hard to make. The entropy of written English text is estimated to be as low as 1.3 bit per symbol (Cover and Thomas, 2006, page 174). A rate of 300 characters per minute would therefore correspond to roughly 400 bits per minute.
- 2.
To be published.
- 3.
Whether automatic eye movements and blinks also display a RP remains to be seen (Shibasaki and Hallett 2006).
References
Aftanas LI, Reva NV, Varlamov AA, Pavlov SV, Makhnev VP (2004) Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: Temporal and topographic characteristics. Neurosci Behav Physiol 34(8):859–867
Allanson J, Mariani J (1999) Mind over virtual matter: Using virtual environments for neurofeedback training. In: IEEE Virtual Reality Conference 1999 (VR’99), pp 270–273
Bayliss JD (2003) Use of the evoked potential P3 component for control in a virtual apartment. IEEE Trans Neural Syst Rehabil Eng 11(1):113–116
Bayliss JD, Ballard DH (2000) A virtual reality testbed for brain-computer interface research. IEEE Trans Rehabil Eng 8(2):188–190
Bayliss JD, Inverso SA, Tentler A (2004) Changing the P300 brain computer interface. CyberPsychol Behav 7(6):694–704
Benovoy M, Cooperstock JR, Deitcher J (2008) Biosignals analysis and its application in a performance setting. In: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, pp 253–258
Bianchi-Berthouze N, Kim W, Patel D (2007) Does body movement engage you more in digital game play? And why? In: Affective Computing and Intelligent Interactions. Lecture Notes in Computer Science, vol 4738. Springer, Berlin, pp 102–113
Blankertz B, Kawanabe M, Tomioka R, Hohlefeld FU, Nikullin V, Müller KR (2007) Invariant common spatial patterns: Alleviating nonstationarities in brain-computer interfacing. Neural Inf Process Syst (NIPS) 20:113–120
Blankertz B, Tomioka R, Lemm S, Kawanabe M, Müller KR (2008) Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Mag 25(1):41–56
Blizzard Entertainment®, Inc (2008) World of Warcraft® subscriber base reaches 11.5 million worldwide. http://www.blizzard.com/us/press/081121.html
Bussink D (2008) Towards the first HMI BCI game. Master’s thesis, University of Twente
Cantero J, Atienza M, Gómez C, Salas R (1999) Spectral structure and brain mapping of human alpha activities in different arousal states. Neuropsychobiology 39(2):110–116
Chanel G, Kronegg J, Grandjean D, Pun T (2006) Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals. In: Multimedia Content Representation, Classification and Security. Lecture Notes in Computer Science, vol 4105. Springer, Berlin, pp 530–537
Chanel G, Rebetez C, Bétrancourt M, Pun T (2008) Boredom, engagement and anxiety as indicators for adaptation to difficulty in games. In: MindTrek ’08: Proceedings of the 12th International Conference on Entertainment and Media in the Ubiquitous Era. ACM, New York, NY, USA, pp 13–17
Chanel G, Kierkels JJ, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Hum Comput Stud 67(8):607–627
Cho BH, Lee JM, Ku JH, Jang DP, Kim JS, Kim IY, Lee JH, Kim SI (2002) Attention enhancement system using virtual reality and EEG biofeedback. In: IEEE Virtual Reality Conference 2002 (VR 2002), p 156
Cover TM, Thomas JA (2006) Elements of Information Theory, 2nd edn. Wiley, New York
Csikszentmihalyi M (1990) Flow: The Psychology of Optimal Experience. Harper and Row, New York
Davidson RJ (1992) Anterior cerebral asymmetry and the nature of emotion. Brain Cogn 20(1):125–151
Doppelmayr M, Klimesch W, Stadler W, Pöllhuber D, Heine C (2002) EEG alpha power and intelligence. Intelligence 30(3):289–302
Fairclough SH (2009) Fundamentals of physiological computing. Interact Comput 21(1–2):133–145
Ferrez P, Millán JdR (2008) Error-related EEG potentials generated during simulated brain-computer interaction. IEEE Trans Biomed Eng 55(3):923–929
Finke A, Lenhardt A, Ritter H (2009) The MindGame: A P300-based brain-computer interface game. Neural Netw 9(22):1329–1333
Galán F, Ferrez P, Oliva F, Guardia J, Millán JdR (2007) Feature extraction for multi-class BCI using canonical variates analysis. In: IEEE International Symposium on Intelligent Signal Processing, pp 1–6
Garcia Molina G, Tsoneva T, Nijholt A (2009) Emotional brain-computer interfaces. In: Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction (ACII 2009). IEEE Computer Society Press, Los Alamitos, pp 138–146
Goodwin NC (1987) Functionality and usability. Commun ACM 30(3):229–233. DOI http://doi.acm.org/10.1145/214748.214758
Guger C, Edlinger G, Harkam W, Niedermayer I, Pfurtscheller G (2003) How many people are able to operate an EEG-based brain-computer interface (BCI)? IEEE Trans Neural Syst Rehabil Eng 11(2):145–147
Hettinger LJ, Branco P, Encarnacao LM, Bonato P (2003) Neuroadaptive technologies: Applying neuroergonomics to the design of advanced interfaces. Theoretical Issues in Ergonomics Science, pp 220–237
Hiltz SR, Johnson K (1990) User satisfaction with computer-mediated communication systems. Manag Sci 36(6):739–764. http://www.jstor.org/stable/2631904
Hjelm SI (2003) Research + design: The making of brainball. Interactions 10(1):26–34
Hjelm SI, Eriksson E, Browall C (2000) Brainball—using brain activity for cool competition. In: Proceedings of the First Nordic Conference on Human-Computer Interaction, p 59
Hwang HJ, Kwon K, Im CH (2009) Neurofeedback-based motor imagery training for brain–computer interface (BCI). J Neurosci Methods 179(1):150–156
IJsselsteijn W, de Kort Y, Poels K (2008) The game experience questionnaire: Development of a self-report measure to assess the psychological impact of digital games. Manuscript submitted for publication
International Organization for Standardization (1991) ISO 9126—Information technology—Software product evaluation—Quality characteristics and guidelines for their use
Ives B, Olson MH, Baroudi JJ (1983) The measurement of user information satisfaction. Commun ACM 26(10):785–793. DOI http://doi.acm.org/10.1145/358413.358430
Jackson MM, Mappus R, Barba E, Hussein S, Venkatesh G, Shastry C, Israeli A (2009) Continous control paradigms for direct brain interfaces. In: Human-Computer Interaction. Novel Interaction Methods and Techniques. Springer, Berlin, pp 588–595
Kaul P (2006) Neurological gaming environments. In: SIGGRAPH ’06: ACM SIGGRAPH 2006 Educators Program. ACM, New York, NY, USA, p 25
Kayagil TA, Bai O, Lin P, Furlani S, Vorbach S, Hallett M (2007) Binary EEG control for two-dimensional cursor movement: An online approach. IEEE/ICME International Conference on Complex Medical Engineering, pp 1542–1545
Keil A, Müller MM, Gruber T, Wienbruch C, Stolarova M, Elbert T (2001) Effects of emotional arousal in the cerebral hemispheres: A study of oscillatory brain activity and event-related potentials. Clin Neurophysiol 112(11):2057–2068
Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067–2083
Krepki R, Blankertz B, Curio G, Müller KR (2007) The Berlin brain-computer interface (BBCI)—towards a new communication channel for online control in gaming applications. Multimed Tools Appl 33(1):73–90
Lalor EC, Kelly SP, Finucane C, Burke R, Reilly RB, McDarby G (2004) Brain computer interface based on the steady-state VEP for immersive gaming control. Biomed Tech 49(1):63–64
Lalor EC, Kelly SP, Finucane C, Burke R, Smith R, Reilly RB, McDarby G (2005) Steady-state VEP-based brain-computer interface control in an immersive 3D gaming environment. EURASIP J Appl Signal Process 19:3156–3164
Lécuyer A, Lotte F, Reilly RB, Leeb R, Hirose M, Slater M (2008) Brain-computer interfaces, virtual reality, and videogames. IEEE Comput 41(10):66–72
Lee U, Han SH, Kim HS, Kim YB, Jung HG, Lee HJ, Lang Y, Kim D, Jin M, Song J, Song S, Song CG, Shin HC (2006) Development of a neuron based internet game driven by a brain-computer interface system. In: Proceedings of the International Conference on Hybrid Information Technology, pp 600–604
Leeb R, Pfurtscheller G (2004) Walking through a virtual city by thought. In: Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2004. IEMBS ’04, vol 2, pp 4503–4506
Leeb R, Scherer R, Lee F, Bischof H, Pfurtscheller G (2004) Navigation in virtual environments through motor imagery. In: Proceedings of the 9th Computer Vision Winter Workshop, CVWW, vol 4, pp 99–108
Leeb R, Keinrath C, Friedman D, Guger C, Neuper C, Garau M, Antley A, Steed A, Slater M, Pfurtscheller G (2005) Walking from thoughts: Not the muscles are crucial but the brain waves! In: Proceedings of the 8th Annual International Workshop on Presence, pp 25–32
Lehtonen J, Jylanki P, Kauhanen L, Sams M (2008) Online classification of single EEG trials during finger movements. IEEE Trans Biomed Eng 55(2 Part 1):713–720
Lin TA, John LR (2006) Quantifying mental relaxation with EEG for use in computer games. In: Proceedings of the International Conference on Internet Computing, pp 409–415
Lin YP, Wang CH, Wu TL, Jeng SK, Chen JH (2009) EEG-based emotion recognition in music listening: A comparison of schemes for multiclass support vector machine. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing 2009, pp 489–492
Liu C, Agrawal P, Sarkar N, Chen S (2009) Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. Int J Hum Comput Interact 25(6):506–529
Lotte F, Renard Y, Lécuyer A (2008) Self-paced brain-computer interaction with virtual worlds: A quantitative and qualitative study “out of the lab”. In: Proceedings of the 4th International Brain-Computer Interface Workshop and Training Course, pp 373–378
Lucas HC (1975) Why Information Systems Fail. Columbia University Press, New York
Mandryk RL, Inkpen KM, Calvert TW (2006) Using psychophysiological techniques to measure user experience with entertainment technologies. Behav Inf Technol 25(2):141–158
Marosi E, Bazán O, Yañez G, Bernal J, Fernández T, Rodríguez M, Silva J, Reyes A (2002) Narrow-band spectral measurements of EEG during emotional tasks. Int J Neurosci 112(7):871–891
Martinez P, Bakardjian H, Cichocki A (2007) Fully online multicommand brain-computer interface with visual neurofeedback using SSVEP paradigm. Comput Intell Neurosci 2007(1):13
Mason SG, Bohringer R, Borisoff JF, Birch GE (2004) Real-time control of a video game with a direct brain-computer interface. J Clin Neurophysiol 21(6):404
McFarland D, Miner L, Vaughan T, Wolpaw J (2000) Mu and beta rhythm topographies during motor imagery and actual movements. Brain Topogr 12(3):177–186
Microsoft® (2009) Project natal. Internet http://www.xbox.com/en-US/live/projectnatal/
Middendorf M, McMillan G, Calhoun G, Jones KS (2000) Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Trans Rehabil Eng 8(2):211–214
Mingyu L, Jue W, Nan Y, Qin Y (2005) Development of EEG biofeedback system based on virtual reality environment. In: Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society. IEEE-EMBS 2005, pp 5362–5364
Morris JD (1995) SAM: The Self-Assessment Manikin. An efficient cross-cultural measurement of emotional response (observations). J Advert Res 35(6):63–68
Mühl C, Heylen D (2009) Cross-modal elicitation of affective experience. In: Proceedings of the Workshop on Affective Brain-Computer Interfaces, pp 42–53
Mühl C, Gürkök H, Plass-Oude Bos D, Scherffig L, Thurlings ME, Duvinage M, Elbakyan AA, Kang SW, Poel M, Heylen D (2010) Bacteria Hunt: A multimodal, multiparadigm BCI game. In: Proceedings of the 5th International Summer Workshop on Multimodal Interfaces eNTERFACE’09, to appear
Müller MM, Keil A, Gruber T, Elbert T (1999) Processing of affective pictures modulates right-hemispheric gamma band EEG activity. J Clin Neurophysiol 110(11):1913–1920
Müller-Tomfelde C (2007) Dwell-based pointing in applications of human computer interaction. In: Proceedings of the 11th IFIP TC13 International Conference on Human-Computer Interaction (INTERACT 2007), vol 4662. Springer, Berlin, pp 560–573
Nacke L, Lindley CA (2008) Flow and immersion in first-person shooters: Measuring the player’s gameplay experience. In: Proceedings of the 2008 Conference on Future Play. Future Play ’08. ACM, New York, NY, USA, pp 81–88
Nelson WT, Hettinger LJ, Cunningham JA, Roe MM, Haas MW, Dennis LB (1997) Navigating through virtual flight environments using brain-body-actuated control. In: Proceedings of the IEEE 1997 Virtual Reality Annual International Symposium, pp 30–37
Nielsen J (1993) Usability Engineering. Morgan Kaufmann Publishers, San Mateo
Nielsen J (1996) Usability metrics: Tracking interface improvements. IEEE Softw 13(6):12–13
Nijholt A, Tan D (2007) Playing with your brain: Brain-computer interfaces and games. In: Proceedings of the International Conference on Advances in Computer Entertainment Technology. ACM, New York, NY, USA, pp 305–306
Nijholt A, van Erp JBF, Heylen DKJ (2008a) Braingain: BCI for HCI and games. In: Proceedings of the AISB Symposium Brain Computer Interfaces and Human Computer Interaction: A Convergence of Ideas, The Society for the Study of Artificial Intelligence and Simulation of Behaviour, pp 32–35
Nijholt A, Tan D, Pfurtscheller G, Brunner C, Millán JdR, Allison B, Graimann B, Popescu F, Blankertz B, Müller KR (2008b) Brain-computer interfacing for intelligent systems. IEEE Intell Syst, pp 76–83
Nijholt A, Oude Bos D, Reuderink B (2009) Turning shortcomings into challenges: Brain-computer interfaces for games. Entertain Comput 1(2):85–94
Obaid M, Han C, Billinghurst M (2008) “Feed the fish”: An affect-aware game. In: IE ’08: Proceedings of the 5th Australasian Conference on Interactive Entertainment. ACM, New York, NY, USA, pp 1–6
Oude Bos D, Reuderink B (2008) BrainBasher: A BCI game. In: Extended Abstracts of the International Conference on Fun and Games 2008, Eindhoven, Netherlands. Eindhoven University of Technology, Eindhoven, The Netherlands, pp 36–39
Palke A (2004) Brainathlon: Enhancing brainwave control through brain-controlled game play. Master’s thesis, Mills College
Picard RW (1997) Affective Computing. The MIT Press, Cambridge, MA, USA
Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191
Pineda JA, Silverman DS, Vankov A, Hestenes J (2003) Learning to control brain rhythms: Making a brain-computer interface possible. IEEE Trans Neural Syst Rehabil Eng 11(2):181–184
Pope AT, Palsson OS (2001) Helping video games “rewire our minds”. Tech. rep., NASA Langley Research Center
Reuderink B, Nijholt A, Poel M (2009) Affective Pacman: A frustrating game for brain-computer interface experiments. In: 3rd International Conference on Intelligent Technologies for Interactive Entertainment. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, vol 9. Springer, Berlin, pp 221–227
Rushinek A, Rushinek SF (1986) What makes users happy? Commun ACM 29(7):594–598
Saari T, Turpeinen M, Kuikkaniemi K, Kosunen I, Ravaja N (2009) Emotionally adapted games—An example of a first person shooter. In: Human-Computer Interaction. Interacting in Various Application Domains. Lecture Notes in Computer Science, vol 5613. Springer, Berlin, pp 406–415
Sander D, Grandjean D, Scherer KR (2005) A systems approach to appraisal mechanisms in emotion. Neural Netw 18(4):317–352
Scherer R, Schlögl A, Lee F, Bischof H, Janša J, Pfurtscheller G (2007) The self-paced Graz brain-computer interface: Methods and applications. Comput Intell Neurosci 2007:9
Shibasaki H, Hallett M (2006) What is the bereitschaftspotential? Clin Neurophysiol 117(11):2341 –2356
Shim BS, Lee SW, Shin JH (2007) Implementation of a 3-dimensional game for developing balanced brainwave. In: Proceedings of the 5th ACIS International Conference on Software Engineering Research, Management & Applications. IEEE Computer Society, Los Alamitos, CA, USA, pp 751–758
Sobell N, Trivich M (1989) Brainwave drawing game. In: A Delicate Balance: Technics, Culture and Consequences. IEEE Los Angeles Council, Torrance, CA, USA, pp 360–362
Solovey ET, Girouard A, Chauncey K, Hirshfield LM, Sassaroli A, Zheng F, Fantini S, Jacob RJ (2009) Using fNIRS brain sensing in realistic HCI settings: Experiments and guidelines. In: Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, pp 157–166
Tangermann MW, Krauledat M, Grzeska K, Sagebaum M, Blankertz B, Vidaurre C, Müller KR (2009) Playing pinball with non-invasive BCI. In: Advances in Neural Information Processing Systems 21. MIT Press, Cambridge, MA, USA, pp 1641–1648
Tijs T, Brokken D, Ijsselsteijn W (2009) Creating an emotionally adaptive game. In: Proceedings of the 7th International Conference on Entertainment Computing. Lecture Notes in Computer Science, vol 5309. Springer, Berlin, pp 122–133
Tonet O, Marinelli M, Citi L, Rossini PM, Rossini L, Megali G, Dario P (2008) Defining brain-machine interface applications by matching interface performance with device requirements. J Neurosci Methods 167(1):91–104
Tschuor L (2002) Computer game control through relaxation-induced EEG changes. Student project report
Tyson P (1987) Task-related stress and EEG alpha biofeedback. Appl Psychophysiol Biofeedback 12(2):105–119
van den Broek E, Janssen JH, Westerink J, Healey JA (2009) Prerequisites for affective signal processing (ASP). In: Proceedings of the International Conference on Bio-inspired Systems and Signal Processing, pp 426–433
van de Laar B, Bos DO, Reuderink B, Heylen D (2009) Actual and imagined movement in BCI gaming. In: Proceedings of the International Conference on Artificial Intelligence and Simulation of Behaviour, pp 9–16
van Reekum CM, Johnstone T, Banse R, Etter A, Wehrle T, Scherer KR (2004) Psychophysiological responses to appraisal dimensions in a computer game. Cogn Emot 18(5):663–688
Vidal JJ (1977) Real-time detection of brain events in EEG. Proc IEEE 65(5):633–641
Vidaurre C, Schlögl A, Cabeza R, Scherer R, Pfurtscheller G (2006) A fully on-line adaptive BCI. IEEE Trans Biomed Eng 53(6):1214–1219
Wang C, Zhang H, Phua KS, Dat TH, Guan C (2007) Introduction to NeuroComm: A platform for developing real-time EEG-based brain-computer interface applications. In: Proceedings of the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 4703–4706
Witmer B, Singer M (1998) Measuring presence in virtual environments: A presence questionnaire. Presence 7(3):225–240
Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain-computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791
Zander TO, Kothe C, Welke S, Roetting M (2009) Utilizing secondary input from passive brain-computer interfaces for enhancing human-machine interaction. In: Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience. Lecture Notes in Computer Science, vol 5638. Springer, Berlin, pp 759–771
Zeng Z, Pantic M, Roisman GI, Huang TS (2007) A survey of affect recognition methods: Audio, visual and spontaneous expressions. In: Proceedings of the 9th International Conference on Multimodal Interfaces. ACM, New York, NY, USA, pp 126–133
Zhao Q, Zhang L, Cichocki A (2009) EEG-based asynchronous BCI control of a car in 3D virtual reality environments. Chin Sci Bull 54(1):78–87
Acknowledgements
This work has been supported by funding from the Dutch National SmartMix project BrainGain on BCI (Ministry of Economic Affairs) and the GATE project, funded by the Netherlands Organization for Scientific Research (NWO) and the Netherlands ICT Research and Innovation Authority (ICT Regie).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag London Limited
About this chapter
Cite this chapter
Plass-Oude Bos, D. et al. (2010). Brain-Computer Interfacing and Games. In: Tan, D., Nijholt, A. (eds) Brain-Computer Interfaces. Human-Computer Interaction Series. Springer, London. https://doi.org/10.1007/978-1-84996-272-8_10
Download citation
DOI: https://doi.org/10.1007/978-1-84996-272-8_10
Publisher Name: Springer, London
Print ISBN: 978-1-84996-271-1
Online ISBN: 978-1-84996-272-8
eBook Packages: Computer ScienceComputer Science (R0)