Abstract
In this paper, an investigation and its results towards brain activity pattern recognition while playing computer games using a non-invasive brain-computer interface (BCI) device is presented. The main aim of the study was to analyse data recorded while participants were engaged in playing popular computer games. The major contribution of the analysis presented is the confirmation of the hypothesis that there is a connection between activities in the brain and the different categories of computer games. Three different popular computer games were used, and the recordings took place under the conditions imposed by two different environments, a noisy one (a typical open-access university computer lab) and a quiet one (a typical controlled-access university computer lab under controlled environmental parameters). Initial results, obtained after analysing the raw electro-encephalography (EEG) recorded data, suggest that there might be a significant connection between the type of activity taking place in the human brain and the type of computer game a player is engaging with.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Brain-computer interface (BCI) technologies constitute complex advanced communications and control methods [1]. Even though studied for decades, it is only for the past years that BCI technology has been more extensively used and its capabilities more closely investigated. Inevitably, this led to the opening of a major research area in the industry as well, rather than just the medical sector. One such aspect of the industry is now emerging to be the computer games industry. Although the number of research groups currently focusing on ways to integrate BCI with computer games is increasing, research in the field still remains largely application-driven. In this field, main interests are based in recording EEG data, that can be later analysed in an attempt to understand in more details the user’s state [2].
BCI-based research nowadays involves more than 100 groups all over the world engaged in a broad spectrum of topics, with more entering the field almost every month [3]. Recent research indicates the fact that BCI has already moved from assistive care to such applications as computer games. The significant improvement in usability, hardware, digital signal processing centred techniques, and system integration is predicted to yield applications in other non-medical areas as well [4].
Results of this trend are already to be recognised in the gaming and entertainment industries, where specific products offer cheap and viable solutions for the general public interested in interacting with this new technology [5–8]. It is a well-known fact that the computer games industry has put a moderate amount of research effort in the field of BCI applications; however, further theoretical studies in the area will offer possibilities to better understand brain activity during gaming sessions leading to more effective (stimulating game-play) devices. Traditionally, games are separated into genres that reflect not exclusively the aspect of the game, but rather, the overall game-play. This can be traduced from a brain activity point of view as a separate mental task for different genres of games. Theoretical analysis of this kind recently made scientific headlines [9] and caused a considerable stir within the computer science community.
The aim of this research is to investigate brain activity during engagement with different computer game genres to understand behavioural patterns. Methodology, analysis and results obtained from processing recorded brain activity data from a number of different users, gathered during play-time are presented. At a later stage thorough comparisons between results obtained were performed. The driving force behind our methodology constituted the assumption that since every computer game genre demands from the user to perform different interaction tasks, can be initially considered to be possible for the brain to respond to these processes in different ways, without loss of generality.
The complete underlining hypothesis emphasising the research direction followed can be summarised as if indeed brain activity is different between different computer game genres, must be at the same time similar between different users engaged with the same type of computer game. As a result, the bottom line was to analyse the recorded data with the aim of identifying, if proved possible, brain activity patterns to confirm or dismiss this underlining assumption.
The rest of the paper is structured as follows. Section 2 presents the necessary background information needed for what comes next in the paper to become thoroughly understood. Section 3 provides detailed information about how the different elements of the experiment were setup. The BCI equipment used, the games setup, the data acquisition, and the testing environments and conditions are explained in detail. In Sect. 4, the brain activity recording methodology is discussed, as well as challenges encountered during recording sessions and how these challenges were met. The filtering approach and data analysis methodology steps are presented in Sect. 5. The processing algorithm, developed specifically for the purpose of analysing the recorded signals, is presented and its functionalities and capabilities explained. Following, our analysis results are presented and their meaning is thoroughly explained. Sections 6, 7 contain the results, ANOVA analysis of data, discussion and conclusions respectively, as well as future directions of our research.
2 Background
Devices currently available in the market incorporating BCI technology capabilities can be categorised in two main categories: (a) Assistive Devices (ADs), and (b) Entertainment and research devices (ERDs). The main purpose of devices classified as ADs is to assist users with various disabilities in completing otherwise difficult or even impossible tasks. An example is “IntendiX”, developed by “g.tec” which allows users to spell by using their brain [10]. Devices classified as ERDs are mainly intended for usage in the entertainment industry (such as in gaming applications) and their main purpose is to assist in expanding research boundaries in various areas. As a result they are not aiming in performing one singular task, as recent review papers regarding BCI systems [11, 12] report. For the benefit of the reader, a brief classification of other BCI areas based on different criteria can be found in [13].
Concepts like electro-encephalography (EEG) patterns, user identification and system adaptation without training remain an issue for many years now. In terms of computer games, an EEG pattern recognition system for serious games has been designed with the purpose of comparing recognition rates for experimental serious games without traditional controllers [14].
A user study in self-paced BCIs with virtual worlds showed that, without training, roughly half of the participants exposed to it were able to control the application by using real foot movements and a quarter of them were able to control it by using imagined foot movements [15].
In a relatively early experiment involving a website based game linked to a BCI system, real-time brain activities from the prefrontal cortex of a rat successfully translated into external device control commands and used to drive the game [16]. Another BCI based 3D game measured the user’s attention level in order to control the movement of a virtual hand, using 3D animation techniques. Was developed for training those suffering from Attention Deficit Hyperactivity Disorder (ADHD) [17]. Researchers are now focusing on the design and implementation of tennis computer games’ avatars requiring the user to supplement only brain activity signals as means of action control commands [18]. This implementation will assist people with movement disabilities in controlling a realistic tennis computer game, otherwise an almost impossible task. For this to be achieved in the most efficient way, studies focusing on the practicality of using the mu \((\upmu )\) brain activity rhythm have been conducted [19].
“Affective Pacman” is a computer game developed to investigate the influence of loss-of-control in the performance of Brain-Computer Interfaces (the frustration level of users while playing the game) [20]. The game’s controls consist of two buttons which rotate “Pacman”. In another study, a Steady-State Visual Evoked Potential (SSVEP) based BCI was used to control an avatar in the computer game “World of Warcraft” [21]. To control the avatar the user had in reality to control four icons. Three of them were used to command the avatar to turn left, right and forward, while the fourth was used to instruct the avatar to perform certain general purpose actions, such as grasping objects and/or attacking other avatars.
Recently, a player satisfaction model based on insights from neurobiological findings as well as the results from earlier demographic game design models was proposed [22]. The model presents seven different archetypes of players and explains how each of these player archetypes relates to older player typologies and how each archetype characterises a specific playing style. Authors conducted a survey among more than 50,000 players using the model as a personality type motivator to gather and compare demographic data to the different “BrainHex” archetypes. In another pilot study, the dynamic EEG patterns associated with long term video game play in healthy human participants were examined based on the theta \((\uptheta )\) rhythm distribution over the scalp [23]. The dynamic brain activity during continuous video game play using the high resolution EEG was also investigated. Participants played a competitive video game, “Mario Power Tennis”, on a Nintendo Game cube while their EEG signals were recorded at evenly distributed time segments [24].
Concluding, despite all the efforts, at the present time BCI systems are slower and less accurate than traditional input interfaces currently available. In addition, BCIs often require training for achieving any level of interaction between the end-user and the BCI-based computer game, something that weakens the overall user experience. Overall, BCIs can provide the end-user with experiences that no other traditional computer game controller can provide. Connecting a user directly with a virtual world has the advantage of offering a more natural way of control and communication. Results indicate that both BCI technologies currently available possess the potential of being used as alternative game interfaces [25]. Although BCI technology cannot by any standards considered to be ready yet, players find this novel way of interaction very exciting and engaging.
3 Experimental setup
This section describes how the different elements of the experiment were setup including: (a) the BCI equipment, (b) the games setup, (c) the data acquisition, and (d) the testing environments and conditions.
3.1 BCI equipment
The vast majority of brain activity monitoring and recording devices developed for the non-medical sector are based on EEG [4], which actually is nothing more than the monitoring and recording of the electrical activity throughout the scalp of the user. Although such devices are available to the general public and fairly easy to use, other incorporating more complex techniques are also currently available in the market. In this study the “g.MOBIlab+” device was used, capable of capturing data from 8 different channels (sensors) placed on the user’s scalp using the well-known 10–20 arrangement system. The real advantage of the “g.MOBIlab+” device comes from the fact that its 8 channels can be customised in accordance to the size of the end-user’s scalp, as well as each application’s specialised parameters.
For comparison reasons, a number of other similar devices in existence, starting by the “Emotiv” headset, developed by Epoch, were examined. The device is a 14-channels device but in [26] is stated that before each trial, participants have to go through a new profile creation procedure using the “Emotiv” control panel, a procedure that takes approximately 30–60 mins. Other such devices like: “MindSet” and “Mind-Wave” are even more limited in capabilities in comparison to “g.MOBIlab+” (mainly, they do not allow for the usage of extra sensors alongside the ones already attached to the headset). Enobio (with 8, 20 or 32-channels) is an alternative device which allows for increased spatial resolution and best-in-class signal-to-noise ratio in wireless systems [27].
Due to its characteristics, “g.MOBIlab+” can be used to record raw data in a variety of environments, making it that way the suitable tool of choice for conducting experiments that involve brain activity measurements and signal recording, either in a noisy or a quiet environment. However, one drawback of this type of a system is the amount of time it takes to setup the sensors cap before actually proceeding with the signals recording activities, but overall this is well compensated by the better signal quality achieved [28].
3.2 Games setup
For the purposes of our experiments and study three completely different popular computer games were considered: (a) the “Minesweeper” game, (b) the “Quake3 Arena” game, and (c) the “Trackmania” game (Figs. 1, 2, 3). The goal was to capitalise on the inherently different environmental parameters. The three games selected in such a way as to represent a different computer game genre each. The fact that they target different audiences played a significant role during the selection phase as well. For example, “Minesweeper” is engaging a very wide range of players, while on the other hand, “Trackmania” target’s a smaller range of players and “Quake3 Arena” an even smaller one. Also, there is a big variation to be observed on visual stimuli. The highest occurs with “Quake3 Arena”, while the lowest with “Minesweeper”. Exactly the same pattern is applicable regarding interaction with the games. Finally, in terms of concentration, “Minesweeper” has higher cognitive workload but it is not clear how much higher or what is the load on the other two games. Additional to the mental tasks required from the player, are the environmental demands and the environmental parameters which are unique for each game. This difference between environmental demands and parameters directly translates into different visual stimuli received by the end-user’s brains from game to game.
“Minesweeper” is considered to be a Puzzle Type (PT) computer game, “Quake3 Arena” belongs to the First-Person Shooters (FPS) category, while “Trackmania” belongs to the Arcade Racing (AR) category. In “Minesweeper” the motivation is to solve a puzzle by using a combination of educated guesses and logical steps, “Quake3 Arena” targets in keeping the player concentrated by aiming and dodging gun fire, while in “Trackmania” the goal is to achieve each time a better lap time from that of your opponent(s) or to beat a pre-set lap time.
Another factor to be taken into consideration is the amount of effort required from the end-user’s side to achieve a satisfactory level of interaction with the computer game’s environment. As an example illustrative of the fact, for two of the computer games used in the experiments, namely, “Trackmania” and “Minesweeper”, participants had to be aware of the full surrounding environment during game-play. In contrast, effectively interacting with the “Quake3 Arena” environment, required from the participants to be constantly and fully aware of only the exact location of the AI-controlled bots. These were the major differences and challenges imposed by the three computer games used for the purposes of brain activity data gathering and analysis, the results of those two procedures are presented in this paper.
3.2.1 Data acquisition
“g.MOBIlab+” is capable of capturing raw EEG signals from 8 different channels/sensors (namely, channel/sensors: O1, O2, T7, P3, Cz, P4, T8, Pz) placed on the participant’s scalp using the well-known and widely used 10–20 system of electrode placement (Fig. 4). Is also equipped with low-noise bio-signal amplifiers and a 16-bit A/D converter (256 Hz), which guarantees excellent data quality and a high signal-to-noise ratio.
The first step in every experimental process of this nature is setting up the “BCI2000” computer software package required to retrieve the actual data from the headset. “BCI2000” is a general-purpose computer software package specifically designed and implemented for BCI research, which was used for recording brain activity data, detecting stimulus presence, as well as brain monitoring purposes. The generally stated goal of the “BCI2000” software package project was to assist in the area of research and the development of applications, with BCI extensions. That goal matched exactly the needs and purposes of this research. Another advantage of using this software package is the fact that is freely available for non-profitable research and educational purposes. Recently has been reported that over 600 laboratories are currently using it for similar to ours research and educational activities, spread out in all over the globe [29].
The recording software package uses successively a high pass filter with a cut-off frequency at 1 Hz, a notch filter with a cut-off frequency at 50 Hz (to reject “mains hum” from power lines) and all data were digitised in continuous recording mode at a Sampling Rate of 256 Hz. The anti-aliasing filtering operation insures that all frequencies which are too high to be digitised by the ADC are rejected. Each recorded epoch expands to a total duration of 66.684 Seconds, with 17,464 frames per epoch. The goal of the overall acquisition setup here is to measure the signals with as little noise as possible and without significant interactions due to measurement.
After the parameters were appropriately set into the application, the recording cap was placed on each participant’s head and the sensors were aligned in accordance to the 10–20 International System as shown in Fig. 5. The overall data recording procedure involved four major steps: (a) sub-procedures followed as necessary to achieve a quality recorded brain activity signal, (b) a total number of 5 complete samples of data for “Minesweeper”, (c) a total number of 5 complete samples of data for “Quake3”, and (d) a total number of 5 complete samples of data for “Trackmania”.
The order in which the games were tested was randomly selected. The main reason behind randomising the recording order was to avoid brain activity reflecting the ever increasing amount of time spent in front of a computer screen to contaminate brain activity data originating from interacting with the computer game itself.
3.3 Testing environments and conditions
The recordings took place in two different environments (a noisy one and a quiet one) as depicted in Fig. 6. There are a number of reasons behind employing two different recording environments, the major ones of which being: (a) to accommodate the participants in a comfortable dedicated computer gaming environment (such as Coventry University’s Games Lab), and (b) for having the ability to effectively observe if similar brain activity patterns occur during game-play even when the participants found themselves under differing environmental parameters; with the second, if turning to be true, constituting further solid ground for validating the final analysis results obtained.
To make the testing procedure as rigorous as possible, all participants engaged with the games under exactly the same conditions (Fig. 7). The testing conditions interrelations between the three computer games were decided on the basis of striking a balance between having as much as possible similar testing conditions between games (difficulty level, time limit etc.) and allowing the unique characteristics of each game to unfold as fluently as possible during game-play.
4 Methodology
This section presents: (a) the experimental design including important issues for capturing and recording EEG data, (b) participants information and, (c) the procedure followed.
4.1 Experimental design
In order for “g.MOBIlab+” to provide an experiment supervisor or the end-user with the ability to account for connectivity issues and corrupted data compensation, comes together with a recording software utility. During sessions, the recording software utility provides real-time raw data observation, correction and adjustment capabilities.
There are various issues in respect to capturing and recording EEG data. The most significant one is noise artefacts that arise through movement of the head during the recording phase. Another one is corrupted data which are coming as a result of a misplaced cap and sensors on the participant’s head (most commonly experienced at the initial stages of the process). To minimise these errors a well verified method for checking the connectivity status between the scalp and the electrodes/sensors was followed [30].
The method involves a simple test that can be performed before starting any data recording session. That way, can detect from a very early stage channels with poor signal capturing performance. After certain adjustments made, further improve the situation by adding extra “g.GAMMA” gel or by readjusting as was though more appropriate the cable connections. The experience gained out of this procedure is suggestive of the fact that the single most important initial step is to ask, as the experiment supervisor, the participant to relax [10].
The next step was to instruct the participant to wink several times and look for variations in the real-time displayed raw EEG waveforms. It is well-known that winking has as immediate result higher waveform amplitudes to be detected and, what’s more, the winking effect becomes almost immediately noticeable, as expected, in waveforms corresponding to channels located closer to the eyes. This process can be taken a step further by asking the participant to bite his/her teeth for a short time. This causes high amplitude artefacts to appear in the raw EEG waveforms [10]. This is marked as one effectively addressing the problem of identifying poor sensor connectivity and poor waveform quality.
Despite all that, it is important to emphasise that although a brain activity waveform may appear to have a comparatively improved quality that alone does not necessarily ensure and its validity. So, it is still possible for all the sensors/channels to respond as expected during the “eye blinking” and “teeth biting” tests but for the waveforms captured from some of the channels to be contaminated with unwanted artefacts. An example of this situation was a waveform corresponding to the “Cz” channel and containing high level noise was not correlating because of that with the rest of the incoming waveforms captured from the other channels. Although at the beginning of the recording session connectivity quality insured by scholastically following the method described above, during the actual recording period the incoming waveforms (raw data) observed to be abruptly corrupted by a high level noise pattern. This was a fairly easy situation to resolve, because the cause for this type of noise is generally recognised to be due to physical factors such as the participant’s discomfort level, causing abrupt movement of the head which results in dislocating some or even all the attached sensors.
In some other cases still, participants may show various types of tics which in general involve rapid movements of the facial muscles as a net effect. A customary practice in all these situations is to request participants to try and limit the disrupting movements to the minimum possible in order for the interaction with the game’s interface phase to commence.
4.2 Participants
Twenty one participants took part in the study performed at Coventry University. Twenty were males with ages spanning between 19 and 26 years old. Ten located in a quiet environment, and eleven located in a noisy environment. At the beginning of each separate game session participants were allowed a few minutes to familiarise themselves with the controls and mechanics of the game about to engage with. All participants had previous experience with computer games and they are considered to be gamers. As a result (and due to the fact that the selected games were popular), nobody expressed any problems in understanding them within the two minutes minimum allowed time.
4.3 Procedure
The methodology followed for data analysis consisted of three major steps: (a) data streaming manipulation, (b) data processing and, (c) feedback delivery. Each of these steps was broken down in a number of appropriate sub-steps for data streaming: (a) channel selection, (b) data filtering and, (c) buffering; a number of appropriate sub-steps for data processing: (a) data pre-processing, (b) feature extraction and, (c) classification; and finally a number of sub-steps for feedback delivery: (a) selection of desired end-user interactions based on classification results and, (b) promotion of end-user interactions based on the same criteria.
The decision was made to use all eight channels provided by the “g.MOBIlab+” device to obtain data from as many as possible active brain locations. Another reason for going along this option was to increase the amount of data used in the data processing phase to 100 % and increase that way at the same time possibilities of achieving very accurate results to the maximum possible. The data collected were then filtered in accordance to the device’s standards by employing the accompanying software package (signal pre-amplification, signal amplification, High Pass Filtering with a cut-off frequency at 1 Hz. Notch Filter with a cut-off frequency at 50 Hz). The anti-aliasing filtering operation and signal digitisation took place as part of the data recording procedure (a build-in pre-processing/processing stage). The data captured were then stored as a collection of row vectors, one corresponding for each recording channel.
As part of the feature extraction and classification stages all the logged data were analysed and fragmented off-line in consecutive epochs of 66.684 s. Then EEG epochs with ophthalmic, muscular and other types of artefacts were preliminarily identified by displaying the channels and manually removing the artefacts by means of visual inspection. The onsets of artefacts were chosen as close to zero crossing as possible. The cut-off points were chosen so that their slopes would match to avoid introduction of artificial changes of direction in the recorded signals. Here it must be noted the necessity of automated artefact removal algorithms for even more accurate results. In that respect, the computerised method described in [31] can be used for further analysis. During the selection and promotion stages of the desired end-user interactions, the EEG epochs strongly contaminated by artefacts that could not be removed with the above mentioned procedure were rejected from the analysis living us finally with three epochs from each user, one for each game per experimental environment.
The selected signals were then grouped into a data set of sixty three logged signals in total, ready for further processing. For the final data processing stage a custom-built processing software was developed based on the \(\hbox {MATLAB}^{\mathrm{TM}}\) programming environment. In parallel with this and for verification purposes “EEGLAB” was used [32].
The purpose behind us building our own processing software to process the logged data was not to emulate functionality and processing capabilities already available in “EEGLAB”, but rather to be capable of exerting absolute control on all functional parameters even on those necessarily lying hidden in “EEGLAB”. Therefore it became possible to easily fine-tune and readjust as was needed specific procedural parameters.
After artefact removal, the data were filtered using a low-pass elliptic filter with an order of 10, pass frequency of 50 Hz, a stop frequency of 60 Hz, and stop band attenuation of 60 dB. The final results were obtained directly from the Time-domain EEG signals. The frequency-domain representation of these signals was obtained after application of a digital FFT-based power spectrum analysis; the Welch technique with a Hamming windowing function and no phase shift. The power density of the EEG rhythms with a 1 Hz frequency resolution, ranging from 2 to 45 Hz was calculated. The final signals were computed by taking the average across each individual channel, per recording environment, per game.
Signal averaging is the technique that allows estimation of small amplitude signals that are buried in noise, it is a technique well justified from past EEG signal analysis applications, and was adopted in this research. It usually assumes the following: (a) signal and noise are uncorrelated, (b) the timing of the signal is known, (c) a consistent signal component exists when performing repeated measurements and, (d) the noise is truly random with zero mean. In real situations, all these assumptions may be violated but the averaging technique has been in general proven sufficiently robust to provide accurate results under minor violation situations of all four basic assumptions.
5 Results
The first observation made is that the set of frequencies lower than 8Hz appear increased in magnitude. Next, Beta waves (i.e. 13–30 Hz) for the signals representing recordings under noisy conditions appear to possess a considerably high magnitude level. Beta waves are generally associated with active attention and concentration. An increased magnitude level can reflect the participants’ attempt to concentrate and focus more on the game’s environment than on the surrounding environment and/or external disturbances.
By isolating Beta waves it can be demonstrated that each game stimulates different magnitude levels, which in general correlate with the noisy and the quiet experimental environments, with “Quake3” to show the highest magnitude levels of Beta waves among the games. The peculiarity with this game is that requires the player to be very context aware in order to successfully avoid “death”.
Players must be aware of any traps in close proximity, enemies, as well as available ammunition in order to progress and achieve the highest score possible. The very determination of performing well can force players to concentrate more on the game; something that directly reflects upon the increasing Beta rhythm activity levels. “Trackmania”, as the second game put to test for the increasing Beta rhythm activity levels, may doesn’t require from the players that much concentration and environmental awareness but, nevertheless, they must prove careful enough not to collide with obstacles while “driving” around the circuit; a more or less equally demanding task. This might result in lower Beta rhythm magnitude levels than “Quake3”, but still higher than those resulting from engaging with “Minesweeper”.
Regarding Alpha rhythm magnitudes, the Alpha rhythm is known as a relaxation indicator. Results suggest that “Quake3” related signals contain higher magnitude activity levels of Alpha waves than when compared to those from the other two games (second graph in every figure). This indicates that although players concentrated more during “Quake3”, found the game to be relaxing at the same time.
Signals related with the O2 sensor (Fig. 9) show trends similar to those identified for the O1 sensor (Fig. 8). The highest magnitude levels for Alpha rhythm are encountered in “Quake3” related signals, followed by “Trackmania” related ones, with those for “Minesweeper” to follow immediately after. This is most indicative of the fact that the relaxation levels are higher for the “Quake3” game environment.
It is worth mentioning, that the Beta rhythm magnitude levels are still higher under the Noisy environment recording conditions, but for “Trackmania” the magnitude levels under the quiet environment recording conditions seem to appear slightly higher. It is difficult to predict why the specific sensor recorded higher Beta rhythm activity magnitude levels under the quiet environmental conditions. One suggestion is that for the participants involved under these environmental conditions the sensor detected higher levels of concentration. One sensor cannot recreate the complete brain’s activity image, however can provide enough for a first conclusion to be reached.
Another important issue to be addressed has to do with similarities arising when a comparison is performed between results coming as an outcome from analysing signals recorded under noisy environmental conditions and those coming as the result of quiet environmental conditions. Although the general magnitude levels are different between them, the signal peeks are following a similar distribution pattern and suggests the existence of general patterns relating to each type of games.
Data recorded from the T7 sensor (Fig. 10) appear to be match different from those recorded from the O1 and O2 sensors. The first thing noticeable is the big difference between the Beta rhythm magnitude levels under noisy and quiet recording conditions. If in the previous case the gap in magnitude between the two types of recording environments was not that great, in this case the gap appears significantly larger. The higher difference can be observed for the “Minesweeper” case (first graph in every figure), where power values extend from approximately 38 units over to 40 units. Although other cases do not project such a high difference, it is still noticeable that the Beta rhythm magnitude levels appear to be higher under the noisy environmental conditions. This indicates that users do not concentrate more on the game environment under noisy conditions, since external disturbances force them at some point to give up trying.
Another interesting result is the lack of high magnitude level values for the Alpha rhythm range of frequencies. Again, the higher magnitude levels appear to occur during “Quake3” recording sessions, followed in magnitude by “Trackmania” (third graph in every figure) and “Minesweeper” recording sessions. A possible reason is the fact that although Alpha rhythm appears at the posterior regions of the head and the sides, the sensor recorded EEG data which translated into Beta rhythm. Of course, another equally possible cause may be the presence of a noise level such that causing the signal to be translated as Beta rhythm activity.
The T8 sensor (Fig. 14) follows the O1 and O2 sensors’ pattern. The analysis shows higher magnitude levels of Beta rhythm under the noisy environmental conditions, but the magnitude gap between recording environments is lower than that appearing in the case of the T7 sensor.
Moreover, the highest magnitude levels for Beta rhythm appear in “Quake3” recording sessions, followed by “Trackmania” and “Minesweeper”. This suggests that “Quake3” requires more concentration from the participant’s side, although the possible cause for this may be attributed to the more complex handling required by the game theme and the level of concentration required to perform well. However, it is important to acknowledge the fact that there is a considerably large error window in every such statement. Alpha levels appear much clearer and higher in magnitude than Beta rhythm levels. Alpha rhythm peeks are located around the 10 Hz mark point in all the signals recorded. “Quake3” appears to contain the highest magnitude levels for the Alpha rhythm band of frequencies, followed by “Trackmania” and “Minesweeper” possessing between them very similar magnitude values. This is the case despite the fact that initial predictions, suggest attributing the higher magnitude levels in the Alpha rhythm to “Minesweeper” (which is the simpler game to successfully engage).
Data recorded from the Cz sensor (Fig. 12) seem to possess different data signatures from those observed in relation to the previously mentioned channels. Alpha rhythm peeks are not as obvious as in some of the previous sensors. Although present in all the signals, it is hard to point out which one of these signals possesses the higher magnitude values. However, higher levels of Alpha rhythm can be observed in signals attributed to “Quake3”, followed by “Trackmania” and “Minesweeper”.
The Beta rhythm magnitude levels tend to follow the previous observations, with higher magnitude values attributed this time to “Quake3”, followed by “Trackmania” and “Minesweeper”. The differences between noisy and quiet recording environmental conditions, although not very obvious, are never the less clearly observable. Beta rhythm magnitude levels seem to reflect the fact that in a noisy environment users need to concentrate more to achieve good results. Attempting a comparison between games, “Quake3” is the game giving the higher Beta rhythm magnitude levels, with them attributed to high concentration required during game-play. Signals recorded from sensor P3 (Fig. 11) appear similar to those recorded from sensor Cz. The Alpha rhythm magnitude values are hard to identify, but they are clearer than in the case of sensor Cz. “Quake3” and “Trackmania” have higher values in the frequencies range of the Alpha rhythm when compared to “Minesweeper”. This seems to be the case for most of the sensors’ signals analysed.
Beta rhythm activity magnitude levels suggest that “Quake3” required more concentration from the participants than the other two games. The differences emerging between the Noisy and the Quiet environmental conditions are again clear in “Quake3” and “Minesweeper” as well. “Trackmania” possesses similar magnitude levels of Beta rhythm in both recording environments. There are a number of reasons for this, with the most important being the recording and/or filtering artefacts persisted during the pre-processing stage. However, the similarity between the Beta rhythm magnitude levels does not necessarily constitute an indication of contaminated data.
Signals recorded from the Pz sensor (Fig. 15) resemble both those from Cz and P3 sensors, as well as those from O1 and O2 sensors. Although not as obvious as in O1 and O2 signals, the Alpha rhythm peeks are much clearer. “Quake3” gaming environment is responsible for the higher magnitude levels of Alpha rhythm waves, followed by “Trackmania” with similar levels under the noisy recording environmental conditions and lower under the quiet recording environmental conditions. The lowest levels can be observed for the “Minesweeper” under both environmental conditions.
The Beta rhythm magnitude levels follow the tendencies encountered in the case of the O1 and O2 sensors. Data recorded from the P4 sensor (Fig. 13) are similar to those recorded from the Pz sensor. The Alpha rhythm magnitude levels are more visible and top magnitude levels can be observed for “Quake3”. “Trackmania” and “Minesweeper” show similar magnitude levels, however the latter show lower magnitude levels under noisy environmental conditions. This alone suggests the behaviour noticed and in the previous signals case, namely, that “Quake3” is indeed the game during which game-play the higher magnitude levels of relaxation occurred. The Beta rhythm magnitude levels are higher under noisy environmental conditions, suggesting that participants needed to focus more on the game-play when faced with an overpopulated gaming environment.
Concluding, the final step was the averaging signals across all channels to offer a better representation of the activity of the brain during game-play for all three games.
As was the case for most of the individual channels situation, an obvious difference in Beta levels was noticed between the quiet and noisy recording environmental conditions.
A reason for this can be that the participants needed to concentrate more on the task at hand. Because of the noisy environmental conditions participants had to try and ignore disturbances from the surroundings in order to perform better during game-play. Another issue about Beta rhythm is the difference in magnitude levels between games. The highest level observed during “Quake3” and as mentioned above a successful player needs to possess quick reflexes capabilities and be capable of keeping up with the fast pace of the game. The game belongs to first person genre, so the player’s gaming environment window is limited by what characters can “see” concentrating more on querying the environment for possible threats and enemies.
Furthermore, the interaction procedure for “Trackmania” requires more active involvement to steer at the right time and be careful not to collide with obstacles. “Minesweeper” is the simplest game among the three when interaction and game-play is considered. Alpha rhythm showed higher magnitude levels for “Quake3”, followed by “Trackmania” and “Minesweeper”, suggesting that participants were more relaxed during engagement with “Trackmania”, with the game showing the lowest relaxation magnitude levels being “Minesweeper”.
Next, we obtained results from ANOVA testing our hypothesis. Setting variable “\(\hbox {S}_{\mathrm{G1}}\)” to represent the “Minesweeper” game, variable “\(\hbox {S}_{\mathrm{G2}}\)” to represent the “Quake” game, and variable “\(\hbox {S}_{\mathrm{G3}}\)” to represent the “Trackmania” game we can formulate the NULL Hypothesis and the ALTERNATIVE Hypothesis for our ANOVA analysis as follows:
From the description of our experimental setup, becomes apparent that our scheme consists of three elements, with raw data obtained from the participants as the Random Variable.
Factors as arranged are typically suggestive of a 3-way ANOVA analysis which takes the form presented in Table 1.
The “Independent” ANOVA variables are then given as:
A | Environment type (Two Levels: Noisy, Quiet) |
B | Type of sensor (Eight Levels: Sensor 1, ..., Sensor 8) |
C | Type of game (Three Levels: Game 1, Game 2, Game 3) |
Three “Independent” ANOVA variables give four ANOVA “Interactions” which are as follows:
AB | Environment Type vs. Type of Sensor |
AC | Environment Type vs. Type of Game |
BC | Type of Sensor vs. Type of Game |
ABC | Environment Type vs. Type of Sensor vs. Type of Game |
The analysis was performed in a multi-way (n-way) Analysis of Variance (ANOVA) for testing the effects of multiple factors on the mean of the input vector. In our case the input vector was consisting of a combination of all the averaged signals as those presented in Figs. 8, 9, 10, 11, 12, 13, 14 and 15, for each of the two environments. The resulted ANOVA table of our analysis is as presented in Table 2.
With X1 representing factor A, X2 representing factor B, and X3 representing factor C (the “Independent” ANOVA variables), p values in the last column of the table (Prob\(>\)F) are suggestive of the validity of our hypothesis. Because the output vector p contains p values for the NULL hypotheses on the N main effects, element p(1) contains the p value for the NULL hypothesis \(\hbox {H}_{\mathrm{0A}}\), that samples at all levels of factor A are drawn from the same population; element p(2) contains the p value for the null hypothesis \(\hbox {H}_{\mathrm{0B}}\), that samples at all levels of factor B are drawn from the same population; and finally, element p(3) contains the p value for the null hypothesis \(\hbox {H}_{\mathrm{0C}}\), that samples at all levels of factor C are drawn from the same population.
The small p-value for \(\hbox {H}_{\mathrm{0A}}\) suggests that at least one A-sample mean is significantly different from the other A-sample means; that is, there is a main effect due to factor A. The same is true for \(\hbox {H}_{\mathrm{0B}}\) and \(\hbox {H}_{\mathrm{0C}}\). For the purposes of this analysis it was chosen a bound for the p value to determine whether a result is statistically significant of 0.05.
6 Discussion
Often, the properties of EEG signals need to be optimised for maximum gain. For example, the sensors need to be placed at optimum positions and all the necessary precautions need to be taken so the recorded waveforms reflect the actual activity taking place in the brains. Also, the external conditions (noise in the environment, quality of the recording device, etc.) play a very significant role.
However difficult to assess all these parameters, taking care of all the necessary conditions can result in accurate enough recordings to be obtained for, after appropriate processing of data, an attempt to be made to draw significant conclusions in respect to how and to what extend engaging with different computer games affects activity within human brains. Thus, a useful insight into brain activity in relation to computer games and how this activity can be used to differentiate brain signals from different computer games can be gained, even to the extent of deciding with a fair amount of accuracy about complex in nature issues like computer games addictiveness or addictive elements in computer games scenarios, game-play, etcetera.
For the multiple games situation considered in the paper, other adjustable parameters include the end-user’s focus level, the recording environment, and the game’s difficulty level. Parameters like these can be chosen based on the particular possible application in mind or the particular element under investigation (for example the games’ difficulty level can be chosen as in targeting particular brain activities).
Traditionally, humans automatically tend to consider external noise as something unwanted which only purpose is the destruction of our precious bunch of data. In recording and analysing brain activity related signals though, noise can be proved to consist a significant factor to be taken into account. If a useful device to perform BCI tasks is to be build, even for experimental only purposes, cannot be thought of as operating in a sterilised environment with no external environmental noise present. Instead of trying to eliminate such kind of a noise from our recorded signals before any attempt to analysis, an alternative route may be to try and understand the effect of the environmental noise to the brain and how is affecting the quality of the outcome of the end task to be performed. Some very interesting conclusions is possible to be drawn especially if the main focus of the investigation is some sort of Plug-and-Play, Brain computer interface related, device.
In summary, evidence strongly suggesting that brain activity follows a different pattern for different categorised computer games was provided. Results suggest a number of influential factors. An environmentally stable, well arranged and managed recording session with all the parameters taken into account can capture all the relevant brain activity on which further analysis can reveal activity patterns and common brain activity characteristics between categories of computer games.
Even though we have actually not covered any of these topics to the minute detail, our point is that even with a simple arrangement, only three computer games involved, and a relatively small group of participants, useful and accurate results can be obtained and accurate conclusions can be drawn as a result if proper conditions and analysis tools applied.
Although the research concentrated on three specific computer games for BCI purposes, there are many other possibilities for future work. For example, best-selling commercial computer games from a number of different games platforms can be considered and investigated under the same methodology for the purpose of identifying brain activity patterns and more. Highly successful commercial games played on different video games consoles can be proved capable of generating different brain activity patterns mainly because of, but not restricted to, the different input devices used by the different games consoles.
7 Conclusions
The aim of this research was to investigate brain activity during engagement with different genres of computer games in an attempt to take a first step into understanding end-user’s behavioural patterns. The initial hypothesis underlining our work was that BCI techniques are capable of differentiating brain signals produced when engaging with different computer games as well as recognising, to a certain degree, different users engaging with the same type of computer game (assessing mental tasks in three different computer game genres). Although it was expected that the computer game genre will be an important factor in defining the activity of the brain, other factors, such as the overall design, the input mechanism and the game mechanics, were expected to contribute significantly to the final results.
Results were obtained from analysing the rhythmic activity of the brain between a frequencies range of 2–45 Hz, focusing on the Alpha and Beta rhythm waves. These waves were of a greater interest than the others because they reflected relaxation levels (Alpha rhythm) and concentration levels (Beta rhythm) which constituted the main focus area of our investigation. These two user states are considered to be the most likely to be influenced by a game-play scenario. Results revealed that the highest Beta rhythm magnitude levels are obtained when engaging with the “Quake3” game. Beta rhythm magnitude levels observed were attributed to the extra concentration required to successfully navigate around the game’s environment, avoiding hazards and trying to survive from enemy attacks. It is fully appreciated that future research in the area has to focus on a one parameter variation situation. Only after multiple studies carried out under this condition will become possible to predict with some good level of accuracy how the brain will react during engagement with different types of games.
Signal analysis confirms the existence of differences in the brain activity during engagement with different categories of games. However, there are still a number of important factors that make impossible any attempt to pinpoint what exactly causes the different activity patterns of the brain to emerge. Results were suggestive of a number of influential factors, such as the interaction procedure, the overall game-play, the surrounding environment, and the presence of opponents. Further studies in the area will almost certainly lead into identifying the type of gaming environment or set of particular actions responsible for triggering different responses in the brain.
References
Tan DS, Nijholt A (2010) Brain–computer interfaces applying our minds to human–computer interaction. Springer, London
Nijholt A, Tan D (2007) Playing with your brain: brain–computer interfaces and games. In: Proceedings of the international conference on advances in computer entertainment technology, ACM Press, New York, pp 305–306
Wolpaw JR (2007) Brain–computer interfaces as new brain output pathways. J Physiol 579:613–619
Van Erp JBF, Lotte F, Tangermann M (2012) Brain–computer interfaces: beyond medical applications. IEEE Comput IEEE Comput Soc 45(4):26–34
NeuroSky Inc. (2013) NeuroSky Brain Wave Sensors For Everybody. http://www.neurosky.com. Accessed 21 May 2013
Mattel Inc. (2013) MindFlex. Accessed 21 May 2013 http://mindflexgames.com. Accessed 21 May 2013
Emotiv (2013) http://www.emotiv.com. Accessed 21 May 2013
Uncle Milton Inc. (2013) http://unclemilton.com. Accessed 21 May 2013
Denworth Lydia (2013) Brain-Changing Games, Scientific American Mind, January/February
g.tec, IntendiX (2013) Personal EEG–based spelling system—advanced user and administrator manual. http://www.gtec.at/Products/Complete-Solutions/intendiX-Specs-Features. Accessed 21 May 2013
Nicolas-Alonso LF, Gomez-Gil J (2012) Brain computer interfaces, a review. Sensors 12(2):1211–1279
Bi L, Fan XA, Liu Y (2013) EEG-based brain-controlled mobile robots: a survey. IEEE Trans Human-Machine Syst IEEE Comput Soc 43(2):161–176
Brunner C, Birbaumer N, Blankertz B, Guger C, Kübler A, Mattia D, Millán José del R, Miralles F, Nijholt A, Opisso E, Ramsey N, Salomon P, Müller-Putz GR (2015) BNCI Horizon 2020: towards a roadmap for the BCI community, brain–computer interfaces, Taylor & Francis, ISSN: 2326–263X (Print), pp 2326–2621 (Online)
Ahn JS, Lee WH (2011) Using EEG pattern analysis for implementation of game interface. In: Proceedings of the 15th international symposium on consumer electronics (ISCE), IEEE Computer Society, Singapore, pp 348–351
Lotte F, Renard Y, Lecuyer A (2008) Self–paced brain–computer interaction with virtual worlds: a quantitative and qualitative study—out of the lab. In: Proceedings of the 4th international brain computer interface workshop and training course
Lee U, Han, SH et al. (2006) Development of a neuron based internet game driven by a brain–computer interface system. In: Proceedings of the 2006 international conference on hybrid information technology, IEEE Computer Society, Vol. 02, pp 600–604
Jiang L, Guan C, Zhang, Wang C, Jiang B (2011) Brain computer interface based 3D game for attention training and rehabilitation. In: Proceedings of the 6th IEEE conference on Industrial Electronics and Applications (ICIEA), IEEE Computer Society, Beijing, pp 124–127
Lopetegui E, Zapirain BG, Mendez A (2011) Tennis computer game with brain control using EEG signals. In: Proceedings of the 16th international conference on computer games (CGAMES), pp 228–234
Mattiocco M, Babiloni F et al. (2006) Neuroelectrical source imaging of mu rhythm control for BCI applications. In: Proceeding of the 28th annual international conference of the IEEE on Engineering in Medicine and Biology Society, IEEE Computer Society, pp 980–983
Reuderink B, Nijholt A, Poel M (2009) Affective Pacman: a frustrating game for brain–computer interface experiments. In: Proceedings of INTETAIN 2009, Springer, Berlin, pp 221–227
Kapeller C, Hintermller C, Guger C (2012) Augmented control of an avatar using an SSVEP based BCI, In: Proceedings of the 3rd augmented human international conference, Article No. 27, ACM Press, Vancouver
Nacke LE, Bateman C, Mandryk RL (2014) BrainHex, A Neurobiological Gamer Typology Survey, Entertainment Computing, Elsevier 55–62
He EJ, Yuan H, Yang L, Sheikholeslami C, He B (2008) EEG Spatio-spectral Mapping During Video Game Play, Proceedings of the 5th International Conference on Information Technology and Application in Biomedicine, in conjunction with The 2nd International Symposium & Summer School on Biomedical and Health Engineering, Shenzhen, China, May 30-31, 346-348
Sheikholeslami C, Yuan H, He EJ, Bai X, Yang L, He B (2007) A High Resolution EEG Study of Dynamic Brain Activity during Video Game Play, Proceedings of the 29th Annual International Conference of the IEEE EMBS, Cité Internationale, Lyon, France, August 23-26, 2489-2491
Liarokapis F, Debattista K, Vourvopoulos A, Ene A, Petridis P (2014) Comparing interaction techniques for serious games through brain-computer interfaces: A user perception evaluation study, Entertainment Computing, Elsevier 5(4):391–399
Vourvopoulos A, Liarokapis F, Petridis P (2012) Brain–Controlled Serious Games For Cultural Heritage, Proc. of the 18th Inl’ Conference on Virtual Systems and Multimedia, Virtual Systems in the Information Society, IEEE Computer Society, Milan, Italy, 291–298
g.tec, g.GAMMAcap EEG Sensor Cap - Instructions For Use, Schiedlberg: Guger Technologies, (2010)
Schalk G et al (2004) BCI2000: a general-purpose brain–computer interface (BCI) system. IEEE Trans Biomed Eng 51(6):1034–1043
g.Tec, IntendiX (2012) Advanced user and administrator manual (Version 2.12.00), p 30, Copyright 2012 g.Tec Medical Engineering GmbH
Moretti DV, Babiloni F, Carducci F, Cincotti F, Remondini E, Rossini PM, Salinari S, Babiloni C (2003) Computerized processing of EEG–EOG–EMG artifacts for multicentirc studies in EEG oscillations and event-related potentials. Int J Psychophysiol 47(3):199–216
SCCN, EEGLAB, Available at: [http://sccn.ucsd.edu/eeglab/], Accessed at: 21/05/2013
Acknowledgments
The authors would like to thank Human-Computer Interaction Lab members for their support and inspiration. A video that describes the system can be found at: youtube.com/watch?v=ncyUL2svi kk&feature=youtu.be.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Bakaoukas, A.G., Coada, F. & Liarokapis, F. Examining brain activity while playing computer games. J Multimodal User Interfaces 10, 13–29 (2016). https://doi.org/10.1007/s12193-015-0205-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-015-0205-4