Keywords

1 Introduction

The ability to engage with an audience on an emotional level is one of the defining traits of many forms of art and media. The Cambridge English Dictionary defines art as “the making of objects, images, music, etc. that are beautiful or that express feelings” [1]. This supports the notion that the author of such works is able to express emotion, be it their own or that of others, through a variety of mechanisms. One might also take the view that the audience members, whilst being able to perceive the emotion of the author, will also have their own response to the stimulus, which may be consistent with the author’s emotional intent or not. Historically, this emotional communication has been a largely one-directional affair: the author creates a piece of work with its emotional intent and disseminates it in some way, the audience experiences the art and has their emotional response to it, and this sequence may repeat itself. However, there is traditionally no recourse for the emotional response of the audience to feedback to the author beyond the writing of reviews, critiques, blogs and other forms of response.

We propose that computer games can be considered a form of contemporary art, much in the same way as video, film, animation and interactive audio-visual installations are now viewed. Certainly, games combine the requisite forms of media cited by the dictionary definition: images and music that are visually pleasing. Games are also recognised for their ability to engage an audience and provoke emotional responses in their audience. Gee [2] advocates that the interpretation of their aesthetics and the story aspects of games qualify them as art. Further, we note that games are critiqued, reviewed, discussed and are points of cultural reference. Smuts [3] shares this perspective, although notably recognises that not every computer game may qualify for the label as being recognised as art. One characteristic that makes games particularly compelling as an art form is the rate of change and development with which the media evolves and its facility to integrate easily with other technological developments, both hardware and software based. Therefore, as a form of interactive art, games are uniquely poised to trailblaze and be the proving ground of new ideas and initiative, which may later be adopted by the wider artistic community in all forms of practice. Melissinos [4] states that “…games are also the only form of media that allows for personalizing the artistic experience while still retaining the authority of the artist,” making the distinction that narrative and content of the work can be tailored to the individual(s) playing. It is on these bases that we consider games as being a platform to develop work that affords interactivity, in terms of audience affect and immersion within the game. Further, we propose that this can lead to develop and enhance a sense of community and connection between human players in a game environment, who are remote to another.

Many years have now passed since Rosalind Picard produced the seminal work that helped to define the field of Affective Computing and its associated challenges, opportunities and potential risks [5]. Key to the domain are the three types of affective technology Picard outlines where computers are able to: recognise; represent and have emotions. Whilst the former of these is of greatest concern in this chapter, the second element of representation of emotion should not be neglected. The field itself has developed, at least on the front of academic and some industrial research, over this time period. However, its mainstream applications in the hands of the audience are still relatively few and far between with exceptions such as the formation of spin-out company AffectivaFootnote 1 from Picard’s MIT Media Lab. Perhaps this is unsurprising, as we have seen the widespread adoption of other paradigm-shifting art and entertainment technologies, such as virtual reality (VR), take a long time to reach the mainstream and gain acceptance from publishers and audiences alike.

A common underpinning element of affective computing is the reliance upon various sensors and mechanisms for the provision of biofeedback; the mechanisms by which aspects of bodily function and response can be identified and manipulated [6]. Whilst not an essential aspect of all affective computing systems, the use of such technologies is an intuitive method that provides indicators of affective state and responses to stimuli, often with the ability to operate in real-time. There are distinct advantages of using such sensors. For example, subjects, in our case games players, do not need to actively focus on providing feedback as they would with self-report mechanisms, rather data can be collected passively and without distracting attention from the medium itself. Additionally, sensors are able to provide objective measurements, whilst asking for self-reports or other information may introduce bias on the part of the subject. Whilst medical grade equipment is desirable in high-resolution research scenarios, at a consumer level there are a range of technologies, some of which can provide viable affective data [7,8,9,10] and be easily integrated or added to gaming devices and apparatus.

We propose that affective gaming is a natural extension of the affective computing field. Such an extension offers game developers, the opportunity to detect, respond to and provoke emotions in players. Computer games, like many forms of art, media and entertainment, offer their players the opportunity to be emotionally stimulated and involved [11, 12]. One of the key features that separates games from other forms of media-based entertainment is its broad non-linearity and its deep and varied levels of interactivity. As such, affective gaming affords the possibility to create gaming experiences that can not only provoke and influence their players’ emotional responses, but that player emotions can also be used as an input to the game itself, allowing it to respond to the player and provide a customised experience. This process is generally referred to as affective feedback [13]. One clear example would be a game that increases its level of difficulty to increase tension in the player, if it were to detect that the current level of difficulty is not sparking the intended response.

The ways in which end users interact with technology is maturing and traditional mechanisms of button pressing, joystick moving and mouse clicking have been ubiquitous and effective. However, these technologies have become a legacy, partly down to reluctance in the market to engage with alternate methods of input, such as motion detection, speech and other modes of interaction. We suggest that, in the coming years, the time will be right to examine both active and passive methods of interaction with games and other digital media. There are multiple possibilities to create experiences that are powerful, intelligent, customisable, adaptive, socially enabling and responsive to their users. These developments represent exciting opportunities to create innovative entertainment and artistic works that enhance and evolve the user experience.

2 Affect, Immersion and Engagement

In Inner Sound: Altered States of Consciousness in Electronic Music and Audio-Visual Media [14], Weinel proposes a conceptual model that considers the ‘affective properties’ of sound and audio-visual media as a central feature. Here affective properties are considered as related to the properties of valence and arousal described in Russell’s [15] circumplex model of affect. In games, sound and music may help to elicit an emotional response from the player. Following Gabriellson and Lindstrom’s [16] meta-study of music and emotion, typical features such as percussive sections of music may be associated with high arousal, while sparser rhythmic sections may relate to low arousal. Major modes may relate to positive valence, while minor modes may suggest negative valence. While these are general correspondences that may not apply in all cases, such typical associations can still be used by games designers to reinforce the affective experience of games. We commonly see this through action sequences that are scored with up-tempo music, or failure or melancholy sections in game narratives that are reinforced with corresponding music. For Weinel, such features help to provide an affective experience that is framed by representational properties, and the overall experience provides a heightened state of multimodal awareness that may encompass forms of experiential knowledge.

Of course, in the gaming context, affective states do not only relate to music. Cajella’s [17] work on ‘player involvement’ breaks down immersion in video games into the categories of kinaesthetic, spatial, shared, narrative, affective and ludic forms of player involvement. While this model takes the affective experience as a distinct component, it is surely a form of involvement that must emerge from all of the others. For instance, kinaesthetic involvement describes engagement through control of a game character, while spatial involvement relates to the virtual environments provided by games. When a player controls an avatar that leaps through the levels of a game, they are engaging kinaesthetically and spatially, but the feeling of excitement that may develop from this also leads to affective involvement. Similarly, for players who enjoy the shared involvement of multiplayer combat games, an added sense of arousal comes from knowing that their own success or failure has a bearing on the emotions of their human competitors. For games that develop complex narratives, as with films, designers may aim to tell a story that has an emotional resonance with the player. Lastly, ludic involvement also generates arousal, as players attempt to evade losing lives or reaching a ‘game over’ state, which inevitably is disappointing. Hence, we can see that in fact, all types of involvement feed into the affective involvement of the player. Sound and music are by no means the only way through which affect can be generated in interactive games, but these are often used to help reinforce and heighten the affective experience of the player.

3 Social Presence and Connection

Weinel’s [14] work on altered states of consciousness also draws some further connections between the affective properties of media and social or shared experiences. This emerges through Rouget’s [18] seminal work on music and trance in various ritual contexts such as those found in shamanic cultures. Rouget draws a distinction between high-energy states such as trance, which could be understood as extreme states of high arousal, and meditative states of contemplation (referred to in the original text as ‘ecstasy’), which conversely may be thought of as low arousal in form. Rouget explores a variety of ritual contexts in which these states may be used in cultures across the globe, and comments on how music can be involved in structuring and producing these states. Yet interesting, high-energy states such as trance are also distinguished as social, involving loud noises, movement and other techniques of sensory overload such as the spraying of liquids; while meditative states are characterised as occurring in darkness, solitude and related to a reduction, narrowing or focusing of the senses.

There has been little work done on considering video games as altered states of consciousness, but clearly this is a popular form of media that perhaps is valued by audiences for its capability to induce a sense of (secular) trance. Is it not possible that as teenagers play popular games such as Fornite [19], the heightened state of excitement they experience, which draws them out of the everyday and into the virtual, is in fact something we might call a state of trance? Various interactive aspects of the media conspire to produce a highly aroused, social experience, in which they are immersed into imaginative virtual worlds of play. In these terms, it seems that there is no shortage of high-arousal action games that aim to produce something we might call a sense of trance. But what then, of more tranquil or meditative states? In fact, there are also various games that specifically aim to induce states of meditation, such as Guided Meditation VR [20]. This title is essentially the VR equivalent of a guided meditation experience, which provides tranquil virtual locations with an instructive audio narrative. More broadly, other single-player games may seek to provide tranquil, solitary experiences. For instance, ABZÛ [21] is a mysterious underwater exploration game that de-emphasises ludic involvement in favour of kinaesthetic, spatial and narrative involvement, as the player navigates mostly peaceful ocean environments with a relaxing soundtrack. Certainly, games such as this could be understood as offering experiences that are characterised as meditative in form.

4 Augmenting Virtual Environments

The field of Augmented Reality (AR) has developed significantly in recent years. From the perspective of a user or consumer, it seeks to combine information from the real world with information not visible in that environment using the standard array of human senses [22]. Potentially, this additional information may be drawn from imperceptible information that exists in that space or by including extra information from another place: real or virtual. Thus far, the majority of AR systems have sought to overlay a user’s everyday environment with additional information. For example, an AR navigation application might overlay information about a car that is being looked at, such as year of manufacture, engine size and so on. Alternately, a real worldview might be overlaid with the user’s schedule for the day. There have been several commercial attempts at developing AR systems of this kind, perhaps most notably the Google GlassFootnote 2 hardware project, as well as an array of smartphone apps developed commercially or as research projects.

However, less common is the ability to apply the same principles to augment a Virtual Reality (VR) space with information from the real world. For example, a virtual tour of a building to be constructed might be presented by replicating the current weather being experienced in the real world by the user. Notable consumer devices that explore these possibilities are Sony’s PlayStation Camera [23] and Microsoft’s Kinect for Xbox One [24]. These platforms have seen video games being developed that insert, or respond to, a live camera feed image of the player in a game environment, giving them the ability to interact by gesture with the game, hence augmenting the VR environment. This builds upon the more established practice of integrating audio, specifically player speech utterances, into game environments, made popular in online multiplayer gaming. It is this latter variation of augmentation that exemplifies some of our concerns in this chapter.

Our motivation particularly stems from the recognition that playing games in virtual or online environments has many differences from their real-world equivalents. For instance, there is a reduced ability to interact with fellow team members and opponents. For example, there is a distinct absence of interaction through facial expressions, gesture, posture and general body language. Even with the presence of speech utterances and emoticons, it is challenging to determine accurate emotional responses and exert forms of emotional intelligence, to determine if other players may be angry, sad, scared, exhilarated and so on. As recognised by Picard [5] in relation to other forms of electronic communication, such as email, this detracts from the entertainment and immersion within the game and also from the ability for players to successfully interact with one another. To address this, we propose that these differences lead to opportunities to improve players’ sense of immersion within a virtual environment and their social experience with other players in that environment. However, the drive for increased immersion and realism in games and virtual worlds leads to causes for concern regarding the health issues associated [25] and thus any experimental approach must be mindful of the ethical issues surrounding such work.

The use of standard biofeedback is fairly commonplace, whereby real-time representations of the underlying signals in question are presented to a person connected to some form of sensor. Perhaps the most familiar manifestation of this would be an electrocardiogram (ECG) machine in a clinical environment, where the person connected to the device would be able to hear and see their heart rate. Similar body sensors allow games to alter their virtual environments based on the user that is playing them. Elaborating, emotion is quantifiable through sensor measurements. A virtual game environment becomes dynamic, if it reacts to emotion. A strong case study for such an application is Dynamic Difficulty Adjustment (DDA) in games.

A game such as those from Electronic Arts’ FIFA [26] series can generate an enormous amount of frustration for players; with a number of players reporting they quit playing the game because they felt it unfairly caused them to lose. In future consoles, controllers could embed heart rate monitoring sensors. The data produced could in turn generate a crude measure of emotion, calculated through computer algorithms that translate raw sensor data to meaningful information. Therefore, the level of emotional distress becomes the value for adjusting the difficulty in the game. This adheres to the theory of flow by Csikszentmihalyi [27]. The channel of flow exists between challenge and achievement. A game that dynamically alters its difficulty based on the player’s emotion, can help keep gameplay within the channel. The Internet of Things (IoT) and serious games provide more detail and background on this topic.

The ability for the affective information about players to be shared with third-parties [28], such as in online multiplayer gaming environments, presents interesting opportunities. Consider the illustrative example of a mock-up interface provided in Fig. 13.1, where the virtual game environment has been augmented with data about the human players controlling the in-game avatars. In this simplistic example, we observe the presentation of raw biosensor data alongside a higher-level summary of affective state. In this exemplar, raw heart-rate data in beats-per-minute is presented numerically in a heart icon on the left of the pop-out window. In the middle is an emoji icon, which might be derived by a range of different sensors or other mechanisms, but with the intention of providing a summary representation of the player’s emotional state. Finally, on the right side of the pop-out a lightning bolt, which fills up with an orange colour, provides an indication of the galvanic skin response (GSR) data values of the player. Although visual paradigms and designs have been adopted for the purposes of communicating these concepts clearly in a written form, there is of course no reason why this information could not be communicated in alternate mode, such as sonification or tactile feedback.

Fig. 13.1
figure 1

Image Copyright © Stuart Cunningham 2019

Multiplayer game mock-up with affective data visualisations.

Full size image

5 The Internet of Things and Games

Research is helping bridge the gap between sensors and games. In 2015, Favorskaya et al. [29] presented their book into smart serious games. The idea behind the term is games that incorporate the benefits of the Internet of Things (IoT) with the applications of serious games. Existing research in the field investigates measuring student engagement through the IoT and serious games [30]. Developing data algorithms forms a key component for the integration of IoT with games. It is possible to exclude them in games where sensors are limited to direct input. An example of such games is Kinect Sports [31], where the Kinect acts as an input device for controlling a player in the virtual environment. However, such games constrict their scope of application, as the data could be interpolated into something grander and more meaningful.

However, this idea can become more abstract, and potentially, more beneficial if we consider the combination of IoT with games technology. Henry et al. [32] produced a framework that illustrates the backbone for merging serious games with IoT, however as the framework did not focus on a case study, it is also valid for the broader spectrum of game technologies. The potential for this line of research is near unlimited.

Figure 13.2 illustrates the untapped potential for this academic domain in the present day. This validates the theories and ideologies presented in this chapter, and produces a stepping-stone to realising them. Research is continuing into understanding the relationship between all aspects of game applications and harnessing their potential.

Fig. 13.2
figure 2

Potential applications for Henry et al. framework on serious games and the Internet of Things

Full size image

Research is emerging into combining IoT with Gamification applications. Crowley et al. [33] proposed a system where users can report issues within their local environment through gamification. Users would be awarded for contributing and could send the information through their smartphones. Research by Islam et al. [34] into advancements of IoT healthcare applications suggested wearables to bridge patient engagement and public health information. In industry, smartwatches have turned this concept into reality. Companies such as Apple utilise sensors to monitor the activity of its user and promote healthier habits by awarding badges and forming competitions with friends that own the same product. This is an example of gamification and IoT in applications. The method for engaging users with their health and maintaining their commitment is gamification, utilising game elements to an application outside the definition of a game.

It is evident that games are much more than just entertainment software, and this remains true for game technology. Research utilises game technology to solve specific problems in domains such as business modelling and smart cities. Maines et al. [35] introduced a third dimension for Business Process Model and Notation, in an attempt to overlay the security requirements of a system in its design stage, therefore increasing its security at implementation. This solution was developed utilising the Unity 3D game engine. If Maine’s research were to introduce IoT, it could assess existing physical and virtual systems and highlight vulnerabilities based on current standards. Tully et al. [36], presented a methodology for rendering large map data for crisis management. This application ties into the principles of smart cities and provides perspective to the future tools that emergency services may use to run more efficiently during a crisis.

When reflecting on the progress research is making in this domain, it becomes evident that sensors and the IoT generate more than novel applications or theoretical frameworks, but a new perspective. This perspective allows all existing research into serious and traditional games to be re-examined with the inclusion of the IoT. The following section describes the technical challenges developers of ‘smart serious games’ solutions face today, and how they may be tackled in future.

Regarding the combination of IoT and serious games, the biggest technical challenge is heterogeneity. As the number of manufacturers producing Internet interconnected solutions increases so does the competition, driving prices down and encouraging exploration. However, this competition generates segregation in the industry and causes development to be cumbersome. There are devices that only work within a manufacturer’s ecosystem, such as Samsung’s ecosystem of Smart appliances, devices that require bespoke bridges to connect with the Internet, and devices with bespoke online portals.

The only method for tackling the increasing segregation in the Internet of Things is standardisation. Wireless charging is an example for the effective use of standardisation. Without the Qi standard [37], devices would require their own type of wireless charger. In such a scenario, large technology companies such as Apple and Samsung would develop their own versions of wireless charging that would only be compatible with their ecosystems. Therefore, wireless charging in cars or in embedded surfaces would become challenging.

6 Measuring Affective Response

Sykes and Brown [28] conducted a small-scale investigation to determine how the affective response of players would be influenced by increasing game difficulty in a remake of the classic arcade game Space Invaders [38]. Their study employed a sample of ten participants who each played the game at three randomised levels of difficulty. During each period of gameplay, the pressure with which players pressed the analogue button to move their game character was recorded. The authors argue that the pressure of button presses may give an indication of the player’s level of emotional arousal, drawing upon earlier findings in the field of sentics [39]. The results obtained gave an indication that at the hardest difficulty level, the pressure applied was significantly different than at the easy and medium levels. As an early work these results are compelling, although a larger sample would be desirable along with more rigorous statistical analysis of the data. There is doubt about the ability of touch universal method to communicate human emotion [40], although for most games players this may be an intuitive and appropriate scenario and might be one that could be learned for each individual over a period of time. Nevertheless, the method employed is interesting, especially since the authors justify the use of button pressure measurement as being something that can be employed in game controllers with analogue buttons, meaning that more elaborate equipment and setups can be avoided. This is hugely beneficial given that the control devices, such as gamepads, touchscreens, buttons and joysticks, are the main physical interaction mechanisms that the player has with a game.

Following the convention of making use of game controllers for affective feedback, Bacchini et al. [41] attempted to integrate several biofeedback mechanisms into a common gamepad design. As such, they augmented an existing console style control pad with sensors designed to detect GSR, skin temperature, pressure/force and heart rate statistics via photo plethysmography (PPG). Thus, the device itself is one that is already well accepted in the gaming community and presents minimal disruption to the ergonomics of the typical gamepad. Their work was undertaken with the specific design intention that such a device would lend itself to affective gaming applications, which might be used for a variety of purposes, including entertainment as well as serious game situations. The gamepad was evaluated using a linear horror game named Death: Unknown [42], which was an independent game available freely to download. The device itself showed promise in being able to successfully detect scare events in the game. Data obtained in the reported trial of 31 participants shows promise in being able to successfully detect expected scare events. However, the analysis makes limited to no use of additional sensor data and is largely focused upon use of GSR data.

Simões et al. [43] developed a serious game to help individuals with Autistic Spectrum Disorder, measuring the levels of independence and performance in the context of a serious game designed to simulate the performance of an everyday task—planning a bus route, making the journey and interacting with others along the way. The key metric to measure anxiety in this scenario was via the use of electrodermal activity (EDA). It is interesting to note that the VR game itself attempts to mimic real life (to a reasonable degree), rather than obtusely stimulate emotional responses.

Kivikangas and Ravaja [44] report upon the affective responses encountered in users playing against one another in a multiplayer scenario compared with playing against the computer. Measurements of emotion were recorded using electromyography (EMG) and EDA sensors. As such, there was no direct affective feedback loop returned to the game nor was there specific measurement of pre-defined game mechanics designed to provoke emotion. Rather, the authors were evaluating the effect of the outcome of playing the game (winning or losing) and how the relationship between each player and their opponent (whether they were friends, strangers or a computer-controlled character) impacted their emotions. Their study involved 33 participants and used the game Duke Nukem Advance [45], a version of the classic first-person shooter game adapted to be played on a portable gaming console; the GameBoy Advance. The findings showed that positive affect occurred every time the player won a game, but also when they lost. The positive responses to victory became stronger when playing against another human, rather than a computer-controlled character. One intriguing anomaly is that the authors note “…a curious effect of negative response to victory over a friend, for which we presented numerous possible explanations, although we cannot say which one would be the most plausible one.” Broadly, the work is supportive of the concept that relationship is an important factor in the intensity of emotional response in players.

An early study of how affective feedback can be used to influence in-game actions used a simple competitive two player racing game, appropriately entitled Relax-to-Win [13]. The notion of incorporating biofeedback has also received attention from a number of large games console manufacturers, with a notable example being the patent held by Sony [46] that depicts various interface and game control devices that have notably been fitted with bio-signals, shown in positions 39a and 39b in Fig. 13.3, designed to be used as inputs to the game. The patent goes on to discuss a range of such signals and systems that may be used to generate data to be sent to the game or interactive software. Such an approach is flexible and avoids over-reliance upon a particular type of signal. The validity of skin response equipment may be brought into question under certain gaming circumstances, especially in demanding scenarios where intense movement of the muscles, usually in the hand, is required [28].

Fig. 13.3
figure 3

Game controller equipped with bio-sensors (points 39a and 39b) in Sony Patent [46]

Full size image

7 Suggestions for Future Work

7.1 Measuring the Efficacy of Affective Games

An initial piece of research to be done would examine the effects of incorporating biofeedback within networked multiplayer video games. This research could use a participatory study to demonstrate the effects of biofeedback technologies on the experience players have of the video game, particularly with regard to their immersion, sociability, experience and overall enjoyment of the game. This would make use of a range of bio-sensors attached to players of the game. For the purposes of a start point, we suggest that indicators of players’ ECG, EDA and electroencephalography (EEG) are selected.

The research could provide a simulated game environment that displays biofeedback information about competitors on the computer screen, much like the mock-up presented in Fig. 13.1. Suitable methods of visualisation and sonification of this data may be used, which would enable participants to perceive the data in an accessible manner appropriate to the video game medium. Participants would play the video-game in a head-to-head or collaborative scenario, comparing the video game experience without biofeedback to the one with biofeedback (one group of participants will play with biofeedback, another group will play without). As such the study would broadly follow the approach of a randomised controlled trial. Both subjective, self-report data, using instruments such as the GEQ outlined earlier could be used to capture player experience alongside other objective measures such as playing time, success metrics in the game itself, video footage of each player and so forth.

7.2 Affective Games and Altered States of Consciousness

In previous work, several prototypes were created which explored the use of consumer-grade biofeedback headsets as controllers for interactive artworks and games. One of these: Psych Dome [47] utilised a NeuroSky MindWave to control the generation of a visualisation and sound in a mobile full-dome. While the design of the sound and visualisation were based on the concept of altered states of consciousness, using biofeedback allowed properties of the simulation to be linked to the brain activity of the person seeing it. This type of interaction is largely non-volitional (that is, the individual tends not to be able to control their brain in such a way that they can choose to influence the signals in a particular way), but instead provides a form of ‘passive’ interactivity that can enrich the sense of interactivity. In a related project, Quake Delirium EEG [48], this passive interaction was again provided for an adapted video game, which aimed to simulate an intoxicated state of hallucination, for which graphical and sound properties were linked to the EEG signals. These projects are based on practice-led research, in which prototypes are developed, but the prototypes can then use empirical research approaches to measure the experience of participants. Eventually such approaches could lead towards the possibility of ‘altered states of consciousness simulations’, as proposed by Weinel [14].

In relation to the latter, it has been shown that there are significant therapeutic benefits that can arise from altered states of consciousness facilitated by drugs such as MDMA, LSD or psilocybin (e.g. Mithoefer et al. [49]; Bogenschutz and Johnson [50]). Technologies such as VR have been widely touted as capable of simulating real-world experiences that could have therapeutic applications—could VR also be used to effectively simulate an altered state? Some early projects such as Guided Meditation VR [20] hint at this possibility, yet there are also good reasons to be sceptical about the benefits such experiences can actually provide when compared with real meditation practices. What is therefore needed is research that takes bespoke designs aimed to simulate altered states of consciousness through games or VR, and measures the response of players, in order to demonstrate the extent to which these designs are effective. Such research can then feed back into the generation of new designs, perhaps allowing us to create virtual experiences that are tailored towards particular kinds of perceptual experiences, eventually simulating altered states of consciousness.

7.3 Measuring the Social Aspects of Multiplayer Affective Games

Research on the level of connection and social presence [51] between players in multiplayer gaming scenarios is underrepresented when compared to the amount of information available about the single player experience. This is especially true when considering the range of sensor data that affective gaming systems can provide. Ekman et al. [52] recognised this, defining the metric entitled physiological linkage and postulated that it may be associated with the social presence experienced by players. The concept itself is simple to understand: the physiological measures of players are likely to display patterns of synchronicity or correlation to one another, meaning that it “…can be used as a measure of the intensity of the interaction between participants” [52].

As explained earlier, when dealing with affect in multiplayer games it would be highly desirable to examine how social elements of the gaming experience contribute to player affect and, in turn, how the feedback of a player’s affective data might influence other players. Accurately determining multifaceted concepts of game user experience such as social presence, immersion and flow can be tough when using only biosensors. Similarly, mechanisms to verify the affective responses predicted from these sensors are also desirable. As such, a challenge exists in being able to measure these complex phenomena of the human condition. It is here where self-report mechanisms become useful and there are several good examples of tools to measure many of these concepts. In their work, which also highlighted objective measures for immersion, such as eye-tracking, Jennett et al. [53] developed a series of self-report questionnaires to be issued to games players. Factors they recognised as signifying immersion in players included: affect; cognitive involvement and detachment from the real-world. Contributing to the level of immersion experience were also factors produced by the game itself, namely, that of challenge and control. Whilst Jennett et al.’s work deals with measuring the experience of the individual, it was not purposely intended to examine social aspects and connection with other human players in the game.

One of the most commonly employed tools in the field of computer games research is the Game Experience Questionnaire (GEQ) and its component instrument, the In-game GEQ [54]. The tool integrates aspects of: competence; immersion; flow; tension; challenge and affect. Additionally, it offers a social presence module, which is designed to “…investigates psychological and behavioural involvement of the player with other social entities…” [54], where these social entities may be other human players or computer-controlled characters. The social presence element of the GEQ consists of a total of seventeen questions that participants rate using a five-point Likert scale. These questions provide scores under three themes of: empathy; negative feelings and behavioural involvement. The first two of these have a strong resonance in affective gaming, leading to useful indicators of how well an affective game might be communicating the emotional state of other players (empathy) and how this information might impact the player observing such information (negative feelings—incorporates elements of influence from others as well as a sense of revenge or retribution). Lastly, behavioural interaction, seeks to determine how the in-game actions of other players would change what the player in question did in the game, as well as providing some indication of how meticulously players observed one another. Consequently, the GEQ appears to be an effective mechanism for the subjective evaluation of experience in affective multiplayer games. Its social presence module certainly takes a step in the right direction to gauge the nature of interaction and influence between players, but it would be useful to consider other instruments that attempt to measure how connected or ‘close’ players feel to one another during gameplay. A solution to this may be to append the In-Game Relatedness items from the Player Experience of Need Satisfaction (PENS) questionnaire [55]. Such a hybrid approach, combining overlapping aspects from a range of game experience instruments is advocated by the work of Denisova, Norden and Cairns [56].

8 Ethical Considerations

There are ethical concerns relating to the combination of serious games and IoT. Games in traditional forms store little information regarding their players—commonly scores, an alias or their real name. The advances in mobile gaming have transformed the data that a game can access. Numerous games require you to sign into a social media account. Once you have signed-in, the company has access to personal information about the player and a list of the player’s friends. Introducing the IoT only increases the amount of information games would retain about its users.

As discussed, an IoT enabled game could record a player’s heart rate to dynamically alter a virtual environment, or help a player progress in it. The idea of a game company holding such personal data could become uncomfortable to players; however, there are arguments to counter this perspective too. As younger generations are introduced to technology from an increasingly early age, their perception of sharing data could be different in comparison to generations that were introduced to technology at an older age.

Irrespective of the discomfort users may have, there is an irrefutable fact that games of this nature will need to store very sensitive information on its users. This raises concerns regarding security. The General Data Protection Regulation (GDPR) law in Europe [57] is a conscious effort to increase liability on developers regarding storing personal data securely. This pressure increases the priority of securing new interconnected systems and the data that flows within it. However, there are still many challenges that the Internet of Things faces regarding security, many of which stem from the lack of standardisation. Therefore, when embedding IoT with Games, these challenges will relate to games too.

Associated with the work discussed here, which proposes to utilise affective data and bio-sensor signals as input to game environments that may be socially oriented, are concerns over psychological effects. Consider the first research activity proposed earlier, for example, which would involve the visualisation and sharing of bio-sensor information in collaborative or competitive games between human players. Certain parallels might be drawn with aspects of the Stanley Milgram’s controversial experiment [58] into behaviour and obedience. If one player of a game is competing against another and sees that their opponent’s heart rate has elevated to abnormally high-levels, will they attribute this situation to themselves and would guilt and blame be attributed to them should their real-life opponent suffer cardiac problems? Perhaps the answer is that the sensor attached to their opponent is simply faulty, but without knowing this, the player might still suffer anxiety or low mood as a consequence. One rebuttal to these fears may be that multiplayer gaming is not a new concept and so simply presenting this information is only making visible a phenomenon that may have been occurring already. Nevertheless, there is a duty of care to be considered once this information has come to light. As recently pointed out in an article by Tidsdale, “…video gaming has been claiming victims for decades” [59] and with the enhanced sense of immersion, social connection and presence that we propose here, we would be wise to approach research in this field with due care and diligence.