Keywords

1 Introduction

Human-Human Interaction (face-to-face interaction) and multi-party interaction are nowadays included in the research areas of Human-Computer Interaction (HCI). The assumption is that knowledge of how such interaction takes place can help to develop digital technology that can real-time support these interactions or to archive these interactions and later help to retrieve useful information from such recorded interactions. In both cases it is necessary that the digital technology is able to have a level of understanding of the interaction. Speech recognition and speech understanding are of course extremely useful when understanding verbal interactions. But not all human-human interactions and multi-party activity are speech only. Moreover, even in the case of speech, we need to understand the role of nonverbal speech (pauses, hesitations, prosody) and it is useful to know about accompanying facial expressions, gestures, and eye gaze behavior. In previous years we studied both global aspects of multi-party interaction [1] and synchrony of face-to-face interaction [2], including subtle aspects such as mimicry behavior [3, 4].

In recent years we have also seen that information that can be obtained from wearable physiological sensors is used to learn about its user. This information is used to inform a particular user about his or her performance during exercises and about health issues during longer periods of activity. We have not yet seen many research attempts that aim at investigating situations where users interact with each other and that physiological information plays a role in the interaction. There is hardly any research that aims at detecting a user’s cognitive or affective state - using other than visual and auditory cues - while interacting with others in a face-to-face or multi-party situation. There are exceptions, and interestingly, in the case of measuring brain activity (Brain-Computer Interfacing or BCI), they can often be found in many interactive interfaces and environments that have been designed by artists for mixed media performances [5–7].

BCI provides sensors and actuators with information about a user’s affective and cognitive state. In addition a user can issue commands by manipulating his or her brain activity or by being receptive to sensorial stimuli that are caused by natural or artificially created events in the environment and then give feedback to the environment. Here we introduce research and trends in BCI and applications that assume multiple users at the same time. These applications will be looked upon in Sect. 2. It will be followed by a section (Sect. 3) with some global thoughts about cooperation and competition in general. A step towards distinguishing useful characteristics of cooperation and cooperation that can help in designing multi-brain applications is taken in Sect. 4. Section 5 connects BCI with research on neuroscience and social interaction. Some conclusions can be found in the final section.

2 Multi-Brain BCI: Applications and Characteristics

Interestingly, already in the 1970s artists looked at brain signals to be used for artistic and playful expression generated from simultaneously measured brain signals of more than one performer equipped with simple EEG devices. Ideas, examples, and applications are often related to being able to measure synchrony in brain activity of two or more subjects. More interest in such multi-brain processing emerged in the present century. In [8, 9] applications of research on multi-brain computing are mentioned and collected. Based on these papers we mention the following application areas:

  • Joint decision making in environments requiring high accuracy and/or rapid reactions or feedback

  • Joint/shared control and movement planning of vehicles or robots

  • Assess team performance, stress-aware task allocation, rearrange tasks

  • Characterization of group emotions, preferences, appreciations

  • Social interaction research (two or more people)

  • Arts, entertainment, games.

Not in all cases (almost) real-time use of measured brain activity is necessary. For example, in non-critical situations task allocation can be done while preparing a next task. Knowing about preferences and appreciations can help to design a next version of a product or interface. However, in this paper the focus is on real-time interactive use of the users’ brain activities. Examples of multi-brain research in all of these applications areas can be found in the literature. For example, joint decision making during meetings and brain storm sessions and assessing team performance is a recurring theme in the research of Berka and colleagues [10]. Joint decision making and joint/shared control of (virtual) vehicles, robots and movement planning in critical situations has been addressed in [11, 12]. In this latter case we look at joint brain activity of event related or externally evoked potentials that are perceived by collaborating users at the same time (exogenous or sensory-driven neural activity), or the users’ synchronized intentional manipulation of brain activity (endogenous activity). For example, for the latter case, joint imagery movement, in order to increase robustness and speed when performing a task in a critical situation.

When measuring group emotions and group preferences it is useful to distinguish between (1) small groups (two or more people) performing a particular task and interacting with each other, a usually (2) large group of people that attend a performance requiring audience participation, or an even (3) larger group of people that are addressed, not necessarily at the same time and in the same location, to evaluate a particular (multi-media) product [13]. In the first situation we are interested in understanding the group process and ways to improve it; this may concern decision making, but it can also concern less task-oriented issues such as social cohesion and empathy awareness; in that way it becomes part of social interaction research. In the second situation we are interested in having one or more persons’ brain activity getting involved in creating or adapting a piece of life media art, maybe also in interaction with an artist/performer who is coordinating the joint life performance [14]. In the third situation we are interested in the statistics that can be obtained and from analysis we learn about preferences that help to improve the product. Measuring audience responses with BCI is less well known than with more traditional physiological sensors (heart rate and skin conductivity sensors), but with the more recently developed EEG wireless headsets with dry electrodes this will certainly change.

Finally, returning to the last bullet in the above list, multi-brain or multiparty BCI is researched in the context of (serious) games. Games usually assume more than one user. But of course, we can also have a user competing or collaborating with the game environment or its artificial agents. Whatever the situation, in game environments designers have the freedom to introduce problems, challenges, and unusual situations. Gamers can be guided into interaction situations where they have to collaborate, compete, cheat, conspire, and form alliances. In recent years these issues have also become part of BCI research. In artistic, game and entertainment applications artists and designers can introduce unexpected events, don’t necessarily have to take care of efficiency issues and can introduce situations where gamers, performers, or audience members have to take unusual actions or unexpected partnerships. This makes BCI an interesting tool in games, entertainment, and performative arts, and, moreover, such applications can result into new research approaches from which new applications areas can follow.

3 Multi-Brain BCI: Thoughts on Competing, Collaborating

Here we are not interested in games of chance. We look at games that require skill and that include competition. Competition can be either among players or against the game system itself. When there are more players then players can cooperate in order to beat the system or to beat another team of players. The composition of teams can change during a game. A game can end when we have reached a particular goal defined by the game. However, it can also be the case that a game or our colleague players continue to offer us challenges and as long as we survive the game continues. We want to consider how BCI can be used in this context. That is, can we identify situations where brain activity of two or more players in the game can be integrated in order to perform a joint task, or can brain activity of two or more players be compared in order to determine who or which team performs a game task better than an opponent or an opposing team? It should be mentioned that although examples usually make references to videogames, there is no need to restrict ourselves to videogames. In fact, examples from traditional BCI research include the movement of artificial limbs and the navigation of wheel chairs. Actuator technology has become part of wearables, tangibles, and robots. Hence, games supported by BCI technology can be played in the real, physical world as well.

Competition and cooperation are issues we want to investigate. These are rather obvious issues in games, but also in many other situations. Whenever we have a situation where there are two or more people we can have agreement and disagreement about how to solve a problem. Some situations ask for cooperation, other situations ask for competition. The situations can be virtual: fighting, survival or achieving a high score in an entertaining videogame. But it can also be about performing with others in a serious game or a real-life situation: for example, a discussion or some other activity that requires verbal, nonverbal, or physical interaction. We already mentioned research aimed at evaluating a meeting participant’s contribution to decision making in a small group meeting [10]. Obviously, this requires an analysis and comparison of his or her brain activity with the brain activity that can been measured from the other participants in the joint activity. More in general, we can have situations where we have multiple users and we can be interested, depending on the application, in the brain activity that is fastest available, the ‘strongest’ brain activity, the most consistent brain activity, the sum or the average of multiple users’ brain activities. Obviously, it is also possible to consider mixtures of these possibilities and putting weights on the various possibilities.

Competition and cooperation are part of life performance art as well. Performative artists can create a piece of work that only becomes alive when audience members or museum visitors discover they can interact with the artistic installation. Performance art can also become alive when a performative artist has designed BCI audience involvement and real-time BCI audience interaction while he or she is performing. When an artistic or entertainment multimedia environment becomes accessible for the audience we can hope or expect that audience members cooperate and compete in order to make changes to the mixed media performance. In these applications multiple users wear a BCI device to give the artistic or game environment access to their affective state. Audio-visualized or other feedback from the artistic or entertainment application has impact on the user’s or users’ involvement and experience. Artists can design their BCI art with a focus on various ways of audience involvement. There can be a joint performance, but there can as well be a designed and implemented environment that allows and assumes feedback from audience members without further involvement of the artist or an environment that not only incorporates affective state information, but also expects explicit commands generated by one or more audience members. [7, 14].

There are many ways of cooperation and many ways of competition. For example, in a game environment gamers or teams of gamers can decide about their strategy and they make decisions about joint actions. However, not everyone in the team needs to agree. An opposing team will have the same problem. Full competition and full collaboration are extremes on a scale and these extremes and everything in between make it difficult to make decisions about how to use and compare brain activity to be detected from multiple users. In addition, competitors can decide to cooperate and when a certain sub goal has been achieved, start competing again. Especially in massively on-line role playing games we see situations where alliance forming occurs. In our view these games are not essentially different from many artistic BCI applications where a designer allows the audience to interact with his or her BCI installation or where the BCI artist performer accepts various forms of BCI audience involvement in the performance. See [15] for many of such examples in the artistic domain, in particular music performances.

At this moment a systematic review of such possible multi-brain applications needs more time and more effort than we can afford here. With ‘more time’ we certainly want to refer to the need of seeing more ‘bottom-up’ applications of BCI research where multiple users are involved. In recent road maps on BCI research [16] there is no or hardly any mentioning of BCI multi-brain research or applications. These road maps have been designed by traditional BCI researchers hardly being aware of applications for other than patients and disabled users.

4 Cooperation and Competition Using BCI: Interactions

It is possible to distinguish more simple examples of cooperation and competition in existing and potential multi-brain BCI applications. That is, artistic applications that don’t involve complicated audience behavior or audience-performer interaction behavior. Or, as has been investigated more extensively, in multi-brain BCI game or training environments. And, of course, as mentioned before, we can be interested in neuro-marketing applications or applications where we want to analyze team performance and contributions of team members by measuring their brain activities. However, before attacking these problems, it is useful to consider more simple examples of cooperation and collaboration. From them we can learn about introducing BCI in games, in particular games that require more than one player.

4.1 Comparative and Interactive Games

Firstly, we can distinguish between comparative games and interaction games. As mentioned above, we are not interested in BCI applications of games of chance. We are interested in games where players have to use game playing skills and where BCI supports a players’ game activity or even determines a player’s game actions at a particular moment or being available to help in interpreting a user’s or gamer’s actions. In comparative games there is no game interaction between players. There is no interfering. A game is played and once having completed the game you can compare your score with the score of others. The game can be played in a social context, others can compete at the same time and your results can be compared in a competitive situation. But it is not the case that in any way your game actions change the state of the game in such a way that your opponent has to deal with an other than neutral situation. Hence, when two or more people compete in a game, it does not necessarily mean that one gamer’s performance has impact on that of the other(s). It may even be the case that a gamer only becomes aware of the performance of another player when the game is ended or when he or she sees the score displayed in a ranking of others who have played the same game.

4.2 Turn Taking Without Control Interference and Without Game Interference

In a game on a pinball machine you can have both. You have a direct opponent and are in a battle to score more than this opponent with the balls you have. Maybe you can increase your efforts when you know about the score of your opponent. But there is no way you can leave the machine in a situation that is to your advantage. That is, making life for your direct opponent more difficult. And there is also a general ranking obtained from previously played games that shows how your performance compares with many others. ‘Tilting’ the machine is a possibility to cheat, but preventing, detecting, and punishing tilting are counter measures that have been introduced by the companies that sold such machines. Playing pinball using BCI (motor imagery) to detect whether the gamer intended to use the left or the right flipper has been discussed in [17]. There is not a continuous brain activity involved in the game. Gamers can compete by taking turns, and once a turn is taken brain activity can be used to control the flippers. Your score depends on your BCI skills, in this particular case, motor imagery.

4.3 Turn Taking Without Control Interference and with Discrete Game Interference

An example of a game with game interference is the Connect Four game. Here we have a vertical grid of 6 × 7 positions. Players take turn in putting coins on these positions by choosing a column to drop a coin. The game ends when a gamer has been able to connect four of his coins, either horizontally, vertically or diagonally. A BCI version of this game allows the users to choose a column to drop a coin using a P300 evoked response [18]. This changes the state of the game. Playing chess using BCI control of the moves is another example. Again, you make a move, your opponent waits until you finished and his or her move is usually dependent on the move you made. In both games this is different from the previous pinball example. There is competition and what a player does is very much dependent on what his opponent has done. But again, there is turn taking. And, each turn leads to a different situation that has to be assessed by the turn taking player. Actions can follow rules from a general strategy point of view. There may be conflicts between such a global view and what an opponent allows you to do. There is however, apart from frustrating your opponent by making moves he or she doesn’t expect or like, an assessment of the game board situation and how it fits in your strategy to beat your opponent. At this moment there is no way of incorporating such thoughts in a BCI that helps you to improve your chess playing capabilities. Nevertheless, we can introduce BCI in chess playing, for example to move a piece from one position to the other. In [19], similar to the P300 Connect Four game and well known spelling applications, a chess player has to pay attention to his or her preferred move while being submitted to a display of possible moves. That is, the chess player uses P300 BCI control to select a start and a destination position among the 8 × 8 possible positions. In [20] imagery movement is used to tackle the same problem. These applications allow users without other communication possibilities than using their brains only, to play chess. Each player has to wait for the other player in order to make his move.

Although in these games BCI is used to play the game, we prefer not to call them BCI games. The reason is that the outcome of the game is not dependent on your BCI skills. You decide about the column (in the Connect Four game) or the chess move which you want to make. No BCI skills are needed for that. How the piece is moved is not in any way essential for the game. Whether you do that by physically picking up the piece, by clicking arrow keys on your keyboard, asking someone to do it for you or by, after having made the decision what move to make, having your brain activated in such a way that it becomes clear where a particular piece should move, does not help you to win. Obviously, we can go into ‘subtleties’ such as the efforts you have to make to have a piece moved from one position to the other on the chessboard goes at the expense of your thinking about what the right move should be. More interestingly is when we introduce time constraints. But then again, in real chess time constraints are meant to be constraints on the time you take about thinking about which next move to make, not about the time you need to make the next move. So, from the point of view of the original chess game, although this introduces an extra challenge, it leads to a different kind of game.

4.4 Turn Taking Without Control Interference and with Continuous Game Interference

In the examples mentioned above we have two different BCI paradigms. Motor imagery to decide between left and right in the pinball machine, P300 to decide among the columns of the Connect Four game, and P300 or motor imagery for deciding the position changes of a chess piece. It is not difficult to think of implementations of these games where other BCI paradigms, maybe taking more time, are used to make such decisions. In the examples a player can perform his or her action without having his action enhanced by other players (cooperation) or thwarted by other players (competition). Although there is turn taking, actions of players in a game can have real-time impact on the actions to be taken by an opponent, that is, by watching what the other player is doing during a turn, his or her opponent can prepare counter measures. For example, in a Pong game we have to look at the performance of our opponent and, in real-time, we have to move our bat up and down to anticipate the right position to return the ball. This example is quite different from the previous ones. We have real-time interaction between players. If we have the movement and speed controlled by BCI, then the players’ BCI skills determine who is going to win. The players’ actions – move the bat up and down - are fully determined by what is perceived on the screen. Evoked brain activity can be used to move the bat to the right position, but there is no need to compare brain activity of two opponents in this game. A version of this game using motor imagery to control the bats has been introduced in [21].

4.5 Continuous Control and Continuous Game Interference

As an example where brain activity of two (or more) players need to be compared in order to have a system to make a decision we can look at one of the first BCI games, the BrainBall game [22]. Two competing players are involved, their brain activities are measured and compared, and based upon this comparison a decision is made about the direction the ball should move. There are many more similar examples of such multi-brain applications in arts and games. In this BrainBall example we have to compare and ‘subtract’ brain activity information in order to determine who is the strongest. In other applications where users cooperate we have to ‘add’ brain activity information from different users in order to decide which way ‘to go’. Which way ‘to go’ can be decided by the ‘strength’ of the brain activity of a group of people (a sum or an average). The brain activity involved can be about synchronized relaxation, event related potentials, evoked potentials, joint motor imagery or joint intentions to move, where the latter can be detected before physical or imagined movements. Various other examples have been discussed in [9]. They include (serious) game applications where two or more than two users participate in situations where their joint brain activity is measured and is used in game, entertainment or arts applications.

4.6 Multimodal, Multi-Brain, Hybrid, and Brain to Brain Fusion

One interesting issue in multimodal interaction research is how and when multimodal information that is obtained from a user has to be fused. Usually a distinction is made between fusion at the signal, the feature or the decision or application level [23–25]. Obviously, one of the ‘modalities’ can be brain waves and there are research examples where brain waves and gaze, brain waves and speech or brain waves and heart rate information is fused in order to get more complete information from a user. Modality switching in a BCI context is described in [26]. In multi-brain applications we usually have the situation that for all participants the same BCI marker contributes to the cooperative or competitive actions. This makes it possible to ‘compare’, to ‘add’ or ‘subtract’. For example, in [12] we have users using motor imagery only and in [27–29] we have examples of multi-brain interaction using ERP only when performing a joint task. Earlier we have experimented with applications that require different BCI markers. For example, make a choice using SSVEP and then have the speed of an activity determined by the user’s level of relaxation [30]. This is an example of sequential multimodality, rather than parallel multimodality [23].

In [31] the notion of hybrid BCIs was introduced to give a name to a BCI using two BCI paradigms or a BCI combined with another system. The authors didn’t embed their observations in already existing multimodal and HCI research. Moreover, they required that “there must be at least one recordable brain signal that the user can intentionally modulate to effect goal-directed behavior” and that the user must obtain feedback. Clearly, this is not the way we have been looking at BCI in this paper and the way BCI has been incorporated in HCI research. We certainly accept multi-brain situations where users do not intentionally modulate their brain signals. They may be aware that there brain signals are recorded and, although not necessarily real-time, that is, no immediate feedback, may lead to changes in their interaction environment. Another example where we need to measure and integrate in parallel two BCI paradigms can be found in [32]. In this paper we have the standpoint that the cognitive or affective state of a BCI user has impact on brain waves that are evoked by external stimuli or by intended modulation, hence, detection of both is needed in order to improve the control that is intended by the user. The BCI system can compensate for the user’s emotional state. More generally, we can think of multi-brain applications where more than one BCI paradigm is included or where multi-brain BCI is combined with, for example, multi-gaze, multi-gesture or multi-heart rate information. Audience participation based on merging heart rate and brain wave information is reported in [33].

Finally, we should mention brain-to-brain computing. Clearly, this involves two or more brains and, when looking at current research examples, it assumes cooperative BCI controlled interactions, where specific patterns of brain activity of one user are delivered, using transcranial magnetic stimulation, to a partner who can complete a particular task [34]. However, there is no attempt and also less reason to compare brain activity of the two participants in such an interaction.

5 Neuroscience and Social Interactions

In the previous sections our aim was to illustrate the various applications and characteristics of multi-brain computing for HCI purposes. In this decade the research area Neuroscience for Social Interaction (NSI) has drawn attention of many neuro-scientists. Clearly, when we talk about social interactions we assume interactions between two or more people. In this research area the notion of hyperscanning was introduced [35, 36]. Hyperscanning is the simultaneous scanning of multiple human brains in order to study brain patterns that emerge during multiparty interaction, for example, when having a conversation, when playing music or during game play. From a HCI point of view it is especially interesting to see how this field can contribute to the multi-brain and face-to-face interaction research topics related to cooperation and competition. How can this NSI research provide knowledge that helps us to interpret correctly commands or preferences issued or obtained from users while they cooperate or compete with others? NSI knowledge can also tell us about joint attention or other information related to performing a joint task that provides us not only with a context to interpret BCI signals, but also the possibility to make changes to the environment based on this information or alert the interactants to start a joint action or issue a multi-brain command. As examples we can mention the possibility to predict the next speaker in a multiparty interaction [37] or the detection of mimicry [38]. NSI helps us to obtain computational models of the interacting mind [39] and clearly, such models are helpful in interpreting minds that cooperate or are in competition.

6 Conclusions

This paper allowed us to discuss the need for a systematic approach to designing and evaluating multi-brain computing. It does not yet allow us to elaborate on the characteristics and parameters that need to be distinguished and introduced when attempting to model various forms of multi-brain interactions, let alone the embedding of multi-brain interactions in a natural multimodal interface and a multimedia display context. This needs further research. We are in need of research that embeds the issuing of cooperating or competing BCI commands in knowledge from the field of neuroscience for social interaction (NSI). Embedding means that we interpret BCI information that we plan to use, whether it is a user intended task control command or information that helps to adapt the environment to a situation that suits the aims of the game or task and the cognitive and affective state of the user. We think that the distinctions we made here can make us more aware of the issues that play a role and therefore help us to introduce a more systematic approach to designing multi-brain BCI applications in the future. Clearly, our observations are rather preliminary and require more elaboration in future research.