The way gaze cues are used in social interactions is by no means irrelevant because they are fundamentally important for understanding social interactions. In this paper, we argue that social conflict is a form of relating and that gaze clues are critical to understanding the underlying cognitive processes in this phenomenon. To learn more about conflict, we created an experimental setting that reduces real life to a mixed-motive game. We analyse the gaze patterns of 22 10- to 12-year-old children in specific game moments that could have been conductive to conflict. Our aim is to understand how subtle forms of conflict unfold, by analysing micro-level behaviours and establishing a link to high-level psychological constructs. Their gazes show that children are being more competitive or cooperative at different stages of the game. Children tend to avoid confrontation by averting face-directed gazes when they are asking for larger profits, and they gaze longer to attempt to persuade the other child.
Dyadic forms of conflict encompass cognitive, affective and behavioural dimensions that cannot be taken singly, when considering any form of social conflict . The interaction between these three dimensions in a continuous feedback loop establishes conflict as a dynamic process that produces complex, goal-directed behaviours . How this process unfolds, however, is still unclear in the literature. Conflict happens at different levels of social interactions, and despite the vast research on conflict in the social sciences, it is not clear what actually happens during this multi-level process. What makes some fights break out and others subside, for example, is difficult to explain unequivocally. This topic has provoked discussion in various areas of research. Understanding the dynamics that underlie conflict processes is a far-reaching social skill in life that can help people cope with conflicts and detect them beforehand.
Given the importance of the conflict phenomenon and the increasing interest in socially intelligent technologies, it is of interest to endow these type of systems with mechanisms for detecting or anticipating conflicts in human-human or human-computer interactions. Not only would it help the agent to act towards prevention or creation of conflict (depending on its utility), but it would also increase the agent’s ability to choose the best strategy to tackle the situation either to find a mediation solution or react appropriately to the interaction. In the multi-agent systems community social conflict has been neglected. Conflict has been addressed as a failure or a synchronisation problem  without regard for its cognitive, affective and behavioural dimensions. Despite that fact, a great deal of work has been devoted to the study of conflict resolution mechanisms, but little has been discussed regarding detection of conflict, which is an essential step for an effectively approaching conflict episodes.
To design agents that are more socially aware and thus able to express and recognise dyadic forms of conflict, it is critical to understand how conflict episodes unfold. For example, how a pacific coexistence can sometimes rapidly shift to an overt manifestation of conflict. In our work, we aim to explore subtleties of how conflict evolves in social interactions in order to better represent the phenomenon in multi-agent systems. To that end, we investigate social signals that could give us insight about the social phenomenon and inform us about the agent’s design. The work described here focuses on gaze patterns; in particular, we are interested in the monitoring function of gaze, which reflects information needs, preferences or expectations and the affect of the participants in the interaction.
Social signals are present in any social interaction, whether between humans or between humans and machines. These signals can mirror agreement, empathy, dominance or conflict and they “make up the very texture of our everyday life” . A wide range of social signals has been studied over the years to create and design embodied agents that are more socially aware. Researchers have looked into social signals not only to unravel patterns in different contexts but also to understand how people make sense of that social interplay. The Social Signal Processing Network of excellence (SSPNETFootnote 1) was created with such aim. The goal has been to accurately sense and interpret social interactions and build systems, based on models of human behaviour, towards agent systems that are likely to be perceived by humans as more natural and more successful in building rapport . These social signals, in the form of non-verbal behaviours, such as head nods, smiles, laughs, posture, and mutual or individual gaze, for example, are a continuous source of information that expresses attitudes, mental states and feelings. Within non-verbal behaviours, gaze cues are considered to be fundamentally important in understanding social interactions  and are often referred to as a mind-reading tool. Gaze is the most rudimentary form of social contact , and it serves myriad functions, such as starting or stopping encounters, expressing attitudes towards others, helping to synchronise speech , assisting in information seeking and signalling underlying interpersonal emotions and attitudes for others to decode . The tight connection between gaze and cognitive and affective aspects in face-to-face human interactions has brought attention to a powerful signal that could act as a descriptor of interpersonal behaviour. For that reason, researchers in human-computer interaction have sought to create more natural experiences with humans by using gaze cues to coordinate conversations [34, 38] or as an attempt to reflect an agent’s inner cognitive processing in a virtual environment , to name a few examples. Furthermore, with the advances in eye-tracking technology, the study of gaze cues may also allow us to predict people’s moral decisions , which is valuable information in human-computer interaction. Additionally, gaze cues can also be used as a process tracing methodology to track cognitive states or stages .
The latter frames the work described in this paper. We attempt to establish links between phases of conflict in a bargaining scenario by analysing moment-by-moment gaze patterns. For this purpose, we analyse the gaze patterns of 22 children (aged 10 to 12 years-old) in a dyadic mixed-motive game (a game with competitive and cooperative incentives) under incomplete information. We created a real-life setting reduced to a mixed-motive game, in which children’s previous experiences and relationship with their interactional partner played a relevant role in the interaction. The competitive nature of the task creates potential for conflict and makes us aware of the gaze sequencing that occurs in this bargaining setting in five distinct situations. We present and discuss similarities and dissimilarities in the players’ gaze patterns of the players within fixed, contextually relevant time-windows: (a) when players establish their expectations for the round; (b) when players adopt a competitive or cooperative stance;and (c) when players make a deal and reveal their profits.
Gaze patterns show that children are more competitive or more cooperative at different stages of the game. When they make cooperative moves, they tend not to look towards their partner when they make concessions but to look more when they expect or anticipate positive reactions. When they make competitive moves, they look at their adversary for longer periods to attempt to persuade the other and make him to concede. In addition, when children attempt to earn a greater portion of profit, they find it difficult to look at their partner.
This paper has three main contributions. First, it presents a datasetFootnote 2 that pretends to reduce real life to a mixed-motive game. The game’s structure creates potential to conflict and is considered natural by the naive subjects. Second, the temporal analysis of gaze creates detailed models of face-directed gaze, in specific contexts, that can be useful in synthesising the behaviour of interactive agents. Finally the temporal analysis of gaze allowed us to gather inside information about players’ intentions that could be used to further analyse their interactions.
Gaze in interaction
Gaze, just as any other social signal has a set of features that characterise it and an associated meaning . These features—signals—are physical characteristics, such as eye-direction, humidity, head movements, eyebrow movements, eyelids or wrinkles, which together characterise the gazeFootnote 3 under analysis . The meaning corresponds to the set of beliefs, goals or emotions that the person’s/agent’s gaze communicates.
Hence, in developing believable artificial agents, it is critical to develop gaze models that generate realistic gaze behaviours, that is, that produce gaze behaviours that reflect the agent’s internal states (beliefs, goals or emotions) and that can be inferred by either humans or other agents . However, to associate cognitive states with gaze motions, it is necessary to understand what laws govern social interactions and the role of non-verbal behaviours in these . Although nonverbal behaviour is polysemous and context dependent, researchers have attempted to establish recurrent correspondences between signals and their meanings. For the case of gaze (and touch), Poggi  developed a lexicon that links signals to meanings and that has proved useful in developing artificial agents that use gaze to communicate . Nevertheless, gaze can be naturally ambiguous because it is influenced not only by context but also by time and by ‘fusion’ with other modalities.
Gaze behaviours have been analysed in great detail to date, but mostly in conversational or collaborative laboratory settings, which in some cases can constrain data ecology. The work of Heylen , for instance, has focused on non-verbal behaviours, in particular, gaze and head movements, in multiparty-conversations between agents and humans. He had found that the use of such non-verbal behaviours had an impact of people’s perception of the agent’s behaviour . Not only did the non-verbal behaviours impact communication of intentions, but also serve impression management. The found patterns have been used to create models for artificial agents that interact with humans and other agents, so they display signs that they are listening and understanding the other in an appropriate fashion. Other domains have also been explored. A recent work by Macdonald  studies the seek and follow of gaze cues in a real-world collaboration setting. The authors’ aim was to understand whether the role of the interactional partner had any effect on gaze cues in a natural setting. They found that the amount of gaze towards the other is conditioned by the participant’s social perceptions of the other person.
Gaze cues have also been used to describe the nature of negotiations. The work of Foddy  studies how humans use gaze in cooperative vs. competitive settings. To our best knowledge, this is the only work that compares gaze cues based on these two motivational orientations in a negotiation. Her structural approach aimed at producing a model for negotiators that could predict the competitive or collaborative nature of the interaction based on the amounts of gaze and mutual gaze in the two different settings. She found that in cooperative settings, people engage in frequent long looks, whereas under competitive conditions people avoid longer eye contact and take frequent short looks.
In contrast, in order to shed light on how a setting that is prone to triggering overt manifestations of conflict will unfold, we seek to more directly observe (and trace) cognitive processes by analysing moment-by-moment gaze cues in specific temporal windows in a mixed-motive game. We take advantage of Foddy’s findings to divide our data and create a base for analysis.
Conflict in theory
Interpersonal conflict has often been associated with tensions, disputes, interference or obstruction of goals and beliefs between interacting parties. Despite being necessary attributes of a conflict episode these factors are not representative of the conflict phenomenon as a whole. The view that is adopted in our research is that conflict is a dynamic process that goes through multiple stages, in a cycle (in line with ), and that incorporates cognitive, affective and behavioural processes (refer to ). The agent’s reasoning process from perception, through diagnosis to resolution is illustrated in Fig. 1, and it unfolds as follows.
First, a set of baseline conditions in each setting determines the potential for conflict (state 1). These may be situational conditions that emphasise a structure of interdependence between the parties or the actual state of beliefs of the agents. Although these conditions are not sufficient to trigger conflict, they act as a fuel for a possible conflict episode. More often than not, potential conflicts simmer unseen at this state and never become active;
I f we are in conflict, we have to be aware of it. For that to happen, some event in the world acts as a trigger and makes the agent aware of the potential conflict (trigger 1). That trigger can be communication with another agent or one’s actions or lack of actions;
To be aware of a conflict or a potential conflict is to be emotional about it . Hence, appraisal is central to this process, as it in any other social interaction. At this stage (state 2 and 3), the agent is feeling deprived of something, or exploited, or perceives a failure in its own expectations. Yet for an overt manifestation of conflict, some change in the world transforms the participant’s perception and emotions pull the trigger, activating the conflict (trigger 2). Conflict emerges (state 4) when at least one potential participant manifests the belief that his goals are incompatible with those of an adversary, implying goal formation.
To sum up, we propose (in ) that for an agent to recognise that it is in a conflict situation, it must be aware of some incompatibility between relevant beliefs or goals; some interference in its plans must have occurred such that it can no longer pursue what it thought to be the best strategy; and finally, that obstruction is attributed to another. It is important to note that potential conflicts may simmer unseen and never become active.
Negotiation is an interactional process that is built into our daily encounters. Whether with our parents, friends or colleagues, we often engage in the process without even being aware of it. Additionally, in mixed-motive negotiations, conflicts are bound to emerge because participants have opposing preferences, and each attempts to maximise his or her own gains. An experimental setting in which the potential for conflict exists, acts as a model of a social interaction that is object of study. For this study, we used a variation of the “Game of Nines”.
The “Game of Nines”
The “Game of Nines” is a mixed-motive bargaining game and it was firstly used by Kelley . This bargaining game was selected because it creates a setting where two negotiators face dilemmas concerning their goals and forms of communication.
- The game :
The game is played with ordinary playing cards and involves three parties: the two players and the house/bank (represented by the experimenter). Each player holds eight cards from one (Ace) to eight, available every round. During each bargaining round, the players had to jointly agree on a possible contract. Each contract corresponded to a card that would be played by player A and one that would be played by player B so that their sum would not exceed 9. For example, if player A plays the 6 card, one possible contract is for player B to play the 3 card (it is also possible to play the cards 2 or 1). The parties’ interests are always directly opposed: what is most profitable for one player is least profitable for the other. In each round, a minimum necessary share (MNS) is assigned to each player, privately, by the house. This MNS value is only known by the person to whom it was assigned. Therefore, the information about the other is incomplete. In addition, for a profitable agreement, the negotiator has to bargain for a value above his MNS (eg., if a player has a MNS equal to 4 and he plays a 6, he will get 2 as a reward), without knowing the extent of the concessions the other can make. If the participants do not reach an agreement in a limited amount of time both get zero. Therefore, it is of mutual interest to reach an agreement, and it is in each person’s individual interest to choose the division that is the most profitable personally (and thus minimally profitable to the other player). The player who makes the most profit wins a cash prize at the end.
- Adaptation :
For the current experimental setting, the basic structure was the same. However, because the negotiators were children, we eliminated the time constraint. Because of that, if the children were not able to reach an aggreement, they both received zero, and “the bank” won as many points as the difference between 9 and the sum of the players MNS values. In the end, both players must individually make more profit than the bank. This establishes that lack of consensus has a cost and also adds a cooperative incentive, ensuring a mixed-motive relationship. The experimenter, however, does not play the role of the bank and is absent from the room. In addition to holding the cards ranging from one to eight, each player also holds a card (the Queen) that allows them to give up if they feel they are not able to achieve reach a viable agreement.Footnote 4 The rules for not reaching a consensus apply here: both participants get zero and the bank makes profit.
The experiment took place over five rounds. In each round, each participant took a MNS (a numbered card ) from an envelope. The child was instructed not to show the card during the trial and to never agree to a value below that number. They were told to negotiate as they saw fit. At the end of the round, each participant had to show the other his or her MNS value. The player with more total points in the end wins the game. The points were not converted to money, but the winner chose a prize between two choices.
What is conflict in this setting?
In this created setting, the children bring their expectations about their partner, their past experiences and the relationship they have built with their partner. These elements, along with the structure of the task, determine the degree of interdependence between them. That is, it establishes the fuel for potential conflict at the beginning of each round.
The task structure is determined by the MNS values that are discovered by each participant when each opens a new envelope. The rationale behind using fixed MNS values (see Table 1) throughout the game is to create the same situation each time a new round begins (with the exception of round 4). Figure 2 illustrates the possible contracts that a player can make in each round considering the opponent’s options. The number of available solutions for the bargaining problem in rounds 1, 2, 3 and 5 is the same, allowing for behaviour comparison. In round 4, under the assumption that fewer options will create more potential for conflict, the participants are faced with a situation in which there is no mutually profitable agreement.
This interdependence creates a certain level of interference in that neither party is can reach her goal independently but it is not sufficient to provoke a conflict episode. As argued before, at least one party must perceive the incompatibility between the two and attribute the interference in achieving the desired goal to the other party. Still, why some conflicts emerge and others subside is difficult to answer unequivocally. The analysis of micro-level behaviours in specific moments of the game may shed light on that issue.
The “Game of Nines” data were collected in a public school in Oeiras, Portugal. In total, 22 children (13 girls and 9 boys) aged 10 to 12 years-old participated in dyadic sessions of the game. All dyads, with one exception, were same-sex participants. The mixed-sex dyad was excluded from the analysis described in this paper. Opt-out consent forms were provided to all of the children’s parents or guardians and all games were video and audio recorded.
Data recording and annotations
Two cameras recorded the interactions. Each of the cameras was directed at one of the children, capturing their face and hand movements. An audio recorder was used owing to the poor performance of the cameras’ microphones, and the three elements were later synchronised. The basic actions of the game (when the players see their minimums; when they bid; when the agreement is set; when they turn over their minimum cards) were manually annotated (post-game) along with gaze and speech, using ELANFootnote 5 software. For gaze, after video streams were segmented, each time-interval was coded as look-at-game, look-at-other or look-elsewhere.
The measurement design for the gaze annotations was not fully crossed, and a subset (one third) of subjects was rated by two coders (because this is a time-demanding task). The resulting estimate of Cohen’s kappa estimates averaged across the annotated variables by coder pairs was 0.84, indicating substantial agreement.Footnote 6 Furthermore, two psychologists annotated the game sessions according to action tendencies because we are looking at the interpersonal effects of emotion (Cohen’s kappa averaged across the annotated variables by coder pairs was 0.674). On average, each game session lasted for approximately 13 min (12 min and 47 s). In total, 2 h and 20 min of video data were annotated.
Action Tendencies in this work are conceptualised in terms of Horney’s theory . In decision making, one can be moving towards, or engaging in more cooperative activities, moving away by taking a passive stance in the interaction, or moving against the other, in which non-cooperative actions are taken. The action tendencies framed here in the context of decision making, have also been used to define a taxonomy of conflict behaviour [12, 48]. In the context of this study action tendencies were coded according to the following scheme:
- Moving towards :
Denotes cooperative activities such as “offer a fairFootnote 7 division (little difference from the player’s minimum, i.e., 1 or 2 points)”; “Accept the other’s offer (whether it is fair or unfair)”.
- Moving against :
Annotations intended to describe movements in which the player took a firm stance in bargaining, such as “make an unfair offer (attempting to gain a large profit, i.e., more than 2 points)”; “refuse to concede to the other’s offer”; “engage in deception”; “Threaten the other (‘If you don’t do ...I will ...’)”
- Moving away :
Annotations corresponded to acts that conveyed refusal to take an action; adopting a passive stance; or avoiding the partner. Therefore, in this context we considered the following events as belonging to this category: “making a concession, that is, playing a card lower than the card the player wanted to play”; “using the ’give up’ card”; “not engaging in the negotiation”; “showing indifference to the other’s claims.”
Before the game sessions, the children completed a sociometric questionnaire and were administered a personality test. The former was applied mainly to ensure that children on opposite poles (neglected vs. popular) or children who did not like each other were not paired together, given the sensitive nature of this experimentFootnote 8, to avoid undesirable effects on the participants. The paired children were from the same class, hence they knew each other and shared a history.
For the experiment, each dyad was collected from their classrooms and bracelets to measure their electrodermal activity were immediately attached to their wrists (the results from this sensor will be discussed elsewhere). Then, the children were conducted to a room that had been made available for the purpose. The participants were sat face-to-face at the opposite ends of a table that had a card board to assist them throughout the game. After the rules were explained, the participants were “walked through” two rounds of the game to learn its mechanics (The pre-game sessions took, on average, 15 min). Subsequently, they were left alone to play the game.
To motivate the participants to do well, we told the players that the person who accumulated more points during the game would win a prize. In the end, both children won prizes, but the winner was able to choose between two options (one item was better than the other).
A game turn
A game turn began when one child opened the envelope containing his or her minimum for that round. The children were instructed to negotiate as they saw fit. When they reached an agreement they were told to turn over their MNS cards and mark their points on a piece of paper provided by the experimenter. In between rounds, the players had to fill in the information about their own points, the partner’s points, their expectations about the their partner’s MNS value and whether they wanted to take points from their partner (fictitiously) to penalise him or her for some action taken in the previous round.
Analysing the temporal dynamics of gaze
Successful decision-making depends on our ability to decode others’ emotions, intentions and beliefs . Humans’ non-verbal behaviours communicate these high-level constructs and allow others to monitor behaviour in real-time for effective cooperation . Gaze, as argued before, is linked to high-level cognitive information and may unveil the course of a person’s cognitive activities . As an attempt to disclose the cognitive processes behind the interactions in the “Game of Nines”, we look at relevant gaze cues in human-human interactions to disclose the motivations behind certain actions. We aim to understand how and when children reduce or increase competition and which processes serve to accomplish it. To that end, we take advantage of gaze descriptors (e.g., mutual gazes, number of face-directed gazes or average gaze lengths) and how these can characterise the nature (competitive/cooperative) of the interaction.
Exploring human-human gaze
Many theories of conflict establish the incompatibility of goals as the fundamental premise behind observations of conflict between dyads . Tomasello , however, argues that adults and children naturally cooperate to reduce competition and harmonise goals with their interaction partners. Thus, to conflict be observed, some breakdown in cooperation must occur. However, how can we discern whether children are in fact cooperating or competing in the setting we created?
In the domain of this experiment, the degree of cooperation or competition could have varied because the nature of the situation was not clear (mixed-motive). The structure created by the MNS values attempted to increase competition, but this is not guaranteed because competition is not sufficient to generate conflict, although it may be an effective igniter. In the current study, we explore how gaze can assist us in unveiling such moments during encounters.
Previous research suggests that gaze can signal a particular intention given that gaze descriptors (e.g., average length of gaze or mutual gaze) show different patterns in both cooperative and competitive environments . Cooperators look more and establish longer gazes and mutual gazes during encounters. Competitors in contrast, display shorter but frequent looks, and they tend to avoid mutual eye-contact. To explore the nature of the interactions in the children’s dyads we analysed these characteristics using the K-means clustering algorithm .
Data clustering is an unsupervised method that is used to explore the underlying structure of a data set. In our case, we attempt to create a partition between situations that appear to be cooperative or competitive, using individual gaze as a channel. Therefore, features were selected based on gaze dimensions that are effective descriptors of the nature of the situation (in ): (a) number of face-directed individual gazes (per second); (b) average length of face-directed individual gazes; (c) percentage of mutual gazes (per second); (d) average length of mutual gazes; and (e) number of individual glances (face-directed individual gazes of short duration,i.e., less than 1 s) in each encounter. Each round was considered an encounter, and thus, 99 cases (20 children, 5 rounds each, 1 case omitted) were analysed. All variables were normalised.
Figure 3 describes the relationships between the variables used to compute the clusters. The identified clusters clearly demonstrated the characteristics of competitive and cooperative settings as described above. The cooperative cluster encompasses 46 cases and the competitive cluster 53 cases. This information allows us to ascertain children’s motivation in a specific round. Using this case labelling as a scaffold, we analyse the temporal dynamics of gaze in specific game moments for a deeper understanding of the intentions behind each action. We consider three possible gaze states for the interaction analyses (Sect. 6.2.1), and we analyse them over time (Sect. 6.2.3) within specific time windows (Sect. 6.2.2).
To access the quality of the clustering partition we used NbClust R package Footnote 9 , which provides twenty-six cluster validation indices to aid the user decision about the number of clusters. Nine out of the 26 indices suggested that the partition into two clusters (against the seven indices indicating three clusters as the best option). To deeper our analysis we looked into five specific indices. A recent study by Arbelaitz et al.  presented a comparison of 30 cluster validation indices and they identified that five indices (Silhouette, Dunn’s, Calinski-Harabasz, Davis-Bouldin, COP Footnote 10, Sdbw) behave significantly better than others. Within those, in our data, the indices Silhouette and Calinski-Harabasz suggested that two clusters is a good partition. The former measures the cohesion of a cluster, therefore is a confidence indicator of the membership of the ith sample in a cluster . It returns a value in the interval [\(-\)1;1], and the closer to 1 the more cohesive is the cluster. The latter is a ratio between intra-cluster and inter-cluster variance, and the higher the ration the better the data partition.
Arbelaitz et al.  verified then when noise is present or clusters overlap the performance of the indices reduces. This also changed when the dataset’s configuration changed (e.g., dimensionality increased or clusters were not so dense). Therefore, they suggest that several indices should be used to quantify the “quality” of the partition. From Fig. 3 we can see that the separation between the two clusters is not very clear and they overlap. Increasing the number of clusters does not help on this issue, but do increases complexity in interpretation given the set of features. This raises some questions, which are discussed in Sect. 7.
Classification of gaze states
For each child, the gaze data were classified into three high-level behavioural states that, which were manually annotated (see Sect. 5.1 for more details):
Look-at-other, corresponded to face-directed individual gazes and was annotated each time a player was looking at the other’s face (4a).
Look-at-game described the time segments of the interactions during which each player was looking at his or her own cards or at the board that helped them during the game (4b).
Look-elsewhere characterised any occasion when the player was looking anywhere other than one of the two places described above. This could have been st the room’s walls or ceiling, or at the researcher, or it was when the player had a glazed look (4a).
From those, one other high-level behavioural states was automatically derived:
Mutual gaze was established when both players were looking at each other’s faces at the same time. This feature is for each pair of interacting children.
Actions that influence gaze
As was already stated, the time and context in which gaze is analysed are relevant. In particular, in terms of conflict dynamics, we are interested in four specific moments: (1) when the parties see their minimums and thus establish their goals and expectations for that negotiation round; (2) during the bid, in which they attempt to negotiate the most profitable deal. At this point, the social interplay could increase the potential for conflict; (3) when they reach an agreement or reject a possible agreement; and, finally, (4) when they reveal their minimums. Here, they perceived how exploited their partner was (or was not) during the round, and this point in the interaction was also a possibly critical moment for increasing conflict potential. During the bid (and in a response to the bid) different types of movements can occur, which correspond to the action tendencies described previously in Sect. 5.1. We consider that the player can move towards the other, move against the other or simply move away. Further analysis of the gaze patterns in these specific moments follows in Sect. 6.2.4. In that analysis we use the terms actor and target to describe the child that is taking one of the said actions and the child to whom the action is directed, respectively.
Gaze probabilistic profiles
To analyse the gaze patterns in different contexts with special considerations of time, we captured temporal gaze profiles across a related class of events (similar to Yu et al. ). In the current study, the relevant class of events is described in the previous Sect. (6.2.2). For example, the event See Minimum has the temporal profile depicted in Fig. 5. All instances of the event under analysis, were aligned at \(t=0\), based on the onset of each instance. The event is then analysed in three fixed temporal windows: before, during and after the event. The events last on average for one second. Each data point in the plot is the probability that the actor will be looking at his or her partner, the game or elsewhere (these are the three considered targets). Thus, each line in the plot represents the probability that the players will be looking at any of the three targets over a period of 3 s (1 s before the onset of the event and 2 s after the onset of the event). This approach allows us not only to have a dynamic picture of the gazes over time but also to aggregate behaviours and analyse general trends.
To statistically compare the gaze patterns over time we conducted a profile analysis. Because we only had 30 data points, time series analysis was not suitable for our purposes (and it would have been unnecessarily complex). Profile analysis is the multivariate equivalent of repeated-measure or mixed-measure ANOVA. It allows us to compare the same dependent variable between groups over multiple time points .
Gaze dynamics within the time-windows
When children open the envelope and see their minimums they build expectations for the upcoming round. Figure 5 shows a probabilistic model of gaze function during the times before, during and after that event. From Fig. 5, we can see that during this time span, children look more at the game (at the board or at the cards in their hands) than at the other player. Surprisingly, the players (with some exceptions) did not manifest their intention to understand what their partner’s MNS values might have been. From the analysis of the videos, gazing at the other during this time span served the purpose of coordination, either to open the envelope or to start negotiation. Initiating an interaction requires gaze sequences (including mutual gaze) . Occasionally, when their minimums pleased them, the children looked down to the game to hide their satisfaction.
Throughout the game, when the children bid, they used cooperative and competitive moves, which can be translated into the action tendencies already described in Sect. 5.1. Most of the work in social decision-making has focused only on the moving towards or against decisions, which represent cooperative and competitive moves respectively. The decision to moving away has never been studied, but it is an important component in social decision making under competitive situations, and therefore, it is considered in our study. The clusters described in Sect. 6.1 allowed us to divide our sample into two sets of cases that reflected a given player’s overall motivation during a certain round. Although we verify that certain gaze patterns fall into the competitive cluster, this does not mean that the player’s actions during the round are restricted to moving against or away from the other. In fact, the three different types of action tendencies can (and are) used in both contexts.
Moving towards Figure 6 shows a probabilistic model of gaze during an action with mostly cooperative intents in two different contexts: when the actor’s overall motivation is to compete or to cooperate. Conversely, Fig. 7 shows the other side of the interaction. The detailed eye shifts of a player with competitive or cooperative motivation when he or she is the target of a moving-towards type of action (it does not consider the actor’s motivation). In the cooperative cluster (cluster 2), the targets (of this type of action) look more at the actors of the action than they do in the competitive cluster (cluster 1), and the patterns are significantly different (\(F(1)=43.321, p=0.000\)). In cooperative settings, people gaze more to demonstrate readiness to help, sensitivity to common interests or orientation towards mutual power and gains . Because this is a ‘positive’ type of action, players may also anticipate positive reactions from others, and people do look more under such conditions . In cluster 1, the target of the action is more likely to be looking at the game (often at the cards he or she is holding), signalling a cognitive effort to better attend to planning and delivering of his or her decision (left-hand side of Fig. 7). The actors have similar gaze patterns in both clusters, but they show lower probabilities of looking at the other player (the target) and higher probabilities of looking at the game in cluster 1 (the difference is significant, \(F(1)=48.530, p=0.000\)). This is likely because it is difficult to make concessions.
Moving against The gaze patterns during moving against show a different type of interaction (see Figs. 8, 9). When the action is performed with cooperative intent, face-directed gazes are established by the end of the sentence to monitor the forthcoming reaction, similarly to what occurs during moving towards in cluster 2 (Fig. 6 right-hand side). Yet the actor’s gaze at the target is more prolonged during moving against for the same cluster (Fig. 8 right-hand side), and this difference is significant (\(F(1)=39.483, p=0.000\)). This may be because people gaze more when they are attempting to be more persuasive . Argyle argues that these higher levels of gaze, when individuals want to persuade others, are seen as credible and trustworthy. When moving against with competitive intent, players tend to avoid the their partners’ eyes (see Fig. 8 left-hand side) more than during moving towards (\(F(1)=970.894, p=0.000\)). This is likely because competitors find it difficult to look at their partners when they are asking for larger profits, because they know they are holding out for more. This pattern was also found in Foddy’s study . With regard to gaze patterns of the action targets they appear to be very similar during moving towards actions. Averting one’s gaze under the competitive intent may mask emotions or signal once again a cognitive effort to attend to decision-making. Previous research supports that gaze aversion is often used to block sources of distractions and allow reasoning . Yet action targets in cluster 2 (cooperative intent) look more at their partners than do those in the same cluster during the moving towards actions (\(F(1)=35.402, p=0.000\)). Gaze here may be acting as a social reinforcer , so that the actor begins to move towards the action target.
Moving away within the competitive set of actions, the moving away tendency is related to appeasement and avoidance (competitive) situations. In this case (Fig. 10), the data for the competitive cluster is not representative (there were only four cases). From the perspective of the action target, there is an increase in looking at the other, which may indicate surprise at said action, because the target’s goals are being fulfilled or because nearly every time a ’give up’ card was used, it had been agreed on between the two players. Regarding the actor of the action, number of gazes at the other increased, and it was higher than the number when the players moved against the other in cluster 2 (\(F=29.352, p=0.000\)). This could have been an act of defiance or simple coordination, because most of times that the ’give up’ card was used, it had been a joint decision.
Signs of agreement or disagreement are important social signals because they relate closely to the children’s goals. From a cognitive point of view, agreement occurs when there is congruence between both person’s mental states (in this case) . Nevertheless, an agreement may be truthful or unwilling, or it may mask disagreement (apparent agreement) . We found no relevant gaze pattern that allowed us to better understand the players’ mental states at the end of the rounds. Players look mostly at the game avoiding looking at the other player.
Another important moment in these interactions is when the children reveal their minimums. We argue that at this point, the underlying cognitive process is related to anticipation. From Fig. 12 we can see an increase in monitoring the others’ acts, for face processing, in anticipation of future behaviour. From another angle, most of the children may not look at the other when they fail to gain the desired profit , whereas the other does tend to look after revealing his or her minimum as a sign of satisfaction.
Signals are produced by a sender (S) and received/sensed and interpreted by a receiver (R) to communicate some goal or belief. S may send signals either to influence R or to provide information to R . A social signal such as gaze also follows this premise, which is used in our analysis.
In this setting, there is a structure of interference and interdependence between the agents/children that is characterised by the MNS values and the rules of the game (see Sect. 4.2). This structure creates the potential for conflict. According to Castelfranchi , an agent has two alternatives to address situations. Agents can adapt their own behaviour or attempt to change the other agent’s behaviour, depending on the nature of the interference (positive or negative). The action tendencies annotated for each video segment (see Sect. 5.1) fit this paradigm. In response to negative interference, agents aim to avoid the obstacles, and thus they merely coordinate by moving away. In the other hand, by employing moving against actions, agents may induce the other to abandon its goals. When presented with a proposal that could favour some of their goals, agents move towards the other by including the other’s actions into their plans to exploit the other individual. The use of these actions may increase or decrease the potential for conflict, and at the same time, it increases and decreases competition. Yet using these actions may depend on the signals that are sent jointly with the actions.
Gaze patterns facilitated identifying the players’ cooperative or competitive motivations throughout the rounds. In contrast to what was expected, the round that was more prone to conflict (Round 4) found higher gaze levels congruent with cooperative environments (see Fig. 13). The implications are twofold. On one hand, to coordinate, children seek more information from their partners, and gaze’s monitoring is active. On the other hand, children may be using face-directed gaze as a form of persuasion and we can not classify these cases as cooperative.
Unsurprisingly, face-directed gaze is not sufficient for understanding the subtleties of mixed-motive interactions, but it can guide further analysis. More gazes, associated with cooperative settings, characterised the last two rounds, in which the players employed more moving against actions (see Fig. 14). Were these interactions truly cooperative and gaze was serving to build a trustworthy environment? Or was gaze being used to induce the other to change his or her goal? This type of eye behaviour is revealed to be particularly important in these mixed-motive situations, serving to anticipate and adapt to any form of possible conflict the other players’ moves.
The data also note that players with competitive motivations do not look much at their partners and they focus more on the task at hand. Face-directed gaze increase when the time to reach agreement also increases, mainly when players are not able to coordinate (see Fig. 13). The behaviour of avoiding looking at the other player can be taken as an expression of avoiding conflict, which contrasts with the gaze behaviour in the cooperative cluster. In fact, Argyle et al.  note that people with high dominance look at the other more in competitive situations, whereas people with high affiliative needs look more in cooperative situations.
An interesting gaze pattern occurs when players withdraw from the negotiation, that is, when they move away. The data show that these players are more likely to be looking at the game (avoiding looking at the other), whereas their partners seek information by shifting their gazes to the player. This may indicate surprise or the need for confirmation, because the player’s goals are being satisfied.
From the analyses of the plots (Figs. 6, 7, 8, 9, 10, 11, 12), it appears unlikely that the gaze patterns only served the function of speech synchronisation. This suggests that the differences in gaze patterns are attributable to the differences in the players’ motivations and types of action they choose to employ. Additionaly, we found no correlation between gaze dimensions and children’s relationship or status variables.
Conclusion and future work
The experimental paradigm employed in this study attempts to create a natural setting that reduces real life to a mixed-motive game. There are several natural negotiation situations that resemble the created setting. For example, similar negotiation occurs every time we bargain for a price with a seller or when we ask someone to make concessions uncertain of what the other is willing to give . In previous research , we verified that children deal with similar issues on the playground. The paradigm is be same, although the the object of dispute changes. That creates an interesting parallel between adults and children when they are confronted with interpersonal conflict. Therefore, it would be interesting to explore if the same patterns of gaze occur between pairs of adults or among children from other age groups, and we hope that the framework presented here may foster future research on the topic. Our focus in this age range is because this study was part of a larger project, the SIRENFootnote 11 project, which aim was to create a serious game to teach about conflict and conflict resolution to children. Our aim is to understand a conflict episode’s underlying dynamics, in order to create more natural agents and ultimately, be more effective in teaching conflict and conflict resolution to children. To that end, we looked for small details in the interactions that could uncover how conflict unfolds.
Small pieces of information (behavioural cues) that describe social behaviour are critical for designing more natural experiences between agents and humans. More important than understanding which signals a person uses to communicate is the context and timing in which these signals are sent and received.
In real life interactions between humans, the way gaze cues are used is by no means irrelevant. Impressions of emotional states and interpersonal attitudes are also based on gaze cues . We can either receive or send information through gaze . Gaze cues carry a great deal of information related to a person’s beliefs about the interacting partner and even the affect towards him or her. In this study, we attempted to connect gaze micro-behaviours to high-level psychological constructs. We began by taking advantage of the gaze dimensions, which are useful descriptors of competitive and cooperative interactions, in order to divide our sample in two distinct sets. Then, we analysed the negotiation moves when the underlying motivation was cooperative or competitive. Cases in the competitive cluster showed, that players with this motivation tended to avoid the eye contact with their partners whether they were the target or actor of the action, irrespective of the type of action. This can be taken as an expression of avoiding conflict, which contrasts with the gaze behaviour in the cooperative cluster. Nadler  argues that face-to-face negotiation fosters rapport, which develops a willingness to cooperate among players. These children appeared to have developed rapport quite naturally which helped them in effective conflict resolution, particularly from round 4 onwards. Although this is a fair explanation, in future work we will explore other social signals, particularly to understand which signals accompany which action tendencies and when. These further analyses may establish whether the gaze dimensions we used were sufficient to characterise the nature of the interactions. For instance, a sign like smiling (and evaluations about whether they are fake or authentic) may help us to distinguish signs of appeasement. Other source of information about how the interactions unfold is the electrodermal signal collected (to be presented elsewhere).
Face-to-face negotiations have been studied before, to understand how people use non-verbal behaviours that could be exploited to automatically predict the respondent’s reactions  or the negotiation outcomes . In , Park et. al study the interpersonal dynamics of the negotiation by looking at the mutual behaviours that occur between the interactional partners. They created a model that merges visual and acoustic mutual features. The work by Nouri et al.  attempts to automatically predict the negotiators strategy and outcomes by looking into conversational structure and acoustic features. However, the temporal aspect of the negotiation is not captured.
Our current analysis based on context and timing facilitates the creation of models based on when and for how long gazing has been displayed by both parties. Furthermore, it also supports the derivation of meaning in a mixed-motive situations, which is valuable for the developing models of cognitive agents. Displaying timely and suitable signals is an ongoing challenge in the social signal processing community having many areas of application (from computer mediated communication to social robots) . For artificial agents, displaying appropriate social behaviour is crucial for more effective communication and natural interactions to convey and sense intentions and maintain natural conversations . This has an impact in applications that deal with sensitive information [24, 40] or are intended for training people  or teach them a skill.
From the perspective of design of agents (in terms of behaviour), we attempted to trace the interactional processes within a mixed-motive game by analysing only the dynamics of gaze. It is our stance that by understanding how processes unfold it will allow us to create agents whose behaviours evolve dynamically over time. For the field of human-agent interaction, the why and how of the intention that are conveyed is critical to developing better agent models that are more capable of producing natural experiences between agents and humans.
More information about the dataset in http://web.ist.utl.pt/joana.campos. It includes instructions on how to access it.
In this work, the term gaze is used to denominate eye-direction or eye-contact, only. It is important to note that we are interested in what a person is looking at in specific moments of the game, and not the direction of gaze per se.
This variation was introduced in one of Kelley’s experiments using this game .
The agreement was computed using the GSEQ software , since it offers a platform for computing Cohen’s kappa when coders first segment, denoting onsets and offsets, and then code the events according to the type of behaviour observed.
Here fairness is used as in the context of social dilemmas in game theory . We say that the players’ offers are fair when they choose to distribute the payoffs in a more egalitarian manner at personal cost. Throughout the rounds if a player attempts to get 1 or 2 points as a reward that will allow his or her partner to obtain a similar payoff.
The results from both questionnaires are beyond of the scope of this paper and are not going to be discussed here.
This measure was not in the used R package.
Abele, A. (1986). Functions of gaze in social interaction: Communication and monitoring. Journal of Nonverbal Behavior, 10(2), 83–101.
Arbelaitz, O., Gurrutxaga, I., Muguerza, J., Pérez, J. M., & Perona, I. (2013). An extensive comparative study of cluster validity indices. Pattern Recognition, 46(1), 243–256.
Argyle, M. (1988). Bodily communication (2nd ed.). London: Routledge.
Argyle, M., Ingham, R., Alkema, F., & McCallin, M. (1973). The different functions of gaze. Semiotica, 7(1), 19–32.
Bakeman, R., Quera, V., & Gnisci, A. (2009). Observer agreement for timed-event sequential data: A comparison of time-based and event-based algorithms. Behavior Research Methods, 41(1), 137–147.
Barki, H., & Hartwick, J. (2003). Rethinking interpersonal conflict. Technical Report. HEC Montréal.
Bolshakova, N., & Azuaje, F. (2003). Cluster validation techniques for genome expression data. Signal Processing, 83(4), 825–833.
Boucher, J. D., Pattacini, U., Lelong, A., Bailly, G., Elisei, F., Fagel, S., et al. (2012). Gaze effects in human-human and human-robot face-to-face cooperation. Frontiers in Neurorobotics, 6, 3.
Brock, G., Pihur, V., Datta, S., & Datta, S. (2011). clValid, an R package for cluster validation. Journal of Statistical Software (Brock et al., March 2008).
Campos, J., Martinho, C., & Paiva, A. (2013). Conflict inside out: A theoretical approach to conflict from an agent point of view. In Proceedings of the 2013 AAMAS.
Castelfranchi, C. (1998). Modelling social action for AI agents. Artificial Intelligence, 103(1), 157–182.
Deutsch, M., Coleman, P. T., & Marcus, E. C. (Eds.). (2006). The handbook of conflict resolution: Theory and practice. Hoboken: John Wiley & Sons.
Doherty-Sneddon, G., & Phelps, F. G. (2005). Gaze aversion: A response to cognitive or social difficulty? Memory and Cognition, 33(4), 727–733.
Emery, N. J. (2000). The eyes have it: The neuroethology, function and evolution of social gaze. Neuroscience and Biobehavioral Reviews, 24(6), 581–604.
Foddy, M. (1978). Patterns of gaze in cooperative and competitive negotiation. Human Relations, 31(11), 925–938.
Frith, C. D., & Singer, T. (2008). The role of social cognition in decision making. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1511), 3875–3886.
Glaholt, M. G., & Reingold, E. M. (2011). Eye movement monitoring as a process tracing methodology in decision making research. Journal of Neuroscience, Psychology, and Economics, 4(2), 125.
Hartigan, J. A., & Wong, M. A. (1979). Algorithm AS 136: A k-means clustering algorithm. Applied Statistics, 28, 100–108.
Heylen, D. (2005). Challenges ahead: Head movements and other social acts during conversations. In Proceedings of the Joint Symposium on Virtual Social Agents.
Heylen, D. (2006). Head gestures, gaze and the principles of conversational structure. International Journal of Humanoid Robotics, 3(03), 241–267.
Horney, K. (1945). Our inner conflicts: A constructive theory of neurosis. New York: WW Norton & Company.
Ingram, G., Campos, J., Hondrou, C., Vasalou, A., Martinho, C., & Joinson, A. (2011). Applying evolutionary psychology to a serious game about children’s interpersonal conflict. Evolutionary Psychology: An International Journal of Evolutionary Approaches to Psychology and Behavior, 10(5), 884–898.
Jones, T. S., & Bodtker, A. (2001). Mediating with heart in mind: Addressing emotion in mediation practice. Negotiation Journal, 17(3), 207–244.
Kang, S. H., Sidner, C., Gratch, J., Artstein, R., Huang, L., & Morency, L. P. (2011). Modeling nonverbal behavior of a virtual counselor during intimate self-disclosure. In H. H. Vilhjálmsson, S. Kopp, S. Marsella, & K. R. Thórisson (Eds.), Intelligent virtual agents. Lecture Notes in Computer Science (Vol. 6895, pp. 455–457). Berlin, Heidelberg: Springer.
Kelley, H. H., Beckman, L. L., & Fischer, C. S. (1967). Negotiating the division of a reward under incomplete information. Journal of Experimental Social Psychology, 3(4), 361–398.
Kriesberg, L., & Dayton, B. (2012). Constructive conflicts: From escalation to resolution. Lanham: Rowman & Littlefield.
Lee, J., Marsella, S., Traum, D., Gratch, J., & Lance, B. (2007). The rickel gaze model: A window on the mind of a virtual human. In Intelligent Virtual Agents, no. 4722 in Lecture Notes in Computer Science. Berlin, Heidelberg: Springer.
Macdonald, R. G., & Tatler, B. W. (2013). The effect of social roles on gaze cue utilisation in a real-world collaboration. In Proceedings of 35th Annual Cognitive Science Society Meeting (CogSci 2013).
Mller, H. J., & Dieng, R. (2000). On conflicts in general and their use in AI in particular. In Computational Conflicts: Conflict Modeling for Distributed Intelligent Systems, with Contributions by Numerous Experts. Springer.
Nadler, J. (2003). Rapport in negotiation and conflict resolution. Marquette Law Review, 87, 875.
Nouri, E., Park, S., Scherer, S., Gratch, J., Carnevale, P., Morency, L. P., & Traum, D. R. (2013). Prediction of strategy and outcome as negotiation unfolds by using basic verbal and behavioral features. In 14th Annual Conference of the International Speech Communication Association (INTERSPEECH 2013) (pp. 1458–1461). Lyon, France: ISCA Archive.
Park, S., Scherer, S., Gratch, J., Carnevale, P., & Morency, L. P. (2013). Mutual behaviors during dyadic negotiation: Automatic prediction of respondent reactions. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on (pp. 423–428). IEEE.
Pärnamets, P., Johansson, P., Balkenius, C., Hall, L., Spivey, M. J., & Richardson, D. C. (2013). Changing minds by tracking eyes: Dynamical systems, gaze and moral decisions. In Proceedings of 35th Annual Cognitive Science Society Meeting (CogSci 2013).
Pelachaud, C., & Bilvi, M. (2003). Modelling gaze behavior for conversational agents. In T. Rist, R. Aylett, D. Ballin, & J. Rickel (Eds.), Intelligent virtual agents. Lecture Notes in Computer Science (Vol. 2792, pp. 93–100). Berlin, Heidelberg: Springer.
Poggi, I. (2001). The lexicon and the alphabet of gesture, gaze, and touch. In A. de Antonio, R. Aylett, & D. Ballin (Eds.), Intelligent virtual agents. Lecture Notes in Computer Science (Vol. 2190, pp. 235–236). Berlin, Heidelberg: Springer.
Poggi, I., & DErrico, F. (2012). Social signals: A framework in terms of goals and beliefs. Cognitive Processing, 13(2), 427–445.
Poggi, I., & Francesca, D. (2010). Cognitive modelling of human social signals. In Proceedings of the 2nd international workshop on Social signal processing (pp. 21–26). ACM.
Poggi, I., & Pelachaud, C. (2002). Signals and meanings of gaze in animated faces. Advances in Consciousness Research, 35, 133–144.
Rehg, J., Abowd, G., Rozga, A., Romero, M., Clements, M., Sclaroff, S., Essa, I., Ousley, O., Li, Y., Kim, C., Rao, H., Kim, J., Presti, L., Zhang, J., Lantsman, D., Bidwell, J., & Ye, Z. (2013). Decoding children’s social behavior. In 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Rizzo, A., Forbell, E., Lange, B., Buckwalter, J. G., Williams, J., Sagae, K., & Traum, D. (2012). Simcoach: An online intelligent virtual agent system for breaking down barriers to care for service members and veterans. In Healing War Trauma: A Handbook of Creative Approaches (pp. 238–250). Routledge.
Robb, A., White, C., Cordar, A., Wendling, A., Lampotang, S., & Lok, B. (2014). A qualitative evaluation of behavior during conflict with an authoritative virtual human. In T. Bickmore, S. Marsella, & C. Sidner (Eds.), Intelligent virtual agents. Lecture Notes in Computer Science (Vol. 8637, pp. 397–409). Springer.
Schröder, M., Heylen, D., & Poggi, I. (2006). Perception of non-verbal emotional listener feedback. In Proceedings of Speech Prosody.
Sigmund, K., Fehr, E., & Nowak, M. A. (2002). The economics of fair play. Scientific American, 286(1), 82–87.
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. London: Pearson International Edition.
Tomasello, M. (2008). Origins of human communication. Cambridge: MIT press.
Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27(12), 1743–1759.
Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D’Errico, F., et al. (2012). Bridging the gap between social animal and unsocial machine: A survey of social signal processing. IEEE Transactions on Affective Computing, 3(1), 69–87.
Vliert, E Vd. (1997). Complex interpersonal conflict behaviour: Theoretical frontiers. Hove: Psychology Press.
Yu, C., Schermerhorn, P., & Scheutz, M. (2012). Adaptive eye gaze patterns in interactions with human and artificial agents. ACM Transactions on Interactive Intelligent Systems (TiiS), 1(2), 13.
This work was partially supported by the European Commission (EC) and was funded by the EU FP7 ICT-317923 project EMOTE and partially supported by national funds through FCT - Fundação para a Ciência e a Tecnologia, with reference UID/CEC/50021/2013. It was also supported by a scholarship (SFRH/BD/75342/2010) granted by FCT.
About this article
Cite this article
Campos, J., Alves-Oliveira, P. & Paiva, A. Looking for Conflict: Gaze Dynamics in a Dyadic Mixed-Motive Game. Auton Agent Multi-Agent Syst 30, 112–135 (2016). https://doi.org/10.1007/s10458-015-9282-8
- Interpersonal conflict
- Gaze dynamics
- Intelligent agent
- Social signal processing