Keywords

Introduction

Recently, the “social brain” (i.e., how the brain works in social context and the mechanisms for our social behavior) has gained focus in neuroscience literature – largely due to the fact that recently developed techniques allow studying different aspects of human social cognition and its brain correlates. In this context, hyperscanning techniques (Montague et al., 2002) open the horizon for human interaction studies, allowing for the evaluation of interbrain connectivity. These techniques represent methods for simultaneously recording signals from different brains when subjects are interacting. In this chapter, we will explore the potentials of functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS), which are techniques based on blood-oxygen-level-dependent (BOLD) signal. We will start with a brief explanation of the BOLD response basic principles and the mechanisms involved in fMRI and fNIRS measurements related to brain function. We will then discuss the foundation of the social brain, based on the first studies, with one subject per data acquisition, to allow for understanding the new possibilities that hyperscanning techniques offer. Finally, we will focus on the scientific literature reporting fMRI and fNIRS hyperscanning contribution to understand the social brain.

Hemodynamic Response and BOLD Signal

The relationship between neuronal activity and hemodynamic response is the basis underlying the blood-oxygen-level-dependent (BOLD) signal. One of the first models based on biomechanical properties related to blood volume, blood flow, and oxygen consumption was proposed by Buxton et al. (1998): the balloon model. This construct is a valid point to address how hemodynamics is related to neural function. Although local increase in metabolism related to neuronal activity increases the consumption of oxygen, the increase in supply of oxygen is higher than required by energy consumption needs. This is due to the fact that the transport from intravascular (hemoglobin linked) O2 to intraneuronal space depends on passive mechanisms related to differences in pressure gradients. We have also to add to that fact the increased blood volume and flow in the dynamics of the process. This equilibrium evolves over time and generates a small initial decrease in oxyhemoglobin/deoxyhemoglobin proportion, followed by a strong increase of this ratio. When the temporal dynamics are considered in this equation – and also based on the observations – the BOLD response associated with neuronal activity is a slow response that reaches its peak around 6 s after a stimulus. Moreover, this neurovascular coupling is associated with several pathways, including neuronal release of vasoactive mediators (e.g., nitric oxide), and pathways related to calcium activity in astrocytes (for reviews, read Longden et al. (2016) and Filosa et al. (2016)). Thus, the BOLD response evaluated with fMRI and fNIRS is an indirect measure of neural activity.

How Functional Magnetic Resonance Imaging Detects BOLD

The blood-oxygen-level-dependent (BOLD) signal was detected in MRI by Ogawa et al. (1990) and was first described in human brains in 1992 (Bandettini et al., 1992; Kwong et al., 1992; Ogawa et al., 1992). For a detailed history of the development of fMRI, we recommend the review article by Bandettini (2012), which celebrates 20 years of the technique. The detection of BOLD signal was possible because the hemoglobin has different magnetic properties when it is oxygenated, due to conformation changes related with the iron-oxygen binding site. While oxyhemoglobin is diamagnetic (low interaction with magnetic induction), deoxyhemoglobin is paramagnetic and generates a distortion of the magnetic field around it. A distortion in the magnetic field causes a faster decrease in the hydrogen nuclear resonance signal. Higher concentrations of oxyhemoglobin relative to deoxyhemoglobin allow a more stable local magnetic field and therefore more signal. Hence, fMRI is a technique that measures relative signals based on the oxyhemoglobin and deoxyhemoglobin proportion. This highlights the importance of baseline control conditions for fMRI experiments as it is not an absolute measure; it is a relative measure. fMRI has a whole-brain coverage with good spatial resolution (in the order of millimeters, and can reach submillimetric resolution using ultrahigh field MR systems), however a relatively lower temporal resolution (in the order of seconds, mainly due to the slow temporal hemodynamic response, although MR systems are capable of acquiring data in the order of hundreds of milliseconds). The precision of MRI is related to gradients generated in the magnetic field, which alter the specific radio frequency absorbed by hydrogen nucleus. Therefore, it is highly sensitive to movement and requires participants to lay down inside the scanner. During fMRI, volunteers enter the scanner bore, a tunnel large enough to host a human body – and as such are not a natural environment, but rather may induce claustrophobia – and image acquisition depends on a head coil to detect the resonance signals. Also, there are several restrictions or exclusion criteria for participating in fMRI experiment, due to the intense magnetic field, such as pregnancy, pacemakers, magnetic prosthesis, tattoos (depending on the pigment used), and other situations that might induce risk for the participant.

How Functional Near-Infrared Spectroscopy Detects BOLD

The BOLD signal in the human brain was detected using fNIRS around the same period it was observed using fMRI (Hoshi & Tamura, 1993; Chance et al., 1993; Kato et al., 1993; Villringer et al., 1993). Therefore, similar celebrative reviews detail the history of fNIRS development (Ferrari & Quaresima, 2012; Scholkmann et al., 2014). As mentioned above, the oxygen bond to hemoglobin causes a conformational change which alters electromagnetic properties of the molecule. fNIRS depends on changes in light abortion in different near-infrared wavelengths related to oxygen bond to hemoglobin. Though different absorption rates of oxyhemoglobin and deoxyhemoglobin are observed in several parts of the electromagnetic light spectrum, near-infrared light is less absorbed by the skull and other tissues between the cortex and the scalp. In this way, fNIRS can be used to evaluate separately the cortical concentration of oxyhemoglobin and deoxyhemoglobin. For these measures, fNIRS uses optodes (similar to electrodes but with optical properties) as sources of light and as detectors of the light that is scattered through the brain tissue. A combination of source and detector forms an fNIRS channel (one source and four detectors could form four channels) located between the optodes. Duo to light scattering and absorption, fNIRS can only detect signal a few centimeters below the scalp, providing mainly cortical signal. It is also important to notice that tissue transparency to light depends on age, in a way that the skull in babies is more transparent than in adults. The recommended distance between optodes for adults is between 2.5 and 4 cm. Positioning optodes closer together (e.g., 0.8 cm) can be used to detect and later filter hemodynamic processes unrelated to local brain activity (Brigadoi & Cooper, 2015). Usually, these optodes are attached to a cap that follows the 10–20 coordinate system (and its variations) of electroencephalography (Jasper, 1958; Oostenveld & Praamstra, 2001). As an advantage, fNIRS can be portable and is less affected by movement, being more suitable for ecological and naturalistic studies. Also, since the volunteers can move to a certain degree, it is more suitable for studies with babies and young children. The temporal resolution of fNIRS systems depends on the number of sources used, since each source has to be turned on separately to avoid mixing signals from different regions in the detectors. Usually, the sampling rate can vary around 4 Hz to 60 Hz, providing fNIRS with higher temporal resolution than fMRI. This is useful for correction of cardiac artifacts, given the higher sampling rate diminishes aliasing artifacts, which are present in fMRI data. On the other hand, fNIRS has lower spatial resolution (in the order of centimeters) and does not have a whole-brain coverage (restricted to cortical signal), and the number of optodes available defines the cortical coverage level of the system.

Types of Experiment Design and Data Analysis for BOLD Studies

In task-based designs, fMRI and fNIRS can be used to identify regions with BOLD signal variation related to task variation (from a baseline control condition to the task of interest), based on BOLD response after a stimulus (event-related design), or due to a block of stimulus (block design). In these types of design, it is important to consider stimulus (or task) sequence, duration, number of repetitions, time between stimulus, and the hypothesis of which brain regions will be related to the task in order to define the design that will provide more statistical power for a general linear model analysis (the most common analysis in this context). For a review of study design, we suggest Amaro Jr and Barker (2006).

Alternatively, there are designs in which the subject sustains a brain state either by continuously performing a specific task or by remaining in resting state (with no specific task, only with the instruction to remain awake and not focus in anything in particular). These designs are used in connectivity studies, which explore signal relation between different brain regions and explore brain organization and communication between areas. For example, resting-state studies allowed the identification of intrinsic brain networks (Greicius et al., 2003; Fox et al., 2005; Damoiseaux et al., 2006). There are several connectivity measures, and they can be applied both to resting-state (Han et al., 2018) and to task-based (as event-related and block) studies as well (Friston, 2011). Some measures are data-driven, like independent component analysis, while others depend on previous hypothesis-driven models, as in the case of dynamic causal modeling. Even more simple calculations, as correlation index, can be used to study organization of brain networks. Graph theory can be applied using these connectivity measures to evaluate the characteristics of these networks.

Hyperscanning Design and Data Analysis

The term hyperscanning was first used by Montague et al. (2002) referring to measuring brain signal from interacting humans. In their study, two synchronized fMRI scanners were used to measure brain activity while pairs of volunteers interacted in a competitive game. Since then, hyperscanning has been performed with several techniques, such as electroencephalography (EEG), magnetoencephalography (MEG), fMRI, and fNIRS (Babiloni & Astolfi, 2014; Zhdanov et al., 2015; Wang et al., 2018). Different from acquiring data from subjects separately, hyperscanning offers the possibility of relating brain activities from different subjects preserving trial-specific characteristics. In other words, even though having a pair of volunteers perform the same task twice while measuring one subject at a time would provide a way of comparing brain activities, the data from each subject would have different specific trial characteristics as performance score or event-specific strategy or brain state during the trial; on the other hand, with hyperscanning, these are preserved, providing more information on brain signals during interaction. Also, hyperscanning can be used to expand the concepts of connectivity analysis from within to between brains, revealing more than what areas have more signal during interaction but also how these areas from different brains coordinate their activity.

There are some important technical aspects to consider when doing hyperscanning:

  • Synchrony between equipment. In order to be able to take advantage of simultaneous recording, it is necessary to have precise synchrony of recordings from the different volunteers. As an example, if there is asynchrony or a lag between recordings, then correlation measures would be shifted; in case of causation methods, this could significantly alter the interpretation of leader and follower. It is also important to notice that sampling rate and the type of signal measured directly influence the required precision of synchrony. In fMRI, usually with whole-brain sampling rate of 2 s (and around 50 ms per slice), a lag of 50 ms (one slice) might be acceptable, also taking into account the slow hemodynamic response; in an EEG setup with 1000 Hz sampling rate measuring fast electric changes, 50 ms would represent a lag of 50 data points, and would not be acceptable. It is also important to notice that if the lag is constant, it can be corrected during analysis by shifting the time series appropriately. However, if the lag is variable and with no clear pattern, then it probably will not be correctable. Intranet- and Internet-based synchrony solutions were developed to allow for triggering MRI scanners in different buildings (Montague et al., 2002). Also, there are software solutions such as lab streaming layer (https://github.com/sccn/labstreaminglayer; Gramann et al., 2014; Ojeda et al., 2014), which allows for synchronizing fNIRS and other types of signals received by a computer. Another option would be to use only one system to get the data from two or more subjects (see Fig. 14.1 for hyperscanning options).

  • Equivalent measures in each equipment. Hyperscanning requires having the same signal quality control routines for all systems used to assure equivalent measures in all participants. This is highlighted in MRI scans, since there is a great variability of sequences and parameters available in different scanners. Therefore, having the same version of scanner for each subject can help assuring comparable measures. A usual alternative in fNIRS is to split the optodes from one system between subjects, so equivalent quality control and synchrony are guaranteed. In the case of MRI, a similar solution would be to use one scanner with a dual-head coil (Lee et al., 2010, 2012; Lee, 2015a, b), although this would have implications for comfort and in the type of interaction between participants.

  • Controlling prior interaction and types of interaction between participants. A few studies have evaluated that the level of relationship (strangers, friends, romantic relationship) could alter the level of interbrain connectivity, with lovers presenting more connectivity (Pan et al., 2017). Therefore, controlling how well participants know each other may be important. Also, during the hyperscanning protocols, participants can try to interact in different ways, for example, a pair might try to use verbal communication, while other might choose nonverbal if no specific instruction is provided. Moreover, eye contact can alter interbrain synchrony (Hirsch et al., 2017; Koike et al., 2019). Since different brain mechanisms could be participating in each type of interaction, it is important to prepare your experiment design and instruction to participants in order to select the type of interaction that is important for your study. Regarding interaction paradigms, they can be classified according to three axes: goal of interaction, divided in cooperative and competitive interactions; temporal structure and sequence of actions in time, which can be turn-based (as observed in chess or card games) or continuous interaction (e.g., coordinated singing); and dependency of actions, which can be in independent (usually in competitions) or interdependent (Liu & Pelowski, 2014). Decision-making in social context, as in the game theory paradigms (Wang et al., 2015), are good examples of interaction paradigms. We will further explore brain mechanism in some of these types of interaction in the section called “Hyperscanning Studies of the Social Brain.”

  • Having a good control condition. Since one of the main goals in hyperscanning is to measure interbrain connectivity and relating it to a specific task or condition, a proper control condition is required. For example, two volunteers receiving the same stimuli or performing the same task (e.g., watching a movie), with the same temporal structure, would have a level of synchrony between their brains, even if these data were recorded one subject at a time (Hasson et al., 2004). As we will discuss below, some studies used this property to study the social brain, recording one participant at a time, but repeating the social task, using recorded videos from one session in the other (Schippers et al., 2009; Lee et al., 2018). To filter this signal similarities or to have an adequate baseline condition, the control task should have all the same cognitive elements and with the same intensity as the main task, except for the social aspect on which the research is focusing.

Fig. 14.1
An image A has 2 brains facing each other at the bottom 2 data acquisition computer are on the top with a line extending from it and reaching fNIRS that branches into multiple wires and plugged into the brains. A flowchart is between the computers for synchrony devices. Image B has a computer, fNIRS branching into wires plugged into 2 brains.

Hyperscanning options: (a) using one data acquisition device for each subject and using synchrony devices to assure synchronized data acquisition; (b) sharing the same data acquisition device for all the subjects and providing synchrony between data from all subjects but limiting distance between subjects and reducing the number of channels (fNIRS). For schematic representation, fNIRS was presented in the image, but similar considerations are valid for fMRI hyperscanning. LSL lab streaming layer

Full size image

The Social Brain: Empathy, Theory of Mind, and Mirror Neuron System

This section will provide some key theories about brain mechanism relevant for social tasks. For this, the following text will concentrate on the first approaches used to unveil the brain mechanisms related to social interactions and build the ground for understanding the advantages that hyperscanning techniques offer for studying the social brain, which will be covered in the next section.

The idea of the social brain refers to the brain mechanism related to social skills, and cognitive processes important for social interaction, as motor coordination (joint action or imitation), affective empathy, and theory of mind (ToM – also called mentalizing). ToM refers to the capacity of understanding perspectives and beliefs from other people (and maybe animals) or, in other words, to the ability of attributing a mental state to others and to oneself as well. The first study to use the term “theory of mind” was evaluating the possibility of ToM in chimpanzees (Premack & Woodruff, 1978). Later, using positron emission tomography, Fletcher et al. (1995) studied theory of mind in humans, related to stories that had a mind state explanation for a character action compared to simply physical causality stories, finding increased activity in left medial frontal gyrus, anterior cingulate gyrus, and posterior cingulate gyrus. The same idea of stories but with different modalities was also used by Gallagher et al. (2000), identifying increased response in medial prefrontal cortex and temporoparietal junction (TPJ). The TPJ is now one of the main regions associated with the social brain, as the medial prefrontal cortex is related to ToM (Frith & Frith, 2003). Based on these results, fNIRS studies usually focus on TPJ and prefrontal regions in the context of ToM, since most fNIRS studies do not have enough optodes for a whole-head coverage. In adults, it was proposed that even without specific instruction (in a video story task), TPJ is engaged and related to spontaneous detection of others’ beliefs (Hyde et al., 2015). Also, the emotional state of the participant seems to affect lateral prefrontal cortex activity during ToM director task, in which the volunteer has to consider the perspective of another person (Himichi et al., 2015). However, evaluation of cognitive empathy and affective empathy in children is very challenging. In these cases, fNIRS inherent characteristics enable specific designs in children. For instance, a study using cartoon stories and verbal stories in 4–8-year-old children showed involvement of medial orbitofrontal regions, as well as dorsolateral prefrontal cortex in these tasks (Brink et al., 2011). In addition, other authors observed greater TPJ relationship with belief detection compared to desire intention in 6–10-year-old children (Bowman et al., 2015).

Another part of the social brain is identifying motor intentions and coordinating motor action (as in imitation). Evaluating mechanisms of imitation with fMRI, the inferior frontal gyrus and the superior parietal lobule were found to be related both to performing a finger-tapping and to observing the finger-tapping, and even more intense activity was found during imitation of movement (Iacoboni et al., 1999). These results were later associated with the mirror neuron system (MSN, Grèzes et al., 2003; Iacoboni et al., 2005; Jeon & Lee, 2018 for a review), which is related to internal representation of motor intentions from others and is formed by inferior frontal gyrus, inferior and superior parietal lobule, borders of superior temporal sulcus, and premotor cortex. The mirror neuron system was also identified with fNIRS in a more naturalistic table-setting task, which involved observation and execution (Sun et al., 2018). The mirror neural system is usually associated with mentalizing, as simulation-based system (Gallese & Goldman, 1998; Frith & Frith, 2006; Mahy et al., 2014), but there is still current debate to define the specific function and limit of each system in each social context and how these systems interact. Canessa et al. (2012) studied human subjects observing pictures with cooperative context (two persons caring an object) and pictures with affective context (two persons holding hands) using fMRI. These authors observed that both conditions seemed to engage the temporoparietal junction, but the ventromedial prefrontal cortex was more related to affective scenes, while inferior frontal gyrus and inferior parietal lobule were more associated with cooperative images. Moreover, the signal in these regions was related to empathy scores of participants. In a posterior study, the effective connectivity of these regions was evaluated indicating significant connectivity in these regions, but with different directionality according to the type of picture (Arioli et al., 2018).

Receiving and processing social feedback are also another important part of the social brain. In an fMRI study, the brain regions related to social influence on rating of emotional images were evaluated showing participation of the prefrontal cortex, borders of superior temporal sulcus, amygdala, and insula (Lin et al., 2018). In this study, participants rated images before and after seeing the average rating given by (simulated) group members, to check the social influence and induced rating adjustment. Another fMRI study evaluated positive and negative feedback of personal traits simulating a hyperscanning competitive group situation, but the feedbacks were predefined by the experimenter, and only one subject was actually being evaluated at a time (Dalgleish et al., 2017). In these experiments, an increased BOLD signal was observed in the ventromedial prefrontal cortex related to positive feedback, while both positive and negative feedback elicited responses in the anterior cingulate cortex and amygdala.

The aforementioned studies had a similar design in the sense that they used single subject tasks to study specific cognitive function relevant for interaction (Fig. 14.2a). However, this method has limitations, as it does not measure the brain signals during an actual interaction. As a first option to overcome this limitation, it is possible to evaluate the effect of an actual interaction on the brain activity of a subject (Fig. 14.2b). For example, Chauvigné et al. (2018) compared brain activity of professional dancers in different types of hand interactions (in a type of joint action), comparing leading and following brain activities and finding activity in the inferior frontal gyrus and premotor cortex during leading and in the ventromedial prefrontal cortex and borders of superior temporal sulcus in following. Rauchbauer et al. (2019) compared brain activity of subjects during conversation with another human to the activity during conversation with robots and observed higher temporal activity in human-human interactions. These results may indicate a starting point to understand human social cognition, as well as the social competence of robots interacting with humans.

Fig. 14.2
A diagram of social brain experiments. Diagram A is the intra-brain analysis and single tasks, diagram B is the single subject brain signal acquisition and multi-subject task, and diagram C is the intra-brain analysis and multi-subject task.

Types of social brain experiments and analysis possibilities. (a) Single subject brain signal acquisition; single subject task. (b) Single subject brain signal acquisition; multi-subject task. (c) Multi-subject brain signal acquisition; multi-subject task

Full size image

Some studies also tried to compare the brain activity of participants that were scanned in different sessions, by recording a video of the first session and using it in the second session with the other volunteer. Lee et al. (2018) evaluated mother-child brain signal similarities during stress condition for the adolescent (the video of the child was recorded, and the mother was scanned observing the video of the adolescent during this stress condition), finding family relationship level impacting similarity in the insula and anterior cingulate cortex. Schippers et al. (2009) also used this video recording technique and found mirror neuron system and TPJ activity during decoding of gestures in a charades game.

Although the scientific evidence covered in the above paragraphs provided the basis for main theories proposed to explain social interaction, they were based on experimental designs unable to probe the relationship between neural systems during dynamic interpersonal interaction. This is necessary to understand how interacting brains regulate their function based on the other person’s behavioral responses.

Hyperscanning Studies of the Social Brain

Key aspects of correlation between brain signals and also brain activity in real interactions can be better evaluated in hyperscanning setups (Fig. 14.2, panel c). A review of hyperscanning with fNIRS reported 20 fMRI and 7 fNIRS hyperscanning studies published up to spring of 2013 (Scholkmann et al., 2013). Based on PubMed and Web of Science search using keywords Hyperscanning and fMRI and Hyperscanning and fNIRS, we found 14 fMRI hyperscanning research articles from the beginning of 2014 to March of 2019 and 33 fNIRS hyperscanning articles, showing the increased applications of these techniques, specially fNIRS hyperscanning. It is important to mention that our literature search found a total of 23 fMRI (9 up to spring of 2013) studies and 40 fNIRS studies, indicating that the search parameters used by Scholkmann were different from ours – perhaps not only due to time differences – especially regarding fMRI hyperscanning. The following subsections will explore hyperscanning experiments with each technique.

fMRI Hyperscanning

The first study using the term hyperscanning was performed using two 1.5 T MRI scanners synchronized by one server through the Internet, with latencies below 300–400 ms, to show the feasibility of hyperscanning studies. After that, the technique was adopted to explore different aspects of social interaction in combination of 1.5 and 3 T scanners (Saito et al., 2010; Krill & Platek, 2012; Fliessbach et al., 2012; Tanabe et al., 2012; Spiegelhalder et al., 2014; Stolk et al., 2014), two or more 3 T scanners (Tomlin et al., 2013; Morita et al., 2014; Trees et al., 2014; Bilek et al., 2015, 2017; Koike et al., 2016, 2019; Shaw et al., 2018; Špiláková et al., 2019; Abe et al., 2019), and also combining 3 T and 7 T scanners (Baecke et al., 2015). Alternatively, Ray Lee and colleagues proposed performing hyperscanning with a dual-head coil designed for hyperscanning studies (Lee et al., 2010, 2012; Lee, 2015a, b). This would provide a good solution for synchrony issues and sequence parameters. As a drawback, an MRI system is already quite small for a single subject, so sharing this little space inside the scanner with another person can be uncomfortable. Other examples of hyperscanning implementation include virtual reality (Trees et al., 2014) and using a brain-computer interface system (Baecke et al., 2015).

Although there are several variations in fMRI hyperscanning techniques, one of the most mentioned in the literature is the eye-cued joint action. In this approach, one of the volunteers has to guide his gaze by the gaze of the other participant. This simple task circumvents some difficulties faced by other interaction tasks in fMRI environment, mostly related to restrictions of body movement. These studies found higher activity during eye gaze cued tasks in the inferior frontal gyrus, occipital cortex, anterior cingulate gyrus/medial prefrontal cortex, temporal cortex, and borders of the superior temporal sulcus (Saito et al., 2010; Tanabe et al., 2012). Moreover, higher interbrain cross correlation, after filtering task effects, was found in the right inferior frontal gyrus (Saito et al., 2010), and this connectivity was diminished in pairs in which one of the volunteers had autism (Tanabe et al., 2012). Interbrain connectivity in the inferior frontal gyrus was also observed during simple mutual gaze, either days after a joint action task (Koike et al., 2016) or without joint attention task execution, but closer to the insular cortex in this case (Koike et al., 2019). With a different approach during eye-cued joint attention mutual gaze, using independent component analysis and evaluating the relation between components involved in the task from each volunteer of the pair, higher interbrain connectivity was detected in the right temporoparietal junction (Bilek et al., 2015). The normal connectivity pattern was also disrupted in patients with borderline personality disorder (Bilek et al., 2017). These examples illustrate the potential of hyperscanning in the context of disorders that affect social interaction (Ray et al., 2017).

Several other social interaction paradigms were evaluated with fMRI hyperscanning, such as joint force, used to study cooperation and motor coordination (Abe et al., 2019), finding higher interbrain connectivity in the right temporoparietal junction, and right temporoparietal junction signal increase during task was related to performance scores. Both Spiegelhalder et al. (2014) and Stolk et al. (2014) have evaluated verbal and signal communication (respectively) and found a similar result: brain signal synchrony in temporal lobes either to areas related to talking, in verbal communication, or to the pair temporal lobes, in case of signal communication.

Game theory tasks are also used in hyperscanning experiments. For instance, the ultimatum game is a turn-based game in which a proposer chooses a proportion of reward distribution between participants and then the responder has to decide either to accept the proposal, and the reward is distributed between participants as agreed, or to reject it, in which case none of the participants receive a reward. Using this type of paradigm, Fliessbach et al. (2012) found striatum and ventromedial prefrontal cortex activity in both participants in more generous proposals and in higher level of acceptance by responders. Another interesting finding is the anterior/middle cingulate cortex interbrain connectivity, which was correlated to reciprocity of the proposer, possibly related to judgment of proposals in situations with advantageous and disadvantageous inequities (Shaw et al., 2018). An important mechanism involved in the social brain decision is the reward mechanism, which seems to be associated with increased BOLD signal when the pair worked together to complete a task, as indicated in a study that used a maze task in which one of the participants saw the maze and gave instructors to the other, which had to drive through the maze without directly seeing it (Krill & Platek, 2012). In a group hyperscanning with groups of five participants at a time, the effect of social influence (provided by feedback about other participants’ decision) on decision indicated that insular response was higher when individual decisions were different from other participants and also predicted the tendency of realignment to the group in the next decision. This insular activity could be related to embarrassment, as suggested by a study that evaluated self-face recognition while being observed by others, which also detected intra-brain connectivity between the anterior cingulate cortex and the dorsomedial prefrontal cortex when being observed (Morita et al., 2014).

As previously mentioned, social interaction tasks can have different structures related to temporal structure of the task, goal of task, and dependency of actions. These can evoke different neural systems, as evaluated by Špiláková et al. (2019). They studied goal and temporal structure effects with a pattern game in which a builder player had to form a pattern with disks in a virtual game table, while other participants could cooperate or try to avoid reaching the pattern (goal effect). In one condition, the players responded simultaneously, while in the other the builder started the turn-based interaction. Higher BOLD signal was found during cooperative tasks in the ventromedial prefrontal cortex, superior and middle temporal gyri, and orbitofrontal cortex, while competitive tasks were related to the dorsolateral prefrontal cortex, supplementary motor area, insula, and cerebellum. Also, simultaneous tasks had higher activity in the temporal lobe, insula, and motor areas. These highlight the importance of choosing the appropriate paradigm according to the desired interaction mechanism to be studied.

Together, these studies exemplify the possibilities of fMRI hyperscanning, using its high spatial resolution to try to identify specific roles for areas within a system. On the other hand, fMRI offers several restrictions for experiment design, which limit more naturalistic tasks.

fNIRS Hyperscanning

The first hyperscanning study using fNIRS technique involved joint action (Funane et al., 2011) in a task in which participants have to synchronize their motor response (usually a button press) after a stimulus. This task and small variations of it are probably the most used paradigms in hyperscanning fNIRS studies because they offer a simple model of interaction. The control condition in these studies usually is a competitive condition, in which the volunteers compete for the faster response after a signal to perform the movement. Using this kind of paradigm studies found increased prefrontal connectivity between brains associated with better performance (Funane et al., 2011) and similar results in the superior frontal cortex (Cui et al., 2012). The coherence between pairs in the right superior frontal cortex was affected but level of intimacy, when comparing lovers, friends, and stranger pairs, with lovers presenting higher coherence and better performance (Pan et al., 2017). Also, the gender of the pairs seems to affect interbrain connectivity and performance, with male-male pairs having better synchrony performance, although contrasting opposed interbrain connectivity results were found in different studies (Cheng et al., 2015; Baker et al., 2016). Interestingly, controlling the feedback after trials and runs could modulate performance and interbrain connectivity, with better performance associated with higher coherence in the superior prefrontal cortex and dorsolateral prefrontal cortex (Cui et al., 2012; Balconi et al., 2018). This effect was also evaluated from mother child dyads, also finding right dorsolateral prefrontal cortex interbrain connectivity related to joint action (Reindl et al., 2018), and different brain mechanisms might be associated with child gender (Miller et al., 2019). Simulated positive feedback could also alter connectivity and performance even in a competitive task (Balconi & Vanutelli, 2017).

A small variation of this task is the joint finger-tapping, in which participants have to synchronize finger-tapping movements, with the control conditions being metronome synchronization. Studies found increased interbrain connectivity in the right prefrontal cortex (Dai et al., 2018) and in the medial prefrontal cortex, when performing source-based analysis (Zhao et al., 2017). Adding an imitation component to the task with a definition of leader and follower to determine the finger-tapping rhythm, higher granger causality from leader to follower was detected in the left premotor cortex (Holper et al., 2012). Prefrontal cortex cortex involvement as in the studies mentioned above was observed by using the joint n-back task, a different type of joint task in which pairs performed dual n-back task (each participant in charge of a n-back) together (Dommer et al., 2012). These studies highlight the possibilities of joint action interaction and finger-tapping tasks, although they might be too associated with motor mechanisms, and therefore other types of tasks could be important to further explore the social brain.

A game theory paradigm with the ultimatum game was also evaluated with fNIRS hyperscanning indicating higher coherence between right temporoparietal junctions when performing the task face-to-face (Tang et al., 2016), probably associated with higher ToM. This is in agreement with studies with eye contact by itself, which can alter interbrain coherence in temporal regions (Hirsch et al., 2017). Face-to-face interaction also has an impact in communication mechanisms, presenting higher hyperconnectivity (interbrain connectivity) in the left inferior frontal cortex than during back-to-back dialogues and even than face-to-face monologues (Jiang et al., 2012). In a group communication, left temporoparietal junction hyperconnectivity distinguished a leader-follower pair, as opposed to two followers during group communication. In a four-group word game, synchrony was observed in frontopolar regions (Nozawa et al., 2016). There are several explanations for the difference in location of the synchrony since the studies used different fNIRS optode positioning and the communication tasks were different, as well as artifact handling during analysis.

In turn-based game interactions, poker game adaptation indicated the importance of temporoparietal junction for ToM when comparing human-human against human-computer competitions (Piva et al., 2017). TPJ was also related to higher-risk decision, which is assumed to engage more mentalizing due to more careful evaluation of the opponent (Zhang et al., 2017). This study further suggested there might be a gender difference in high-risk situations, with women presenting higher TPJ hyperconnectivity than men. Pattern game studies found higher mentalizing during competition and therefore increased hyperconnectivity in the inferior parietal lobule and further different inferior frontal gyrus participation, while borders of the superior temporal sulcus showed hyperconnectivity during both competitive and cooperative games (Liu et al., 2015, 2017). On the other hand, comparing obstructive and cooperative Jenga game indicated more hyperconnectivity in the right dorsolateral prefrontal cortex during cooperation (Liu et al., 2016). Therefore, competitive characteristics in some task-based games are completely different from competition control conditions in joint action or in the Jenga obstruction example, due to different strategies and different mentalizing requirement in each type of competition. These results suggest that rather than unified cooperation and competition mechanisms, different mirror neuron system, empathy, and ToM mechanisms might mediate cooperation and competition according to specific task design.

Working together to solve problems is a common situation in social interaction, and it is related to creativity. Studies with realistic resented problem task, in which participants have to provide as many solutions as possible to solve a realistic problem, showed higher hyperconnectivity in the dorsolateral prefrontal cortex and in the right temporoparietal junction associated with higher cooperation, either comparing to a competitive task or when evaluating creativity levels of dyads and also in the context of an experimenter acting as a participant and providing feedback for the ideas proposed by participants (Xue et al., 2018; Lu et al., 2018, 2019; Lu & Hao, 2019). Interestingly, when comparing creativity levels between pairs, low creativity individuals formed efficient dyads in cooperation with high interpersonal neural synchrony and good performance in the task, while other pairs with at least one participant classified as creative did not show the same cooperation and connectivity. Also, positive feedback enhanced interaction and cooperation, and negative feedback seemed to disrupt interaction.

Considering other close to real-life tasks, teacher and student interactions were evaluated showing higher interbrain connectivity in the left prefrontal cortex when information was transferred from teacher to student (Holper et al., 2013). Also, a feasibility study suggested that it is possible to perform the teacher-student experiment with a child student, finding student’s prefrontal signal related to teacher’s TPJ (Brockington et al., 2018). The same article explored the possibility of recording data from four students in a class and showed hyperconnectivity between students when they were paying attention to the teacher. Last, they showed that it is possible to perform these measures also combining with eye-tracking information. In another example of a real-life situation that can be studied with fNIRS hyperscanning, interbrain connectivity was evaluated between client and counselor during psychological counseling, indicating hyperconnectivity in the right TPJ compared to a chatting control situation, possibly related to required mentalizing for psychological counseling (Zhang et al., 2018).

Music is also highly related to social interactions and therefore could be a good task for fNIRS studies. A song-learning task found increased hyperconnectivity in the inferior frontal cortex, with directionality from teacher to learner (Pan et al., 2018). Similar results were observed related to cooperative singing and humming, regardless of face-to-face or non-face-to-face execution of the task (Osaka et al., 2014, 2015). Also, the right inferior frontal cortex seemed to be more associated with humming. Pairs of violinists in a leader-follower context presented higher temporoparietal junction and somatomotor signal when playing as follower (Vanzella et al., 2019), and a feasibility study suggested hyperconnectivity between violinists (Balardin et al., 2017). Another feasibility study suggested that multibrain hyperscanning could be represented as a multibrain network and evaluated as a graph and as feasibility proposed a data collection in nine participants simultaneously drumming (Duan et al., 2015).

A different approach to fNIRS hyperscanning was proposed by Duan et al. (2013). In their study, the authors have designed a neurofeedback platform. Neurofeedback has been studied as a tool for improvement of cognitive functions, especially in the context of brain disorders, though there is still debate about the efficacy and proper use of neurofeedback (Thibault et al., 2016; Kadosh & Staunton, 2019). Therefore, hyperscanning neurofeedback might present an opportunity for investigations on enhancement of social abilities or treatments of disorders which affect the social brain, but these possibilities should be addressed carefully.

These studies highlight the advantage of fNIRS for naturalistic and closer to real-life tasks, given its tolerance to movement, and portability (in some systems). However, it is also important to notice that some disagreement in the fNIRS hyperscanning literature can come from different optode position, given that studies with systems which allow whole cortical coverage are scarce, and possibly can be due to smaller spatial resolution, which implies studying larger cortical areas as a single region, compared to the specificity that fMRI studies can provide. Further, different preprocessing steps (mainly dealing with noise) could affect hyperconnectivity results, and there is still search for better analysis processes.

Conclusion and Future Perspectives

We have discussed the possibilities fMRI and fNIRS offer for studying the social brain. While fMRI provides high-resolution whole-brain coverage, fNIRS offers a great opportunity for real-life task studies. Studies recording one subject can help elucidate basic mechanisms that are engaged and combined during real interaction, as they offer easier possibilities for isolating and controlling cognitive aspects of the experiment. Meanwhile, hyperscanning provides an integrated look that can unveil new interactions between basic mechanisms and possibly new mechanisms (or better models) for social interaction related to different contexts. Moreover, hyperscanning can offer a new perspective in search for biomarkers and in understanding diseases and disorders that affect the social brain (Ray et al., 2017). Further studies should also focus on combining fMRI and fNRIS techniques such as EEG, eye-tracking, and possibly even MEG, to provide more information and higher temporal resolution, preserving spatial resolution, although newer analysis methods and hypothesis-driven models are also required for better use of the data (Koike et al., 2015). Another potentially interesting combination for fNIRS and fMRI hyperscanning are autonomic response measures, since there seems to be autonomic coupling during cooperation (Vanutelli et al., 2017). Controlling these autonomic responses in hemodynamic response-based systems could help interpret the data (Kadosh & Staunton, 2019).