Keywords

1 Introduction

Ever since Picard [12] coined the term “affective computing”, machines have gained great opportunities to monitor and analyze user reactions and respond adequately. Thanks to advanced technology, various systems are capable of detecting and recognizing the emotional states of users and moreover, they can try to influence them. Potential applications in the field of HCI are extremely wide: from serious, such as enriching communication in remote education, social robots in health care, safety systems at cars, to entertainment, e.g. interactive computer games or immersive storytelling in interactive VR experiences. One potential application is attitude formation. One can imagine, for example, an automatic training system that helps to shape proper eating habits or a virtual environment for working with children with ADHD. A system equipped with the possibility of monitoring the level of arousal (even with the use of relatively simple measures, e.g. galvanic skin response - GSR) could use the evaluative conditioning effect to shape user’s attitudes.

2 Theoretical Context

Carberry and de Rosis [2] described four main areas of affective computing development. They are: analysis and characterization of affective states, especially those exhibited in natural interactions, and analyses of the relationships between cognitive and affective processes (e.g. in learning environments); automatic recognition of affective states (e.g. facial expressions, sounds, or physiological responses); adapting system response to affective states of a user; last but not least, designing affective virtual agents (virtual humans that are able to use the information on user state to exhibit proper affective answer, or to influence user’s state in enhanced HCI). Affective computing methods have been used in many fields, including for example training and learning environments. As emotions have impact on motivation, reasoning or decision making it is feasible and necessary to also look at the consequences of eliciting affective states by the elements of interface (like conversation agents, virtual humans), not only detecting and responding to (see e.g. [9]). With the above in mind, we would like to check to what extent interface users can be influenced by affective information, for example in a conditioning procedure. More precisely, we want to trace the process of forming attitudes in a situation where the user encounters emotional stimuli, the source of which is a computer system.

The purpose of the current studies was to determine the dynamics of emotional processes in attitude formation in evaluative conditioning (EC). EC is defined here as a change in the evaluation of a specific stimulus (CS - conditioned stimulus) that can be attributed to its repeated presentation with another affective stimulus (US - unconditioned stimulus; cf. [6, 7]). In three experimental studies (N = 94) we investigated the relationship between the level of arousal generated by the US and the effectiveness of evaluative conditioning, the extinction of this effect, and the effectiveness of revaluation of the affective stimulus. Previous research has suggested an effective transfer of the arousal value of the affective stimulus US to the conditioned stimulus CS, vaguely suggesting that the evaluative conditioning does not only involve declarative ratings of liking, but also occurs at the arousal level [4]. However, to this point, only one paper investigated this hypothesis analyzing only the declarative arousal ratings of subjects. Key aspects of this project were (a) to trace how arousal affects the size of the effect of EC, (b) to determine possible asymmetries between positive conditioning (US+) and negative conditioning (US−), (c) to determine the effect of the arousal value of the US stimulus on the ability to extinguish the conditioned evaluative response or change its sign, and (d) to determine the degree of intentional control over the acquisition and expression of conditioned affective responses depending on the arousal value of emotional stimuli [1, 3, 5]. It is worth emphasizing that in our case the computer system receives information about the user’s physiological arousal and uses it in real time to adjust the stimulation.

3 Experiments

3.1 Method

Participants

Ninety-four participants volunteered to participate in the experiment (46 Female) ranging in age from 18 to 35 years (M = 24.8, SD = 4.21). Participants were individually tested.

Materials and Procedure

All three experiments used neutral human faces as CSs and affective images differing in sign (positive vs. negative) and arousal level (low vs. high) as USs. CSs were selected from a validated set of facial expressions [11] and USs from IAPS [8]. The latter were pre-selected based on available standardization data to maximize differences on two orthogonal dimensions: affective value (positive vs. negative) and arousal level attributed to the images (low vs. high). Therefore, we managed to create four groups of USs used in the studies: positive – highly arousing, positive – little arousing, negative – highly arousing, and negative – little arousing.

All subsequent tests were conducted in a standard EC procedure with two stages at its core. After filling in a consent form, participants were instructed to observe CS-US pairs appearing several times in random order (presented together in Experiment 1 or in rapid succession in Experiments 2 and 3). We used eight different CS-US pairs. In the second stage, participants made evaluative judgments of the CS stimuli using a scale from 1 – very negative to 9 – very positive.

GSR and EMG were continuously measured during both stages of the study with a BIOPACK MP150 set. Galvanic skin response (GSR) refers to dynamic changes in sweat gland activity that correlate with the intensity of one’s emotional state (i.e. emotional arousal). Electromyography (EMG) measures muscle response and was used in two facial muscles: the corrugator supercilli that is typically active when frowning (negative emotion), and the zygomaticus major that is active when lifting the corners of the lips when smiling (positive emotion). The EMG of both muscles is believed to reflect positivity and negativity of the emotional state of the participants.

Participants were thanked and debriefed after experiment completion.

In each study, within-group manipulations and random assignment of CS-US pairs were used. The description of subsequent experiments includes only those elements of the procedure that differed from the standard study design mentioned above.

Results

In the first (N = 31) and second (N = 29) studies, the goal was to determine the relationship of physiological arousal to the size of the effect of evaluative conditioning. Selected US stimuli with low and high arousal values were presented in pairs with neutral CS stimuli. We hypothesized that high-arousing stimuli would lead to a stronger evaluative conditioning effect compared to low-arousing US. Our results revealed an average EC effect of higher evaluative arousal for positively conditioned CSs compared to negatively conditioned ones, F(1,59) = 4.54, p < .05. Furthermore, declarative scores of arousal gathered after presentation of negatively conditioned CS were found to be higher than ratings after presentation of positively conditioned CS, and declarative arousal increased after conditioning, F(1,59) = 6.29, p < .01 (Fig. 1). Finally, the study showed that the level of physiological arousal after the presentation of the CS stimulus increased during the conditioning stage with increasing number of presentations of US-CS pairs and persisted throughout the postconditioning stimulus ratings. The results support the endorsed hypothesis that the magnitude of the conditioning effect varies with the arousal elicited by the US stimulus and that there is a transfer of arousal between the US and CS both at the declarative level and at the level of the physiological response that is a marker of arousal.

Fig. 1.
figure 1

Level of arousal in response to conditioned CS stimuli as a function of the affective value of the US (pos - positive, and neg- negative) and the level of arousal elicited by the US (high vs low).

Full size image

In the third experiment (N = 34), the objective was to determine the mechanism responsible for the transfer of arousal between the affective stimulus US and the neutral stimulus CS. In this study, we manipulated the level of arousal of the affective stimuli (low and high arousal) and the order of presentation of the stimulus during the conditioning phase (CS-US or US-CS). The hypothesis tested was that physiological arousal transfer is more efficient when the affective stimulus US (causing arousal increase) is presented first, followed by the neutral stimulus CS, compared to the opposite order of stimulus presentation. We did not assume such a difference for the declarative measure of arousal because we assumed that transfer at the declarative level of the measure does not depend on the order of presentation of the CS and US stimuli. The results confirmed the hypothesis that the transfer of arousal at the declarative level does not depend on the order of presentation of the stimulus. On the contrary, we did not obtain the expected effect of greater arousal transfer at the physiological response level for US-CS pairs compared to CS-US pairs.

We also hypothesized that the level of arousal of conditioned CS stimuli would depend on the temporal interval between US and CS such that transfer would be greater for CS presentations at the peak of the arousal response to US compared with CS presentations earlier and later relative to the highest response to US. The results of this study confirmed the previously obtained effect of physiological arousal transfer between US and CS at both the declarative and physiological response levels. Furthermore, the transfer of arousal at the declarative rating level did not depend on the temporal interval between the US and CS. For physiological arousal elicited by conditioned CSs, a higher transfer was observed for those CSs presented at the peak of the US response (during conditioning). The latter effect was present only for negatively conditioned CSs, suggesting (as in previous studies) the existence of an asymmetry in responses to positive and negative stimuli.

4 Discussion

The project succeeded in accomplishing most of the stated substantive goals attributed to the primary research. Firstly, a novel line of research was conducted on the transfer of arousal (at the declarative level and at the level of physiological reactions) in evaluative conditioning, which is an important addition to the knowledge on the mechanisms of attitude acquisition and their dependence on stimulus characteristics. Second, we examined for the first time how the procedural characteristics of EC (CS-US presentation order) are related to the arousal value of affective stimuli. And third, new experimental procedures were developed to enable complex research on the role of arousal in attitude acquisition.

Let us consider the many possible applications of the above discoveries. Virtual environments in which the user communicates in a natural way are becoming more and more common. This applies not only to experimental laboratory environments, but also to commercial and entertainment applications. Both simple interfaces (like chatbots [10, 14]) and more complex environments with some or many virtual humans (e.g. [13]) can be effective. More research is needed on this phenomenon.

5 Limitations

One of the basic limitations of the interpretation and application of the above-described results for a HCI and affective computing is the nature of the experimental procedure used. On the one hand, it allows for precise control of distorting variables, on the other, it does not resemble natural situations in which the system could try to influence the participant’s affective arousal. It would benefit from moving it to a more realistic setting, e.g. during an interaction between participants using social media.

Another limitation is the use of precise equipment that records physiological responses incomparably more accurately than commercially available sensors. The use of arousal peaks detection algorithms developed by us requires uninterrupted registration with millisecond precision, which may turn out to be impossible in the case of e.g. commercial VR HMDs, game pads or watches.

6 Further Directions

In the times of the Internet of Things and the widespread use of measuring devices, affective information will gain more and more importance (both the knowledge about the affective state of the system user and the ability to influence it). Understanding the processes of evaluative conditioning and algorithms to effectively detect a user’s affective states in real time based on the available information is essential for finding applications as well as for deepening the basic understanding of the subject. For example, modern consumer VR systems are equipped with sensors that capture large amounts of data for simulation purposes. This data can also be used to infer about affective arousal, and this information can then be used in turn to modify the content of the simulation. Subsequent research should focus on users in their natural environments, on the daily relationships of users and technology. If your headset knows about your emotions, why not use it to help you. It is especially important in times when we experience isolation and more and more of our daily relationships are mediated by information systems. Not to mention the replacement of relationships with humans with relationships with machines.