Keywords

1 Introduction

The ability of technical systems’ to recognize, respond to and even influence affective states of users is of growing importance for the human-computer interaction (HCI). Especially regarding interactions with personalized assistive technology that individually adapts its functionality to its users, like for instance in the case of Companion-systems [1, 2], the consideration of users’ affect is crucial. When assisting a user in his or her everyday life, the complexity of tasks is very high and thus the risk to encounter critical dialog situations inducing negative emotions increases. In such cases, system’s adequate reaction to users’ needs is beneficial for comforting users, for keeping the interaction going and for improving users’ attitude towards the system.

Affective interfaces that provide emotional support by applying affective interventions have been shown to be capable of relieving emotional states like stress [3] or frustration [4], enhancing the problem solving capabilities [5] and improving the willingness to further participate in a given task [6]. Although these studies demonstrate that synthetic emotions expressed by technical systems can help frustrated users, it is still poorly understood why this is the case [7]. In these studies, interventions’ effectiveness was measured by applying questionnaires (ratings of frustration, valence and arousal and ratings of the interaction in general on predefined scales), observing user behavior or analyzing user performance data. Besides these ratings that provide quantitative measurements of users’ states or users’ evaluation of the system, the question of what was going on in users’ minds during the interaction and the intervention remained unanswered.

The aim of the present study is to help filling this research gap by exploring how users individually experienced an affective intervention, which was given to them in reaction to a critical dialog situation of HCI and hence to better understand what makes such an approach valuable. It is not the intention to either evaluate the applied system, or to test the effectiveness of the applied intervention. Instead, in an account of basic research, it is the aim to understand subjective sense-making processes regarding the affective intervention.

In order to design affective interventions in a way that supports each individual user, it is important to understand how users experience these and what they wish for regarding their design. In previous studies, we were able to demonstrate that users experience interactions with technical systems highly individual and that their experience was influenced by individual sense-making processes, which helped users to understand nature and behavior of their counterpart and thus to gain safety in the interaction [8–10]. Because individual sense-making processes are mostly represented on an implicit (i.e., barely consciously reflected) level of awareness, an adequate methodology is needed for their assessment. For this purpose, established methods of user research, which mostly rely on structured questionnaires or interviews with pre-formulated items, are suitable to only a limited extent. Instead, by using open question formats, narrations can be evoked in users, which in turn make it possible to access the initially hidden sense-making processes. The content of such processes is crucial for the acceptance of technical systems and especially for affective interventions. Thus, in this study, the sense-making processes underlying users’ experiences of an intervention will be investigated. For this purpose, a qualitative research methodology that allows to consider users’ individuality and the implicitness of their sense-making will be applied.

2 Empirical Investigations

The present study builds on a widely standardized empirical experiment in which a critical dialog situation of HCI was established. In reaction to this critical situation, an affective intervention was given to the participants. Subsequent to taking part in the experiment, participants were interviewed.

2.1 Wizard of Oz Experiment

In order to simulate a computer system capable of speech and individualized reactions, the empirical experiment was designed as a Wizard of Oz study [11]. The system was represented solely by a computer screen with a graphical user interface (without any interface agent) and a male machine-like sounding computer voice. The only possible way for the participants to interact with the system was via speech. In cooperation with the system, participants had to pack a suitcase for a holiday trip by selecting items from a catalogue depicted on a screen in front of them. At a certain point during the packing procedure, participants were informed about the actual weather conditions at the destination of their trip (‘weather barrier’), which were contrary to what was suggested in the beginning. Therefore, participants were required to repack their suitcase under increasing time constraints, what was intended to cause feelings of stress and frustration. In reaction to this critical situation, an affective intervention was given to the participants (for a detailed description of the whole experimental design see [12, 13]).

2.2 Affective Intervention

The affective intervention was given to the participants right after the weather barrier. It consisted of three consecutive components (cf. Table 1), which basically refer to the common factors of psychotherapy (activation of positive resources, actualization of what is to be changed, active help for coping with the problem, motivational clarification), which were formulated by Grawe [14]. The intervention was realized as a solely text-based speech output and differed from the rest of the experiment in that the system firstly referred to itself as ‘I’ and secondly directly asked for participants’ feelings. This was meant to change the level of interaction (from a primary factual to a more personal level) and to help participants in reflecting on their critical situation as well as to offer support for recovery.

Table 1. Intervention components and corresponding speech outputs
Full size table

2.3 User Interviews and Sample

The interviews aimed at investigating how participants experienced the interaction with the system and how they were affected by the intervention. For conducting the interviews, a semi-structured interview guide, including open narration generating questions, was used [15].

In total, there were 35 participants (17 female) who took part in the empirical experiment, received the affective intervention and were interviewed subsequently. They were between 18 and 75 years old (two age groups: 18-28 and 60 +) and had different educational backgrounds. By investigating such a heterogeneous sample, a wide range of experiences should be grasped. This corresponds to the rationale of qualitative research, which can be seen in the maximization of variance and in the generation of hypotheses rather than in testing those.

2.4 Analysis Procedure

Initially, the audio records of the interview sequences dealing with the experiences of the intervention of all 35 participants considered for this study were transcribed (which resulted in 232 transcript pages) and the transcripts were imported into the analysis software ‘MaxQDA’ [16]. Following the procedures of ‘summarizing qualitative content analysis’ [17], at first the text was broken down into ‘meaning units’ (MUs), which are segments of text that contain one main idea and are understandable by themselves [18]. Next, these MUs were assigned to the most suitable of five predefined categories (so-called ‘first level categories’: experience of the context, experience of the system, experience of the relation to the system, experience of the intervention and self-related experience). The assigned MUs were then compared to each other and grouped according to similarities. These groups constituted a set of subcategories (‘third level categories’), which in turn could be arranged into main categories representing a higher abstraction level (‘second level categories’).

As a result of this procedure, a category system with 5 first level, 13 second level and 58 third level categories was worked out. With the help of this category system, users’ subjective experiences were structured and patterns of individual sense-making processes regarding the affective intervention could be identified. Next, these patterns will be described in more detail.

3 Results

The analysis revealed a variety of subjective sense-making processes. Especially, the shift from a factual to a personal level of interaction was experienced with a lot of astonishment by the users. At first, the intervention was experienced as “surprising” (07-AJ)Footnote 1, “unusual” (28-CM) and “strange” (29-SG) and in a sense “it fell out of the context” (14-RT). For some users aim and purpose of the intervention were not clear, which led to some uncertainty. Being addressed on a personal level prompted users to question if this is even possible. For some users it was clear that a computer system is based on facts (“for me computer programs are facts and no statements of feelings”, 01-VL), that it functions according to its programming and that it is not capable of comprehending feelings (“when a computer asks me if I am disappointed, sad or funny, it can maybe classify the words into categories, but nevertheless it cannot comprehend them”, 05-KK). Others instead were convinced that the system can individually process information and react adequately (“you always think that a computer is a predefined program, but when it answers that individually, it is odd […] it is admirable or strange that a computer can respond as we do”, 08-BP).

After this initial astonishment and ambiguity concerning system and intervention, users tried to make sense of system’s intervention behavior (cf. Fig. 1).

Fig. 1.
figure 1

Users’ sense-making of the intervention and the resulting attitude towards interacting on a personal level.

Full size image

The sense making regarding the intervention differed significantly between the users, but in abstracting the individual experiences, three main tendencies could be identified.

Some users normalized system’s behavior, which means that they considered system’s self-referencing and addressing of users’ emotions as an ordinary part of the interaction, which was not different from the rest. For these users it was simply a “program point” (17-VK) and a “task” (27-CD) that had to be worked off. They found nothing special about being asked after their feelings by a computer program (“I just answered and that’s it”, 24-BS). By normalizing system’s behavior, these users in a way negated system’s interest in their emotional state (“no matter what I say […] the computer is not interested”, 05-KK). This way they ruled out the possibility of interacting with the system on a personal level.

Other users specified system’s behavior by clearly identifying it as, for instance, a “documentation of feelings” (03-MR), a “critical evaluation” (18-JK), a “review of the actual situation” (13-CK) or a “confirmation of changed requirements” (14-RT). For these users, the initial uncertainty regarding the intervention vanished as soon as they had figured out its aim and purpose. In a sort of routine they recognized the intervention as something they know from prior experience, for example as a situation in which they have been tested or asked for their opinion.

A third group of users anthropomorphized the system because of the behavior it displayed. For these users the system revealed “human traits” (09-EG), which resulted in not recognizing it as a machine any more (“it is a computer, I didn’t associate that, I felt that it was a person sitting there and asking me”, 30-KM). In the case of anthropomorphization, the users regarded the intervention as a confirmation for system’s personhood. This interpretation helped users to regain safety in the interaction, because interactions with human-beings are familiar and thus predictable. Moreover, anthropomorphizing the system enhanced the collaboration with it (“it understood me, it noticed that I had problems and asked after these, I think this deepens the collaboration and the relationship with the program, too”, 26-SS).

The different ways of making sense of the intervention influenced users’ attitudes towards further interacting with the system. Users, who normalized system’s behavior, wanted to distance themselves from any form of personal interaction (“I know it was programmed to be nice to me, but I don’t need that from a machine, that is not useful for me”, 11-HG). For them the system is a “service provider” (11-HG) and being too personal with it is unconducive for the interaction. Instead, for these users the system should just function and not be occupied with users’ feelings.

On the contrary, users who anthropomorphized the system because of the intervention were inclined to accept a personal level of interaction. They felt “comfortable” (06-UK) with the system and “chitchatted” (09-EG) with it. For them, a personal level of interaction with the system improves the interaction experience and helps to cope with negative emotional states.

Users, who specified system’s behavior, either tended to distance themselves or to take part in a personal interaction. For these users, the functionality on a factual level is most important, but they also value a more intimate level of interaction (“when it’s going well on a factual level, then the emotional side is safe for me, too”, 25-SP). But still there a concerns like participant YD puts it: “it is strange to tell an unemotional machine how I feel, because it can probably not emphasize with my situation” (19-YD). Thus, these users remain somewhat undecided with regard to their preferred level of interaction with the system.

4 Discussion

In an account of basic research, the aim of the present study was to understand how users conceptualized and experienced an affective intervention, which was given to them by a system. Above relating to ratings on predefined categories or measurements of effectiveness, different ways of how users made sense of the intervention have been worked out. These indicate that a personal level of interaction, in which users’ emotional state is addressed, is not advantageous for all kinds of users and that it can also have negative effects on users’ willingness to further participate in the given task.

With the intervention, there was a shift from a task related factual level of interaction to a more reflective personal level. In different studies, Reeves, Nass and Moon [19, 20] were able to show that people readily engage in communicative behavior as they know it from human-human interaction, as soon as a computer gives basic social cues (like speech-based communication). With regard to the affective intervention, the research presented here confirms this tendency of users. However, the sense-making processes that have been worked out indicate that users reacted in individually different ways to the offer of a personal communication. While some users anthropomorphized the system (‘anthropomorphization’), others did not react to the social behaviors of the system (‘normalization’) or identified it as something detached from the system (‘specification’). Moreover, the data revealed that some users (those who normalized or specified system’s behavior) denied any form of personal interaction with a technical system. Especially for the further development of assistive technology, which aims at reacting adequately to needs and states of its users (like in the case of Companion-systems that are meant to provide an emotional dimension of the interaction in addition to a task-oriented formal one [1, 2]), this is an important fact that needs to be considered. Only if such systems are able to react to users’ needs and actual emotions on an appropriate interactional level, systems will be perceived as individualized assistants and partners and thus will be able to support users in processing their affect and in reaching their goals.

The affective intervention applied in this research was intentionally kept relatively short and was given to the participants regardless of their current emotional state. This way it was possible to gather a differentiated view on experiences, since participants were not tempted to interpret the intervention in a certain way and moreover, the sense-making processes of all kinds of participants were included in the analysis. To validate the results, further research is needed. It will be interesting to find out, how users’ sense-making processes change depending on (a) the critical situation they are in (e.g., stress, frustration, cognitive overload or boredom), (b) the time frame of the interaction with the system (single vs. multiple interactions) or (c) the representation of the system (e.g., anthropomorphic vs. artificial system voice; interface agent vs. no agent). Also different kinds of interventions (e.g., personal vs. factual) should be tested against each other to further investigate to which extent an intervention influences the experience of a system as a whole.

The implications of the present study can primarily be seen in generating an understanding for the importance of individual sense-making processes in HCI. Already a relatively simple intervention was able to evoke a broad range of interpretations and moreover clarified that a ‘one type fits all’ solution is not inevitably appropriate for all kinds of users, because users draw on their own sense-making processes to arrive at reasonable explanations. These sometimes may seem pretty far-fetched from reality, but they enable users to participate in an efficient interaction with the system and thus to achieve their goals. Regardless of the best intentions of designers, in the end the user will decide about acceptance or denial of an intervention or even a system as a whole. This decision is based on individual experiences and attitudes, which are only to some extent consciously reflected. In HCI contexts, investigating such implicit ways of experience is only in its beginnings. Here, the user is mostly viewed as a computable variable, whose ways of experiences can be best described in quantified categories. The present study wants to contribute to a modified view on the user and his or her individual sense-making processes. In the future, more research based on an interpretative qualitative methodology will be needed to investigate user experience in HCI and thus to gain a differentiated view of users’ interpretations regarding the system and its behavior. With regard to affective interventions, the sense making processes that have been worked out in this study can serve as a starting point for future research.