Cognitive Processing

, Volume 13, Supplement 2, pp 487–497 | Cite as

Analysing user’s reactions in advice-giving dialogues with a socially intelligent ECA

  • Nicole Novielli
  • Irene Mazzotta
  • Berardina De Carolis
  • Sebastiano Pizzutilo
Research Report


In this paper, we investigate the user’s reactions to received suggestion by an Embodied Conversational Agent playing the role of artificial therapist in the healthy eating domain. Specifically, we analyse the behaviour of people who voluntarily requested to receive information from the agent, and we compare it with the results of a previous evaluation experiment in which subjects were not properly motivated to interact with the agent because they were selected for evaluating the system. This study is part of an ongoing research aimed at developing an intelligent virtual agent that applies natural argumentation techniques to persuade the users to improve their eating habits.


Human computer interaction Analysis of user behaviour User’s reaction to persuasion Natural language user interfaces 


Embodied Conversational Agents (ECAs) (Cassell et al. 2000) are a good interaction metaphor for involving users in a human-like conversation. Using verbal and non-verbal communicative signals, an ECA has the potential to establish a relationship with a user. Several studies have shown how ECAs can be successfully employed in the role of counsellors (Marsella et al. 2003), personal trainers (Bickmore 2003), or healthy living advisors (de Rosis et al. 2006), as well as in other domains where it is important to build a relationship with the user (Bickmore and Cassell 2005).

The more the ECA appears ‘believable’, ‘competent’, and ‘natural’, the more users have the impression of interacting with a human partner rather than just using a tool. Under this perspective, the agent has to be able to observe the user in order to be believable and successful. First of all, it has to prudently wait to get knowledge about the user attitudes (values, interests, goals, etc.) before planning the interaction strategy to adopt in a given circumstance. Second, it should observe the user’s reactions so as to understand whether and how to correct inappropriate choices and to integrate the information provided with other data.

This study well fits in the scope of an ongoing research aimed at developing an ECA that applies natural argumentation techniques to persuade users to improve their behaviour in a domain of its competence (Mazzotta et al. 2007a). With ‘natural’ argumentation, we intend the ECA’s ability to integrate rational arguments with the common sense, a friendly style, and emotional ingredients that are used in human–human communication. Among the various aspects that are considered in designing an ECA, we are interested in verbal rather than non-verbal communication forms. In particular, we are working on both generating the natural language persuasion message and recognizing and interpreting the user’s reactions.

The aim of the present research is to investigate the user’s reactions rather than to evaluate the effectiveness of the applied persuasion strategy. In our opinion, two aspects of the users’ reactions are of particular relevance:
  1. (a)

    the user’s reaction to the suggestion received, in order to understand whether and how to correct inappropriate strategy and to integrate the information provided with other data that might increase the level of user’s persuasion;

  2. (b)

    the user’s social attitude towards the agent, in order to adapt the interaction style accordingly by demonstrating social intelligence.


Particularly, we analysed the reactions of people who voluntarily asked to interact with the ECA and compare them with the reactions of subjects selected for dialoguing with the ECA in a previous evaluation study. In fact, one of the main limitations of our previous study was that we could not be sure of the generality of our findings: the subjects involved were not completely representative of the population of potential users of our application since they did not spontaneously request to interact with the agent and were not specifically interested to receive ECA’s suggestions.

The paper is structured as follows: in section ‘Motivation’, we provide the motivation that justify this work; then, section ‘The user’s reaction’ introduces the meaning of users’ reactions in term of reaction to persuasion attempt and social attitude; in section ‘Collecting empirical data’, we describes the Wizard of Oz tool and the experimental settings used for collecting the human-ECA dialogues with subjects that spontaneously requested to interact with the agent; hence, in section ‘Analysing the user’s reactions: voluntariness versus non-voluntariness’, we compare the behaviour of users in the study described in section ‘Collecting empirical data’ with not motivated subjects of the previous evaluation study. Conclusion and final remarks are reported in section ‘Final remarks’.


Persuasion may be seen as ‘human communication designed to influence others by modifying their beliefs, values, or attitudes’ (O’Keefe 2002). Factors related to the user, the context in which the persuasion dialogue occurs, and the source of information provided are crucial for the success of a persuasion attempt. A successful virtual agent playing the role of an artificial therapist needs to model not only the cognitive components of the user’s state of mind but also some extra-rational factors such as affective states and attitudes: this is important to enable the ECA to adopt the most appropriate behaviour during the interaction (e.g., using small talk, including affective elements in the language and displaying expressions of emotions in the agent’s face). The ECA should therefore observe the user’s reactions during the dialogue, to decide whether to put itself in the shoes of a traditional therapist or to make appeal to artifices of various kinds (such as irony, colloquial language, and admitting its knowledge limits) in reciprocating the social attitude of the user. This is an important aspect because users are not necessarily keen to establish a friendly relationship with an artificial agent, either due to their background and expectation (Clarizio et al. 2006) or interaction mode (Novielli et al. 2010).

In a previous research, we investigated the interaction between emotional and non-emotional aspects of persuasion, from the viewpoint of both the system, when reasoning on the persuasion attempt, and the user, when reacting to it. Results proved the importance of a-rational factors in both generating persuasion strategies and modelling the user’s reactions (Mazzotta et al. 2007a). Moreover, different studies have proved how it is possible to recognize signs of ‘warm’ and ‘cold’ social attitude (Andersen and Guerrero 1998) by mean of linguistic analysis of user’s individual dialogue moves (Novielli et al. 2010).

One of the main limitations of our previous studies was that the subjects involved did not spontaneously request to interact with the ECA because they were selected for evaluating the system. The non-voluntary nature of the interaction could probably have affected in some way the user’s behaviour that could have been not motivated to dialogue with the ECA, nor necessarily interested to the dialogue topic.

To validate the findings of our previous research, we have collected a new corpus of human-ECA dialogues with people who spontaneously requested to receive information about healthy habits from the agent. The dialogues have been analysed in terms of both reactions to received suggestions and social attitude towards the agent. Lastly, we have compared the speech-based corpus collected in our previous study with the new data set collected in this new set of dialogues.

The user’s reactions

The more an agent succeeds in achieving a natural interaction with the users, the more they are expected to show some signs of social relationship with it (Nass et al. 2000; Oviatt and Adams 2001; Darves and Oviatt 2002; Dahlback et al. 1993; Clarizio et al. 2006; de Rosis et al. 2006). A believable ECA should be equipped to recognize communicative and social behaviours of users, in order to accordingly adapt the dialogue and the interaction style. In particular, the ECA has to decide whether to act as a traditional therapist or rather to reciprocate the user’s social attitude. Moreover, observing the user’s behaviour during the interaction enables the agent to verify the appropriateness of the adopted persuasion strategies and, if necessary, to revise them.

In the following sections, we briefly introduce the meaning of persuasion attempt and then we describe the dimensions along with we investigate the behaviour of users in term of users kind of reactions to the persuasion attempts and their social attitude towards the agent.

The persuasion attempts

As said in the section ‘Introduction’, this work is part of an ongoing research aimed at developing an ECA that applies natural argumentation techniques to persuade users to improve their behaviour.

PORTIA (Mazzotta et al. 2007b) is a user-adapted persuasion system capable of simulating the persuasion process used by humans to convince someone to perform a given action. The ECA plays the role of the Persuader, the user the role of the Receiver. The system is based on the theory of a-rational persuasion (Miceli et al. 2006) and is relied on a user model that includes a specific knowledge (that is facts acquired and updated during the dialogue) and a generic knowledge (criteria to infer the user goals and abilities in conditions of uncertainty). Its key point is the separation between reasoning and argumentation phases in the persuasion process (Walton 1990): it first performs a what if kind of reasoning on the user’s mental state to select the most promising persuasion strategy or the appropriate response to the user’s reaction; then, it translates the selected strategy into a ‘discourse plan’ by combining Walton’s argumentation schemes (Walton 1996) and Rhetorical structure theory (Mann et al. 1989), in order to produce a rhetorically coherent natural language message.

Persuasion moves combine discourse plans representing different argumentation schemes: for examples, Argument from Consequences is combined with Argument from Evidence in the move ‘You should try to increase the proportion of fruit and vegetables in your diet! They proved to be very effective for health, which seems to be a value to you. I’m sure you may do it if you wish!’, which includes a suggestion supported by an evidence about some user attitudes.

Regardless of the strategy applied, if the dialogue is natural in its developing, users will tend to not a-critically accept the system’s suggestions and the ECA should be equipped so as to recognize these reactions and dynamically adapt/revise the dialogue strategy while the interaction goes on. In adapting its answer, the ECA should be equipped to recognize two main aspects of the users’ reaction: their social attitude towards the ECA and their reaction to its suggestion.

Recognizing the user’s reaction to persuasion attempts

Literature on argumentation in agent systems proposes several languages for the formalization of the argumentation exchanges between Persuader and Receiver (Prakken 2006). The communication language among artificial agents proposed by Cohen and Levesque (1990) neatly stated the semantics of ‘illocutionary acts’ (Austin 1962; Searle 1975) in terms of the effects the Speaker intends to achieve: the hypothesis was that this effect always consists in ‘communicating own mental state’, with the Speaker’s ‘sincerity’ as a strong assumption about communication conditions. The existing work on communication language among artificial agents is of primary importance in the immediate interpretation of a given sentence in terms of an agent’s beliefs and intentions. Though, given the particular nature of the persuasive dialogues, this kind of communication language needs to be enriched in order to allow the representation of this kind of dialogues, including also the wide range of possible users’ reaction to the suggestion received. Indeed, a suggestion move (or persuasion attempt) may be criticized by Receiver in several ways: by questioning the goal premises, by attacking them with counter-arguments alleging that one or more of them are false, by undercutting the inferential link between premises and conclusion with critical questions, by rebutting the practical reasoning inference with counter-arguments asserting that the conclusion is false, or by putting forward a proposal arguing for a different action and contending that the arguments for this opposed proposal are stronger (Walton 2006).

We grounded the analysis of the user’s reaction to a persuasion attempt on Walton’s argumentation schemes and their later interpretations and refinements (Verehij 2003; Walton and Gordon 2005). Argumentation schemes are common types of defeasible arguments, evaluated by a set of related critical questions (CQs) (Walton and Godden 2005). The majority of argumentation schemes are focused on ‘persuading to believe’ (Argument from Evidence, Argument from Expert Opinion, and Argument from Position to Know). Among the few of them which are aimed at ‘persuading to do’, the most commonly applied are the Argument from Consequences and the Practical Reasoning. CQs can be seen as ‘representing additional relevant factors that might cause an argument to default’ (Walton and Reed 2003). They are used in everyday conversational arguments ‘when a user is confronted with the problem of replying to an argument or making some assessment of what the argument is worth and whether to accept it’ (Walton and Gordon 2005). From the viewpoint of the Receiver, CQs are questions that inquire about the conditions or circumstances that tend to challenge premises of a suggestion or the suggestion itself. Starting from the analysis of the critical questions of these schemes, we defined a set of users’ reaction to system’s persuasion move, which enables us to define a method to recognize the level of user’s acceptance of the system’s suggestions.

Figure 1 is one of the most commonly used Walton’s argumentation scheme called Argument from Consequences.
Fig. 1

Walton’s argumentation scheme: argument from consequences (Walton 1996, 75–77)

Recognizing the user’s social attitude towards the ECA

By social attitude, we intend ‘the pleasant, contented, intimate feeling that occurs during positive interactions with friends, family, colleagues, and romantic partners’ according to the definition of interpersonal warmth introduced by Andersen and Guerrero (1998). Among the large variety of verbal and non-verbal markers of interpersonal stance (Scherer et al. 2004) proposed, we focus on forms of expression with language in human–human communication (Andersen and Guerrero 1998; Polhemus et al. 2001; Swan 2002).

According to this definition, users display their social attitude towards the agent through various ‘signs’ in the employed language, that is by introducing colloquial style, friendly greetings, farewells or humour as ‘offers of sympathy’, questions about the agent’s ‘private life’ and self-disclosure to establish a common ground, positive or negative comments as a demonstration of interest in the dialogue. This attitude is influenced by the interaction mode (text vs. speech) and can be recognized with statistical language processing methods (Novielli et al. 2010) and by combining prosodic or linguistic characteristics for recognition (de Rosis et al. 2007).

Collecting empirical data

In this section, we describe more in detail the corpus of human-ECA dialogues collected through a Wizard of Oz study on which we grounded our empirical analysis. As said in the previous section, we use the same approach adopted in our earlier researches.

The Wizard of Oz tool

To collect the human-ECA dialogues, we employed a Wizard of Oz (WoZ) tool in which the application domain (in our case, the healthy eating) and the ECA’s appearance may be set at the beginning of the simulation. According to Zimmerman et al. (2005), female agents are preferred when acting as therapists or medical advisors. We therefore employed a 3D realistic agent shaped as a youthful woman named Valentina (Fig. 2).
Fig. 2

A smiling expression of Valentina

The WoZ tool supports two interaction modalities: text input, in which users can interact with the ECA with keyboard and mouse, and spoken input, in which the ECA is displayed on a touch screen and users can use a microphone to talk to it and a touch screen to send other commands. Subjects’ moves are completely unconstrained: they can just answer the agent’s questions and hear its suggestions or can take the initiative to make questions and comments of any kind. More details about the WoZ tool may be found in (Clarizio et al. 2006).

The corpus

We collected thirty speech-based dialogues with an ECA from subjects between 21 and 28 years of age, equally distributed by gender, that we hypothesize might be naturally interested in receiving suggestions about healthy eating, such as, customers of a gym.

For 2 weeks preceding the experiment, the system was sponsored in the gym through advertising (posters and flyers): Valentina was presented as a virtual therapist in the diet domain for promoting personalized healthy eating suggestions. Thus, subjects involved in the study spontaneously requested to interact with Valentina to receive suggestions about the diet.

Agent’s moves were organized into categories: self-introduction (in which the agent introduced itself and described its role), questions about the subjects’ eating habits (what they used to eat, what they liked, etc.), suggestions (advantages or disadvantages of various meal components or combinations), general comments (‘Good question!’, ‘You are right!’,…), and farewell. At every dialogue step, the wizard selected its next move from among those available on his server-side, by following an established dialogue plan: after initial self-introduction, information about the eating habits of the subject was collected, before providing tailored information and suggestions about healthy eating and justifying them.

At the end of interaction, the agent’s moves were automatically stored in a log-file, while the users’ turns were manually transcribed. Overall, 860 dialogue moves were collected.

The user’s reaction to persuasion attempts

Each dialogue was analysed so as to isolate the ECA’s persuasion attempt move and all the users’ reactions that followed it. All the moves were translated into formal logic representations and formalized as speech acts. To analyse the user’s reaction to the suggestion received, both the suggestion and the arguments, which support it, had to be considered.

Ten distinct persuasion attempts have been observed with different frequencies in our corpus and most of the examples of receivers’ reaction prove how our users applied the CQs of the Practical Reasoning schema while they reacted to the suggestion received. Table 1 shows some examples of users’ reaction to a single suggestion move. The first column is an instance of a suggestion move received by the users followed by some of their possible reactions. Both system’s suggestion and users’ reaction are extracted from the corpus. The second column is the argumentation scheme used to analysed the arguments. Particularly, selected examples demonstrate that, although the system adopted a purely rational persuasion strategy, users introduced various a-rational elements in their CQs (see for example, U3).
Table 1

Examples of users’ reaction to a system’s suggestion where users apply the CQs of the Practical Reasoning schema

System suggestion move

Practical Reasoning schema

International research demonstrated the importance of fruits and vegetables in a correct diet. It recommends a daily assumption of a portion of row and a portion of cooked vegetables and two or three portions of fruits. Precooked food helps in controlling the portions.

<PREMISE>Bringing about Sn is my goal</PREMISE>

<PREMISE>To bring about Sn I need to bring about S1</PREMISE>

<CONCLUSION>Therefore, I need to bring about S1</CONCLUSION>

Examples of users’ reaction

Critical questions

 U1: uhm… but I don’t like fresh fruits: how may I substitute them?

 <CQ>Are there an alternative possible of action to bringing about S1 that could also lead to the goal?</CQ>

 U2: But I know fresh food is better than precooked products

 <CQ>Is S1 the best (or most favourable) of the alternatives? </CQ>

 U3: But… a sin of gluttony is better than any healthy and balanced diet!

 <CQ> Do I have goals other than S1 that may be better to achieve and that should have priority? </CQ>

 U4: I can’t eat vegetables because I suffer of colitis

 <CQ> Is it possible to bring about S1 in the given circumstances?</CQ>

 U5: Are you sure that precooked food is not dangerous for health?

 <CQ> Would bringing about S1 have known bad consequences that ought to be taken into account?</CQ>

It is not always possible to classify the reactions found in our corpus as one of the critical questions defined in Walton’s schemes. Several examples of perplexity, requests of more information, provision of information about their own situation, or clear objections were founded in our corpus. Table 2 summarizes the user’s reactions recognized in our corpus. This set was identified starting from Searle’s classification of speech acts (1975) and Kibble’s studied on communicative acts in persuasion inner dialogues (2006).
Table 2

The markup language for user’s reaction to the system’s suggestion

Comm. Act




U does not express any clear opinion



U ask the truth value of a fact

Do you think my diet is correct?


U asks for more information about some topic

How could I substitute fruits?


U asks the system to justify its statement

And how do you know it?


U provides some evidence about his/her attitudes or behaviour

I eat meat, fish, vegetables, lots of fruits…


U declares to agree with the evidence provided by the system

Right, I agree


U declares to disagree with the evidence provided by the system

No, you’re wrong. I don’t agree


U presents an exception that falsifies the system argument

I love unbridled life, with light aversion towards healthy food.


U argues about the truth value of a premise of the suggestion

Are you joking? So you mean I have to bring a fruit bag with me, at work?


U declares to agree with the received suggestion

Understood! So I should try to do it?


U commits him/herself to apply the received suggestion

Ok, I will do it


U declares to not be persuaded by the suggestion

So many portions of fruits? I’ve eard contrary theories on this topic


U refuses the suggestion

But… a sin of gluttony is better than any healthy and balanced diet!


U presents an exception that falsifies the suggestion

I don’t want to avoid sweets at all

Let us consider that S and U represent, respectively, the System and the User, and the upper-case Greek letters Φ and Ψ were used, respectively, to represent speech act themselves and propositional content of the speech acts. Examples of speech acts in our corpus were the following:
  • Suggest(S, Ψ) for the System’s persuasion attempt (e.g., Suggest S ShouldDo(U,EatingVeetableg))

  • AskInfo(U, Φ), AskJustify(U, Φ), Inform(U, Φ), or Rebuttal(U, Φ) for the user’s reactions (e.g., AskJustify U (Suggest S ShouldDo(U,EatingVegetable))).

It is worth noting that the examples of I-Rebuttal, Object, and Reject communicative acts show some forms of irony and colloquial style in users’ reaction.

The user’s social attitude towards the ECA

Different signs of social attitude could be observed in our corpus. As said in section ‘Introduction’, we focus on language of users and the set of signs were defined starting from language features. Table 3 shows the signs of social attitude defined and some examples recognized in our corpus.
Table 3

Our markup language for signs of social attitude



Friendly self-introduction

The subjects introduce themselves with a friendly attitude (e.g., by giving her name or by explaining the reasons why they are participating in the dialogue)

 Oz: Hi. My name is Valentina. I’m here to suggest you how to improve your diet.

 S: Hi, my name is Isa and I’m curious to get some information about healthy eating

Colloquial style

The subject employs a current language, dialectal forms, proverbs, etc.

 Oz: Are you attracted by sweets?

 S: I’m crazy for them.

Talks about self

The subject provides more personal information about self than requested by the agent

 Oz: Do you like sweets? Do you ever stop in front of the display window of a beautiful bakery?

 S: Very much! I’m greedy!

Personal questions to the agent

The subject tries to know something about the agent preferences, lifestyle, etc., or to give it suggestions in the domain.

 Oz: What did you eat at lunch?

 S: Meat-stuffed peppers. How about you?

Positive or negative comments

The subjects’ comment on the agent behaviour: its experience, its domain knowledge, etc.


  Oz: I’m sorry, I’m not much an expert in this domain.

  S: OK Valentina, don’t worry: so far, I found your suggestions very useful.


  Oz: Good bye.

  S: What are you doing? You leave me this way? You are rude!!

Friendly farewell

This may consist in using a friendly farewell form or in asking to carry-on the dialogue.

 Oz: Goodbye. It was really pleasant to interact with you. Come back when you wish.

 S: But I would like to chat a bit more with you


No signs of social attitude.

Humour and irony

The subjects make some kind of verbal joke in their move

 Oz: Did you ever try to ask yourself which are the reasons of your eating habits?

 S: Unbridled life, with light aversion towards healthy food

Analysing the user’s reactions: voluntariness versus non-voluntariness

As said in section ‘motivation’, we performed the study described in the previous section in order to overcome the main limitation of our previous studies, that is the non-voluntary nature of the human-ECA interaction. These dialogues indeed were collected in the scope of our previous studies with thirty subjects recruited from the student population at the University of Bari that were instructed on the evaluation procedure and were aware of interacting with a prototype rather than with the final version of the application. Subjects were representative of the potential users of the system in terms of age and background. Though, they did not spontaneously requested to interact with the system, and therefore, they probably were not necessarily interested in the ECA suggestions on the specific topic of healthy eating. This may be seen as a limitation of our old study. Therefore, we designed and performed the study described in this paper in order to verify whether it was possible to either generalize our previous findings or there were differences in the behaviour of more motivated users that spontaneously requested to interact with the application. As previously explained, we decided to recruit the population of this new experiment among people who usually go to the gym, since we assumed it was easier to find people naturally interested in receiving suggestions about healthy eating in such a context.

This section aims at verifying whether there are differences in the behaviour of the users involved in our previous evaluation study and those involved in the new study. In particular, we want to investigate whether subjects that spontaneously requested to interact with the system showed a different level of social attitude towards the agent and reacted in a different way to its suggestion. We assumed, in fact, that people who spontaneously requested to interact with the ‘Valentina’ agent were more likely to have an information seeking goal rather than just enjoying or challenging the application.

Henceforth, D1 will denote the corpus of dialogues with subjects involved in our old previous study while D2 the more recent one with subjects in the gym scenario.

Overall, we collected sixty speech-based dialogues with an ECA from subjects aged between 21 and 28, equally distributed by gender. 1,762 dialogue moves were collected: 902 in D1 and 860 D2. It is important to highlight that the two groups presented exactly the same features: the only difference is the fact that people involved in the gym experiment spontaneously expressed their intention to receive suggestion about diet from ‘Valentina’.

Moreover, the ECA had exactly the same visual and audio behaviour during the interaction with both groups, and also the set of dialogue moves available to the wizard was the same (78 moves overall).

Investigating the relationship between social attitude and goals

We have performed both qualitative and quantitative analyses of our dataset. We have annotated our new set of dialogues (D2) by applying the same method that has been used in (Novielli et al. 2010) for analysing the previous corpus (D1). To recognize the linguistic signs of social attitude listed in Table 3, we applied a Bayesian classifier (de Rosis et al. 2007) in which an input text is categorized as ‘showing a particular sign of social attitude’ if it includes some word sequences belonging to semantic categories that are defined as ‘salient’ for the considered sign. Bayesian classification enables associating with every string (segment or full move) a value of a posteriori probability for every sign of social attitude.

Moreover, we computed for every subject the following measures of social attitude:
  • the percentage of social moves, that is the proportion of moves in the dialogue that was labelled as displaying at least one sign of social attitude. This proportion can be considered as an overall index of social attitude: the more frequently the user displays signs of social attitude in their dialogue turns and the more likely they can be considered as displaying the intention of establishing social relationship with the ECA;

  • the level of involvement as a function of the average number of subject moves in a dialogue and of their average length in characters.

  • the degree of initiative, as a function of the percentage of questions raised by the subject over all the dialogue moves.

Table 4 reports the label distribution in the two datasets, while Table 5 shows the measures of social attitude and the results of the t test on D1 and D2. Results show that the subjects belonging to the two groups actually demonstrate some differences in their behaviour while interacting with the ECA. The average number of moves per dialogue is not significantly different from a statistical point of view: we observe an average length (in moves) of 30.7–28.7, respectively, in D1 and D2. On the contrary, the move length and the percentage of social moves seems to be significantly lower in the D2 indicating that the subjects in the new group made generally shorter and less social moves (see Table 5).
Table 4

: The label distribution in the two corpora


D1 (%)

D2 (%)

Friendly self-introduction (fsi)



Colloquial style (cstyle)



Talks about self (talks)



Questions about the agent (qagt)




 Positive (pcomm)



 Negative (ncomm)



Friendly farewell (ffwell)



Table 5

: Measures of social attitude




Two-sided t test

% of social moves



P = .000007

Dialogue duration



P = .3981

Move length



P = .0000008

Avg % of questions



P = .8360

These results are also reflected in the different distribution of signs of social attitude reported in Table 4, reflecting a different social attitude in the two groups of users. In particular, the subjects in the new group (D2) demonstrate to be less keen to spend time in establishing social ground with the agent: they perform less moves in which they provide information about themselves (‘Talks about self’) and spend also less time in agreeing/disagreeing with the agent’s suggestions (‘Positive/Negative Comments’). Though, a common phenomenon to both D1 and D2 is the highest percentage of moves tagged as ‘Talks about self’ among all signs of social attitude. This may be interpreted as the intention of the users to actively participate, in both cases, in the advice-giving process by enabling the agent to provide user-tailored suggestions.

Overall, we observe differences in the behaviour of subjects in D1 and D2. Though, we need to be cautious. If it is generally true that subject in D2 is less warm towards the agent, this might not be true, for example for people who are particularly extroverted or are enjoying the interaction with the ECA (Bickmore and Cassell 2005). Also in D2, in fact, there are subjects who show a particularly warm attitude towards the agent, as demonstrated by the following examples in Table 6. This evidence is a further confirmation of our claim about the need to adapt the interaction style to the specific user’s characteristics.
Table 6

: Examples of warm social attitude in D2

Adjoint pairs

Signs of social attitude

Oz: And so? Tell me something more about you!


Subject: Valentina, I’m fine! What about you? What do you think about my diet? Is it good? I never eat fried food.

Colloquial style

Talks about self

Questions about the agent

Oz: I would suggest you to talk with an expert.


Subject: (by laughing) Ok, I will see an expert. But an expert in what, diet? I would rather see an expert… in wine!

Humour and jokes

Oz: You should avoid sweets or at least limit them only to particular occasions. I would suggest you to prefer fruit ice creams or biscuits with jam and to avoid sweets with whipped cream that are rich in fats or calories.


Subject: But those are not sweets! I would rather prefer to drop sweets from my diet! I like the cream cakes, I like the sweet stuffed, I like the pistachio, hazelnut, trifle ice cream. I don’t like fruit ice creams!

Negative comments

Colloquial style

Talks about self

Observing the user’s reaction to a system’s move

We analyse our new set of dialogues by applying the same method used for D1 (Mazzotta et al. 2007a) by asking three independent raters to annotate dialogue pairs (System move–User move), after segmenting complex moves into individual communicative acts. Results of the annotation (for both D1 and D2) are reported in Table 7. It shows that the overall users’ behaviour, in terms of reactions to suggestions, is similar in the two groups.
Table 7

The users’ reactions to the system’s persuasion attempts in the two corpora


D1 (%)

D2 (%)































In particular, we observe that, in the majority of cases, users of both groups react to a persuasion attempt by asking more information (53 and 48% of ASK, respectively, in D1 and D2). This data confirm our previous observation about the degree of initiative of users. Another interesting result is the different percentage of CONFIRM (9% in D1 and 19% in D2): subjects who spontaneously requested to interact with Valentina show more interest in the healthy eating domain since they often confirm the information received by the system.

The present annotation will be used in our future work to develop an automatic approach for users’ reaction recognition. In particular, we plan to combine contextual information about the dialogue history and the users’ cognitive state of mind with the results of Latent Semantic Analysis (LSA) (Landauer and Dumais 1997) of each dialogue turn in order to automatically infer communicative acts during the dialogue. LSA has been already employed to extract the semantic of students’ dialogue turn (Graesser et al. 2000) and might help us in recognizing the communicative act (Novielli and Strapparava 2010).

Final remarks

The study is part of an ongoing research aimed at developing an ECA playing the role of an artificial advisor in the healthy eating domain by applying natural argumentation techniques. In our previous research, we have demonstrated that users are not necessarily keen to establish a friendly relationship with an artificial agent. This may depend on their background and expectations (Clarizio et al. 2006): users with background in humanities demonstrated a more natural attitude towards the agent and made longer dialogues with respect to people with background in computer science; computer scientists, on the contrary, tended to be more formal and cold towards the agent and often adopted a ‘challenging’ attitude, as their main goal was to investigate the limits of the application and the actual ability of the agent to successfully manage the dialogue. The interaction mode also plays an important role in determining the users’ social attitude towards the virtual therapist (Novielli et al. 2010): the findings of our previous investigation agree with the results of the study by Oviatt et al. (1994) according to which spoken input entails longer utterances than written input. In particular, we observed that in the spoken interaction modality, users displayed a warmer social attitude and employed a richer language.

In this paper, investigate the role of voluntariness in determining the users’ reactions of people interacting with an ECA (Valentina). In our previous research, we perform the evaluation of a prototype of our system with people who did not spontaneously ask to interact with the ECA. This constituted one of the main limitations of our work. In fact, even if subjects involved in our previous experiment were similar in age and background at the potential users of the system, at the same time they were not properly motivated to interact with the agent. This causes that their behaviour may not reflect the one of the motivated users (the final user of the application). Therefore, in this work, we decided to make further investigations involving subjects who were interested in receiving information about healthy eating.

To this purpose, we recruited a new population of thirty subjects by advertising our dialogue system in a gym. People were told that they could have the opportunity to talk with an artificial therapist able to provide personalized suggestion about diet and healthy eating. We choose this scenario because we assume that it is easier to find subjects potentially interested in healthy eating among people who regularly go to the gym.

In this study, we compared the behaviour of the users involved in our previous studies with those who spontaneously requested to interact with the application with the goal of receiving suggestions about diet. We analysed their behaviour in terms of social attitude and reactions to the system suggestion and find that, overall, there are not considerable differences. The most significant difference was in term of percentage of social move: subjects in the gym group show less signs of social attitude in their moves indicating their less propensity to establish a social ground with the agent in favour of more goal-oriented interaction.

Though, we must acknowledge that some subjects in the new group show a friendlier attitude probably because they are particularly extroverted or the ECA behaviour matches their needs or expectations. Particularly, the first hypothesis is supported by the similarity between our signs of social attitude and some of the language features that characterize extraversion (Gill and Oberlander 2002): according to that study, extraverts tend to use a ‘relaxed’ and ‘informal’ style, make less use of the first-person singular pronouns and express a ‘positive affect’ more frequently; these linguistic markers are similar to those we included among our signs of social attitude.

This strengthens the basic assumption underlying our research, that is adapting the dialogue (both the persuasion strategy and the interaction style) is crucial for an effective persuasive interaction. Of course, in building the user model, several features should be taken into account from the permanent user’s characteristics (e.g., personality traits) to more transient variables that change during the interaction (e.g., how much the user shows he is enjoying interacting with the agent). That is, a successful virtual agent that plays the role of an artificial therapist has to model both the cognitive components of the user’s state of mind and some extra-rational factors such as affective states and attitudes.



We want to acknowledge Enrica Pesare for her cooperation in collecting and analysing the data and the ‘Body Energy’ fitness centre in Mola di Bari (IT) for kindly enabling us to perform the new WoZ experiments.


  1. Andersen PA, Guerrero LK (1998) Handbook of communication and emotions. Research, theory, applications and contexts. Academic Press, New YorkGoogle Scholar
  2. Austin JL (1962) How to do things with words, 2nd edn, 2005. Harvard University Press, Cambridge (Mass)—PaperbackGoogle Scholar
  3. Bickmore T (2003) Relational agents: effecting change through human-computer relationships. PhD Thesis, Media Arts & Sciences, Massachusetts Institute of TechnologyGoogle Scholar
  4. Bickmore T, Cassell J (2005) Social dialogue with embodied conversational agents. In: van Kuppevelt J, Dybkjaer L, Bernsen N (eds) Advances in natural, multimodal dialogue systems. Kluwer Academic, New YorkGoogle Scholar
  5. Cassell J, Joseph S, Sott P, Churchill E (2000) Embodied conversational agents. The MIT Press, Cambridge. ISBN 0-262-03278-3Google Scholar
  6. Clarizio G, Mazzotta I, Novielli N, de Rosis F (2006) Social attitude towards a conversational character. In: Proceedings of the 15th IEEE international symposium on robot and human interactive communicationGoogle Scholar
  7. Cohen PR, Levesque HJ (1990) Intention is choice with commitment. Artif Intell 42:213–261CrossRefGoogle Scholar
  8. Dahlback N, Joensson A, Ahrenberg L (1993) Wizard of Oz studies: why and how. In: Proceedings ACM international workshop on intelligent user interfaces. ACM Press, New York, pp 193–200Google Scholar
  9. Darves C, Oviatt S (2002) Adaptation of users’ spoken dialogue patterns in a conversational interface. In: Hansen J, Pellom B (eds) Proceedings of the 7th international conference on spoken language processing (ICSLP2002)Google Scholar
  10. de Rosis F, Novielli N, Carofiglio V, Cavalluzzi A, De Carolis B (2006) User modeling and adaptation in health promotion dialogs with an animated character. J Biomed Inform, Special Issue on ‘Dialog systems for health communications’ 39(5):514–531Google Scholar
  11. de Rosis F, Batliner A, Novielli N, Steidl S (2007) ‘You are Sooo cool, valentina!’ recognizing social attitude in speech-based dialogues with an ECA. In: Paiva A, Picard R, Prada R (eds) Affective computing and intelligent interaction, springer LNCS 4738, pp 179–190. ISBN 978-3-540-74888-5, ISSN: 0302-9743 (Print) 1611-3349 (Online), doi:10.1007/978-3-540-74889-2_17
  12. Gill AJ, Oberlander J (2002) Taking care of the linguistic features of extraversion. In: Gray W, Schunn C (eds) Proceedings of the 24th annual conference of the cognitive science society, pp 363–368Google Scholar
  13. Graesser A, Wiemer-Hastings P, Wiemer-Hastings K, Harter ND, Person, the Tutoring Research Group (2000) Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interact Learn Environ 8(2):129–147CrossRefGoogle Scholar
  14. Kibble R (2006) Dialectical text planning. In: Proceedings of CMNA 2006. In the scope of ECAIGoogle Scholar
  15. Landauer TK, Dumais ST (1997) A solution to Plato’s problem: the latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psycholog Rev 104:211–240CrossRefGoogle Scholar
  16. Mann WC, Matthiesen CM, Thompson SA (1989) Rhetorical structure theory and text analysis. Information Sciences Institute Research Report, pp 89–242, ISI/RR-89-242Google Scholar
  17. Marsella SC, Johnson WL, La Bore CM (2003) Interactive pedagogical drama for health interventions. In: Hoppe U (ed) Artificial intelligence in education: shaping the future of learning through intelligent technologies. Amsterdam, IOS Press, pp 341–348Google Scholar
  18. Mazzotta I, Novielli N, Silvestri V, de Rosis F (2007a) ‘O Francesca, ma che sei grulla?’ Emotions and irony in persuasion dialogues. In: Basili R, Pazienza MT (eds) Proceedings of the 10th congress of the Italian association for artificial intelligence on AI*IA 2007: artificial intelligence and human-oriented computing (AI*IA ‘07). Springer, Berlin, pp 602–613Google Scholar
  19. Mazzotta I, de Rosis F, Carofiglio V (2007b) Portia: a user-adapted persuasion system in the healthy-eating domain. IEEE Intell Syst 22(6):42–51. ISSN:1541-1672Google Scholar
  20. Miceli M, de Rosis F, Poggi I (2006) Emotional and non emotional persuasion. Appl Artif Intell Int J 20(10):849–880CrossRefGoogle Scholar
  21. Nass C, Isbister K, Lee EJ (2000) Truth is beauty: researching embodied conversational agents. In: Cassell J, Prevost S, Sullivan J, Churchill E (eds) Embodied conversational agents. The MIT Press, Cambridge, pp 374–402Google Scholar
  22. Novielli N, Strapparava C (2010) Exploring the lexical semantics of dialogue acts. J Comput Linguist Appl 1(1-2):9–26 ISSN 0976-0962Google Scholar
  23. Novielli N, de Rosis F, Mazzotta I (2010) User attitude towards an embodied conversational agent: effects of the interaction mode. J Pragmat 42(9):2385–2397. ISSN: 0378-2166Google Scholar
  24. O’Keefe DJ (2002) Persuasion: theory and research, 2nd edn. Sage Publications Inc., Beverly HillsGoogle Scholar
  25. Oviatt S, Adams B (2001) Designing and evaluating conversational interfaces with animated characters. In: Embodied conversational agents. MIT Press, Cambridge, pp 319–345Google Scholar
  26. Oviatt SL, Cohen PR, Wang M (1994) Towards interface design for human language technology: modality and structure as determinants of linguistic complexity. Speech Commun 15:283–300CrossRefGoogle Scholar
  27. Polhemus L, Shih L-F, Swan, Karen (2001) Virtual interactivity: the representation of social presence in an on line discussion. In: Proceedings of the annual meeting of the American educational research association, SeattleGoogle Scholar
  28. Prakken H (2006) Formal systems for persuasion dialogue. Knowl Eng Rev 21:163–188CrossRefGoogle Scholar
  29. Scherer KR, Wranik T, Sangsue J, Tran V, Scherer U (2004) Emotions in everyday life: probability of occurrence, risk factors, appraisal and reaction pattern. Soc Sci Inf 43(4):499–570CrossRefGoogle Scholar
  30. Searle JR (1975) A taxonomy of illocutionary acts. Language, Mind and Knowledge, Minnesota Studies in the Philosophy of Science, Minnesota, pp 344–369Google Scholar
  31. Swan K (2002) Immediacy, social presence and asynchronous discussion. In: Bourne J, Moore JC (eds) Elements of quality online education, vol 3. Sloan Center For Online Education, Nedham, MA, pp 157–172Google Scholar
  32. Verheij B (2003) Dialectical argumentation with argumentation schemes: an approach to legal logic. Artif Intell Law 11(2–3):167–195CrossRefGoogle Scholar
  33. Walton DN (1990) What is reasoning? What is an argument? J Philos 87(8):399–419CrossRefGoogle Scholar
  34. Walton DN (1996) Argumentation schemes for presumptive reasoning. Erlbaum, MahwahGoogle Scholar
  35. Walton D (2006) How to make and defend a proposal in a deliberation dialogue. Artif Intell Law 14(3):177–239CrossRefGoogle Scholar
  36. Walton D, Godden D (2005) The nature and status of critical questions in argumentation schemes. The uses of argument. In: Hitchcock D, Farr D (eds) Proceedings of a conference at McMaster university, pp 476–484. Ontario Society for the Study of ArgumentationGoogle Scholar
  37. Walton D, Gordon TS (2005) Critical questions in computational model of legacy argument, IAAIL workshop series, international workshop on argumentation. In: Dunne PE, Bench-Capon T (eds) Artificial intelligence and law. Wolf Legal Publishers, Nijmegen, pp 103–111Google Scholar
  38. Walton D, Reed C (2003) Diagramming, argumentation schemes and critical questions. In: van Eemeren FH et al (eds) Anyone who has a view: theoretical contributions to the study of argumentation. Kluwer, Dordrecht, pp 195–211CrossRefGoogle Scholar
  39. Zimmerman J, Ayoob E, Forlizzi J, McQuaid M (2005) Putting a face on embodied interface agents. In: Proceedings of the conference on designing pleasurable products and interfaces. Eindhoven Technical University Press, pp 233–248Google Scholar

Copyright information

© Marta Olivetti Belardinelli and Springer-Verlag 2012

Authors and Affiliations

  • Nicole Novielli
    • 1
  • Irene Mazzotta
    • 1
  • Berardina De Carolis
    • 1
  • Sebastiano Pizzutilo
    • 1
  1. 1.Intelligent Interfaces, Department of InformaticsUniversity of BariBariItaly

Personalised recommendations