About the participant sample
The sample consisted of responses from 207 valid participants. Of these, 62% were male and 38% were female. Mean age was 27 years (SD = 8). The participants used chatbots on a variety of messaging platforms, notably Facebook Messenger (79%), Skype (54%), Kik (38%), Viber (12%), Slack (10%) and Telegram (4%). Further, 65% reported using chatbots daily or weekly, 48% reported having used chatbots for 3 or more years and 40% had experience with Google Assistant.
The chatbots reported as the most recently used by the participants reflected a wide range of chatbots: for productivity purposes, marketing, customer service and entertainment. The most reported recently used chatbots were virtual assistants such as Google Assistant (18%), Siri (7%), Alexa (4%) and Cortana (4%) and chatbots for social chatter such as Cleverbot (11%), Eviebot (3%), Mitsuku (1%), SimSimi (1%), Zo (1%) and the no longer operative Smarterchild (3%). The frequent mention of Google Assistant as a recently used chatbot may be due to its availability on the Android operating system.
Positive chatbot experiences
Participants reported a broad set of positive chatbot experiences. The reports were on average 105 characters long (SD = 90). Nearly all reports provided sufficient detail to identify one or more characteristics of the experience.
Although we asked the participants to report on a specific positive episode, not all of them did. Rather, 45% reported on a specific episode, whereas 22% reported on their overall experience with a specific chatbot. The remaining participants made high-level reports of chatbot attributes regarded as positive without mentioning a specific episode or chatbot. To provide a feel for what the participant reports look like, we include the following two quotes, which exemplify reports of a specific episode (P37) and reports of a specific chatbot (P40).
I actually recently interacted with a chat bot about a complaint I had with a company. The chat bot informed me the correct way and persons to send my information to. It was quick and easy and I really appreciated this since I was already quite annoyed (P37)
I get Cortana to tell me a joke—on Windows 10 (P40)
Following Hassenzahl's [23] pragmatic-hedonic framework for user experience, the participant reports were analysed with regard to whether they reflected pragmatic attributes of an interactive system, such as usefulness and usability, and/or hedonic attributes, such as facilitating evocative or stimulating experiences.
In the user reports, we found an appreciation for both pragmatic attributes and hedonic attributes in participants' detailing of positive chatbot user experiences; 42% of the participant reports reflected pragmatic chatbot attributes and 36% highlighted hedonic attributes. In addition, 20% of the reports reflected codes that are not directly related to pragmatic or hedonic attributes. The most frequent of these additional codes concerned the social aspects of an interaction (7%) and the chatbot's character as humanlike (4%).
The distribution of codes for the participant reports of positive chatbot user experiences is provided in Table 1. Details concerning each coding category are provided following the table.
Table 1 Coding categories for positive chatbot user experience reports, with associated attribute type, descriptions and frequencies (N = 207) Pragmatic attribute: help and assistance (34%)
The participant reports strongly reflected the importance of perceived usefulness or practical value for positive user experience. When asked to report a particularly good episode, participants often reported on getting assistance or help from the chatbot. A number of the reported episodes concerned customer service support and also instances of training or coaching through the chatbot. Other episodes concerned personal assistance, such as setting reminders for tasks or getting help with a specific task at hand, as in this quote:
I asked what good places there was around me to eat and it brought up a list and i chose from it. Now the place is one of my favorite places to eat at (P21)
The instrumental or pragmatic characteristics of chatbots were clearly apparent in the reported episodes, where task achievement and efficiency in particular were highlighted as important in numerous participant reports. Participants reported receiving help in situations where they were pressed for time because of an urgent problem or a circumstance where they needed information quickly. Participants also made particular note that the assistance was efficient and easily accessible, as in the following example:
The chatbot for customer support for my wireless carrier was a great help! I didn't have to wait on a representative to become available, I was able to find out what I needed to know about different plans and their pricing. So much better than sitting on hold or waiting an hour for someone to message me back (P23)
Pragmatic attribute: information and updates (8%)
While help and assistance for a particular task was by far the most frequently reported category of positive user experience episodes, some participants instead reported on the pragmatic benefit of chatbots for general information searches or more routine updates, such as news reports and weather forecasts. These participants highlighted the chatbots' support for retrieving general online information or daily updates, rather than getting help in a particular situation. Participants reported gathering information through a chatbot that they would otherwise be able to access through a search engine. Participants also reported using chatbots for doing their everyday checks of information important to their daily routines, for example in the following quote:
I use google assistant to do simple tasks on my phone every morning when I am waking up and need to know the time and weather […] (P59)
Hedonic attribute: entertainment (29%)
In their reports, participants indicated substantial appreciation for hedonic chatbot attributes. When reporting on such non-pragmatic aspects of chatbot user experience, participants typically highlighted the entertainment value of chatbots. Entertaining chatbot episodes were presented in ways that indicated they were seen as stimulating and contributed to the participant feeling happy and engaged. Participants used words such as ‘fun’, ‘entertaining’ and ‘cool’, as in the following example:
It was funny. I asked it if it liked me and it asked me if I like me (P207)
Participants who reported the entertainment value of chatbots typically referred to situations where they engaged in small talk with a chatbot. That is, they often did not have a particular task to be resolved but rather saw the chatbot as a means of involving themselves in a pleasing activity. Specifically, they reported that the chatbots’ ability to be funny and witty was a source of pleasure, or they reported that the chatbot was something they could joke with or turn to when bored. An example of such use is reflected in the following quote:
Chatbot and I just kept talking random things, that when looked at after made some sense. it was fun (P128)
To our surprise, quite a few participants who reflected on hedonic aspects of the user experience reported on the use of chatbots by children. Some of these reports were from their own childhood; for example, participants reported that conversations with chatbots as a child were a source of entertainment in the company of their friends, or a source of relief as a teenager when they were bored. Other participants described experiences as parents—observing their own children engaging with chatbots, either on the initiative of the parent or through the child's own initiative. The finding that chatbots serve as a source of stimulation and engagement for children is interesting, as it suggests the potential of chatbots to stimulate playful social interaction for and possibly also among children.
My earliest memories of artificial intelligence are with an online chatbot called SmarterChild. I remember it being pretty funny sometimes, witty and intelligent, almost like it was a real person behind the character typing his responses (P132)
Hedonic attribute: inspiration and novelty (8%)
Some of the participants who highlighted hedonic chatbot attributes in their descriptions of good chatbot experiences reported an inspirational episode or a general sense of novelty in chatbots.
Among the participants who reported on the chatbot as inspirational, some described the episode as ‘eye-opening’ or described how they were able to talk to the chatbot about a topic that engaged them, such as pets or food. Such reports in part reflect the potentially evocative character of chatbots and in part reflect the potential of chatbots to adapt to topics with which the user identifies, as in the following participant quote:
I had a pleasant conversation about my life with a chatbot. I talked about my family and my feelings (P182)
Some participants also reported being excited or engaged by their perception of chatbots as a novel and fascinating way to interact with computers. In these reports, some participants explained how they saw it as amazing to actually have a conversation with a computer, and some also described how they had tried to test the degree to which the chatbot is able to act like a human. The following quote exemplifies participant reports belonging to this coding category:
I had an interesting experience trying out artificial intelligence through small talk with a chatbot. You could tell it wasn't human but it was interesting nonetheless (P83)
Other attributes: social and humanlike (11%)
While the coding categories for pragmatic and hedonic attributes are reflected in most of the participants’ reports, some reports referred to attributes that did not readily fit into the pragmatic-hedonic framework but were still relevant to user experience. In particular, this was the case for participant reports about the social value they received from using the chatbot (7%) as well as the perceived experiential benefit of the chatbot being humanlike (4%).
Social value typically involved enjoying a social situation with the chatbot. In these reports, the participants described how they appreciated the social interaction with the chatbot. That is, even though they were aware that the chatbot is a machine, the social interaction was seen to hold value in itself, as in the following example:
Chatbot helps me get my day moving when I don't talk to anyone (P141)
For some participants, the chatbot was used to support social interaction with other (human) users, as for example in group chats. Here, the chatbot could serve instrumental purposes in a social interaction, such as providing linked content on topics of conversation or helping to get conversations or groups started.
I used chatbots to send links to websites mentioned in Skype conversation. It was very convenient way to make the conversation more efficient (P45)
The identified humanlike attribute was in many ways associated with the social attribute. Here, participant reports explained how chatbot characteristics that are almost human may contribute positively to user experience. The humanlike character of chatbots was noted by some participants, but fewer than might have been expected given that this is often seen as a prominent chatbot attribute.
I use to ask them all kinds of questions till they had a whole conversation with me told me where they were from an how they worked as a waitress at a bar it was such a funny conversation i actually thought the chatbot was a real person (P34)
In reports that emphasize the social or humanlike attributes of chatbots, these attributes were often discussed together with hedonic or pragmatic attributes. We nevertheless found it important to single out the social aspects of chatbots as reflecting a distinct attribute outside the group of hedonic attributes because the pragmatic-hedonic framework does not specifically address social interactions with interactive systems. It should be kept in mind that our coding scheme allowed any user report to have multiple codes associated with it, so this represented no challenge in terms of coding.
Negative chatbot experiences
We asked the participants to report on poor or unpleasant chatbot user experiences. In these reports, the most frequent characterizations of the chatbot and the user experience involved pragmatic (23%) and hedonic (16%) attributes, with pragmatic attributes being the most prevalent.
The distribution of codes for the participant reports on negative chatbot user experiences is provided in Table 2. Details concerning each coding category are provided following the table.
Table 2 Coding categories for negative chatbot user experience reports, with associated attribute type, descriptions and frequencies (N = 207) Before we present the findings, it should be noted that the participants were much less inclined to report negative experiences than they were to report positive ones. In fact, 41% of the participants reported that they had not had a bad experience with a chatbot (first wave of data collection) or they had not stopped using chatbots (second wave of data collection). It should be noted that no participants skipped this question (mandatory question); rather, the participants who reported not having negative experiences did so in their own words.
This lack of negative episodes was a surprise to us as chatbots are still a relatively immature type of interactive system. One possible explanation for this lower frequency of reports of negative user experiences is that the respondents are relatively early adopters [40] and are hence more tolerant of technical issues or interaction breakdowns in this emerging technology. Another explanation may be a participant response bias, where study participants might hesitate to make negative assessments of a technology under study [10]. It could also be that different phrasings of our request for negative experiences, for example asking for negative experiences in general, would have increased the frequency of negative experience reports.
Nevertheless, the number of reports actually detailing negative user experiences was more than sufficient to establish coding categories and provide insight into chatbot attributes that potentially drive such user experiences.
Pragmatic attribute: interpretation issues (11%)
Not surprisingly, the chatbot attributes reported to drive poor user experiences often concerned pragmatic attributes—specifically, usability. In particular, interpretation issues were prominent. A substantial number of the participant reports of negative episodes concerned the perceived challenge of making the chatbot understand what the user was trying to tell it.
Interpretation issues could be framed as a general frustration that chatbots sometimes need questions or requests to be repeated, or that they are not able to correctly interpret the user input at all. For example, participants detailed how they had to adapt their way of expressing themselves in order to be understood by the chatbot, or they reported that they stopped using chatbots that were unable to understand their input.
Each time I answered a question it would not read my response or respond asking me to repeat my answer. I eventually got annoyed and exited out of it (P43)
Pragmatic attribute: unable to help (11%)
Linked to the issue of interpretation, poor chatbot user experience was often reported when the chatbot was unable to help. In usability terms, participants reported on low effectiveness in the chatbot. Such low effectiveness may be deeply frustrating, as it may completely compromise the potential benefit of the chatbot. The reported lack of help could be due to the chatbot providing an answer that was too generic, and in some cases due to the chatbot's answer being irrelevant to the participant's task at hand. Some participants also noted that they did not trust chatbots or suspected that the chatbot in question provided false information. The following participant quote exemplifies the experience of users when chatbots are unable to help:
When I ask a question, some of the info they provide are completely irrelevant. Still kinks to be ironed out. It's still easier to just Google search for answers to your questions (P68)
Pragmatic attribute: repetitiveness (4%)
A third pragmatic chatbot attribute reported to generate negative user experience was the repetitiveness found in some chatbots; that is, the experience that the chatbot just keeps reiterating the same questions or responses. Participants reporting on repetitiveness described chatbots that ask the same thing over and over without making progress towards the intended task goal. This may be in the context of customer support or information seeking, for example where the chatbot is not able to derive the information needed from the participant in order to progress in its routine. Some participants also reported on chatbots that repeatedly suggested taking the same steps to resolve a problem, even though those steps have already proven unfruitful.
There have been many occasions where a bot has either looped around with its inquiries, and/or not had the info I needed. It made me feel like I had just wasted a good deal of time (P14)
Hedonic attribute: strange or rude responses (7%)
Whereas pragmatic aspects dominated the reports of negative episodes, some reports reflected the hedonic or emotionally charged aspects of the user experience—in particular with regard to a tendency for chatbots to provide strange or rude responses. Rude responses are understood as chatbot responses that are at odds with what the participant sees as acceptable; strange responses are understood as chatbot responses that are seen as off-topic—though not in a task-oriented context. Some participants reported that rude and strange responses were embarrassing, possibly because they were seen as breaking with the promise of intelligence that natural language interaction suggests.
When i asked a question at a friends house and it brought up an answer that was not only irrelevant but dirty (P66)
In a sense, strange or rude responses from chatbots may be seen as somehow related to chatbots being witty and funny. Different users may even interpret the same chatbot response as either funny or embarrassing.
Hedonic attribute: unwanted events (6%)
In addition to strange or rude responses, the participant reports contained references to unwanted events as a negative hedonic chatbot attribute. Participants reported on chatbots that had contacted them at times or in ways that were unsuitable, chatbots taking actions they did not want them to take or chatbots presenting unwanted content. In particular, participants reported on chatbots initiating contact at times when the participant would rather not be disturbed, as in the following example:
I was sleeping one time and the chatbot started texting via messenger and i got really angry because i was sleeping real good and it woke me up (P21).
Such unwanted contact was typically reported to be disturbing, likely because it was experienced as the chatbot invading the user’s private life or as the user losing control of how and when the chatbot interacts with them.
Likewise, unwanted actions from chatbots were seen as frustrating and could potentially hold substantial negative consequences:
I used the chatbot to send an email and it sent it to the wrong person in my contacts (P18)
Hedonic attribute: boring (4%)
In addition to the two hedonic chatbot attributes discussed above, some participants noted that their negative chatbot user experience was due to the perception of the chatbot as boring. Chatbots are an emerging technology, so it is not surprising that a chatbot may be perceived as boring once the novelty wears off. As such, the perceived boring character of some chatbots can be seen as a likely outcome, considering that the novelty of chatbots was found to be a driver of positive user experiences for some participants.
In particular, boredom or lack of experiential value over time were seen as important reasons to stop using chatbots, as seen in the following participant quote:
I stopped using MurphyBot because the novelty of it wore off. Since it mostly was used to search for and meld images together, it got boring pretty quickly (P79)
Age differences
Because participants often mentioned childhood events when sharing positive episodes of chatbot use, we wanted to investigate whether age was related to which chatbot attributes the participants highlighted in their reports.
We had not initially planned this investigation. However, reports of childhood memories or experiences of enjoying talks with chatbots suggested the relevance of doing such an investigation. For this purpose, we conducted two bivariate correlation analyses to investigate the degree to which age predicts whether a participant will report pragmatic or hedonic chatbot attributes respectively when describing positive user experiences. We applied the Spearman rank correlation, as the age data was found to be non-normal following a Shapiro–Wilk test of normality.
In these analyses, we found that greater age was positively associated with the tendency to report on pragmatic chatbot attributes, that is, help and assistance and information and updates (Spearman's Rho = 0.31, p < 0.001). In contrast, greater age was negatively associated with the inclination to report on hedonic chatbot attributes, that is, entertainment and novelty and inspiration (Spearman's Rho = − 0.25, p < 0.001).
In other words, older participants tended to report on pragmatic chatbot attributes, whereas younger participants tended to highlight hedonic chatbot attributes. Effect sizes were medium following Cohen’s rules of thumb [15].