Keywords

1 Introduction

The term ‘user experience’ is often discussed in HCI literature, however, there are in fact many different definitions of user experience [1]. The origin of the term is usually attributed to Donald Norman, who sought to broaden the focus of usability research to encompass emotions, thoughts, and context in addition to the traditional focus on task performance. While some definitions refer to user experience as a specific episode, we prefer to consider user experience as an accumulation of a series of episodes. Thus, we prefer the UPA definition, “every aspect of the user’s interaction with a product, service, or company that make up the user’s perceptions of the whole.” We consider User Experience as a journey that consists of numerous episodes in which the user interacts with technology or otherwise thinks about the technology or technology collateral.

Although this framework might be generally accepted by the HCI community, relatively little has been done to describe or understand how this journey unfolds. Our intent in this study is to examine the temporal aspect of user experience; how the user experience unfolds over time, and how individual episodes might contribute to an overall assessment of experience.

Another aspect of our research is to focus on the domain of wearable devices; specifically, smart watches. It is widely reported that smart watches and other wearables (such as fitness bands), have a high rate of abandonment. An often described scenario is that a user will acquire a wearable device, use it for a while, lose interest in using it, and put it away in a drawer. We are interested in determining why users do this. What is it about the user experience that leads a user to stop using it? What are the differences between users who continue to use a device, and those who stop? Presumably, people purchase a device with the expectation that they will use it. What experiences lead them to abandon the device? What are the differences between people who continue to use a smart watch and those who stop using it?

2 Background and Related Work

2.1 Theories of User Experience

Donald Norman proposed a three tier theory of user experience. In this framework, experience occurs at three levels: visceral, behavioral, and reflective [2]. The visceral level is the emotional reaction that the user experiences in the moment of interaction. It is the ‘gut reaction’ that users have, and can be elicited by the aesthetic design of a device, the touch and feel, animation, etc. Visceral reactions may be sub conscious and out of our control. The behavioral level constitutes the thoughts and behaviors we experience during interaction with the technology. At the behavioral level, it is the operations and conscious, intentional actions that affect our reactions to the experience. The reflective level occurs after the experience itself. Reflective experience is the combination of our memory of the experience and the interpretation and value judgments assigned to that memory. As pointed out by Norman and others, the reflective, remembered experience can be quite different than the in-the-moment experience [3–5].

2.2 User Experience Over Time

However, as we have already pointed out, user experience over a period of time is the culmination of many experiences. There have been relatively few studies that examined user experience over time. Kujala and Miron-Shatz [6] studied 27 mobile phone users, finding that the memory of emotions felt (both positive and negative) were overestimated, and that positive peak episodes were related to emotional responses and usability evaluation.

Karapanos et al. [7] proposed a framework of user experience over time, based on a study of 6 Apple iPhone* users over the period of one month. In analyzing respondents’ daily diary entries, they proposed that three phases occur: Orientation, Incorporation, and Identification. In each of these phases, different forces are at play, and different product qualities assume more influence. In the Orientation phase, Familiarity is the driving force, while aspects of attractiveness and learnability play key roles in assessment of the experience. During Incorporation, Functional Dependency is the key driver, and usefulness and long-term usability are key to experience assessment. Finally, users enter the Identification phase, in which Emotional Attachment is the driving force, while personalization and social considerations assume more focus in evaluation.

Another recent study tracked smart watch usage in detail using wearable cameras to capture watch interactions of 12 participants [8]. The intent was to understand how smart watches are integrated into everyday life. This study found that the flood of messages that users received were generally seen as being brief enough that they did not take users out of the flow of most activities, and this aspect was seen as a positive by most user.

However, one question which is not addressed by any of the longitudinal studies is which types of users are likely to find longstanding value in the device, and which types of users do not. In this study, we attempt to find answers to this question.

3 Methods

3.1 Recruiting and Data Collection

We knew that even with a highly anticipated, well publicized release of a smart watch, it is a challenge to locate a large sample of users who actually would purchase the device and who would agree to participate in a long term study. Therefore, we used dscout*, a consumer research company to recruit from their panel of users people who had pre-purchased the Apple Watch prior to the release of the product. The only recruitment criteria was that they could prove that they had actually pre-purchased the watch, and that they agreed to answer the survey questions and perform certain ‘missionsFootnote 1’ throughout the length of the study. The initial recruit population consisted of 137 users. We wound up analyzing the data from 90 of those users who responded to all three of the detailed surveys at pre-purchase, 30 days, and 120+ days. A subset of 47 users also provided detailed, step by step data on their out-of-box experience within the first 24 h of receiving the watch, and 30 of those users also provided detailed videotaped and written evaluations of at least 12 features of the watch (a suggested list of features was supplied, but users were free to select their own features, as long as there were at least 12 selected).

We collected a wide variety of data through the four months of the study, including videotaped ‘snippets’ (performed by the participants using their cell phone) and on-line surveys. The surveys contained open ended questions as well as scale measures. Here are some examples of the open ended questions:

  1. 1.

    “In a couple of sentences, tell us why you decided to purchase the Apple Watch.” (asked prior to receiving the watch)

  2. 2.

    “Tell us what’s working or going well in this moment with the Apple Watch. Please be specific.” (asked in the first 24 h of receiving the watch)

  3. 3.

    “Tell us what’s NOT working or not going so well in this moment with the Apple Watch. Please be specific.” (asked in the first 24 h of receiving the watch)

  4. 4.

    “In a few sentences, describe a specific experience with your Apple Watch that really stands out to you. What were you doing with the Watch? Where were you? This experience can be either a positive or a negative one, just as long as it’s significant!” (asked after using the watch for about 30 days)

In this paper, we will focus on a few measures: the Net Promoter Score [9], the most memorable experience, how often they wear the Apple Watch at 120+ days, and how well the watch has met initial expectations.

3.2 The Net Promoter Score

Our interest in following the user experience over time was to assess how the user experience affects the overall assessment of a smart watch. While we collected information about usability and usage, we also included the Net Promoter Score as one of our benchmark assessment methods. The Net Promoter Score (NPS) has been touted as a single score that can predict product success [9], and, while there is some controversy about its validity [10], it is generally accepted as a valuable measure. We used this because the success of smart watches as a product category is a major focus of our research, and because its definition and use is not explicitly tied to User Experience measurement, and can act as a more objective standard of product assessment. The NPS is an 11 point scale (0 to 10) in which the respondent is asked, “On a scale of 0 to 10, how likely are you to recommend this product (or brand) to a friend or family member?” We asked users to rate their Apple Watch on this scale at several points during the study.

3.3 Population

Of the 90 users, 27 were females and 63 were males. Subjects all resided in the US. Sixty subjects purchased the Sport Watch and 30 subjects purchased the more expensive Apple Watch. Subjects ranged in age from 18 to 58.

4 Results

4.1 Overall Experience Findings

At 30 days and 120+ days of owning their watch, subjects were asked to report whether they were wearing their watch more often, about the same, less often, or had stopping using the watch altogether. The data are shown in Table 1. Of the 90 subjects, 6 people reported that they had stopped using their watch versus 0 at 30 days, and 18 people reported using their watch less often than in the first week, versus 13 people at 30 days. This is a statistically significant difference (Chi Square, 8.12, p = 0.044).

Table 1. Frequency of using the Apple Watch at 30 days vs 120+ days
Full size table

Table 2 shows the average Net Promoter Score for each of the categories of user. For those who stopped using the watch, the NPS average was 2.83 versus 8.62 for those who reported using the watch more often.

Table 2. Average Net Promoter Score by use of Apple Watch at 120+ days
Full size table

Subjects were also asked an open ended question at 120+ days, “We know it’s been a while, but after owning your Apple Watch for over 90 days, how does it stack up to your initial expectations? In a few sentences tell us - is it meeting them, exceeding them or falling short and WHY?” The 90 responses were analyzed in categorized in the following way. If the answer was unequivocal, such as “The Apple Watch has exceeded my expectations and the reason is ..”, responses were categorized as such. There were two other types of responses: 1. Those who stated “exceeded”, “met”, or “did not exceed”, but then went on to describe aspects that countered their claim, 2. Those who did not explicitly state whether expectations were met or not, but made other statements that required interpretation.

The authors conducted a session with three other HCI experts to code the responses. Dividing up the responses, two experts independently rated each response. Any disagreements were discussed between all of the experts and a unanimous decision was obtained. Table 3 shows the results.

Table 3. Apple Watch expectations assessment at 120+ days by number of responses and average Net Promoter Score.
Full size table

4.2 Types of User Experience Journeys

As we looked at the quantitative and qualitative data provided by the subjects, we began to see patterns in the types of journeys described. We conducted detailed user journey analyses on 30 subjects who had provided both detailed feedback on the first 24 h, and detailed feedback on at least 12 features of the Apple Watch, in addition to completing the surveys at pre-purchase, 30 days, and 120+ days. For these 30 subjects, we had a rich amount of quantitative scales, videotaped snippets of their experiences, and open ended responses to a variety of questions throughout the four months of use.

We identified three types of user journeys: 1. The Communicator, 2. The Tool Techie, and 3. The Detractor.

The Communicator.

The communicator is best represented by Micah. Micah lives in Wisconsin, and previously used a Jawbone UP. Prior to receiving the Apple Watch, Micah expected to use his watch to help him stay connected to friends and family. He was a bit apprehensive about the battery life of the watch. After the first 24 h, Micah was delighted with the watch appearance, saying it was “sexy, sleek, and sophisticated.” He had already tried Apple Pay* and Siri* voice recognition, and was impressed. However, his first 24 h were not without some frustration. He was disappointed that there was no sleep tracking (unlike his Jawbone), and some of the apps did not install correctly during setup. At 30 days, Micah’s had changed his focus to the ability to stay connected via his watch. He reported that notifications filled him with excitement and delight, and phone calls made on the watch were clear and without distortion. The experience that most stood out to him was the episode in which he was able to share a digital “touch” with his mother, calling it a “magical moment.”

With Micah and other “Communicators” like him, it was clear that the convenience of staying current with text and email, combined with new communication features such as digital touch, were what stood out to him and kept him engaged with his Apple Watch throughout the study. Micah’s expectations were well met in providing him with a convenient method of staying connected, while his concerns about battery life and activity were alleviated. After four months, Micah still gave his Apple Watch a ‘10’ on the Net Promoter Score.

The Tool Techie.

The tool techie is best represented by Alex. Like Micah, he still uses his Apple Watch after four months, and gives it high ratings. However, the reasons why Alex loves his watch are quite different. Prior to receiving his watch, Alex expects the watch will be a convenient tool for easily accessing quick snippets of information. Unlike Micah, he does not list “staying connected” as one of the reasons for purchasing the watch. He expresses a bit of concern over lack of support for third party apps. For Alex, it is important that the watch provide a multitude of convenient tools and information.

In the first 24 h, Alex is impressed with the physical appearance of the watch, but most likes the Hue app that he installed, allowing him to change the light settings in his house, and the New York City public transit app, which allows him to see bus schedules from his watch. He is also excited about the activity tracking capabilities. He reports that “Everything works great, and it seems like I’m constantly finding new things I can do with it.”

At 30 days, Alex cites Maps and Navigation and Siri as his ‘top features’, in addition to the Hue app. He feels that the remote control app for his Apple TV* is ‘futuristic’, but a little clumsy to use. He likes the text messaging for short messages, but reports that he winds up using his phone for most meaningful communication. Still later at 120+ days, Alex lists Fitness Tracking, Apple Pay*, and Passbook as his most important features.

One aspect we noticed with Tool Techie user types is that, although they usually wound up liking and continuing to use the watch, their reasons for purchasing the watch often did not line up with the reasons they wound up liking it. Alex, for example, did not have much in the way of clear expectations, but was simply anticipating lots of ‘cool tools’.

The Detractor.

The detractor is a person who is ultimately disappointed in the watch, and either stops using it or greatly reduces use of it. In all, about 27 % of users fell into this category. Tito represents a typical detractor. Prior to receiving the Apple Watch, he states that he hopes the Apple Watch is better than his Pebble* watch. Thus, unlike the communicator and the tool techie, his expectations are lower and have a bit of a negative slant.

In the first 24 h, Tito’s reactions, however, are quite positive. He likes the packaging and finds it easy to put on and comfortable. He is also impressed with the accurate voice recognition of Siri. However, like many others, he is a bit disappointed by the length of time required to do the app update during setup. At 30 days, he has found few capabilities that he really likes or finds important. The feature he mentions as ‘nice’ is the ability to set a timer using Siri (voice). He finds value in receiving text messages, but sending them is ‘tricky’, and navigation via maps, in his opinion, ‘sucks’. Most tellingly, his most memorable experience lacks any sort of interactivity:

“I took my Apple Watch off for a day. Then, I kind of forgot about it. It ended up underneath the seat of my car when I found it a week later.”

At four months, Tito had stopped using the watch.

For the most part, detractors all had different reasons for not liking their watch. All claimed that the watch had fallen short of their expectations, but initial expectations were often vague for these people. What is clear, however, is that all detractors failed to find a compelling usage for the watch that overrode the irritations and negative episodes. Their final assessments were not so much negative, but rueful:

“(It is) Falling short (of my expectations). I haven’t worn it for 2 months. I’m not heartbroken tho, it just seems like a flop to me. I’m perplexed. I bought the original iPhone immediately and loved it. Not the same here.”

4.3 The Most Memorable Experience

After 30 days, we asked all 90 users the following question:

“In a few sentences, describe a specific experience with your Apple Watch that really stands out to you. What were you doing with the watch? Where were you? This experience can be either a positive or a negative one, just as long as it’s significant!”

We analyzed all 90 open ended responses and categorized them, looking for themes of the experience that most stood out to users. Our intention was not to identify the most liked or disliked features, but to identify themes that characterized the content of these stories. As a result, we have identified key value propositions for smart watches.

Micro Interactions.

Micro interactions were often described by subjects as quick updates that appear on the watch that requires minimal attention and time. The quote below is typical of these types of episodes.

“Sometime in that first week of having the watch, I was doing dishes. I got a text message, I was able to look at it, used Siri to respond to it, and never actually had to stop what I was doing. The Apple Watch shines in these moments - small, quick interactions that would otherwise take me out of the moment.”

Clearly, a key aspect of the value of smart watches is that they allow for quick interactions that are prohibitive on smart phones. In a recent study, the average interaction time for smart phones was 38 s, while the average time for smart watches was 7 s [8]. One of the conclusions of that study was that smart watch interactions, unlike smart phone interactions, don’t generally distract users from their current task because of the brevity of most interactions.

Unexpected Features.

Many episodes were described in which the user discovered a feature that they either did not know about, or that appeared in a surprising way. For example:

“My real stand out experience was using the maps app. My girlfriend and I were traveling by car to a restaurant we had never been to before. I asked for directions on my iPhone and the maps app started to give me directions, however to my surprise my watch started tapping me on the wrist - it too was giving me directions even though I hadn’t asked for them on my watch! I thought this was really clever and a good example of how the watch can help you in your daily life.”

Delightful Interactions.

Delightful interactions were characterized by non-essential features that were nevertheless memorable in their novel interaction style. These interactions often appeared as novel effects or animations that enhanced an otherwise mundane experience (Fig. 1).

“I think my most significant experience with the Apple Watch was when I first opened it, turned it on, and paired it with my iPhone 6 Plus. I was wowed by the ability to pair the two devices via the camera. I was ready to go through a more tedious process.”

Social Status.

Some people described experiences in which the watch features or functions were not central – rather, the experience was in the reactions users received from other people.

“The biggest thing that stands out to me was when I was walking in the mall with it on and some random guy walks up and asked if it was the new Apple Watch. When I said “yes” he started hollering and making a big scene about how awesome it looked. That was when I felt the best about making my decision to buy the watch.”

While some may argue that the Apple Watch in particular may be different in this regard because of the high publicity around the watch and the brand value of Apple*, we would argue that social status is an aspect of many new technologies, such as owning the latest large UHD TV or a new car.

Fig. 1.
figure 1

The pairing process for the Apple Watch involved ‘capturing’ a novel animation with the iPhone.

Full size image

Discreet Updates.

Discreet updates were similar to micro interactions, except the central aspect mentioned was that the interaction was not noticed by others, and the discretion afforded by the device was the central aspect of the watch.

“I was in class during a presentation and i also needed to communicate with my brother on a time sensitive matter, i was able to do it discretely with my watch without attracting the attention of my strict professor, that was when i knew i made the right purchase.”

5 Discussion and Conclusion

While 27 % of the Apple Watch users reported using their watch less or not at all after 120 days, this may be in fact a very positive score for smart watches overall. To our knowledge, there have been no similar data reported on other smart watches or wearable devices. Anecdotally, the consensus seems to be that there is widespread abandonment of a variety of wearable devices – more than we have seen in this study.

The other way to look at it is that 73 % reported using the watch the same or more often than they initially used it. The most typical feeling expressed by these respondents is exemplified by the following response:

“…it’s completely integrated into my life. It’s lost the “wow” factor on me because I’m used to it by now, but that’s not to say I’m not in love with it. I’m not as excited about it anymore because I’m accustomed to using it.”

We found some evidence that supports the framework proposed by Karapanos et al. [7], that identification occurs, at least for those whose initial expectations were met. The contribution of this work is to point out some of the aspects in which technology fails in the incorporation phase. This occurs when users do not find compelling value that leads users to integrate the technology into their daily routines. Sometimes this was because an intended usage did not meet expectations, or because of an unforeseen barrier to usage arose. For example, some users reported ongoing connectivity problems or very slow responsiveness, things that were not expressed as concerns initially.

We identified three types of users and their user experience journey for the Apple Watch: Communicators, Tool Techies, and Detractors. Communicators bought the watch for the potential to enhance their ability to stay in touch, and were generally very satisfied with the value provided. Tool Techies weren’t always so sure what they would find valuable in the watch, but found a variety of valuable features, from fitness tracking to payment, to IOT apps such as Hue. The Detractors were generally less enthusiastic than others before purchase, and several reported previous bad experiences with other technology. Detractors simply failed to find enough compelling value to want to use the watch on a daily basis.

We have only begun to touch on an understanding of the user experience journey for wearable devices, or for technology in general. The current study was in some ways unique given the unique nature of Apple products and the Apple enthusiasts who buy their products. Much more research into other brands and types of wearables needs to be done to generalize the findings here, but at least there is now a baseline against we can begin to compare with other products.