Keywords

1 Introduction

Studying user experiences in the wild has seen an increase in popularity within the field of Human-Computer Interaction (HCI) [34]. Research in the wild is aimed at understanding behavior and technology use in people’s everyday lives outside the confinements of the laboratory [34]. Using data collection methods from the domain of lifelogging complements classical ethnographic approaches typically used to study in the wild phenomena [34]. Lifelogging research emphasizes the use of technology to make participants reflect on and report events from their everyday lives. Such technology could involve mobile contextual sensors, e.g. GPS location data to track where participants have been over the course of a day. However, wearable cameras, particularly the SenseCam seems to be the most emphasized technology to support data collection in Lifelogging studies, see e.g. [2, 5, 20, 23, 25, 32, 36]. SenseCam is worn around participants’ neck and captures an image every 30 s or when the user chooses to take a photo manually. Images taken through a wearable camera have proven to be very effective cues for study participants in recalling and describing past events [36]. This makes such technology useful in complementing traditional in-situ observations or interviews [34].

Lifelogging technology enable researchers to continuously collect data within natural settings [27], and this have become more feasible with the availability, price and pervasiveness of new wearable sensor technologies [34]. As a result, studies using wearable physiological sensors to measure e.g. galvanic skin response and heart rate have emerged within the HCI research community, although still to a much lesser extent than the SenseCam, cf. [2, 8,9,10, 31, 37]. Furthermore, such physiological sensors enable researchers to measure the key user experience dimension of emotions [4, 17]. Given the current level of wearability, such physiological sensors seem well suited to study user experiences in the wild. However, we have not been able to find any HCI studies combining the use of a wearable camera and physiological sensors to capture images at emotionally charged events.

This study extends previous work by exploring how study participants appropriate and experience wearing a lifelogging tool that uses a physiological sensor to automatically activate image capture from a wearable camera. We describe the design of our tool and study its usage in the wild. To emphasize the emotional dimension of daily experiences and to engage participants in daily reflections, we employed a provocative design approach in developing the tool. We designed the tool such that the camera can only be curated through emotional reactions, i.e. users cannot control when to the camera takes a photo. Rather, photos are taken when the physiological sensor detects an increase in excitement.

The strength of provocative design is that of challenging existing norms, e.g. by triggering dilemmas through interaction design, or designing something well-known in a very different way. Recent studies used provocative design to motivate people into reflecting on their behavior, see e.g. [33, 34]. However, it is crucial that the provocative design is perceived strange enough, but not too strange, in order to be effective [33].

In the remainder of this paper we present related work on using physiological sensors to support data collection in the wild. We then describe considerations on our lifelogging tool based on provocative design, followed by a description of our study method and results. Finally, we discuss our findings and conclude on these.

2 Related Work

In this section we outline studies that emphasize design of lifelogging tools and report on participant experiences in using these. Lifelogging denotes the collecting of data for self-monitoring and reflection on personal information [26, 27]. Lifelogging tools gather data about people’s daily life using for instance wearable cameras, physiological sensors or smartphone sensors combined with pc or mobile apps to visualize the data [11, 27]. In this study we are particularly interested in studying the use of lifelogging tools based on gathering data from physiological sensors. Such sensors indicate emotional states of participants where e.g. Galvanic Skin Response (GSR) sensors measure changes in arousal [10, 18]. Emotions are relevant to consider in relation to studying behavior in the wild. This is because emotions are weaved on the basis of stimuli perceived through our senses and our following reactions. We thus use our emotions to plan our actions in order to cope with changing situations in daily life [35]. This does not only apply to intense or life-threatening situations, but also in more subtle cases such as interacting with products as stated by Forlizzi and Battarbee [17]: “Emotion affects how we plan to interact with products, how we actually interact with products, and the perceptions and outcomes that surround those interactions”.

2.1 Making Sense of Data

A critical point to consider is how participants make sense of physiological data. Using e.g. heart rate data to infer the physical state of our body is commonly known by people utilizing tools related to support quantified-self purposes. Yet, such tools are developed with physical exercise purposes in mind. The use of physiological data to get insights on our emotional states is, however, more limited.

Be Open for Interpretation.

Ståhl et al. created a lifelogging diary tool named the Affective Diary [37]. The Affective Diary uses a GSR sensor to measure participant’s emotional arousal. The Affective Diary also collects contextual data from participant’s smartphone about sent and received text messages, photos taken and people nearby (via Bluetooth scanning). Participants in the study were able to make sense of the contextual data from their phone, which they frequently referred to when explaining their diary to the researchers during the interviews [37]. The physiological data was visualized using abstract colored human shapes where e.g. a red color signified high arousal and blue low arousal. Findings revealed that some of the participants could not make sense of the abstract human shapes and color scheme. Ståhl et al. suggest that tools based on affective data should be designed to enable participants to interpret the data themselves rather than dictating what should be interpreted [37].

Provide a Condensed View.

Pavel et al. Designed a lifelogging tool to support lifestyle management [31]. The article emphasize the ordering and display of data gathered from wearable sensors and a pc in a way that should be meaningful to the user. To this end the data is combined into stories about the user’s day. This is done by categorizing the collected data, for instance in relation to what the user was doing, emotional states as well as physical and social contexts. Participants could also manually add data for the stories in the form of notes about events they found interesting. The stories represent a condensed view of the collected data with text and background images of where the story has taken place. Above the stories are icons that users can press to get more specific details about the data used in the story. The study showed that participants found it valuable to have a condensed view of the collected data [31]. Using the stories as a condensed alternative to the detailed information available from the data sources, helped users understand the essence of the data. This stimulated reflections about their behavior. The study also showed that it varied what sort of data the participants found relevant to include in their story. This depended on the event they experienced [31].

Similarly, the study by Kelly and Jones deals with designing a lifelogging tool to enable participants to more effectively interact with the large amounts of heterogeneous data, which are collected through wearable and mobile sensors [23]. In their study, participants wore a GSR sensor, to collect physiological data about their emotional arousal. Participants also wore a lifelog camera to collect contextual data, which was stored together with activity data from mobile phones and pc activity in a combined lifelog. The physiological data from the GSR sensor were then used as cues to extract contextual data from the users lifelog for self-reflection. The contextual data items were categorized into minimum, medium and high GSR measurements. Results from the study show that items correlating with high GSR measurements were perceived as most usable for self-reflection. This indicates that physiological data, e.g. high GSR measurements, is useful for highlighting the most important contextual data. This in turn may be used to present a condensed view of lifelog data [23].

2.2 Improving Data Richness in Recalling Events

Arvola et al. studied the use of wearable lifelogging technology to support self-reflection [2]. Participants in that study wore a lifelogging camera, which took a photo every 30 s and an activity tracker that collected data about participant’s heart rate during the day. To examine how sensor and camera data would increase richness of self-reported reflections, participants were not allowed to access data until the end of the study. They were asked to self-report at the end of each day about their experiences, which is based on free recall [2]. From studies in psychology we know that such an approach (very similar to the Day Reconstruction Method) suffers from a significant memory recall bias [12]. Therefore, at end of the study participants were told to compare their free recall notes from each day with the lifelog camera photos labeled with timestamps [2]. This made participants recall a considerable amount of extra details about their experiences. Afterwards they were also allowed to compare the notes and photos with the heart rate data, which was visualized using a graph with time stamps. This resulted in recollecting further details about their experiences. This shows that using different data types to complement each other supports user reflection and increases data richness. However, as the lifelog camera took a large number of photos each day, it was considered too time-consuming to go through all the data during the reflection process. Arvola et al. therefore suggest reducing the data volume [2]. This is in line with the condensed view proposed by Pavel et al. [31] and Kelly and Jones [23] mentioned above.

2.3 Engaging Participants

One of the aims of this study is to explore how participants engage in data collection through their appropriation and experience in using a provocative lifelogging tool. While a few lifelogging studies have touched upon how users engage in data collection, more research is needed. In the following we outline discussions from previous studies on this, but also introduce provocative design as a potential approach to further motivate participants into collecting data in the wild.

Engaging Through Activation.

Participants in the Pavel et al. study on lifestyle management (mentioned above), embraced the opportunity of engaging in the data collection themselves during the day. This was done by adding notes to the stories made within the lifelogging tool. They liked that they could consciously personalize the data, which they were to use for reflection later [31]. This suggests that lifelogging tools should engage users in the data collection, yet this should not be too time consuming as it would diminish participant motivation [28]. This is also in line with the Affective Diary study by Ståhl and colleagues, in which it was found that participants preferred to interpret data themselves rather than having a tool dictate what should be interpreted [37].

Engaging Through Provocation.

Provocative design has emerged as a way of using artefacts that stand out in order to study behavior in the wild [34]. Provocative design aims at using such artefacts to challenge existing norms, hereby engaging participants in reflecting on their behavior [33, 34]. The focus of provocative design studies is on producing knowledge and not the immediate development or refinement of specific artefacts [38]. This fits well with the aim of our study, as we want to study participant behavior and experiences surrounding the use of a lifelogging tool. At the same time, this tool needs to motivate and engage participants into reflecting on their behavior.

The behavior that the design is trying to challenge is what Bardzell et al. describes as the conceptual provocation of the design [3]. By (slightly) stirring up normal routines and beliefs, provocative designs can encourage study participants to reflect upon their actions [33]. It is key that the artefact design is strange enough to fulfil its purpose so that it does not readily blend into the everyday routines [3, 33]. Yet, it should not be so provocative that participants find the artifact weird and rejects it, i.e. a slight strangeness is the key [13]. Also, it should not be fantasy but instead relatable as a plausible next step from the current available artefacts [14]. The recent studies by Bardzell et al. and Raptis et al. have shown that provocative design is efficient in making participants reflect upon their actions through an object [3, 33].

Raptis et al. recommends that the provocative designer embraces design authorship, meaning that not all design decisions have to be mapped directly to requirements or user needs [33]. Instead, design decisions can also be based on the curiosity and intuition of the designer. Rogers and Marshall similarly states that using provocative design to study phenomena in the wild involves deploying a technology, that have been primarily developed by the researchers [34].

2.4 Existing Lifelogging Tools

There is a wide range of consumer apps and devices available for lifelogging on smartphone app stores. While we do not intent to provide a comprehensive list here, we do highlight some of the most popular tools. Journaly is a lifelogging app that supports both manual and automatic functionality to add entries. In a daily entry, the app can automatically add user’s photos, mobile sensor information about location, sleeping patterns, driving and walking [21]. Several entries can be added to the journal on the same day, if the user does so manually. The app uses a timeline with date/time, pictures, and weather information.

Optimized is a lifelogging app with focus on psychological state and social interaction [1]. The concept is to track information about sleep, exercise, social activities, people and how time spent on these activities correlates with the user’s current mood. The functionality is mainly based on manual entries, which are visualized on a timeline or on a graph. Users manually adjusts their mood on a scale from zero to one hundred.

SenseCam developed by Microsoft has also been widely used throughout research studies, cf. [16, 29, 36]. This wearable camera takes a picture up to every 30 s in the default setting, but also includes built-in sensors, which can help in filtering this vast amount of data. SenseCam includes a light detection sensor as well as a sensor that identifies when a person is standing in front of you [19]. So far we have not seen any studies using SenseCam to automatically capture images of emotionally charged events.

3 Design of a Provocative Lifelogging Tool

This section describes our considerations in terms of creating a lifelogging tool. We sought inspiration within related research as well as existing commercial tools, yet aimed to differentiate our tool by applying a provocative design approach.

3.1 Conceptual and Functional Provocation

The conceptual provocation of our lifelogging tool relies on the philosophy of hedonism and the ideal of being present in the moment. It is about showing the world what our lives are truly about, doing what our emotions tell us to do, and not being controlled by technology to put up a façade that lives up to societal norms and expectations. We denote our tool “In the Moment” and the concept is in Fig. 1.

Fig. 1.
figure 1

Conceptual sketch of In the moment.

Full size image

The conceptual provocation manifests itself through the functional provocation of the tool. Users are not in conscious control of the technology, which in our case is a wearable camera. Rather, photos are taken the moment users feel emotionally excited. Thus, In the Moment automatically takes a photo every time participants have an increase in emotional arousal. They can therefore stay in the moment with no option of controlling the technology to put up the right façade.

3.2 Inspirational Sources

The design is inspired by the Journaly app. In the Moment is designed to get participants to reflect on and write about their thoughts in a photo diary consisting of the photos taken while wearing the tool. The diary has its own timeline showing the images taken. Compared to Journaly, In the Moment takes a provocative approach for collecting data to support participant reflection. We also condense the amount of photos shown to participants in the diary. Thus, In the Moment does not show all the pictures taken during the day for participants to reflect upon. This is similar to the functionality featured within SenseCam, in which a built-in light sensor and a sensor to detect people can be used to decide when photos are taken. We used a GSR sensor instead to detect emotional excitement rather than relying on light and co-presence. At the end of the day, we selected the 10 photos taken at points in time with the highest GSR levels. This decision is based on the findings of Kelly and Jones [23]. In that study, data items such as photos or contextual information, which correlated with high intensity GSR measurements, were perceived most usable for self-reflection [23]. Pavel et al. [31] also found that a condensed view of data enabled their participants to extract the essence of the data, which in turn stimulated reflections on behavior.

3.3 Technical Implementation

A physical prototype of In the Moment was implemented using a Narrative clip lifelogging camera [30] and an E4 wristband to collect GSR data [15], see Fig. 2. We also developed a pc application to visualize the data. The PC application was preferred over e.g. a smartphone or tablet app, because the Narrative clip takes a vast number of photos, which would be too cumbersome to transfer wirelessly.

Fig. 2.
figure 2

Left - Narrative clip lifelogging camera, Right - Empatica E4 wristband.

Full size image

The E4 wristband measures participants’ GSR level four times per second, which is recorded in an accompanying app on the user’s smartphone and then transferred to the E4 webserver. The PC Application accesses the E4 webserver to find the 10 most exciting emotional reactions (states of arousal). It does so by selecting the largest increases in GSR readings (spikes).

The Narrative clip camera automatically takes a photo every 20 s and adds a timestamp in a log file. Timestamps of the 10 most exciting emotional reactions obtained from the E4 wristband are compared with the timestamped photos from the camera. The pc application then shows the photos taken in closest temporal proximity to the timestamp of each of the 10 emotional reactions. The 10 photos are ordered by the timestamps which are also visualized next to each photo. Each photo can be commented in order for researchers to gather qualitative self-reflection data.

4 Study Method

The In the moment prototype was used for an In the Wild study. This section describes the participants and procedure of the In the Wild study.

4.1 Participants

The In the Moment tool was used by three participants in the wild. One was female and two male. Participants were included on the basis of their varying occupation, family structure (kids/no-kids) and spare time interests. Participants volunteered to participate and did not receive any gifts or money for their efforts, they did, however, express interest in finding out what sort of photos the tool would visualize to them, based on their emotional reactions to situations in their everyday lives. All three were used take photos with their smartphone on a daily basis and two owned a smartwatch. The latter is relevant as the E4 wristband has a similar form factor to a smartwatch.

John (age 32) did not have any children at the time of the study, but was soon to be a father. He worked as a forklift driver and lived with his girlfriend in their house. He never misses a home match on the local football stadium. He also considered himself as gaming a lot in his spare time. Andreas (age 30) had two small children of the age 2 and 5. He was a student living with his girlfriend in a house and likes watching sport on TV. Marie (age 28) also has two small kids below 6 years old. She worked as a teacher and lived with her boyfriend in a house and goes horseback riding in her spare time. She also enjoys reading.

Given the purpose of this study, this rather small sample size is appropriate in order to build an in-depth and in the wild case study of exploring how participants engage in data collection through their appropriation and experience in using a provocative lifelogging tool. Thus, the study is not designed to be representative of a large population, but rather to sample some of the engagement and appropriation strategies employed here.

4.2 Procedure

The participants each had the tool for three consecutive days and were each interviewed on the day after their three-day period with the tool. During the interviews they went through their photos and comments from each day with the moderator and were also asked questions about their experiences with the tool and their own and others behavior while they wore the tool.

To try and make the In the Wild study more natural, the participants were told to use the tool as they saw fit. They were not given any specific tasks to do with the tool and there were no expectations to how much they would use the tool, or which situations they would use the tool in. They were also told it was optional if they wanted to write a comment for a photo in the PC application.

Due to the automatic nature of data collection, in particular the image capture, we considered the ethical framework suggested in [24]. That framework deals specifically with ethics in relation to automated wearable cameras. In order to live up to the framework, we obtained informed written consent from our participants informing them on the nature and type of data collected during the study. We also dealt with privacy and confidentiality issues by e.g. configuring the data capture such that it was only the participants and the researchers that had access to the data. We also followed the recently introduced GDPR regulative as the study was conducted within the EU.

4.3 Data Analysis

The interviews were recorded on audio and transcribed. Transcribed data was then coded by one of the authors to review and categorize the data into themes. Given the novelty of our tool and the exploratory purpose of our study, we opted for using open coding based on the method described in [7]. Within the data, we emphasized themes related to the appropriation and experience of using our lifelogging tool.

5 Results

This section presents our findings on how participants appropriated and experienced using our In the Moment tool.

5.1 Hiding the Tool from Others

The participants approached In the Moment differently by making the tool fit into their everyday practices, particularly in order to stay within the boundaries of existing social norms.

In Public.

John said he had been thinking about how other people would react, when he was wearing the tool in their vicinity. He tried to mask the tool by making it visually blend in with his other devices: “I have thought about it in general because I knew it was there and what it did. For instance, when facing a bus driver, where I wonder if people behave differently because it looks like there is a camera pointed at them. (…) When I was using my headset, I tried to arrange the cord for the headset in a way that could make it look like it [In the Moment tool] was connected to the headset so people wouldn’t notice.” This is similar to Andreas who choose a specific set of clothes to mask the camera when he wore it to school (see Fig. 3), as he did not like to have a camera pointed at him: “I wore a sweater also, so the camera was marginally visible on my white t-shirt (…) to make it more discrete. A white camera on a black shirt is very noticeable. (…) I don’t like it when people takes photos in my face”. In other situations, Andreas decided to put the camera part of the tool completely away, and for specific events he decided he would not wear the tool at all. His reasoning was that he was afraid of people’s reaction against him wearing a camera in public. He also added that he was unsure about the legal issues for photography in certain places: “I put the camera away, when I delivered my little girl at the daycare, so the personnel would not see it (…) I don’t know, maybe they would think it was a little weird if I was wearing a camera.” and “I had planned on wearing it [In the Moment tool] to the circus, but then my girlfriend and I talked about whether this allowed at all. (…) we were going together with my girlfriend’s work colleagues, so it could be weird if I was wearing a camera”.

Fig. 3.
figure 3

Andreas tried to make the camera part of the tool less noticeable by wearing a white shirt to fit with the white camera and by zipping his jumper almost all the way up.

Full size image

Marie also tried to make the camera less noticeable. While at a trip to the zoo she did this by attaching the camera to the strap on her bag near her waist. She was afraid that people would think she was monitoring them if they could see the camera: “Fewer people look at your stomach than your face when you are at the zoo (…) The goal was to hide it. The reason is, that when a person is wearing a camera, it is to prove that another person is doing something illegal (…) For instance at the zoo it could be to document that the animals are living under poor conditions”. However, it was not only the feeling of monitoring others that was on Marie’s mind. She also felt monitored herself while wearing the tool at home, even when she was by herself: “It is nice to have a day at home for yourself but with a device like that you are not completely alone. (…) Because the photos may be seen by others I don’t feel alone”.

At Work.

Marie said that she chose not to wear the tool to work, because she did not want to have to explain why she was wearing a camera. Also, it was a technology that was unknown to the many people: “At first, I planned on using it at work but then I remembered it takes photos of others, and I did not want to have to explain it. (…) Because no one knows it is possible, that the watch and camera is connected. It sounds a little flighty and I doubt that people will believe what I tell them”.

With Family.

Andreas used the tool on a leisure walk with his family. He chose to wear the tool, even though there would be other people around. His reasoning is that it is okay to wear the tool in public as long as he is doing a private activity: “We went for a walk one day while I was wearing it where I did not think about it (…) we talked to some people but when we go for a walk we are private. We are doing what we feel like so if people approach us it is their own choice. However, if you approach others with a camera it could be stepping on their toes”.

5.2 When Others Become Aware

As the only participant, John sometimes wore the tool without trying to disguise it. This resulted in mixed reactions from the people who noticed the camera.

Initial Skepticism.

The tool made John’s girlfriend feel uncomfortable, because she was afraid there would be unflattering photos of her: “The first day I used it [In the Moment tool] we went out for dinner and were sitting opposite each other like we always do. She was afraid that it would take a lot of photos while she was stuffing her face with food”. He also wore the tool to a family get-together without telling them that he was participating in the study: “I couldn’t help thinking, for instance yesterday, when I was at my parents’ house for a barbecue if people were not themselves because they could see there was a camera. But that was only in the beginning. When I explained what it was, I don’t believe people thought about it”. Their initial questions about the tool were e.g. how the tool was recording their behavior: “My sister-in-law dislikes having her picture taken, so when she first noticed it. she turned away until I told her it was not something that was taking pictures or video constantly”.

Playfulness.

John wore the tool while he was on a visit to his girlfriend’s parents. He told them what the tool did and why he was wearing it. When he was about to leave, a family member became playful and tried to affect what the tool would photograph: “When we were about to leave, my father in law started to talk about the tool. He tried to startle me to try and make me have a reaction. He acted out in front of the camera”.

5.3 Excitement When Unboxing Photos

During the interviews the participants talked about the expectations they had while using the tool.

Own Expectations.

John was looking forward to seeing which situations he had reacted to when he opened the application. He was expecting that the tool would photograph something that he would not have photographed himself: “I was excited to see at the end of the day what sort of photos it had taken. I am a used to taking good photos with my phone and here I had to remember it could be anything. Something totally random. I was hoping there would be something good, like a good situation”. However even though John was looking forward to seeing photos he would not have taken himself, he actually tried to use the tool to take photographs of specific situations, as he would normally have been able to do with a manually operated camera: “There was a situation where my little niece was being a brat. I turned around a bit to make sure the camera was pointing in her direction to see if I had a reaction to the situation. (…) I am used to be able to photograph things that I want a photo of”.

When Marie wore the tool during the second day, she started having expectations about which situations that the tool would react to. For instance, she explained a situation where she was stressed because she was busy and hot: “It gets really cool when you wear it the second time, because then I could predict that something would be photographed. (…) I am doing the dishes and I start to get a stomach ache. Also, I have the sun in my face, it is very hot, and I am sweating plus my son is refusing to put his pajamas on. I am really stressed, and I just need to lie down. While this was happening, I thought to myself that there would probably be photos of this, because I am mentally on the edge. And it did also take a photo of the situations”. Thus, it turned out that photos of these situations were among the most arousing top ten.

Expectations of Others.

It was not just John who had expectations about which situations the tool had photographed. His girlfriend wanted to look through the photos with him, as she had expectations of her own: “My girlfriend wanted to look at the photos in the application together with me. When she saw there were no photos of her she got disappointed because she expected me to have an emotional reaction while we were doing things together”.

Want More Photos.

Participants knew that the tool would only show photographs of the top 10 situations in which they had the strongest emotional reactions. Yet, John was in conflict with himself about wanting to see more photos: “I wish there were more photos. I know it is only supposed to show the photos where you have had the strongest reactions. But I wish there would have been a bit more. Of course, then there would just be lots of photos. There should be a reason they are taken”. Related to this, Marie specifically said she had expected to see photos about a particular situation she experienced as very enjoyable. She was surprised that there were no photos of this: “Here from half past seven to a quarter to nine I went for a long walk. It was a lovely walk in the sun where I stopped to look at some horses playing in the field, but there were no photos of this at all. I had expected that there would be photos of this because it was a very pleasant experience”.

5.4 Making Sense of Lifelog Data

At the end of the day, participants went through the photos in the pc application. During the interviews they talked about the process of making sense of the photos and writing comments for these.

Writing Comments.

John had written comments for most of his photos the first day as illustrated in Fig. 4, but almost none the second day. Even though he mostly wrote comments for his photos on the first day, he could still recall situations on the photos from the second day, when he talked about them during the interview. He said that his reason for not writing more comments the second day was a lack of time: “There are some of the photos from the day before where I can remember exactly, like you know it makes sense that I had a reaction (…) It was a bit late that night and I didn’t know which comment I could write. I wanted to write something that was fit for the photo. Something to sum up the essence of the photo, why it could have taken the photo (…) If I had more time I am sure I would have written a comment”. He also said that he was used to being meticulous when writing comments for photos he posted on social media, as he wanted his comments to be entertaining: “I might be a bit damaged by social media. Because you have this idea that you must come up with something funny or fitting for the photo (…) Maybe it was because I knew that someone [the interviewer] would be looking at it”.

Fig. 4.
figure 4

Example of a photo where John commented: “arranging a dinner date with my girlfriend”.

Full size image

Using Timestamps.

Figure 5 shows one of the photos from John’s second day, for which he had not written a comment. The way he made sense of the photo was by looking at the timestamp of the photo and then describing what was happening outside of the border of the photo at the time the photo was taken: “You can’t see it but just above the top of the photo is the television and here [points to timestamp] we have been watching a football match with my favorite team. And they didn’t do too well”.

Fig. 5.
figure 5

Example of a photo where John also used the timestamp to explain what happened.

Full size image

This is similar to Marie who also made sense of some of her photos using timestamps. Below is an example where she relates the timestamp from the photo combined with her and her boyfriend’s posture at the dinner table: “My boyfriend is sitting like this and I am sitting like this [points to photo]. We are both sitting in a relaxed position. It is 30 min later [points to timestamp]. We are done with dinner and no screaming kids. It is just relaxed. I can’t remember that we were talking about anything special”.

Using External Data Sources.

Additionally, Marie also made use of a messenger log on her smartphone when describing a photo during the interview. On the photo she was working on her laptop, which could be seen in front of her. In her hand, she was holding her smartphone with the messenger app open. While going through the photos during the interview, she picked up her smartphone to show a comparison with the data from her messenger app with the photo and timestamp data from the pc application: “I had missed my little boy all day but here I just received some photos that he is having a wonderful time which makes me very happy. (…) I know this because I can see that I received the photos one minute before and here I am forwarding them to my boyfriend [picks up her smartphone to show the data from her messenger app]. I received the photos from my mother in law 12:05 and 12:06 I forwarded these to my boyfriend”.

5.5 Self-reflection

The participants seemed to recal their experiences and reflect on these by using the tool.

Recalling Details.

There were several photos where John could be seen using his smartphone. When talking about these photos he was able to remember details of what he had used the phone for. There were for instance a photo of him on the bus writing a text message: “I am trying to find a good comeback (…) we typically send funny gif files to each other while texting. I must have found something good since it [In the Moment] reacted to this”.

Using the photos to recall his experiences Andreas was able to remember what happened in everyday situations that he otherwise had forgotten about: “Here I am about to put my coffee cup and drinking bottle in the car on my way to school when I realize that I forgot my keys (…) it is funny because it is something from everyday life that you forget about again shortly after. (…) I could remember right away when I saw the photo, but I had forgotten about the episode before that”. However, in some cases Andreas had trouble recalling a situation from a photograph and could not make sense of why it was selected. He instead hypothesized that he had an emotional reaction to internal stimuli: “Some of the photos does not make sense and it could be because I am thinking about something that has caused the reaction”.

Becoming Aware of What’s Important in Life.

Andreas was positively surprised to see, that there were several photos of situations where he is doing activities with his kids. This was because photos were taken based on his emotional reactions: “…like my comment here were I wrote ‘daddy is proud’. It is great to see that I had a physical reaction to this (…) It is situations like this that I forget about fast, but that I can see are a big part of my life”.

When asked if there were photographs that she would not have thought about taking herself, Marie refereed to some of the photos as joyous photos: “The joyous photos they surprised me, but it is very pleasant to see that I have reacted so strongly on the things that make me happy. That is a pleasant eyeopener.” One of the photos she refers to as a joyous photo is a situation where she is having dinner with her family: “Here we are sitting down and there is food on the table. We are doing this thing where my little boy learns to tell us what he has experienced during the day instead of us asking him about it (…) I am happy that it photographed this because this is not a situation that I had thought about. But I can see now that it took the photo because the situation made me happy. (…) It was a positive surprise”.

6 Discussion

In the following we highlight and discuss our findings from the perspective of using a provocative lifelogging tool to support in the wild studies. This discussion takes a participant centered view on the topic in terms of how they experienced and appropriated using our In the Moment Tool. Our participants were encouraged to use the In the Moment tool like they wanted to and as much (or little) as they saw fit for their daily lives. This differs from related in the wild studies of lifelogging tools, where participants have been asked to use the tools for the entire day [31, 37].

6.1 Breaking Social Norms

The result of our study shows, that the participants in some situations chose not to wear the tool, for instance to work, or in public places or at specific events, because they did not want to break social norms. Due to the camera and the fact of being unable to consciously control this, participants had concerns about others feeling monitored if they realized there was a camera present at all times. One of the participants said that she also felt monitored by the tool herself while she wore it at home, yet she kept using the tool. It was not so much about participants themselves feeling monitored, but more about how others would react to this. Referring to a situation where a participant wore the camera when being around other people, he thought about the behavior of others and whether or not they behaved differently because of this. When around family some of them initially behaved differently as the camera made them feel uncomfortable. On the other hand, one family member took a playful approach to appropriate the tool, e.g. by scaring the participant who wore the camera in order to force a picture being taken.

It is key that the design is strange enough to fulfil its purpose so that it does not readily blend into the everyday routines [3, 33]. Yet, it should not be so provocative that participants find the it weird and rejects it, i.e. a slight strangeness is the key [13].

This suggests that the provocation worked as intended, in particular since participants chose to wear the camera in spite of the fear of breaking social norms. Our participants were at times confronted with initial skepticism by family members or their partner while wearing the camera. None, however, experienced confrontation by strangers in public settings. This may be explained by the efforts made in making the tool blend in with the clothing, i.e. hiding the tool. This highlights the importance for researchers to discuss with their study participants how to handle social situations while wearing data collection tools of this automatically curated nature. Such a discussion should be initiated before conducting the study, e.g. as part of the session in which consent forms are signed. We refer to the work of Kelly et al. for a further discussion and guidelines on how to deal with data gathered from automated wearable cameras [24]. Although participants chose to wear the camera in most situations, it should be noted that there were a few examples of participants leaving the tool at home. This was case when they doubted the legality of using the camera, which follows one of the recommendations suggested in [24].

6.2 Recalling Details

Generally, we found that participants were able to vividly recall details about their past experiences by using the photos with timestamps in the pc application. Participants expressed that they were able to recall being excited at the time that photos were taken, albeit with some exceptions. This is similar to Kelly and Jones’ findings where data items, that correlated with intense GSR readings, were perceived as the most usable for self-reflection and self-awareness [23].

Participants also added additional data sources by themselves to help recollection. This could for instance be opening up a message service on their smartphones to view previously sent and received messages at specific points in time. This was an unintended advantage with our design, yet supported within previous studies. Pavel et al. for instance show that participants include the data sources they deem relevant in order to reflect on past events [31]. The study by Arvola et al. [2] also indicates that complementary data sources increase richness of self-reflection descriptions. This furthermore supports the recommendation in [37] on designing lifelogging tools such that data is left open for interpretation without dictating how participants felt at given moments.

Thus, lifelogging tools that provide complementary cues seem to reduce the memory recall bias known from free recall settings such as the widely applied Day Reconstruction Method (DRM) [12]. We also highlight, that DRM was developed by Kahneman and colleagues [22] as a more feasible solution to the Experience Sampling Method (ESM), in which participants receive prompts multiple times per day at which point they should report self-reflection data. DRM only requires participants to report their self-reflection data at the end of the day, yet at the cost of introducing a memory bias. Given the cued recall nature supported by lifelogging tools, these seem to be a valid, and, perhaps more precise alternative, to DRM when conducting studies in the wild.

6.3 Encouraging (Unexpected) Reflections

Our participants were able to use our lifelogging tool to self-reflect on behavior in everyday life that meant something special to them, behavior which some expressed never to have thought of as explicitly before. They were able to use the tool to make broader reflections in what makes them happy, which is a direct effect of letting the camera be curated through emotional reactions. As an example, one of our participants reflected on photos of him doing activities with his kids. Since the photos were taken on basis of emotional reactions, he expressed feeling good about having a reaction in that situation, and that he usually forget about such situations after a short time. Similarly, another participant noted that she was surprised and pleased to find a picture of her reacting strongly to a joyful moment. This indicates a highly positive effect of our conceptual and functional provocation related to not being able to consciously control when the camera takes photos. This finding is in line with one of the aims of provocative design in encouraging and motivating participants to reflect upon their daily practices [33, 34].

Further indications of how the provocative design encouraged participants is the excitement of unboxing photos at the end of the day. One participant expressed looking forward to seeing which situations he reacted to. He also found that the tool became “cool” to use during the second day as he was engaging himself in predicting when a photo was taken. Not only did this participant show interest in the lifelogged photos, in one case his partner wanted to look at which photos were taken. Another participant expressed the urge to unbox even more photos.

6.4 Study Limitations

Although we did not empirically compare our provocative design against existing lifelogging tools, our findings in several areas are in line with related work. We further extend previous studies by exploring the potential value of a provocative design to support data gathering in the wild. In our study such provocation led to participant reflections that they did not expect themselves as well as inducing the feeling of excitement when about to view the photos taken.

Also, instructing the participants in using the tool as they saw fit, might have contributed to a more natural behavior during the three days they used the tool. While this supports the notion of studying phenomena in the wild, this may also have resulted in participants sometimes choosing not to use the tool in specific situations. This limits the results as the tool did not collect data during an entire day.

Additionally, one participant felt that someone else was in her house, even though she was alone. This is likely because she knew that the photos would be included during the interview. This could also have affected her behavior while wearing the tool. The same could apply in relation to the self-reflection comments given for the photos, which would also be touched upon during the interviews. Commenting on photos for the purpose of a research study was also commented on by one participant. He said that he was used to thinking meticulously of something funny to write when commenting photos on social media for others to see. Being used to think hard about what to write and knowing that the comments would be read by others as part of the study influenced him in a way where he chose not to provide comments on one of the days. This would have taken too much time as he came home late from work.

In the HCI research community we also see a movement towards conducting longitudinal studies in the wild [34]. As this perspective was not included in our exploratory study it is a relevant next step to consider. Also, the study included three participants, which limits the representativeness of a large population. However, the study was not designed for this purpose but rather to sample some of the engagement and appropriation strategies employed when introducing a provocative lifelogging tool to capture user experiences in the wild.

6.5 Future Implications

Our study indicates that lifelogging tools based on the use of physiological sensors and a wearable camera have the potential of supporting continuous data collection to study phenomena in the wild. However, given that the camera of our In the Moment tool could not be consciously controlled, participants in some cases exhibited caution in order not to break social norms. Yet, participants used the tool actively and reported reactions of being positively excited and surprised over the photos taken. This furthermore demonstrates the potential value of basing lifelogging tools on physiological sensors that continuously measure emotional states. We believe our main finding relates to demonstrate the potential value for participants to self-reflect on photos taken outside their own control. This is also supported by Boucher et al. [6], who discusses the use of cameras for cultural probe studies: “Perhaps most importantly, probe returns are most revealing when they are spontaneous and unedited, whereas most common digital devices allow review, editing and deletion”. This study thus provides an example of how to use an emotionally curated lifelogging tool in the wild and the value hereof. Given the novelty of our study, we believe this contributes with a starting point starting point that can inform and inspire future studies to explore the use of provocative design to motivate participants to self-reflect and report on daily experiences.

7 Conclusions

Research in the wild is receiving increasing attention as a way of studying participant experiences and behavior in natural environments. This study contributes by exploring how participants appropriate and experience wearing a provocative lifelogging tool using a wearable camera that takes photos based on participants’ most intense emotional reactions. The design rationale aimed to promote unconscious curation of images, which was done in order to further motivate participant reflection.

The tool was employed by three participants in the wild, who were asked to use the tool freely as they saw fit. At the end of each day they were asked to use the lifelogging tool to view the photos while reflecting over their daily experiences.

Our findings show that participants were able to make sense of the photos presented within the tool and participants could vividly recall details about their experiences. Given the emotional curation of images participants reported that the tool enabled them to reflect on their experiences and become more aware episodes that are special to them and makes them happy in their lives, e.g. doing activities with their kids. However, the study also showed that the participants were concerned about breaking social norms, e.g. by not being able to control when photos of others were taken. They appropriated the tool by making this blend into their clothes and in some cases even chose their clothing to mask the camera. We believe our main finding is that of demonstrating how provocative design, in our case not being in conscious control of the technology, seems to motivate participants in reflecting of their daily lives. They found it exciting to unbox the photos taken during the day to explore which situations they reacted to emotionally. Our findings have relevance for research practice as they illustrate the potential value of using physiological sensors to engage participants in self-reflection.