1 Introduction

1.1 Overview

Social media excels in the here and now, marshalling millions of images, videos and comments and sharing them across social groups and platforms [1]. For individuals, this results in growing archives of material that are often fragmented across different websites, yet issues may arise when people wish to forget and remove certain content [2]. Social media companies are well aware of the value of these data archives, but mechanisms that support meaningful reflection or re-interpretation of a user’s personal digital data are minimal [3]. Inspired by ‘slow technology’ principles, we are involved in a large research project that explores new ways to repackage personal digital data to better support the processes of reflection, reminiscence and identity management. Slow technology is design philosophy coined by Hallnäs and Redström [4] in which technology is often repurposed and given new meaning. It takes a broad agenda that encompasses the following: (i) designing for slowness, solitude and mental rest, (ii) designing interactive systems to be used across multiple generations and lifespans and (iii) designing for slower, less consumptive lifestyles and practices [5]. In this study, we designed a system to support users in selecting three images to reflect on, sourced from their Facebook data. We also looked at the effect of printing and mounting these in order to explore their value both as tangible artefacts and as a means of meaning making in digital personhood.

1.2 Background and study aims

A range of social media platforms exist to promote the collection, transmission and reminiscence of personal digital data; however, each platform has its own particular culture that can, in turn, shape the creation of a digital persona. It is quite natural, then, for one individual to create multiple digital identities or personae across different social media, and this fragmentation raises a number of interesting issues about how well we know our digital selves.

In this paper, we focus on the way digital photography is used in the construction of our online identity. The act of capturing digital images—often taken on our mobile phones—is now part of everyday life [6], and these images now take centre stage on social media [7]. The increasing rate at which photographs are being uploaded to social media, and in particular, the preponderance of ‘selfies’ [8], demonstrates that we are no longer taking photographs simply for personal consumption [7, 9]. Digital photography has changed the way that people archive and share personal digital information and has led to the creation of a number of different platforms and techniques that support the presentation and representation of self. Here, we explore the life narrative elements made possible by a photography archive and we introduce a novel elicitation technique for personal reflection—the triptych.

The triptych is a set of three images that originated as a form of religious iconography in classical art. The use of triptychs to tell a story is considered more recent and is often attributed to Hieronymus Bosch who broke away from a traditional hierarchical structure in the triptych in order to depict narratives that were often richly detailed [10]. The artist Francis Bacon famously used triptychs to ‘convey’ rather than to ‘tell’ a story as he wished that each viewing would elicit some unique response [11]. More recently, the triptych has gained a foothold in popular culture, used to create memorabilia depicting weddings and graduations. Previous work has emphasised the importance of physical photographs for remembering [12] and their affordances when placed in different environments—mementos in a family space can emphasise those bonds, whereas photographs placed in a private space afford immersive reminiscing [13].

We were interested in the use of photography triptychs as an extension of previous work on ‘slow technology’. Examples of work exploring slow technology principles have utilised photographs [14], ‘Cow Cam TV’ [15], online applications to postpone e-mail delivery [16] and printed social media content [17]. Particularly, in the cases described by Odom et al. [14] and Thomas and Briggs [17], work focuses on a purposeful repackaging of digital data—data that can be transformed into something of personal value, to encourage reminiscence and raise awareness of the fleeting nature of social media content.

More specifically, we were interested in how digital photography might support the narratives we create to make sense of our everyday lives. These narratives are often considered a fundamental human behaviour [18] and they are increasingly constructed with the support of personal media archives—in particular, with photographs [9]. The use of digital data to build such narratives and to support personal reminiscence has become commonplace, with examples of services including Timehop, My Social Book, Intel’s ‘Museum of Me’ and Facebook’s ‘Year in Review’—all of which utilise photographs from social media to some extent. We recognise that there is immense potential here for personal archiving, storytelling and life review [17, 19, 20] but we also recognise that very few platforms allow users to fully explore the different kinds of personal meaning available in their data. In examples whereby these kinds of platforms have been assessed [e.g. 18], the difficulties and flaws in these biographies are far more evident than the successes. In Thomas and Briggs [3], participants wanted more ownership of the content and more of an opportunity to take an active role in the narration process. Furthermore, the process of creating a film gave a sense of creepiness more akin to a eulogy than a celebration of the life of the subject.

We know that people can find archival value in their own data, but also that they seldom curate past content [21]. We also know that people struggle to make sense of the vast archives that are available and can have difficulty reconciling past and present selves [21], recognising that there are many different narratives embedded in their own data. It makes sense, then, to assume that some of these digital sense-making tools would be welcome, but users often express reservations about the kinds of automated biographies that result from existing systems. Automated social media content from services such as Intel’s Museum of Me offer only shallow representations of self [3], but some automated biographical services do allow the transformation of digital data into a physical artefact. In an earlier study exploring this transformation [17], participants were introduced to the online service ‘MySocialBook’, a platform that enables people to curate Facebook data into a scrapbook which is then printed. The process of building the book encouraged participants to review and sort their social media data and resulted in positive feelings of serendipity regarding the posts they encountered and a deeper appreciation of online support from friends. Something changed in the value of the data as it moved from the digital to the physical sphere and we were interested in pursuing this as one focus of enquiry.

In this study, we turned to one of the simplest forms of narrative structure—the triptych. We wanted to see how easily people could generate a simple narrative based upon the selection of three photographs in their social media data. We also wanted to see how we might support this process by developing a novel computer interface that would allow participants to generate triptychs from their social media content automatically. Earlier work has shown that it is possible to select sequences of three pictures on the basis of a simple narrative or emotional search and that these pictures are then perceived as more story-like by human evaluators than similar pictures selected at random [22]. Previous work has also demonstrated the value of physical over digital content and the value of physical photographs to invoke memories [12]. In the current study, we wished to know more about the ways that such automated processes might support or hinder personal reflection, asking: are triptychs meaningful as a form of ‘slow’ personal data capture? Can we design an interactive system to support the generation of personal triptychs? Does this add value over and above the manual experience?

2 Study 1

We first designed and ran a study to capture the process of manually creating a triptych using only social media photographs.

2.1 Method

The study was advertised via posters, social media and snowball sampling on a UK university campus. Potential participants were asked to e-mail the researcher if they wanted to participate, and the ten participants who did so were sent a follow up e-mail explaining that in order to take part they needed to provide three images from social media (Facebook or Instagram) as well as the associated comments/tags that were posted at the time. Participants were simply instructed that their three chosen images had to ‘tell a story’. In preparation for their visit, the researcher printed the three chosen images and mounted them in a triptych frame. Participants then visited the lab for a face-to-face interview lasting around 40 min. In the interview, participants were asked to talk about their choice of images and explain their order, how easy the selection process was and what kind of story the images told. At the end of the interview, participants were given their framed photographs to take home (see Fig. 1). Time between e-mailing images and interview was usually around 3 weeks.

Fig. 1
figure 1

Examples of manually curated triptychs

3 Study 2

Our second study looked at the use of a novel computer application to generate triptychs automatically from the photos and posts in an individual’s Facebook account. We wanted to see how participants reacted to triptychs generated from their data automatically and to explore the process of creating a triptych narrative using our application. This prompt was intended to encourage participants to consider the volume and manageability of their digital data and enable us to explore whether principles of slow technology could be meaningfully applied to this process. It should be noted that the prototype system was meant to serve here as a prompt for engagement with study participants, not as a self-standing piece of design work.

3.1 System design and development

We designed and built a novel computer application for this study, to generate triptychs automatically from the photos and posts in an individual’s Facebook account. The application has two distinct parts: a back-end data extraction and analysis module which runs once for each user to create a corpus of data from their Facebook account and an interactive web-based front-end which allows users to create and manipulate triptychs generated from their own data. The back-end system architecture is shown in Fig. 2.

Fig. 2
figure 2

Architecture diagram for the back-end data extraction and analysis module

The back-end module uses the Spring Social Facebook APIFootnote 1 to access all the posts and photos the user has shared. From each of these, it retrieves the image URL, the text of the caption and the time stamp. Next, each data unit is enriched with semantic annotations based on the caption text. The back-end module does not do any image processing or visual analysis of the images themselves, so pictures without captions are skipped. TweetNLP [23] takes the caption text and returns a set of tokens and part-of-speech tags. It was developed using real social media text, so it can handle emoticons like :-) and embedded URLs and hashtags. The sentiment (positive, negative or neutral) of each caption is determined using Sentistrength [24], a sentiment analysis tool designed for short web texts. The text is also processed by AlchemyAPIFootnote 2 to identify locations, other entities (such as people and companies) and themes from a predefined taxonomy. Last, a latent semantic analysis (LSA) model of the complete corpus is built using Gensim [25]. This model provides a numerical measure of the text similarity between each pair of captions in a user’s data.

The interactive front-end provides a graphical user interface for triptych creation that loads in the corpus and allows the user to select and manipulate the images (Fig. 3) and to set the narrative target (Fig. 4). Each triptych requires a ‘narrative target’ to guide the search. The narrative target is a powerful tool for shaping the triptych, incorporating both semantic features (sentiments, locations, entities and themes) and free text; but it can be as simple as specifying that we want to begin with a negative sentiment and end with a positive sentiment. The interface for setting the target presents all the themes, locations and entities found in the user’s data, ordered by frequency so that common entries appear earlier in the list. A different target can be set for each of the pictures. Once the narrative target has been set, a triptych can be generated by pressing the ‘Choose again’ button. The generation process uses dynamic programming to select a set of three pictures with captions that match the given targets, whilst also taking into account similarities between the captions themselves, to produce the most coherent triptych possible from the input.

Fig. 3
figure 3

The web-based user interface for the computer task. The highlighted buttons mean a to choose a new set of three images, b swap the positions of two adjacent images and c switch between using the search algorithm to select images and choosing at random

Fig. 4
figure 4

Setting a narrative target. Search text can be specified, as well as multiple semantic features. Each picture in the triptych can have a different target. The highlighted buttons d bring up the edit box and e reject the chosen picture

The story generation algorithm is a modified version of the Viterbi algorithm and was inspired by the unit selection process in speech synthesis where many thousands of units are fitted to an utterance structure to produce fluent-sounding speech. Two cost functions are optimized to produce an output. The first, ‘target cost’, represents the fit with the target. The second, ‘join cost’, represents how similar two adjacent units are. A visual representation of the algorithm is presented in Fig. 5. The targets for the three images are shown in a row along the bottom of the diagram. Every available data unit is shown in a column above each target, making a grid. Only four data units are shown in the example, but the process is the same for thousands of units. The goal is to trace a path through the grid to identify the best sequence of images to use in the triptych.

Fig. 5
figure 5

The story generation algorithm. When calculating the cost of choosing unit u4 at step 3, the target cost for comparing u4 with target 3 is combined in turn with the join cost for each unit from step 2 and the path cost for the best path leading to that unit. Here, the path indicated by the darker arrows (u3–u1–u4) is chosen because it has the lowest overall cost

The algorithm proceeds from left to right. At every step, the cost of choosing each of the available units is calculated using the target cost at this step, combined with the lowest cost for joining to one of the units from the previous step. At step 1, only target cost is relevant. Units which share annotations with the target and have similar text will have a low target cost. At steps 2 and 3, the join cost and previous path cost must also be included. A pair of units will have a low join cost if they have similar text and annotations and similar time stamps and appear in the correct temporal order. However, a unit with a low join cost can still be rejected if the cost of the best path to reach that unit is too high, for example if it was a poor fit to the target in the previous step.

The grid-based visualisation in Fig. 5 can also be used to describe what happens when an image is rejected by the user, as mentioned in Fig. 4(e). For example, if the user rejected the unit from the middle position in the triptych, then the system would present the next best unit to replace it by excluding that unit from consideration, locking the remaining options and recalculating the path with the lowest overall cost. This is known as ‘unit reselection’. It allows users to modify automatically generated content without requiring an understanding of the underlying technology: they can simply reject a particular unit and allow the system to select the next best unit to fit in its place automatically.

3.2 Method

Five of the original participants from Study 1 took part in a pilot, using the computer-based system to compose a new triptych. Then, 15 new participants were recruited to create triptychs for the first time, using the computer program. Recruitment was via an advertisement placed on a student job website based at a UK university seeking people who often upload photographs to Facebook for a study into storytelling in social media. Participants selected a time for a face-to-face session lasting around 60 min. The session consisted of a computer-based task, followed by a short interview where participants were asked about their experience of using the computer program and how it compared to the way they normally interact with their social media pictures. A summary of both participant groups can be seen in Table 1.

Table 1 Participant information

Participants provided access to their own Facebook data for the duration of the study. Posts and photos were imported into the application and three triptychs were generated automatically: one was composed of randomly selected pictures (our baseline), one had a narrative target specifying a neutral sentiment throughout and the third specified an emotional arc from negative to positive sentiment. Participants were asked to rate the three automatically generated triptychs using the rating scheme from Aylett et al. [22]. They were not told how the triptychs were created, and the order of presentation was varied between participants. All the ratings were completed on paper before the participants were shown the computer application.

Next, the participants were shown how to guide the program to generate new triptychs: by changing settings, choosing search terms, or by rejecting a particular picture and allowing the program to choose the next best. This was an exploration phase with no set task beyond becoming familiar with the user interface (Fig. 3). Participants were first shown how to reorder the images on screen and how to reject and replace either all images or a single image. Initially, the application was configured to select images randomly (not using the story generation algorithm), providing participants with an opportunity to explore their data in an unstructured way. After a short time, the story generation algorithm was activated so that the selection of images was no longer random. At this stage, the narrative target was introduced and demonstrated, and participants were given the chance to see how different targets affected the generated triptychs. Throughout the exploration phase, participants were encouraged to flag any triptychs they liked and wished to keep.

Once a participant was comfortable using the interface, the application was reset and the initial three automatic triptychs were presented again. The participant was asked to start from one of the three and use all the features of the interface until they had created a story. Specifically, their task was to ‘make a triptych that tells a story – it doesn’t have to be dramatic or life-changing’. The participant rated their own triptych using the same scheme as before, then the session concluded with a short discussion about whether using the application had changed how they felt about their pictures and how the experience differed from the way they normally interacted with their social media pictures.

4 Results

Participant interviews were audio recorded and transcribed, and a thematic analysis was conducted. Each data corpus was coded by two members of the four-person research team. Following Braun and Clarke’s stages of coding [26], it was first important to familiarize with the data, which meant reading and re-reading the transcripts. This generated initial codes through constant comparisons between the data and allowed us to search for key themes, whereby patterns and repetition emerged in the data. We identified three overarching areas that map onto the design of the study: creating narrative, identity management and the value in transformation. These are summarised in Table 2 and then described more fully below, followed by a more focussed summary of key issues arising from the use of automated as opposed to manual image selection. Note that for those participants who took part in both automated and manual studies, we have labelled their quotes with ‘both’ and then which Study (1 or 2) the quote specifically comes from. All names used are pseudonyms.

Table 2 Summary of thematic categories

4.1 Creating narrative

Creating a coherent narrative from digital data was quite a different experience for our participants. The only instruction was to choose three images to ‘tell a story’. Those manually selecting images already had a story in mind to tell, and the pictures they chose suited the narrative:

I’d already conceived of this series of photographs as a little photographic journey [Derek—Study 1]

For participants in the manual study, there was a strong focus on time as a narrative guide. Individual photographs would be chosen to represent time changing, with the oldest appearing on the far left or top of the triptych (depending on portrait or landscape shots), with more recent photographs following:

I guess like every relationship, our relationship changed over time, so I thought it might tell a good story to have the pictures. Then, yes, so I just found some, and then I sorted them by time, so it would be a nice story [Doris—both—Study 1]

Sometimes manual triptychs contained images which all depicted a different event or time, but the narrative was that of life changes, such as people leaving a workplace, or friends at the start of a new job. Earlier work exploring social media sites as an archive has suggested that websites such as Facebook are thought of as reliable data stores, providing a better narrative of the past than archives held on personal devices [19]. We concur with this, participants in Study 1 were quick to comment that simply browsing through Facebook provided a coherent enough narrative to enable them to identify suitable photographs for their triptych:

I made the decision for these three photographs before I came here, because I put them on Facebook in this order [Derek—Study 1]

It was a consistent theme across my Instagram. It wasn’t too difficult once I’d got in my head what story I was wanting to tell [James—Study 1]

Here, our participants are using the social media site as an organising repository, and with the ability to place images in folders, the act of navigating their back catalogue of photographs becomes more manageable, perhaps. For participants in the computer-based study, narratives were suggested by the categories identified by the computer program:

I think it was probably easier because it gave you ideas of what could be a story without you having to think of what might count as one [Elsa—both—Study 2]

One of the main differences for participants who took part in the computer-based study was the presentation of unexpected images. We found the automatically generated triptychs in Study 2 made participants consider alternative narratives they may not have chosen when manually creating a triptych:

In the first study [manual] although I did do a bit of browsing I kind of had in my head that I would like a picture of my partner on there and that I would like a picture of where I was born. Whereas this [computer-based] was a lot more about generating unexpected narrative. About those narratives that you have maybe forgotten about and thinking about them in a slightly different way. As opposed to just telling a narrative that you would already say and just using social media to illustrate that [Mike—both—Study 2]

In contrast, this process could also restrict participants in terms of their ability to imagine alternative narratives:

The system presented me with my options. I don’t have them all in front of me, the way I do on Instagram and Facebook, to pick things consciously. I can’t quite envision them and picture what my other options are […] So yes, if I had all of my pictures in front of me, I wouldn’t choose these [Michelle—both—Study 2]

In line with literature on cultural life scripts [27], we were interested to see what kinds of life events were chosen to be depicted in the triptychs. Much work has identified cultural norms—culturally shared representations of major transitional life events. The most commonly occurring include starting school, falling in love, marriage, having children and a parent’s death. We found a subtler set of life events that were documented within the triptychs. Participants chose photos of partners to signify relationships, photos of colleagues to represent new employment or places to represent moving house. On the whole, participants organised their narratives around expected events and relationships that we see in other work which required people to repurpose their social media data into a story [19]. Whilst there were some instances of life script narratives, the nature of the task restricted options for depiction of events such as starting school, as one participant pointed out:

I could have had some from school maybe. Because the period that I was at school is a weird one because it’s slightly rubbish quality pictures mostly, on Facebook you’ve got this selection of grainy pictures of people taking a picture of a picture with their mobile so it’s a bit weird [Mike—both—Study 1]

We asked about the order of images, as participants in the computer-based experiment had the chance to reorder using the computer application, and participants in the manual triptych experiment were specifically asked which order they wanted their images in the frame. We were surprised to find that some participants felt very strongly about the order of photographs:

I just had that idea in my mind of emotional timeline. It would still have the impact of seeing them like this. It would still be nice to see them but it would just have less impact I think. It wouldn’t make me sad. It wouldn’t be that big, but it would still matter and make a difference [Laura—both—Study 1]

Yes, it wouldn’t make sense I don’t think. It would go from having some sort of sense to me, to it being just a collection of pictures. Yes, so say if that one was first it would just feel like here are a collection of pictures about whatever [Mike—both—Study 1]

There were some exceptions of course, where people did not object to a change in position (in one instance the researcher accidentally arranged a triptych incorrectly), but this was often the case with abstract triptychs which depicted unrelated images.

4.2 Identity management

The creation of a triptych began a process of reflection on identity and self-presentation. Participants in the manual study could provide a snapshot of who they were, or a glimpse into their interests for onlookers, but they were aware of the limitations of the source material:

This kind of is how I would like to project myself, as somebody who takes nice pictures and is interested in the countryside and that kind of thing. So in some ways I think this does say stuff about me, but maybe you’d need to know a bit about me first [Derek—Study 1]

Others have talked about the extent to which a social media account such as Facebook can provide a complete picture of an individual [28] and certainly we know that people feel able to curate their own social media in order to tell a reasonable story about their identity [17]. However, they often point out the limitations of more automated content creation [3]. We found that participants in both manual and computer-based studies were able to manipulate content in order to generate a successful result, but would also recognise that both methods involve a degree of personal editing of material. It may be that proactive engagement with social media data using methods such as triptychs may force the assessment of online identity in a unique way.

The overarching desire for an aesthetically pleasing triptych sometimes meant that participant identity was not necessarily reflected well in their chosen photographs:

If I was trying to encapsulate my personality more, I probably wouldn’t go for something like this. This is probably more around my appreciation of where I’m from. If it was to do with my personality, what would I pick? I’m really sad. I’d probably pick TV shows, movies and stuff [James—Study 1]

This comment hints at an important point. Our online identities are comprised of both self-generated and other generated posts. For example, people may reveal political beliefs by reposting news reports, or share their music tastes by linking YouTube videos. Indeed, in a recent study of the way Instagram has influenced family photography, externally generated content comes to take on a vital role of providing a counterpoint to the bland images of family life [29]. Again, we see that people are eclectic in their use of material in order to manage the process of self-presentation online—almost as a form of personal theatre in the sense described by Goffman [30]. However, whilst more nuanced content on social media, for example film/music ‘likes’ and location check-ins, may portray a better sense of who we are, that content is typically not included in ‘slow media’ practices where participants are making careful selections from their repository. Participants do not really want to see horoscope posts from years ago, content from online games they have played or reminders of ex-lovers when revisiting social media content [17]. Our participants also recognised their triptychs were mostly made up of positive images, because that is what people tend to post online.

The act of revisiting social media posts in both manual and automated ways provided the opportunity for participants to think about their identity and how they have changed online:

If somebody actually took it into their head to go through my Facebook photos, they would probably end up quite far back, and kind of to almost a different person than I am now [Carys—Study 2]

It brought back memories that I wouldn’t normally bring back. I wouldn’t normally look at things that I posted from high school. So it reminded me of the person who I was back then and what things interested me back then [Libby—Study 2]

In some cases, participants reflected on the lack of meaningful content in their triptych images and the fact that their identity does not perhaps get truly reflected in social media data:

I suppose it kind of made me feel a little bit sad that I hadn’t recorded more things [Ingrid—Study 2]

My thought that I will leave from here with will be that we’re creating someone, not necessarily ourselves through social media, and what we are showing is probably not, you know – us. Our life is not as perfect as the photos we are choosing [Florence—Study 2]

Comments like this show the power of the triptych in enabling reminiscence and that slow technology certainly provides opportunity for the creator of the data to think about who they are and about how they represent themselves online. The above quotes both come from participants in the computer-based condition, showing that even a fleeting experimental setup can make people re-think their digital profiles and desire a more meaningful experience. Previous work in this area has also shown that automated editing of social media data can result in less meaningful experiences than if participants had been able to edit content themselves. Exploring an online tool which compiles social media data into a short film known as ‘Museum of Me’, researchers have demonstrated that whilst the process of allowing an external entity to collate digital content into filmic form has novelty value, more in depth assessment of identity reveals these tools to produce shallow representations of the self [3].

What is also noticeable from our data is the omission of ‘selfies’. Selfies have become commonplace as a means of sharing identity information with others [31, 32] but are sometimes said to reflect a form of narcissistic practice [33]. When asked to design their own triptych, for display, our participants often had strong feelings about such issues:

I just find it really narcissistic when you just have pictures of yourself printed. If it was pictures of me with my friends, maybe, but then I don’t think I’d appreciate it as much on the wall [James—Study 1]

I was contemplating doing a story around where I’ve travelled, and it was more like me in different locations, but then, I thought, ‘Mmm, is that something I would actually have on my wall though, a picture of me at this place?’ I would have it on social media, but I wouldn’t necessarily print it off, because I just don’t really like looking at pictures of myself [James—Study 1]

Here, we see that the nature of the triptych moved people away from choosing selfies in this study, despite their inclusion on social media. The printed and framed triptychs certainly made participants consider content differently to the computer-generated triptychs in the second study, which were only presented to participants on a computer screen. Choosing a photo of themselves was uncommon for participants, and if they did, it was defended in case it was construed as vain:

[I will] put it up in my room, and cringe a little bit, at the middle one, and then John will cringe at that one. Because people might think that I am a bit vain. I’m not that vain, honest [Evie—Study 1]

This example is worthy of note, but was not the only instance of participants refraining from the selfie. Comments like this surfaced despite participants being able to take the framed photographs home after the interview, thus controlling the audience that would see it. Work exploring the phenomena of ‘selfies’ has found a distinction in age related to posts on social media, with teens aged 13–19 posting more than adults aged 25–39 [32, 34]. The lack of selfie choices in our own data, then, may relate to the slightly older age of our participants.

4.3 The value in transformation

One of the key messages from participants who took part in both triptych experiments was that of delight and excitement at being given the opportunity to review social media images. For our manual triptych participants, the images were bestowed with new personal and sometimes private meaning once transformed into print, in part because of the new weight given to the selection of the three images. We considered the act of printing and framing social media images in this context to be novel. We found that the finished, polished nature of the picture frames for manual triptych participants gave new meaning to the content—it became a ‘declaration’ of the importance of the frame’s content, which was not experienced by the computer-based study participants:

I think putting a frame round it is symbolic because you say this is worth something now; this is something that’s significant […] Yes, it’s just thinking about this as now public. It’s almost a declaration of this being important. I think particularly pictures of you in relationships, it’s a declaration of this is now something important [Mike—both—Study 1]

Once they had completed a manual triptych, participants felt they would not alter the images as somebody else had arranged the frame’s contents. They were taken with the idea of it being a ‘gift’, albeit one based on their own images, and this influenced their responses:

There’s the slightly reflexive notion that this is now part of this story as well, part of the research project that I’ve taken part in, printed for me by someone else. So in some ways actually I think of this as a gift, so that’s probably why I wouldn’t touch it [Derek—Study 1]

These comments echo previous work showing that the reuse of Facebook content in crafting exercises can change the perception of the material’s value [19] and in particular, the act of printing a digital photograph can change its meaning for the owner, resulting in a heightened sense of value and a deeper reflection of its content [35]. Researchers have also emphasised that designing for anticipation is key to the implementation of slow technology principles [35]. We saw a similar anticipation of receiving the final framed photographs from our participants, and their delight with the finished triptych was evident:

I’ll never change them. They’re so pretty. Thank you [Michelle—both—Study 1]

Oh it looks really nice, that’s really cool [Derek—Study 1]

Oh that is so good. They look so pretty [Laura-both-Study 1]

It’s absolutely awesome. Oh, is that for me? That’s brilliant [Michelle-both-Study 1]

In accordance with our guiding research principles of designing for slower, less consumptive lifestyles and practices [5], it is necessary to consider the value, if any, of this practice of transforming digital to physical artefacts. The act of reviewing social media photographs encouraged reminiscence when selecting which photographs to include, but also reminded participants of what they had stored online:

So, first of all I really enjoyed it because it’s not so often that you really scroll back and look, ‘Oh, that was, and this was, and this was’ because sometimes you just forget what happened [Mary—Study 1]

In this case, physical objects held new value, encouraging self-reflection and re-visitation of the past—principles of slow technology [16]. What we also find here is that participants were encouraged to review digital photographs, which led to a new-found appreciation of physical photographs. Related work by Petrelli and Whittaker [12] has also shown the greater value in physical over digital artefacts. It is also important to recognise the triptych format—selecting three photographs—differs from other slow technology research utilising photographs. It is more commonplace for participants to be presented with a single image, a method designed to elicit an emotional response and slow down the consumption of photographs [35]. We also uncovered feelings of surprise and delight when participants experienced the computer-based triptych system. Technology assisted in the spontaneous reminiscence of events that might have been missed with the more deliberative process of designing a manual triptych:

It brought back good memories. And it just, like, it reminded me of photos that I completely forgot I posted. [Libby—Study 2]

It made me want to explore more of those pictures that I have online [Laura—both—Study 2]

It has more mystery to it especially the random option because like I said I didn’t even remember that picture, it just brought things back to me that I even forgot about. It is pretty cool; you remember stuff you posted. [Laura—both—Study 2]

I think this experience was more a mixture of the expected and the unexpected I guess. In the first study although I did do a bit of browsing I kind of had in my head that I would like a picture of my partner on there and that I would like a picture of where I was born. Whereas this was a lot more about generating unexpected narrative I think [Mike—both—Study 2].

However, the automatic presentation of images for participants in the computer-based study sometimes caused unexpected or unwanted reminiscence, and images appeared that participants would not care to remember—prompting comments about reviewing data after the experiment had ended:

It reminded me that there are some photos that I should probably take off that are from, like, years ago that I forgot were there. When they’re trying to make a story and they put in an old random picture of a party you were at three years ago, like, that’s not part of my story. I need to review that [Claire—Study 2]

The feeling of serendipity was experienced by a number of participants, usually in the context of seeing photographs that they had forgotten about, or after witnessing the program linking events together that they had not considered as ‘linked’ before. Researchers have shown that people underestimate how much they will enjoy rediscovering their past experiences [36] and here is a prime example. We also see that the selection of images by the computer program reduced effort for the participant and made it easier to assess if images were suitable for a triptych:

It is definitely more convenient, because it presents my pictures to me, so I don’t need to go to my Facebook and scroll through my lifeline or whatever and look for pictures. […] When I did that before, you just scrolled through your page [Doris—both—Study 2]

As the printed manually selected triptychs were given to participants at the close of the interview, a natural discussion occurred around what would happen to them, where they would go and who would be privy to see them. For participants who identified their triptychs as being some kind of declaration, the choice of where to display the pictures was difficult. One participant in particular reflected that now the images were framed, it altered where they might go.

I knew the pictures were going to be taken but actually seeing them suddenly they are there and it looks like something that would be in someone’s house and you think – suddenly there’s just the thought of, ‘Okay, how do I feel about that being, because it’s been made a thing and how do I – suddenly if people see that’, or whatever [Mike—both—Study 1]

This participant was also acutely aware that their relationship status was more obviously displayed in the triptych (they included a picture of their new partner), which needed more consideration of public display.

I mean I might share it with people in the office because I know them a bit better and it’s a quieter office, but certainly if it was a big, open plan office it would be a bit weird maybe. I think also because I’ve got a picture of me and Tom [boyfriend] on there it immediately is a bit – well, I don’t know [Mike—both—Study 1]

These considerations were also weighed against the knowledge that the images they selected for inclusion were presented in their Facebook albums anyway, yet the change from a digital to physical format made participants question whether they could be publically displayed. Here, we see tensions between online and offline identities, and once photographs were printed and had potential to be hung on a (potentially public) wall, participants in turn reviewed the content more closely.

The introduction of technology to the triptych curation process also prompted reflection of who might see the images, but in an online context. When choosing images in the computer-based triptych experiment, participants still considered how others might perceive their images. One stated they would only include pictures of non-humans because of privacy concerns:

All three pictures, even the one of the cows, I posted them ‘cause they meant something to me, so I went with that one. I also think I went with that one ‘cause there weren’t any humans in it, so I was kind of like a bit more comfortable with that [Carys—Study 2]

Just makes me think about what I’m uploading, I guess. Like, what is that saying to people who are looking at it? [Edith—Study 2]

The introduction of an online tool to create a triptych made participants consider their data from a different perspective, something which was not encountered when participants manually browsed their own social media accounts in private to source images:

It’s interesting to see the reality of my life through social media versus what I know of my actual life, so I think that’s quite interesting ‘cause I’m again seeing it from that other person’s perspective, so I’m seeing things slightly different from what I have experienced [Edith—Study 2]

This self-reflection and consideration of the audience is not a unique finding, having been identified in other work exploring Facebook curation [17]. It is worthwhile to acknowledge that the computer-based tool encouraged participants to look at their social media content in a new light, one in which they may be more likely to carefully craft their digital identities on social media in the future.

4.4 The automated versus manual generation of triptychs

In this final section we consider some of the design issues associated with the use of an automated triptych generator, comparing the user experience of these participants to those in the manual condition.

4.4.1 Serendipity

During the automated triptych study it became clear that participants enjoyed the novelty of viewing the selections, however some participants relished the ‘random’ nature of the way photographs were presented, whilst others felt it provided a more structured way to consider the triptych construction:

I really like this whole random result that it’s pulled out, because there are nice things to be said behind each and every one in those pictures [Michelle—both—Study 2]

Before, it was quite random [manual triptych study], you just scrolled through your page. Here, you get this extra information. It’s like the categories and stuff that you can put in, so it’s maybe a bit more targeted [Doris—both—Study 2].

A number of participants expressed delight in the way that they could be surprised by the selection of photographs in the random system—often alluding to the fact that some elements of their personal narrative had been rediscovered.

…it has more mystery to it especially the random option because like I said I didn’t even remember that picture, it just brought things back to me that I even forgot about. It is pretty cool, you remember stuff you posted. It is pretty cool, I like this programme [Laura—both—Study 2].

Some were related that I didn’t imagine would be. It was quite interesting to see the links that I hadn’t even thought of. I think it kind of makes you look at your pictures in a bit of a different kind of… When you’re just looking through them in the order that you posted, you don’t really think about what… [Elsa—both—Study 2].

4.4.2 Changing Display

Many participants noted that the changing ‘shifting’ nature of the display for the automatically generated photographs presented to participants was enjoyable:

I wasn’t too worried about the order thing, the left and right. I did try it for the sake of experiencing how it works, and it was very nifty, very nice. I liked the feeling of shifting [Michelle—both—Study 2].

This idea of shifting is reminiscent of the way digital photo frames have been designed to display a changing selection of images, instead of deciding on one static photograph to go in a frame. HCI researchers have adopted this kind of technology to aid with reminiscence and grieving in the event of death [e.g. 37], but we see here that ‘shifting’ is a desirable feature of a triptych presentation for our participants. This feeling is in contrast to the way that the manual triptych participants felt about their framed photos- the ‘finality’ of the frame meant that they were reluctant to change the images at all.

4.4.3 Limitations

We witnessed disappointment for some participants whose generated triptychs did not necessarily contain photographs they expected, or who had a limited repository because they did not add captions to their data [the automated program requires images to have text captions in order to include them in the study].

There were pictures on there [computer programme] that I was hoping that would come up but didn’t. I think because I have a habit of not putting stuff in the description. I was hoping that the place that I went to recently, there might be some pictures relating to that but there weren’t [Mike—both—Study 2].

Some of the automated system users also found the number of choices they could make to be overwhelming:

Yes, it’s just that there are so many themes and choices. It tends to absolutely startle me [Michelle—both—Study 2].

Further, the automated program would sometimes categorise photographs incorrectly, which could cause either frustration or amusement:

I chose ‘pets’ [the category]. Not sure why my mum and auntie are pets! [Elsa—both—Study 2].

Finally, participants in the automated condition also seemed nervous that they would ‘lose’ a good triptych selection and this made them more likely to accept an automated triptych early in the session. This was in spite of participants being told they could reselect items again if a subsequent set was undesirable. For example, one participant accepted the first triptych that was generated, for fear of another set of images appearing that would be even worse:

So should I stay with those? [Michelle]

Only if you especially like those three. If not, we can move on. This is still the exploratory phase [researcher].

I don’t, but I know that there are some that I really don’t like, that are just incredibly random [Michelle-both-Study 2].

I didn’t really want to choose again although I could have. Just because I quite liked the original set up. I was a bit scared that if I pressed choose again that it would get rid of them all and I would have to be clicking around forever to get it back [Mike—both—Study 2].

We would suggest that any future system should allow users to ‘bank’ their triptychs in order to compare options at a future time.

5 Discussion

We began this study asking the following questions: (1) are triptychs meaningful as a form of ‘slow’ personal data capture, (2) can we design an interactive system to support the generation of personal triptychs and (3) does this add value over and above the manual experience? We will reflect on these research questions in light of our findings, before turning to limitations and future work.

The biggest difference between participant experiences was the delight that emerged from viewing printed and framed images. Here, we see the value of transforming digital to physical, and the reflective practices that go on once personal data is presented in a different medium. The reactions from our participants of delight and serendipity when reviewing their triptychs would seem to suggest that curation of social media photographs in this way does elicit the slow technology ethos—supporting experiences of personal reflection through and about technology in everyday life [16]. The creation of the triptych itself was deemed simple enough—most participants had a narrative in mind—however, the uploading of content onto social media prior to this study meant that photographs were already grouped in a meaningful way for some people. In addition, we also found that the act of reviewing digital data for participants in Study 2 also encouraged reflection—perhaps to a lesser extent—but acts of reminiscence were still noticeable.

Our work draws upon the slow technology literature [e.g. 35], but also relates to a new body of work that considers the effectiveness of social media for personal archiving. This work includes a consideration of commercial systems deployed to encourage reflection via social media [e.g. Timehop, Facebook Lookback], where there is a concern that automated systems may only create shallow representations of self [3]. It also includes a careful assessment of the ways in which people draw upon social media data in order to construct their own personal archives [38, 39]. Some new systems are designed explicitly to promote the careful curation and subsequent prompted recall of everyday life and there is evidence that such systems may have benefits in terms of promoting wellbeing [40] and providing a valuable reminiscence experience [17]. Our study does not promote the logging of personal data, but does explore how photographs in particular can be used to create a brief narrative.

In particular, we were keen to understand the way automated search may change the user experience when selecting personal digital data. We felt that designing a prototype computer program that could choose three images for people to reflect on compared to the manual creation of a triptych would have vastly different outcomes. What we have learned is that we have developed a tool which might be able to assist in the selection of triptych images for reflection and that it has certain value in creating a different starting point for the selection of images than that typical in manual creation.

Participants were more than capable of using the interface to create meaningful triptychs, and the two approaches (manual versus computer-based) simply offer a different way to choose personal data. Although the computer program was designed to optimize sentiment (in one condition), participants were still given the opportunity to select and revise images they were not satisfied with. This ‘automated’ process then still lends itself to the experience of surprise and serendipity, but in a different way—offering smaller but sometimes surprising editing choices from a large set of data.

5.1 Limitations and future work

We recognise that in the automated triptych study, users still had quite a lot of choice in the way they could select and edit the output. In some ways, this reduces our ability to make a clear distinction between manual and automated triptych generation at present. Further, we are mindful of the fact that the participants in the manual condition made their image selections at home (they were asked to e-mail the researcher the three images they had chosen from their social media account). This meant that the ‘manual’ participants could take their time with the task. In contrast, participants in the automated triptych condition were invited into the lab and were under more time pressure. In other words, we must be cautious when comparing the two participant groups, as some of the differences could be ascribed to constraints within the experimental procedure.

We noticed that some users had a very small set of images to select from, as our computer system could only work with those images that had associated captions. This may have had a profound effect on the triptych experience for those taking part in the computer-based study. Indeed, participants noted that if the program could include images they were tagged in, or those that did not have captions, their breadth of image choice would be very different and the whole experience would most likely be enhanced.

As we noted earlier, participants in the computer-based triptych experiment often expressed concern when selecting their images. Whilst they wanted to play with the system and see what other images may be presented, they were hesitant because of the risk that a favourite image might disappear from the display. For this reason, some participants commented that they would have liked to explore the application, but felt that they would be safer sticking to the first ‘liked’ selection rather than risk losing it in the exploration process. They suggested that a design improvement would be to allow the system to save interim choices as this would give a better opportunity for the full exploration of personal data—something which will be implemented in future work.

What we have not uncovered in this study is any lasting effect of the value of the triptych. The responses from our participants were ‘in the moment’ and perhaps could have been a result of surprise—we do not know if the pleasure derived from revisiting these social media images lasted beyond the study timeframe (4 weeks), or indeed if those photographs even remain in their picture frames. To this end, we plan to revisit Study 1 participants, asking what has happened to their triptychs and how they feel now they are in situ away from the lab. In terms of future work, we would also like to explore the possibility that automatically generated triptychs might be more valued than personally crafted stories, partly because of the effort involved but also because of the relative novelty of the triptych presented. In keeping with other ‘slow media’ work where participants are not necessarily in control of the images or output that is generated [14], we feel it is worth exploring the value of such serendipity further.