Within two weeks of creating their Digital Self (see Section 4), participants were invited to one of six events held at the University campus. As we wanted to ensure participants were strangers to each other, and we were capturing how Digital Selfs might be used in initial, individual interactions, we ran the study in a controlled manner. Such an approach is a standard and valid technique that has been used in prior work on initial interactions (e.g. (Maynard and Zimmerman, 1984; Douglas, 1990; Tidwell and Walther, 2002; Nguyen et al., 2015)), and we based our procedure from this. In our study strangers were able to mingle freely with others and choose whose digital content they wanted to view (or not).
To ensure participants did not meet beforehand, they were directed to different entrances in the building and met by an experimenter, before being taken to individual rooms. Participants were first briefed on the purpose of the study, and were asked to complete a consent form. Participants were then shown a sheet containing a picture of each other person taking part in the same event, and were asked if they had previously had a conversation with any of them. If they had that participant would have been excluded from the study. However, no participants reported they had. This ensured that participants were strangers to each other.
Participants were then provided with an HMD headset (an EPSON BT-200, shown in Figure 5 (left)). This presented a set of facial images (see Figure 6 (A)). During the event these were the faces of the other participants, but a ‘dummy’ set of faces were used to familiarise participants with the HMD. By using the handheld touchpad of the HMD, participants could click on a face and view that person’s Digital Self (as created in Part 1) (see Figure 6 (B)). Again, ‘dummy’ Digital Selfs were used to familiarise individuals with the device. Users could view only one Digital.
Self at a time to avoid cluttering their visual view. A button was provided to return to the facial images to select a different Digital Self. Whilst we piloted a number of different approaches to automatically select a Digital Self (e.g. through facial recognition or markers) and switch between recent ones, these would potentially constrain how the Digital Selfs could be employed in multi-party interaction, something we wanted to investigate as part of this study. Whilst manual selection will not scale to large groups, it is important to help us understand how Digital Selfs are used in multi-party interaction before applying automatic systems that will constrain their use.
When participants were comfortable using the HMD, they were told they would go to another room to interact with the other participants. In line with previous work on initial interactions amongst strangers, we avoided providing a specific task to participants. Participants were told that they should ‘get to know each other’. This is a typical task used in initial interaction studies of face-to-face interaction (Tidwell and Walther, 2002). Participants were instructed that they could interact as much or little as they wanted, and could use the Digital Selfs as much or little as they wanted (including not at all).
When all participants were ready, each was taken to the same room. This was a seminar room, having approximately 7 m × 4 m open space in the middle where the participants were able move without restrictions. Each participant was directed to a location around the open space, so all participants were equidistant but were stood at least 3 m from each other. This placed participants in the far social distance of each other (Hall, 1966, p.123). If participants started closer to each other than this, it would have appeared rude if they did not interact. In far social distance individuals can choose whether or not to interact with others (Hall, 1966, p.123). This ensured participants were all at a “browsing” stage (as discussed in Section 5.2.2).
During the event each HMD logged interaction with the Digital Self applications, such as opening and closing Digital Selfs. We also recorded a wide-angle video of the room (showing how participants moved in the space). To record interpersonal interactions, each participant wore a MeCam Classic wearable video camera around his or her neck (see Figure 5 (right)). Whilst participants could interact for as long as they wanted, we ended each session after 45 mins. Previous studies on initial interactions studying one-to-one case have been as short as a couple of minutes (Douglas, 1990), but we wanted to increase the length as there are more interactions to study in multi-party settings, and we were interested in how the interaction developed after this initial phase. No participants stopped earlier than 45 mins.
After the 45 min session, participants were asked to complete a Likert based questionnaire (1..7 scale, 1 = strongly disagree, 7 = strongly agree). This covered use of the Digital Self, interaction with other participants, and the interaction situation itself. As Nguyen et al. (2015) have suggested that the usefulness of topic suggestions is influenced by how introverted/extroverted a person is, we also administered a ‘Big Five’ personality trait questionnaire (John and Srivastava, 1999) to measure personality type. Participants then took part in an audio recorded group interview. This covered their overall experience, and how the Digital Selfs were employed. Overall each session took 1.5 h.
Interviews were transcribed and coded using a framework approach (Ritchie and Spencer, 1994), using stages of face-to-face interaction (browsing individuals, ice-breaking and supporting conversation) as initial codes. Room videos were analysed to identify how and when participants formed, joined and left groups. From the worn video cameras we carried out a first pass to identify where participants started or joined conversations, and where they incorporated Digital Selfs into those conversations. We then transcribed and analysed these sections in more detail. On-device log files were used to determine when Digital Selfs had been opened and closed. We then triangulated between these data sources to understand how Digital Selfs were used in conversation. Questionnaire responses were graphed and one-sample t-tests were used to statistically compare responses to the neutral Likert score.
Overview of events
Figure 7 provides an overview of how Digital Selfs were employed within the three stages of conversation across all events. Participants employed Digital Selfs at all stages. However, use of Digital Selfs did not dominate interaction. 11 out of 23 participants used Digital Selfs to browse individuals before approaching, forming initial subgroups with them, and starting face-to-face interaction. Subgroups can be seen from Table 1. Twelve participants moved directly to interaction without accessing Digital Selfs first. We discuss this more in Section 5.3. Digital Selfs were mostly accessed and used at the beginning of the Events (see Figure 8), and in 5 out of 9 of the initial subgroups content was employed as a ‘ticket’ to support ice-breaking. Overall, Digital Selfs were useful for this role, and helping to start conversations. Participants agreed with a statement: ‘The other person’s Digital Selfs helped to initiate conversations.’ with a mean score of 6.0 (S.D. = 1.4), which differs significantly (One sample t-test, p < .01) from the neutral score (4).
Use of Digital Selfs were not confined to ice-breaking. In addition to purely providing ice-breakers (see Section 5.3), the topics from Digital Selfs were rich and supported strangers to get to know each other. Participants were positive towards the statement: ‘I found the other persons’ Digital Self useful in getting to know him/her’ with mean value of 5.1 (SD = 1.3), which differs significantly (One sample t-test, p < 0.05) from the neutral score (4). Participants were comfortable using both topics from the Digital Self to incorporate into the conversation, as well as choosing topics from outside the Digital Self. 19 participants incorporated at least one topic from a Digital Self into conversation, whilst 21 incorporated at least one topics outside a Digital Self. As such, Digital Selfs supported conversation but their use did not dominate it.
Although we used a more controlled study design, participants did not feel under pressure to use the Digital Selfs. Having a more controlled study allowed us more detailed study of how Digital Selfs were employed. Our method was based on prior studies of face-to-face interaction (Tidwell and Walther, 2002; Nguyen et al., 2015), but raises issues if the events were too artificial.
To the statement ‘I felt that the situation for conversations was natural.’ participants responded with a mean score of 5.0 (SD = 1.8), which differs significantly (One sample t-test, p = 0.032) from the neutral score (4). To the statement ‘I enjoyed the conversations with the other persons.’ with a mean score of 6.1 (SD = 1.1), which differs significantly from the neutral score (4) (One sample t-test, p < 0.0001). We can conclude that participants did not find the situation unnatural as participants were able to mingle freely, and interact (or not) as they wished. Similar to a real event, participants were also able to remove the HMDs (two participants did this) and leave the event at any time (none did).
Browsing for individuals
Eleven participants viewed Digital Selfs before initially approaching another participant. Figure 9 illustrates how many times and how often each participant accessed other’s Digital Selfs before their first interaction, as well as how these initial interactions formed the first subgroups in each event. Participants accessed Digital Selfs largely to gain some idea of who the other people were before initiating interaction (P18:‘It’s something I like to have, information about the other person before starting talking, that’s why I find concept itself fascinating, because it’s public information in a way’).
Whilst half of the participants viewed Digital Selfs before face-to-face interaction, this was often not to identify who to talk to. Only two participants, P3 and P17, took time to browse the Digital Selfs of all other participants before deciding on which group to join (P17: ‘I did look one-by-one at their profiles, digital profiles and based on those I decided which group, which pair to join.’). P17’s browsing can be seen in Figure 9 and also as a function of time in Figure 10. At time point A, P17 joins the pair P13-P14 after viewing everyone’s Digital Selfs. In other cases participants moved towards the nearest person, and it was this individual’s Digital Self that was accessed. It is likely that the ‘cost’ of joining the nearest group was lower. P16: ‘Yeah cause I was like this is a shorter way than going over there. So then I just took up the Digital Self and walked over there.’. This may be in part due to some events having only 3 participants as most of the lack of pre-browsing was in three person events (see Figure 9).Footnote 2 In smaller groups there is less reason to browse before interaction, as a conversation requires at least two participants anyway. However there are other reasons. 12 out of 23 participants chose not to access the Digital Self before interacting face-to-face. Participants found having to access the Digital Self too demanding at the start, and approaching a random person was easier for initial interactions (P15: ‘I first talked to a person because there was too much things going on that I could concentrate on. The pictures, the person, and I was like I don’t know what to do with all these things.’).
The smaller group size we had clearly impacted on the amount of pre-browsing of Digital Selfs before deciding which group to join. Small groups were in part necessary due to both the number and availability of the HMDs, as well as the previous discussion on having Digital Selfs manually, rather than automatically, selected. We discuss this more in future work. However for small groups individuals access Digital Selfs more to support initial interactions than to make a choice over who to interact with.
Post initial interaction browsing
After the initial subgroups had formed, it was common for participants to browse the Digital Selfs of participants in other subgroups. Participants regularly accessed the Digital Selfs, contributing to the regular and more frequent switching observed during the early stages of the events (see Figure 8 for an average of switching instances over all events). This was used as a way to determine both at the start and during the event if it was potentially beneficial to join a different subgroup. In Events 1 and 4, where both subgroups merged during the event, every participant checked at least one person from the other subgroup’s Digital Self before the merging occurred. Digital Selfs were used as a means of evaluating the potential of interaction with others, com-pared to continued interaction with the person an individual was already talking to. This “sneak viewing” was possible as the Digital Selfs were delivered through private displays, so could be accessed without awareness by other members of the subgroup, helping to avoid any obvious sign that a participant wanted (or was considering) ending the current conversation (P5: ‘I had difficulties to see exactly where everybody was looking. So that helps also on the point that the other one can go through the pictures and somehow it’s not that rude, because you can’t see it.’).
Browsing of Digital Selfs can help individuals identify relevant others, and participants were positive in how self-curated representations could help them express themselves (P20: ‘...it [Digital Self] gives additional layer of way you can express yourself.’). However, participants also highlighted potential dangers if the Digital Self itself appears uninteresting to them and would thus discourage, rather than encourage, interaction (P9: ‘It’s a good way to start a conversation, but in a way it could affect people that they see each other’s Digital Selfs before starting the conversation. So they might think that OK, that person isn’t interesting so I won’t talk to him or her.’).
Participants usually moved towards the nearest person, forming the first conversational groups (see Figure 9 and Table 1). Nine initial groups formed across all events. In 5 of these, Digital Selfs were referred to within the first 10 s from the start of the conversation. In this way the Digital Self was considered as publicly available common ground, acting as ‘tickets’ (P2: ‘I usually don’t start any conversation with anyone in the social events, so I think it would help me to start if I know something’, and P5: ‘We have the same background somehow, so it was much easier to ... make up some stuff, because you had the pictures and you had the texts.’). Digital Selfs in general were perceived as useful for obtaining tickets (P22: ‘You have certain amount of topics which you know that the other people are interested, so when you start a conversation on a certain topic, which someone is interested, definitely that person will respond positively because you know that he or she’s interested in the topic, so it will be a very good start-up for the conversation. So that was really helpful.’). Digital Selfs was also found to be useful to help those who would normally not interact with stranger to do so (P7: ‘I don’t usually talk that much with strangers, or any people. So it was a nice experience.’). There is evidence that topic suggestions would be more useful for introverted individuals (Nguyen et al., 2015). However, in a standard personality test (John and Srivastava, 1999) our participants had a mean extraversion score of 3.0 (SD = 0.9) on a scale of 1 to 5. We did not find a statistically significant correlation (r = −0.17, p = 0.44) between the usefulness of the Digital Self and an individual’s extraversion score.
Whilst the Digital Self was generally perceived as useful for starting conversation and familiarising strangers, five participants did not feel that they really needed it (P9: ‘I thought the digital self was quite unnecessary. It felt like it was only in the way. And that I really wanted to get the conversation going just by myself and that it was only sort of like disturbance on the side...But I don’t think the Digital Self changed the conversation that much...it felt like it would have gone the same way even with or without the Digital Self.’). It was largely the content of the Digital Self that impacted its perceived usefulness. Content that presented very clear information, or which did not lend itself to discussion, tended not to be used. For example, P9 interacted with P12, who had only one image in his Digital Self (P9: ‘I opened it in the beginning, but then I think we already had started talking about things that were there [in the Digital Self] and it didn’t really give anything new to the conversation.’). Therefore we believe how an individual constructs his or her Digital Self has a greater impact on its use as an ice-breaker, than an individual’s level of extroversion.
Digital self use supporting conversation
Accessing during conversation
After initial ice-breaking, Digital Selfs continued to be accessed and referred to during conversation. Digital Selfs were used to provide new topics to continue the conversation. For example, when silent moments occurred or discussion on a topic naturally came to an end (P6: ‘When the conversation is really going good, and you are finding the topics of mutual interest, then you don’t look at the digital self. But if you think you are running out of the topic, then you might go to the digital self of the other person so that you are going to find new things. But it only helps when the conversation is detracting and you are not finding anything new to talk. But if the conversation is really good, you don’t care about the digital self at that point’, P9: ‘And at one point or another I think when there was a silence I checked them again.’). However, this also extended to ongoing conversation, where participants would access the Digital Selfs of other members of their subgroup to identify new topics to ‘pivot’ the conversation towards, and away from the current topic that may interest them less (P12: ‘After the conversation started, after having ice breakers and when the conversation was going, then I went to see the Digital Self, and I guess it, during the conversation, I guess for two times, I saw it so that if I can dig for a new topic of conversation.’). Time point C in Figure 10 illustrates this behaviour, when a topic from the Digital Self was incorporated into conversation. P16 cues content from a Digital Self into conversation after considering it for one minute. This causes other participants (including the owner of the Digital Self) in the same subgroup to access the referred to content and continue conversation from it. In this way all parties switch to the Digital Self, using it to act as a common ground to pivot the conversation topic. Another practical reason returning to Digital Selfs was obtaining names of others when available.Footnote 3 It is common that people do not remember the names of other people they meet, especially where there is more than one name to remember (McWeeny et al., 1987). The Digital Selfs that contained names helped in this (P1: ‘I think it was useful to remember the name, if I could see it, for example your name I can remember, but I already forget your name.’).
When participants chose to access and browse the Digital Self of others was largely driven by their current role in the conversation. Participants who were active (either speaking or being directly addressed) focused on the conversation and did not change from the Digital Self they currently had active. Only when participants were unaddressed recipients did they access and switch between Digital Selfs (P5: ‘I found myself looking at them when I wasn’t part of the conversation, when other people were talking then I took some time to browse through them and read the text.’). Being active in the conversation required full attention by participants, and accessing the Digital Self was considered to be too demanding (P10: ‘I wanted to listen to the person at the time, but then look at the pictures, I’m a simple human being so it’s very hard to concentrate on just one [picture in Digital Self]’). In group conversations, participants had more time to view Digital Selfs (P14: ‘Especially if you want to watch the other person or listen to what they’re saying and look at the Digital Self, then it was too confusing. But in a group you get more time to look at the Digital Selfs. But even then you cannot talk and look at it at the same time.’). Whilst manual support to interact with a Digital Self (supporting the ‘sneak viewing’ previously discussed is important), when engaged in conversation, Digital Selfs require an automatic approach, sensitive to the cur-rent conversational role of the participant.
The topics that participants discussed were similar to those identified from non-augmented conversation between strangers (e.g., interests, where the person lives, what they do, education, occupation, social relations, places they visited and travelling) (Svennevig, 2000). The Digital Selfs did not substantially change the ‘topic space’, but did widen it, making more topics visible and accessible, allowing involving (deep) topics (Svennevig, 2000, p.91) to emerge. Thus, the Digital Selfs provided possibilities for participants to move out from conventional interview style discussion (e.g. ‘What do you do?’ and ‘Where do you come from?’ (Svennevig, 2000, p.91)) towards deeper rich topics that supported more meaningful disclosure and conversation on areas participants were already interested in. For example, transcripts in Figures 13 and 14 illustrate how the conventional interviewing questions were skipped by incorporating questions about the images in a Digital Self.
Whilst participants incorporated topics from the Digital Self, they did not feel under pressure to do so. Often, they felt that coherence in the conversation, and supporting its natural evolution was more important than dynamically changing the topic (P13: ‘I thought in the beginning, that 45 minutes is a really long time to just come up with conversation topics, but in the end it wasn’t. Cause once we got the conversation going and we joined as a big group, then it just went on.’). This revealed a tension, where participants would have liked to talk about a highly relevant shared interest, but did not want to disrupt the flow (P16: ‘Conversation was ongoing somewhere else. You had swimming and I love swimming, I mean if we were just the two of us in a room, that would have been the first thing to pick.’). In addition, some Digital Selfs that contained only basic information became quickly exhausted, and participants found there was nothing new to incorporate from them, so there was no need to try to pivot to something new (P9: ‘I used it in the beginning, but then I think we already had started talking about things that it was already going and it didn’t really give anything new to the conversation.’). It may also be the case that after some time the Digital Self has supported conversation through setting talk and onto rich topics, that conversation becomes self sustaining, and Digital Augmentation (at least in the static representation we used here) becomes less useful.
Digital Selfs were also used when an individual joined a pre-existing group (in addition to when participants ‘sneak viewed’ Digital Selfs outside their current group). Whilst we witnessed a limited number of participants who joined an ongoing conversation (6 participants over all events, 3 in Event 1 and 3 in Event 4), in five instances Digital Selfs were accessed. This was either by the person who joined the group (allowing him or her to integrate with the group) or by other members of the group (to find out more about the new attendee).
Figure 11 illustrates the former, with P3 joining the group P5 is a member of. P3 has looked at P5’s Digital Self, where P5 has written his name, using it to integrate into the group. Other members of the group then access P3’s Digital Self to find out more about him. Thus we argue that Digital Selfs facilitated establishing common ground between newcomers and original members of conversation, and Digital Selfs enable integration of newcomers into conversation through Digital Selfs.
Cueing digital self information
When Digital Selfs were used to open conversation, they were most often explicitly referred to, with participants ‘cueing’ that they were referring to them (P11: ‘So I see from your profile that you like to avoid political matters. Why is that?’). As the HMDs excluded participants directly showing what they were referring to, they either cued the use of the Digital Self verbally, or pointed towards the HMD (indicative gestures (Clark and Brennan, 1991)). When participants referred to particular content within a Digital Self, this was done verbally using the layout, as illustrated in Figure 12. Participants had to engage in management work to ensure everybody understood which Digital Self and which content in it was being referred to. The importance of this was highlighted in cases where participants did not explicitly cue information. In such cases this caused a breakdown in the conversation with the other parties, necessitating work to repair the conversation through re-establishing common ground (Clark and Brennan, 1991). Repair was often accomplished by providing the missing cueing of where the information came from. For example, Figure 13 illustrates this.
Such issues were largely caused by asynchronous opening of the Digital Selfs, and can be seen in Figure 10, where the same Digital Self (P13’s) is open only once (at point D) for every participant, except P13 herself. Prior work, focusing on one-to-one conversation has not identified these issues, since there are no ‘alternative’ visualisations to view. Whilst sensing technology and control mechanisms could be employed to ensure all participants in a group are viewing the same Digital Self, as discussed in Section 5.2.2, this may also have negative impacts. However, an ability to quickly synchronise the Digital Self all participants see would remove much of the management work of cueing content.
Media use from digital selfs
When Digital Selfs were incorporated into conversation it was usually done through images. Par-ticipants did not just ask about contents of the images in Digital Selfs, but also how the images were taken or created. 49% of all images in the Digital Selfs were selected for topics in the conversation, whereas only 18% of text instances were referred to.
Text data was often found to be basic and self-explanatory, and was not felt by participants to provide a rich topic of conversation (P13: ‘His was self-explanatory, so I didn’t feel the need to talk about them anymore.’). However, this also extended to more general adjectives describing a person (e.g., ‘easy-going’, ‘energetic’ and ‘optimistic’) which were also left out from conversation. Participants wanted to make their own interpretations of other people based on interaction, rather than be told what a person was like (P11: ‘It’s not so interesting to get to know new people if you already know something about him or her, so you don’t have to dig all the specialties about the person if you already see the things he likes or doesn’t like.’).
Images, on the other hand, provided much richer, more ambiguous conversational possibilities. Asking about images in another person’s Digital Self (see Figure 14) were the most common source of ‘tickets’. Images were both concrete enough to formulate a reasonable question about (P23: ‘I have used like the picture of the pyramids. The other people will be interested like, oh, what is this pyramid. So it tends to initiate more conversations.’), yet ambiguous enough that they were seen to stimulate a rich conversation (P10: ‘Picture [in Digital Self] was puzzling at first, but it was something to start the conversation with.’ and P13: ‘The information alone was not enough, but when paired with the individual during the conversations it helped a lot.’). One participant who used images described these issues (P14: ‘My content [in Digital Self] was quite simple, but there’s a long story behind it.’).
As discussed, where individuals did not use Digital Selfs as ice-breakers, it was in part down to use of basic or textual content in the Digital Self (See Section 5.3). This extended to instances where the Digital Self was not used in conversation as it was too basic. Whilst we saw substantial usage of Digital Selfs, we would expect greater usage if the Digital Selfs were richer, incorporated more ambiguous content and were more image based. As discussed in Part 1, such rich and ambiguous Digital Selfs were more valuable in supporting conversation.
We chose to present Digital Selfs on Head-Mounted Displays, and whilst these are not the only possibilities (see Section 2.2.2), they provide good collocation between the face and Digital Self. In our study, their private nature also supported ‘sneak viewing’ of other’s Digital Selfs.
However, participants did raise issues with the current generation HMDs that we used, finding at times the relatively thick glass between the user and their eyes distracting (P9: ‘I wanted to see the person’s eyes when I’m talking to them, then the glasses are, you know, the screens are kind of a little bit covering up the eyes.’). This also meant that participants could appear distant (P15: ‘Yeah it felt like, like talk to a person then actually looking like there and like yeah. Like because when you have conversation you have to have the eye contact and be like present, so it felt weird that the other person is somewhere in this weird different world.’). We did identify a few instances where this lack of eye-contact created confusion between participants on who was being addressed. An example of addressing problems is seen in Figure 15. Whilst this is an issue of current HMD technology, the eye-contact will be become clearer as the technology evolves. Overall, these issues did not significantly disrupt face-to-face interactions, with participants neutral (mean score 4.4, SD = 2.0) towards a statement ‘The smartglasses did not distract me from the conversations’.