Augmenting Multi-Party Face-to-Face Interactions Amongst Strangers with User Generated Content
We present the results of an investigation into the role of curated representations of self, which we term Digital Selfs, in augmented multi-party face-to-face interactions. Advancements in wearable technologies (such as Head-Mounted Displays) have renewed interest in augmenting face-to-face interaction with digital content. However, existing work focuses on algorithmic matching between users, based on data-mining shared interests from individuals’ social media accounts, which can cause information that might be inappropriate or irrelevant to be disclosed to others. An alternative approach is to allow users to manually curate the digital augmentation they wish to present to others, allowing users to present those aspects of self that are most important to them and avoid undesired disclosure. Through interviews, video analysis, questionnaires and device logging, of 23 participants in 6 multi-party gatherings where individuals were allowed to freely mix, we identified how users created Digital Selfs from media largely outside existing social media accounts, and how Digital Selfs presented through HMDs were employed in multi-party interactions, playing key roles in facilitating strangers to interact with each other. We present guidance for the design of future multi-party digital augmentations in collaborative scenarios.
KeywordsDigital self Face-to-face interaction Familiarisation Head-mounted display Strangers
On-line social and digital media have blurred the distinction between on-line and off-line relationships between people. Connections that initially occur face-to-face may be managed on-line (e.g. via Facebook), whilst connections initially made on-line may transition to face-to-face (e.g. connecting with someone on Tinder (www.gettinder.com)). It is also common for relationships to be managed dynamically between both on-line social media services and face-to-face. Within CSCW and wider HCI research, this interwoven nature of on-line and face-to-face interaction has led researchers to consider how both can be blended together. In particular, by provoking and supporting face-to-face interactions between strangers through increasing the visibility of an individual’s on-line social and digital media in the physical environment. Whilst historical work, such as (McCarthy et al., 2004), presented information about a nearby conference attendee on a large screen display as a way to provide ‘tickets’ for interaction (Sacks, 1992, p. 265), more recent work has focused on wearable devices, such as head-mounted displays (HMDs) (Nguyen et al., 2015), digital badges (Jarusriboonchai et al., 2015), or bracelets (Chen and Abouzied, 2016).
There is significant potential in this work and implications for CSCW. Individuals have been found to be open to interactions with others in a wide range of contexts and physical places (Mayer et al., 2015, 2016). Everyday social encounters, where individuals will meet without an a-priori purpose, are one of the most common situations in which we encounter people in everyday life, and have high potential for face-to-face digital augmentation to be useful (Svensson and Sokoler, 2008). This can range from interaction at a party or networking event, to short conversations on the street or in shops. Although only a minority of these are likely to lead to a deeper relationship (Rubin, 1974), even short social interactions can improve health and wellbeing (Holt-Lunstad et al., 2010). More widely, CSCW has identified a range of situations where it is important that strangers form a relationship with each other quickly, irrespective of their similarity, such as in ad-hoc teams. Team performance has been identified to improve if members are more familiar with each other before a team is formed (Littlepage et al., 1997). For example, Wong and Neustaedter (2017) have identified the importance of such familiarity amongst flight attendants. A situation where teams must form quickly, with limited time to ‘get to know’ each other. Digital augmentation may be an effective way to support, or speed up, this process.
However, existing face-to-face augmentation work has significant limitations that restrict its potential. Firstly, such work treats users’ social and digital media as a raw resource that can be algorithmically mined to identify and present shared interests between individuals (Nguyen et al., 2015; Abouzied and Chen, 2014; Chen and Abouzied, 2016). Whilst the results of this can be presented in low-fidelity (such as a flashing bracelet (Chen and Abouzied, 2016)) to limit disclosure, other than providing access to a social or digital media account individuals have no control over what media is selected and presented to others about them. This increases concerns over unwanted disclosure of information, leading to loss of face (Goffman, 1969; Farnham and Churchill, 2011). Algorithmic matching also assumes the purpose of the face-to-face interaction is known a-priori, so the algorithm can be tailored to match on the most relevant, similar, information between individuals. Yet, particularly in the everyday social interactions that are most common for strangers to interact in (Svensson and Sokoler, 2008), the purpose of interaction may not be know a-priori. Moreover, by presenting only similarity between two individuals the diversity of face-to-face encounters may be limited, as individuals see only a ‘filter bubble’ of those they are similar to (Pariser, 2011). Finally, existing work (Terveen and McDonald, 2005; Abouzied and Chen, 2014; Chen and Abouzied, 2016) has studied only one-on-one face-to-face interaction. Not all face-to-face interactions involves only two people, rather groups form, separate and re-form dynamically (Bussmann and Schweighofer, 2014). Matching two individuals simply provides a ‘ticket’ to interaction (Chen and Abouzied, 2016), but ignores the prior step of how an individual might browse a room and identify the most interesting people to interact with face-to-face, such as when a person starts in a new workplace and must ‘get to know’ his or her colleagues. Such social interaction and establishment of common ground has been shown as important for effective teams within an organisation (Lykourentzou et al., 2017).
2 Related research
In elaborating on our argument we consider two perspectives. Firstly, we consider existing work to provoke or support real world interaction through the presentation of social and digital media. We then further elaborate on the three stages of face-to-face interaction (browsing individuals, ice-breaking and supporting conversation).
2.1 Digital media in face-to-face interactions
There has been diverse work in considering how to incorporate digital and social media into face-to-face interaction. Existing work is often piece-meal, focuses on only one stage of face-to-face interaction (browsing individuals, ice-breaking or supporting conversation), and its impact is often not evaluated. In discussing this work we consider two key axes (automatic vs. manual selection of media, and visualising nearby individuals vs. face-to-face augmentation).
2.1.1 Automatic vs. manual media selection
Existing work has largely focused on automatic selection of media from user social and digital media accounts. Users provide access to their accounts and media is automatically selected by the system. Whilst this can be directly presented (McCarthy et al., 2004), it is more common that media from two individuals is processed to identify and present their similarity. Jarusriboonchai et al. (2015) did this in a very simple way by presenting interests that two participants had ‘liked’ in their public Facebook account profiles. More sophisticated matching algorithms have been used by Nguyen et al. (2015), who presented potential conversational discussion topics between two individuals wearing HMDs. Suggestions were the output of a matching algorithm run largely against two participant’s Linked-In social media accounts. In all such work, individuals have no input, other than providing initial access to their account (or not), to control what is used or revealed about them. However, as shown by Goffman (1969), individuals are multi-faceted, performing a facet of self during face-to-face interaction. The props (which include digital face-to-face augmentation) individuals incorporate into their performance can either support or undermine this performance. Facets can be incompatible with each other, causing a detrimental impression if they were to ‘leak’. Individuals can go to great lengths to keep their facets separated (Goffman, 1969). Facet management has also been identified in how individuals present and manage their on-line identities in social media services. Farnham and Churchill (2011) identified that individuals use multiple social media services to present different facets of self to different groups.
Whilst automatic selection has potential advantages, it does not take these issues into account, and has not considered how individuals would wish to use digital and social media to augment themselves in face-to-face interaction. Existing studies make these decisions before participants are involved, who are given no input into the process. This increases the risk that unwarranted information is disclosed, which may damage interaction. Just because two individuals match on a particular interest or topic does not mean they would want it disclosed.
Existing matching work has, implicitly at least, considered privacy. On what users are matched can be obfuscated. Chen and Abouzied (2016) developed a matching system where individuals wore a bracelet. If the matching score between two individuals crossed a threshold the bracelets participants wore flashed the same pattern and colour. No information on what participants were matched on was provided. However, more often privacy is considered through either presenting only very basic information, (Jarusriboonchai et al., 2015), or focusing on specific, largely pro-fessional networking, scenarios (Nguyen et al., 2015). Where individuals would be expected to present a more general, professional facet of self.
Additionally, algorithmic matching focuses on finding the similarity between two individuals, ‘weighting’ data to prioritise and match on the most relevant information, given the current situation. This assumes there is a clearly defined purpose or goal to the interaction that can be predetermined. However, as Mayer et al. (2015) have identified, the situation can significantly alter who individuals would wish to connect with. Individuals may wish to connect with those they are similar with, or with those that they are otherwise dissimilar to.
An alternative approach is for users to curate media and decide what should be presented. In this way individuals are choosing what media to represent themselves with, can have control over how their identity is disclosed (in-line with existing understanding of face-to-face interaction (Goffman, 1969) and identity management in social media (Farnham and Churchill, 2011)), and can decide on what information is most important to connect with others on. Persson et al. (2005), for example, developed a system allowing users to curate ‘profiles’ on feature phones that could be broadcast over Bluetooth to others in the nearby geographical area. Whilst users were keen to browse others in the area, a low number of users meant others were often not found.
Whilst this approach removes many of the issues with automatic selection and matching outlined above, there is significantly less study of how such manual curation and presentation of media would impact single or multi-party face-to-face interaction, or how individuals would choose to represent themselves. Recent work tends to focus on novel technical prototypes with only informal evaluation (Kan et al., 2015; Kao and Schmandt, 2015; Devendorf et al., 2016). Only our prior work (McGookin and Kytö, 2016; Kytö and McGookin, 2017), which motivated this work, has investigated user views on how they would wish to use existing social and digital media in augmented face-to-face interaction. It revealed that the same issues of identity, faceting and boundary regulation (Lampinen, 2014) existing in face-to-face interaction and existing social media use, and unconsidered in automatic matching work, also exist in consideration of how such media should augment face-to-face interaction, as well as how individuals use such representations in strictly one-to-one interaction. However, this work has not yet extended to investigate the impact in multi-party interaction. We do so in this paper.
2.1.2 Nearby vs. face-to-face
Existing work focuses on either increasing awareness of suitable others in the nearby environment or focusing strictly on conversation in face-to-face interaction. Commercial work, for example the Tinder (www.gettinder.com) and Grindr (www.grindr.com) dating services, consider ‘nearby’ to be at the city or municipality scale, presenting relevant individuals at least several kilometres away. Chen and Abouzied (2016) developed an LED bracelet to support networking amongst strangers at a conference. The bracelets of two users, who were algorithmically matched, would flash the same pattern and colour when users came within 20 m of each other. Chen and Abouzied (2016) noted that whilst the face-to-face interaction the bracelets provoked was often valuable, it could be difficult for individuals to find their matched partner, with only 15% of identified matches resulting in face-to-face interaction. Work such as Nguyen et al. (2015), on the other hand, focuses specifically on how face-to-face interaction itself may be augmented. They presented algorithmically determined conversational topics between two strangers via a pair of Google glass. Topics were found to be useful in starting and sustaining conversation.
However, there is not a strict delineation between what is and is not face-to-face. For example, Jarusriboonchai et al. (2015) presented matched interests between two stranger’s Facebook profiles presented on badges (mobile phones worn around the neck). Whilst these functioned as ‘tickets’ in face-to-face interaction, they could also potentially act to support awareness of others. However, Jarusriboonchai et al. (2015) studied only situations with two individuals, so did not investigate this. The distinction is therefore how nearby individuals are, rather than if they are strictly face-to-face when media is presented. In this work we consider face-to-face to be within sight of an individual. This means that within the same room or immediate physical proximity. As such, our definition for face-to-face interaction is close to Goffman’s definition for ‘Social situation’. That is, two or more people being in one another’s immediate presence and sharing the same spatial environment so that they have possibility to mutually monitor each other (Goffman, 1963, p.18). By this definition both Jarusriboonchai et al. (2015) and Chen and Abouzied (2016) (and of course (Nguyen et al., 2015) as well) would be considered face-to-face interaction, whereas Tinder would not, as the scanning nearby people could be done without sharing same spatial environment.
2.2 Stages of face-to-face interaction
In considering face-to-face interaction in multi-party conversation, there are three key stages: browsing individuals, ice-breaking, and supporting conversation. Existing social psychology research has provided understanding of how each of these phases works without digital augmentation. Existing work within the HCI and CSCW communities, such as discussed in Section 2.1, has supported some of these stages, with a greater or lesser understanding of the social underpinnings. None supports, or has been evaluated in, multi-party interaction and conversation, with only Chen and Abouzied (2016) and McCarthy et al. (2004) being investigated in multi-party settings (though they still focus on one-on-one interaction). We outline these three stages in more detail and highlight existing work that supports them.
2.2.1 Browsing individuals
When considering our context of multi-party interaction with strangers, individuals first need to identify others who they would like to interact with and if those people are open to interaction. Relevance can be deduced on physical ‘props’ (such as what individuals wear or carry), or how people interact (e.g., language), but the possibilities to spot relevance are limited based on only these properties. In certain situations the relevance of a person is obvious (e.g. a waiter in a restaurant), but in more social situations relevance is much more difficult to detect. To have a ‘safe’ choice, people tend to choose others that belong to the same group (e.g., same religion or country), especially, if belonging to that group is rare among all other people nearby (Goffman, 1963).
Whilst there is a large body of work on using a computational approach (i.e., automatic social matching, see Section 2.1.1) for finding a relevant individual in face-to-face interaction, little consideration has been given to user-curated approaches. Systems using user-generated content to allow individuals to express some element of Self have been largely limited to technical prototypes or have been only informally evaluated (see Section 2.1.1).
Finding a relevant individual and desire to talk with her/him does not guarantee that interaction occurs. Starting interaction between strangers is an example of a situation where persons have restricted rights, and mutual willingness is required. As Goffman (1963, p.124) argues: ‘...acquainted persons in a social situation require a reason not to enter into a face engagement with each other, while unacquainted persons require a reason to do so.’. Thus, a ‘ticket’ (an accepted reason known to the parties) is needed to ‘break the ice’ (Sacks, 1992, p.265). A ticket provides common ground between individuals and allows an interaction to occur. For example, a ticket might be something an individual carries (such as a book) or something in the environment (such as being on a broken down bus) (Sacks, 1992, p.265). Location and context have significant influence over obtaining a license to start interaction. For example, being present at a party with friends is an example of an ‘open-region’ environment (Goffman, 1963). In such environments all individuals are allowed to interact with others.
However, in public situations, such as at bus stops or queueing, individuals have much more limited rights to start interaction and require a ticket (Sacks, 1992, p.265). If individuals wish to interact with others, and an obvious ticket is unavailable, tickets can be synthesised such as by using subtle cues to express willingness to interact. For example, by asking time or directions. Non-verbal cues can also be used, the classic example being dropping something on purpose for others to pick-up (Goffman, 1963). As such, the availability of tickets and ways to express willingness to start interactions with strangers are generally limited in public settings (Goffman, 1963), and thus different approaches have been taken within HCI to support ice-breaking.
Within HCI, work on supporting ice-breaking has been studied mainly in professional contexts (e.g., such as at conferences (McCarthy et al., 2004)). Borovoy et al. (1998) used an LCD badge displaying simple ‘Memes’ that could be exchanged to incorporate digital information into ice-breaking, presenting overall trends of interaction on large-screen displays. Most other early work also used large screen displays, such as McCarthy et al. (2004), who showed information about an individual as he or she stood in line at a coffee station at a conference. The public display of such information was found to be an issue, as some users would have liked to regulate who could see information about them (McCarthy et al., 2004).
Jarusriboonchai et al. (2014) studied how proactive interaction (i.e., devices which identify and interact with each other before the user) could trigger conversation between strangers. They carried out a wizard-of-oz. style study where two strangers were brought into a room where two mobile phones started to make synchronous sounds or ask audible questions about the strangers. They found that mobile phones triggered conversations, but interaction was addressed towards the mobile phones rather than each other. Beyond standard mobile and wearable devices, there has been significant work on novel devices. For example, augmenting everyday objects (such as mugs (Kao and Schmandt, 2015) or handbags (Pakanen et al., 2016)), as well as e-textiles (Devendorf et al., 2016; Kan et al., 2015). Such work however is often conceptual or focused on technical prototypes, and does not investigate use in face-to-face interactions.
2.2.3 Supporting conversation
After ice-breaking, conversations often start with ‘setting talk’, consisting of safe, neutral topics (e.g. discussing the weather) (Maynard and Zimmerman, 1984). However, these are quickly exhausted (Svennevig, 2000, p. 222), limiting conversation unless participants can move to deeper topics (Svennevig, 2000, p.91) which are of interest to both parties.
To allow a deeper relationship to form, participants must disclose information to reduce un-certainty (Clatterbuck, 1979). Disclosure is a gradual process (Altman and Taylor, 1973). What participants are willing to disclose to others is not fixed, but is a dynamic process of boundary regulation that takes place during conversation (Lampinen, 2014). This is influenced by a number of factors, including prior relationship and context, but also more transient factors such as personal mood at the time. Successful ‘navigation’ through this process helps to deepen the interpersonal relationship, increasing social attractiveness (Douglas, 1990), and allowing more meaningful interpersonal relationships to form (Altman and Taylor, 1973).
Structurally, conversation is a turn-taking activity, where the next turn is allocated by the current speaker, or by self-selection (Sacks et al., 1974). Whilst in one-on-one conversation there is only a speaker and addressee (who will take turns at these roles), in multi-party situations there may be one or more unaddressed recipients (Gibson, 2003; Traum, 2004). This role arises when the speaker addresses the speech to a particular person directly (e.g., through saying the addressee’s name (Sacks et al., 1974), or directing gaze towards an addressee (Jovanovic et al., 2006)). The speaker and addressee are aware of the presence of unaddressed recipients, but un-addressed recipients are not recognised as part of the ongoing dialogue. Hence, in contrast to eavesdroppers, unaddressed recipients are involved in conversation, but they usually need to wait their turn until the speaker and addressee are satisfied with the current dialogue (Branigan, 2006). Thus, in multi-party situations, participants need to pay more attention to their current role in conversation, which typically shifts many times as the dialogue proceeds (Gibson, 2003).
A final important difference between one-on-one and multi-party interaction occurs when progressing conversation to both new topics (topic progression) and maintaining and developing the current topic (topic maintenance). Multi-party interaction provides more opportunities to do this, and as such there is often less coherence and predictability (Korolija and Linell, 1996). In large part this is due to participants being able to join and leave existing conversations dynamically (Traum, 2004). Joining a conversation causes a situation where communicators are participating at different conversational backgrounds (i.e., newcomers are not aware of what has been said before), leading to a breakdown of common ground. Newcomers must then either explicitly initiate a repair (to help establish common ground) (Clark and Brennan, 1991), or wait until their understanding of the conversation develops to become active participants (Branigan, 2006).
Whilst there is good understanding on how individuals engage in conversation, there is little understanding of how digital face-to-face augmentation can be used to support it. Nguyen et al. (2015), who studied how strangers can be supported in one-on-one conversations using algorithmically matched topics from individuals Linked-In social media accounts have considered this topic. Three topic suggestions were delivered through an HMD and updated every 2.5 min in a 15 min session. They found that topic suggestions were perceived somewhat useful in general and that they were more useful for introverts than extroverts. They also argued that the timing of topic suggestions should have been matched with points where the current conversational topic was exhausted.
Whilst the topics suggestions were somewhat useful, Nguyen et al. (2015) did not study be-yond one-on-one interaction. Therefore, issues around the use of augmentation in multiple-party interaction discussed above, were not considered. Any algorithmically matched system would need to be re-run as individuals joined or left a conversation, by definition changing the dis-played information. Having a user curated representation of self that would persist across these events may help individuals join conversations that are on-going. Moreover, as topic progression and topic maintenance are more difficult to predict in multi-party situations (Korolija and Linell, 1996), detecting the moments when the current topic is exhausted and changing it (as suggested by Nguyen et al. (2015)) is much more challenging.
2.3 Research questions
RQ1: How do individuals choose to represent themselves to strangers with Digital Selfs?
RQ2: How are Digital Selfs used at each stage of interpersonal interaction between strangers in multi-party settings?
3 Study outline
Twenty-three participants (nine female, aged 18–42 years, M = 27.7 years, SD = 5.6) took part. Participants were recruited from both existing mailing lists and flyers placed on campus. Twenty were students, and the majority were Finnish (17/23). In part 1, participants created a Digital Self to represent a facet of themselves they would want to present to a stranger (a person they had not previously met) during face-to-face interaction. In part 2, participants took part in an augmented multi-party face-to-face event. Participants were randomly assigned to one of six separate events. Allocation was based on common availability for all participants in the event. In each, participants were equipped with a head-mounted display (HMD), and could access the Digital Selfs of all other people in the event as and when they wished. Participants were compensated with two movie tickets (approx. Value 20 euros) on completion of both parts of the study.
4 Part 1: creating digital selfs
4.1 Experimental setup
In the first part of the study we asked participants to create a Digital Self that they would be happy to use to augment their visual appearance in face-to-face interaction with strangers. Participants were told that the Digital Self would be used in Part 2 during face-to-face conversations in a small group of people, and it would be up to participants to “get to know each other”. We left this task deliberately vague, as meeting a stranger without a clear a-priori purpose is the most common, and potentially most useful, scenario where a Digital Self might be beneficial (Mayer et al., 2015; Svensson and Sokoler, 2008).
The Digital Self comprised of a single Microsoft Powerpoint slide (that would later be con-verted into an image). Although this limits the Digital Self to a static image, Powerpoint does provide an easy and flexible tool to support free-form creation of the Digital Self in content, layout and form (text, images etc.). We asked only that participants use English for any text they included and they keep the black background to better support display on the HMD. Otherwise participants were free to design the Digital Self in any way they chose, using as much or little media as they wished. We supplied a template Powerpoint presentation with a blank black background slide.
Part 1 was conducted remotely. This was done so that participants had time to consider what they wanted to include, and could access digital and social media accounts on their own devices, without needing to remember passwords and login on a device supplied by the experimenters. Participants were given several days to complete their Digital Self. Once participants had completed their Digital Self, they e-mailed the Powerpoint slide to us, completed on-line questionnaires about the created Digital Self and demographics, and were interviewed over Skype about the Digital Self they created.
Interviews were transcribed and thematically grouped using a framework approach (Ritchie and Spencer, 1994), with the choice of content, visual form and sources of content used as initial codes. The visualisations were also analysed according to the type and amount of content, and the visual representation used (images or text).
4.2.1 Images vs. text
4.2.2 Image source
4.2.3 Incorporating ambiguity
Participants were explicit in desiring to incorporate ambiguity into the media they chose and its representation. For example, providing facts in text (such as name, age, and interests - see P5, in Figure 3) makes it obvious how a Digital Self should be interpreted. Showing images, particularly more artistic images, requires interpretation to understand, and dialogue to uncover what they mean or say about individuals (e.g. P9’s Digital Self in Figure 3). There were a number of factors that drove this choice, and why individuals chose to incorporate more or less ambiguity in the media they chose. For example, participants did not wish to be misinterpreted, or for others to interpret their Digital Self poorly (P21: ‘I guess people first look at the photo and then get impression immediately after looking at that photo, and I thought adding photo would create bias. So, they were better off just what I wrote.’). More commonly however, was that because the Digital Self would be used in conversation, media interpretation could be supported through conversation. This led to the use of images (P9: ‘It’s easier to give a not so simple impression of yourself in such an abstract picture. So ‘cause I know I can rely to the conversation so I don‘t have to have all the basic information about myself in the picture.’). In this way participants could provide a more ambiguous representation that may not make sense when viewed in isolation, but in conjunction with conversation could provide rich discussion. In this way such images could support boundary regulation (Lampinen, 2014). Participants may be open to discussing a topic, but the ambiguity of the representation in the Digital Self allowed the depth an individual would wish to talk about it to be dynamically determined through face-to-face interaction. This ambiguity was also a reason why media was largely sourced from outside existing social and digital media accounts (P1: ‘And it‘s kind of like with the social media, when you meet someone if you’ve already seen their Facebook profile, there’s really not that much small talk questions that you can ask about them because you know the basic stuff... rather it might be more interesting if there were some visual cues or some elements that I might be interested in asking about those people.’).
4.3 Part 1 conclusion
In creating a Digital Self, participants tended towards avoiding concrete textual facts, such as might be derived from a social media account, favouring using images to represent a more ambiguous interpretation of themselves. Such representations being seen as both a way to support dynamic disclosure of information (by disclosing more or less information about the media through conversation), but also as a way of opening up rich topics of interaction that support conversation rather than concrete textual facts about someone. This is in contrast to algorithmic matching work which presented textual detailed and concrete topics to discuss (Nguyen et al., 2015).
5 Part 2: events with strangers
5.1 Experimental procedure
Within two weeks of creating their Digital Self (see Section 4), participants were invited to one of six events held at the University campus. As we wanted to ensure participants were strangers to each other, and we were capturing how Digital Selfs might be used in initial, individual interactions, we ran the study in a controlled manner. Such an approach is a standard and valid technique that has been used in prior work on initial interactions (e.g. (Maynard and Zimmerman, 1984; Douglas, 1990; Tidwell and Walther, 2002; Nguyen et al., 2015)), and we based our procedure from this. In our study strangers were able to mingle freely with others and choose whose digital content they wanted to view (or not).
To ensure participants did not meet beforehand, they were directed to different entrances in the building and met by an experimenter, before being taken to individual rooms. Participants were first briefed on the purpose of the study, and were asked to complete a consent form. Participants were then shown a sheet containing a picture of each other person taking part in the same event, and were asked if they had previously had a conversation with any of them. If they had that participant would have been excluded from the study. However, no participants reported they had. This ensured that participants were strangers to each other.
Self at a time to avoid cluttering their visual view. A button was provided to return to the facial images to select a different Digital Self. Whilst we piloted a number of different approaches to automatically select a Digital Self (e.g. through facial recognition or markers) and switch between recent ones, these would potentially constrain how the Digital Selfs could be employed in multi-party interaction, something we wanted to investigate as part of this study. Whilst manual selection will not scale to large groups, it is important to help us understand how Digital Selfs are used in multi-party interaction before applying automatic systems that will constrain their use.
When participants were comfortable using the HMD, they were told they would go to another room to interact with the other participants. In line with previous work on initial interactions amongst strangers, we avoided providing a specific task to participants. Participants were told that they should ‘get to know each other’. This is a typical task used in initial interaction studies of face-to-face interaction (Tidwell and Walther, 2002). Participants were instructed that they could interact as much or little as they wanted, and could use the Digital Selfs as much or little as they wanted (including not at all).
When all participants were ready, each was taken to the same room. This was a seminar room, having approximately 7 m × 4 m open space in the middle where the participants were able move without restrictions. Each participant was directed to a location around the open space, so all participants were equidistant but were stood at least 3 m from each other. This placed participants in the far social distance of each other (Hall, 1966, p.123). If participants started closer to each other than this, it would have appeared rude if they did not interact. In far social distance individuals can choose whether or not to interact with others (Hall, 1966, p.123). This ensured participants were all at a “browsing” stage (as discussed in Section 5.2.2).
During the event each HMD logged interaction with the Digital Self applications, such as opening and closing Digital Selfs. We also recorded a wide-angle video of the room (showing how participants moved in the space). To record interpersonal interactions, each participant wore a MeCam Classic wearable video camera around his or her neck (see Figure 5 (right)). Whilst participants could interact for as long as they wanted, we ended each session after 45 mins. Previous studies on initial interactions studying one-to-one case have been as short as a couple of minutes (Douglas, 1990), but we wanted to increase the length as there are more interactions to study in multi-party settings, and we were interested in how the interaction developed after this initial phase. No participants stopped earlier than 45 mins.
After the 45 min session, participants were asked to complete a Likert based questionnaire (1..7 scale, 1 = strongly disagree, 7 = strongly agree). This covered use of the Digital Self, interaction with other participants, and the interaction situation itself. As Nguyen et al. (2015) have suggested that the usefulness of topic suggestions is influenced by how introverted/extroverted a person is, we also administered a ‘Big Five’ personality trait questionnaire (John and Srivastava, 1999) to measure personality type. Participants then took part in an audio recorded group interview. This covered their overall experience, and how the Digital Selfs were employed. Overall each session took 1.5 h.
Interviews were transcribed and coded using a framework approach (Ritchie and Spencer, 1994), using stages of face-to-face interaction (browsing individuals, ice-breaking and supporting conversation) as initial codes. Room videos were analysed to identify how and when participants formed, joined and left groups. From the worn video cameras we carried out a first pass to identify where participants started or joined conversations, and where they incorporated Digital Selfs into those conversations. We then transcribed and analysed these sections in more detail. On-device log files were used to determine when Digital Selfs had been opened and closed. We then triangulated between these data sources to understand how Digital Selfs were used in conversation. Questionnaire responses were graphed and one-sample t-tests were used to statistically compare responses to the neutral Likert score.
5.2.1 Overview of events
. An overview of how participants in each event formed initial subgroups. Apart from event 3, all subgroups eventually combined into one discussion group. No subgroups split and reformed.
Number of subgroups (number of participants in each
(F = female,
subgroup in brackets and first subgroups bolded).
M = male)
5 (1F, 4 M)
(2,2, 3, 5)
3 (1F, 2 M)
4 (1F, 2 M)
5 (3F, 2 M)
(2, 2, 3, 5)
3 (2F, 1 M)
3 (1F, 2 M)
Use of Digital Selfs were not confined to ice-breaking. In addition to purely providing ice-breakers (see Section 5.3), the topics from Digital Selfs were rich and supported strangers to get to know each other. Participants were positive towards the statement: ‘I found the other persons’ Digital Self useful in getting to know him/her’ with mean value of 5.1 (SD = 1.3), which differs significantly (One sample t-test, p < 0.05) from the neutral score (4). Participants were comfortable using both topics from the Digital Self to incorporate into the conversation, as well as choosing topics from outside the Digital Self. 19 participants incorporated at least one topic from a Digital Self into conversation, whilst 21 incorporated at least one topics outside a Digital Self. As such, Digital Selfs supported conversation but their use did not dominate it.
Although we used a more controlled study design, participants did not feel under pressure to use the Digital Selfs. Having a more controlled study allowed us more detailed study of how Digital Selfs were employed. Our method was based on prior studies of face-to-face interaction (Tidwell and Walther, 2002; Nguyen et al., 2015), but raises issues if the events were too artificial.
To the statement ‘I felt that the situation for conversations was natural.’ participants responded with a mean score of 5.0 (SD = 1.8), which differs significantly (One sample t-test, p = 0.032) from the neutral score (4). To the statement ‘I enjoyed the conversations with the other persons.’ with a mean score of 6.1 (SD = 1.1), which differs significantly from the neutral score (4) (One sample t-test, p < 0.0001). We can conclude that participants did not find the situation unnatural as participants were able to mingle freely, and interact (or not) as they wished. Similar to a real event, participants were also able to remove the HMDs (two participants did this) and leave the event at any time (none did).
5.2.2 Browsing for individuals
The smaller group size we had clearly impacted on the amount of pre-browsing of Digital Selfs before deciding which group to join. Small groups were in part necessary due to both the number and availability of the HMDs, as well as the previous discussion on having Digital Selfs manually, rather than automatically, selected. We discuss this more in future work. However for small groups individuals access Digital Selfs more to support initial interactions than to make a choice over who to interact with.
Post initial interaction browsing
After the initial subgroups had formed, it was common for participants to browse the Digital Selfs of participants in other subgroups. Participants regularly accessed the Digital Selfs, contributing to the regular and more frequent switching observed during the early stages of the events (see Figure 8 for an average of switching instances over all events). This was used as a way to determine both at the start and during the event if it was potentially beneficial to join a different subgroup. In Events 1 and 4, where both subgroups merged during the event, every participant checked at least one person from the other subgroup’s Digital Self before the merging occurred. Digital Selfs were used as a means of evaluating the potential of interaction with others, com-pared to continued interaction with the person an individual was already talking to. This “sneak viewing” was possible as the Digital Selfs were delivered through private displays, so could be accessed without awareness by other members of the subgroup, helping to avoid any obvious sign that a participant wanted (or was considering) ending the current conversation (P5: ‘I had difficulties to see exactly where everybody was looking. So that helps also on the point that the other one can go through the pictures and somehow it’s not that rude, because you can’t see it.’).
Browsing of Digital Selfs can help individuals identify relevant others, and participants were positive in how self-curated representations could help them express themselves (P20: ‘...it [Digital Self] gives additional layer of way you can express yourself.’). However, participants also highlighted potential dangers if the Digital Self itself appears uninteresting to them and would thus discourage, rather than encourage, interaction (P9: ‘It’s a good way to start a conversation, but in a way it could affect people that they see each other’s Digital Selfs before starting the conversation. So they might think that OK, that person isn’t interesting so I won’t talk to him or her.’).
Participants usually moved towards the nearest person, forming the first conversational groups (see Figure 9 and Table 1). Nine initial groups formed across all events. In 5 of these, Digital Selfs were referred to within the first 10 s from the start of the conversation. In this way the Digital Self was considered as publicly available common ground, acting as ‘tickets’ (P2: ‘I usually don’t start any conversation with anyone in the social events, so I think it would help me to start if I know something’, and P5: ‘We have the same background somehow, so it was much easier to ... make up some stuff, because you had the pictures and you had the texts.’). Digital Selfs in general were perceived as useful for obtaining tickets (P22: ‘You have certain amount of topics which you know that the other people are interested, so when you start a conversation on a certain topic, which someone is interested, definitely that person will respond positively because you know that he or she’s interested in the topic, so it will be a very good start-up for the conversation. So that was really helpful.’). Digital Selfs was also found to be useful to help those who would normally not interact with stranger to do so (P7: ‘I don’t usually talk that much with strangers, or any people. So it was a nice experience.’). There is evidence that topic suggestions would be more useful for introverted individuals (Nguyen et al., 2015). However, in a standard personality test (John and Srivastava, 1999) our participants had a mean extraversion score of 3.0 (SD = 0.9) on a scale of 1 to 5. We did not find a statistically significant correlation (r = −0.17, p = 0.44) between the usefulness of the Digital Self and an individual’s extraversion score.
Whilst the Digital Self was generally perceived as useful for starting conversation and familiarising strangers, five participants did not feel that they really needed it (P9: ‘I thought the digital self was quite unnecessary. It felt like it was only in the way. And that I really wanted to get the conversation going just by myself and that it was only sort of like disturbance on the side...But I don’t think the Digital Self changed the conversation that much...it felt like it would have gone the same way even with or without the Digital Self.’). It was largely the content of the Digital Self that impacted its perceived usefulness. Content that presented very clear information, or which did not lend itself to discussion, tended not to be used. For example, P9 interacted with P12, who had only one image in his Digital Self (P9: ‘I opened it in the beginning, but then I think we already had started talking about things that were there [in the Digital Self] and it didn’t really give anything new to the conversation.’). Therefore we believe how an individual constructs his or her Digital Self has a greater impact on its use as an ice-breaker, than an individual’s level of extroversion.
5.3.1 Digital self use supporting conversation
Accessing during conversation
After initial ice-breaking, Digital Selfs continued to be accessed and referred to during conversation. Digital Selfs were used to provide new topics to continue the conversation. For example, when silent moments occurred or discussion on a topic naturally came to an end (P6: ‘When the conversation is really going good, and you are finding the topics of mutual interest, then you don’t look at the digital self. But if you think you are running out of the topic, then you might go to the digital self of the other person so that you are going to find new things. But it only helps when the conversation is detracting and you are not finding anything new to talk. But if the conversation is really good, you don’t care about the digital self at that point’, P9: ‘And at one point or another I think when there was a silence I checked them again.’). However, this also extended to ongoing conversation, where participants would access the Digital Selfs of other members of their subgroup to identify new topics to ‘pivot’ the conversation towards, and away from the current topic that may interest them less (P12: ‘After the conversation started, after having ice breakers and when the conversation was going, then I went to see the Digital Self, and I guess it, during the conversation, I guess for two times, I saw it so that if I can dig for a new topic of conversation.’). Time point C in Figure 10 illustrates this behaviour, when a topic from the Digital Self was incorporated into conversation. P16 cues content from a Digital Self into conversation after considering it for one minute. This causes other participants (including the owner of the Digital Self) in the same subgroup to access the referred to content and continue conversation from it. In this way all parties switch to the Digital Self, using it to act as a common ground to pivot the conversation topic. Another practical reason returning to Digital Selfs was obtaining names of others when available.3 It is common that people do not remember the names of other people they meet, especially where there is more than one name to remember (McWeeny et al., 1987). The Digital Selfs that contained names helped in this (P1: ‘I think it was useful to remember the name, if I could see it, for example your name I can remember, but I already forget your name.’).
When participants chose to access and browse the Digital Self of others was largely driven by their current role in the conversation. Participants who were active (either speaking or being directly addressed) focused on the conversation and did not change from the Digital Self they currently had active. Only when participants were unaddressed recipients did they access and switch between Digital Selfs (P5: ‘I found myself looking at them when I wasn’t part of the conversation, when other people were talking then I took some time to browse through them and read the text.’). Being active in the conversation required full attention by participants, and accessing the Digital Self was considered to be too demanding (P10: ‘I wanted to listen to the person at the time, but then look at the pictures, I’m a simple human being so it’s very hard to concentrate on just one [picture in Digital Self]’). In group conversations, participants had more time to view Digital Selfs (P14: ‘Especially if you want to watch the other person or listen to what they’re saying and look at the Digital Self, then it was too confusing. But in a group you get more time to look at the Digital Selfs. But even then you cannot talk and look at it at the same time.’). Whilst manual support to interact with a Digital Self (supporting the ‘sneak viewing’ previously discussed is important), when engaged in conversation, Digital Selfs require an automatic approach, sensitive to the cur-rent conversational role of the participant.
The topics that participants discussed were similar to those identified from non-augmented conversation between strangers (e.g., interests, where the person lives, what they do, education, occupation, social relations, places they visited and travelling) (Svennevig, 2000). The Digital Selfs did not substantially change the ‘topic space’, but did widen it, making more topics visible and accessible, allowing involving (deep) topics (Svennevig, 2000, p.91) to emerge. Thus, the Digital Selfs provided possibilities for participants to move out from conventional interview style discussion (e.g. ‘What do you do?’ and ‘Where do you come from?’ (Svennevig, 2000, p.91)) towards deeper rich topics that supported more meaningful disclosure and conversation on areas participants were already interested in. For example, transcripts in Figures 13 and 14 illustrate how the conventional interviewing questions were skipped by incorporating questions about the images in a Digital Self.
Whilst participants incorporated topics from the Digital Self, they did not feel under pressure to do so. Often, they felt that coherence in the conversation, and supporting its natural evolution was more important than dynamically changing the topic (P13: ‘I thought in the beginning, that 45 minutes is a really long time to just come up with conversation topics, but in the end it wasn’t. Cause once we got the conversation going and we joined as a big group, then it just went on.’). This revealed a tension, where participants would have liked to talk about a highly relevant shared interest, but did not want to disrupt the flow (P16: ‘Conversation was ongoing somewhere else. You had swimming and I love swimming, I mean if we were just the two of us in a room, that would have been the first thing to pick.’). In addition, some Digital Selfs that contained only basic information became quickly exhausted, and participants found there was nothing new to incorporate from them, so there was no need to try to pivot to something new (P9: ‘I used it in the beginning, but then I think we already had started talking about things that it was already going and it didn’t really give anything new to the conversation.’). It may also be the case that after some time the Digital Self has supported conversation through setting talk and onto rich topics, that conversation becomes self sustaining, and Digital Augmentation (at least in the static representation we used here) becomes less useful.
Digital Selfs were also used when an individual joined a pre-existing group (in addition to when participants ‘sneak viewed’ Digital Selfs outside their current group). Whilst we witnessed a limited number of participants who joined an ongoing conversation (6 participants over all events, 3 in Event 1 and 3 in Event 4), in five instances Digital Selfs were accessed. This was either by the person who joined the group (allowing him or her to integrate with the group) or by other members of the group (to find out more about the new attendee).
Cueing digital self information
Such issues were largely caused by asynchronous opening of the Digital Selfs, and can be seen in Figure 10, where the same Digital Self (P13’s) is open only once (at point D) for every participant, except P13 herself. Prior work, focusing on one-to-one conversation has not identified these issues, since there are no ‘alternative’ visualisations to view. Whilst sensing technology and control mechanisms could be employed to ensure all participants in a group are viewing the same Digital Self, as discussed in Section 5.2.2, this may also have negative impacts. However, an ability to quickly synchronise the Digital Self all participants see would remove much of the management work of cueing content.
5.3.2 Media use from digital selfs
When Digital Selfs were incorporated into conversation it was usually done through images. Par-ticipants did not just ask about contents of the images in Digital Selfs, but also how the images were taken or created. 49% of all images in the Digital Selfs were selected for topics in the conversation, whereas only 18% of text instances were referred to.
Text data was often found to be basic and self-explanatory, and was not felt by participants to provide a rich topic of conversation (P13: ‘His was self-explanatory, so I didn’t feel the need to talk about them anymore.’). However, this also extended to more general adjectives describing a person (e.g., ‘easy-going’, ‘energetic’ and ‘optimistic’) which were also left out from conversation. Participants wanted to make their own interpretations of other people based on interaction, rather than be told what a person was like (P11: ‘It’s not so interesting to get to know new people if you already know something about him or her, so you don’t have to dig all the specialties about the person if you already see the things he likes or doesn’t like.’).
As discussed, where individuals did not use Digital Selfs as ice-breakers, it was in part down to use of basic or textual content in the Digital Self (See Section 5.3). This extended to instances where the Digital Self was not used in conversation as it was too basic. Whilst we saw substantial usage of Digital Selfs, we would expect greater usage if the Digital Selfs were richer, incorporated more ambiguous content and were more image based. As discussed in Part 1, such rich and ambiguous Digital Selfs were more valuable in supporting conversation.
5.3.3 HMD issues
We chose to present Digital Selfs on Head-Mounted Displays, and whilst these are not the only possibilities (see Section 2.2.2), they provide good collocation between the face and Digital Self. In our study, their private nature also supported ‘sneak viewing’ of other’s Digital Selfs.
Our study has generated significant new insight into how individuals choose to augment them-selves with digital media in multi-party interaction. By curating representations themselves, and studying their use in multi-party situations, we have gone significantly beyond the current state of the art that has, so far, largely focused on automatic selection of media in strictly one-to-one interactions. In framing our discussion, we do so around our two research questions.
6.1 RQ1: how do individuals choose to represent themselves to strangers with digital selfs?
Participants strongly favoured images over text content to incorporate in their Digital Self. Only a minority of participants chose text only Digital Selfs. Additionally, the majority of images participants chose to include came from outwith existing social and digital media accounts, with over 70% being sourced from a Google image search. This is surprising given the emphasis of existing work (Chen and Abouzied, 2016; Jarusriboonchai et al., 2015; Nguyen et al., 2015) which both focuses on matching or using social media from existing accounts, and presenting those matches to users as text. Our results indicate that whilst these techniques identify common interests, they are unlikely to represent those aspects of self that individuals would wish to present publicly to strangers.
The use of such images was mostly due to participants wanting to express ambiguity in their Digital Self, and images were a good way to do this. Participants considered that the interpre-tation of the image provided only some insight into its meaning, and how it represented them. This meaning could be further disclosed through conversation, facilitating the use of the Digital Self in boundary regulation (Lampinen, 2014), and allowing an individual to dynamically man-age disclosure through conversation. From Part 2 of the study there is also evidence that such ambiguity in presentation through images stimulates and supports conversation more than provid-ing simple textual information. Such Digital Selfs were found to be useful through all stages of conversation. The lack of use of Digital Selfs by some participants was at least in part due to the information in Digital Selfs being textual and unambiguous (name, likes, personality, occupation, etc.). Participants found these to be unstimulating, and often didn’t use them or incorporate them into conversation. Whilst this is not the only reason participants chose not to use Digital Selfs, those that provide only basic facts or interests are unlikely to enhance interaction. Unlike prior work, such as Nguyen et al. (2015), and because we had multiple parties and covered all stages of conversation, there was less explicit focus on use of the Digital Self as the ‘task’, and participants felt able to not use it when they felt it wouldn’t help them. Further study to show this is required, but participants should be encouraged to create richer and potentially more ambiguous Digital Selfs if they are to have greatest benefit. In avoiding automatic selection, identifying only those who are similar, self curation avoids isolation from others that are dissimilar and potentially contributing to a ‘filter bubble’ (Resnick et al., 2013). As with online media, digital augmenta-tions with the same automatic recommendation algorithms may simply keep like-minded people together, whilst as noted by Mayer et al. (2015), individuals may often be open to meeting others who are dissimilar. In many work situations it is often necessary to work with others who are dissimilar, and understanding those differences may be as important as identifying similarities.
Using these richer and more ambiguous representations may also reduce the barrier to interaction with others, providing not only a ticket to commence an initial interaction, but opportunities of setting talk that have greater potential to lead to richer interactions. Such media may pro-vide accelerated opportunities to establish better awareness of others and common ground with them. Within CSCW, this may be a practical approach where ad-hoc teams must form quickly and effectively in a short time period. For example, Wong and Neustaedter (2017) have identified how cabin crew must quickly learn about and trust one another in a safety critical environment. Similarly Lykourentzou et al. (2017) notes how having a deeper relationship with team members before those teams are formed enhances task performance. Although we have focused on more social scenarios, and the media chosen may well be different, the use of ambiguous media may provide an effective way to support this relationship forming between team mates faster.
6.2 RQ2: how are digital selfs used at each stage of interpersonal interaction between strangers in multi-party settings?
Existing work has largely focused on supporting individual stages of interaction, and evaluated work has focused only some of these (e.g. (Chen and Abouzied, 2016; Jarusriboonchai et al., 2015; Nguyen et al., 2015)). Existing solutions are unlikely to support all three stages. For example, whilst Jarusriboonchai et al. (2015) badges present simple information to support ice-breaking, they are unlikely to be useful during conversation. Nguyen et al. (2015) focused only on supporting conversation, and although their system provided new topics to discuss, it did not support evolving conversation beyond basic interaction. Our work found that Digital Selfs were employed, and were useful at, all stages of conversation. Most importantly, and unlike prior work, they are also useful after the initial ice-breaking phase and may support moving onto rich topics. Whilst Digital Selfs were still accessed when participants had established conversation, the frequency of accessing and switching reduced during the course of the study. This indicates that conversation had reached richer topics of which there was more to talk about. This is unlike Nguyen et al. (2015), which shows more referrals and use of topic suggestions as their study progresses. We argue Digital Selfs supported richer conversation, which did not need augmentation to sustain, whilst towards the end of interaction Nguyen et al. (2015) automatic topic selections were used to simply keep talking rather than those conversations being insightful. Again, this highlights how Digital Selfs might support individuals who must work together (e.g. (Wong and Neustaedter, 2017)) to form better relationships based on aspects of their interest, increasing common ground between them, and if not forming closer personal relationships, at least gaining a better understanding of others.
Whilst participants used Digital Selfs at all stages of interaction, not all participants used them at every stage. It was not the case that some participants did not use Digital Selfs at all. Participants used Digital Selfs when they felt it would be beneficial, and felt able to ignore it when they did not. There were multiple reasons why or why not this was done, but the Digital Self did not dominate the interaction, nor replace existing conversational practices. If participants were engaged in an interesting conversation, the Digital Self would sit ‘on the side’, with topics participants found and wished to introduce left until a suitable point emerged. Participants were comfortable using content as an ‘ice-breaker’. Either starting conversation directly with it, or introducing themselves before. Our event is an example of an ‘open-region’ (Goffman, 1963), where there is an assumption, due to the context, that interaction is permissible. This is true in all other formal studies of face-to-face augmentation (Nguyen et al., 2015; Chen and Abouzied, 2016; Douglas, 1990; Maynard and Zimmerman, 1984). It is challenging to study such systems outside of an ‘open-region’ environment, where such assumptions cannot be made (e.g. at a coffee shop), but there would significant value in doing so and understanding how this might impact their use in ‘ice-breaking’. Digital Selfs also worked as ‘advertisements’ for participants, as they guided which conversation to join. Digital Selfs played a key role in establishing common ground when participants (newcomers) joined conversations. Newcomers accessed the Digital Selfs of the members of their ‘new’ conversation group before joining, and newcomers were integrated into conversations through Digital Selfs by incorporating a conversation topic from that newcomer’s Digital Self. Again, in relation to more general CSCW, there is potential to support on-boarding of new team members of colleagues faster into existing groups. A potential future avenue of investigation is to consider how Digital Selfs perform in creative environments, such as academic departments or maker spaces, where individuals are open to new collaborations with others. Existing work has investigated public displays (Bilandzic et al., 2013) for this, but there is value in considering how co-located representations of self (such as current ideas or work) might support this, and leverage on the ‘advertisements’ identified by participants.
Whilst participants did browse others before starting interaction, and again before merging groups, we observed much less of this behaviour than we expected. We argue that in small multi-party gatherings there is less need to be selective on who to talk to. In the 3 person groups we observed fewer instances of browsing Digital Selfs before moving to ice-breaking, although Digital Selfs were used during ice-breaking for these groups. In the 4–5 person groups we saw more evidence of this ‘pre-browsing’, and we would expect more in even larger groups. All participants started interaction at the same time at the beginning of the study. This comes from existing study methods (Nguyen et al., 2015; Douglas, 1990) that we used in our study, and a desire to ensure everyone started from the same position. However, it is somewhat unlike how individuals would enter such gatherings (such as parties), where individuals would arrive in a staggered order. Therefore there would not be the situation that all participants would arrive at the same time (although multiple people may still arrive at the same time). From participant comments, dealing with the Digital Self at the start of the event could also be demanding. Participants found it too much to both browse the Digital Selfs and decide who to talk to. We carried out our study in a controlled way to better consider how it related to existing one-to-one studies (such as (Nguyen et al., 2015)), and to ensure we focused on interaction with strangers. However, there is value in study of applying Digital Selfs to pre-existing events (which may have a mix of friends and strangers) to better understand how they are used for initial browsing of others and choosing who to interact with.
To do this it is necessary to consider automation support to access and move between Digital Selfs. Such issues, due to the focus on one-to-one interaction where there are no alternative augmentations have not been uncovered in prior work. We allowed only manual selection of the Digital Self as we did not want to constrain how they were used by enforcing an automatic system to switch between them. In many cases manual selection was important. For example, the ability of participants to ‘sneak view’ the Digital Selfs of individuals outside their current group, or where an individual was an ‘unaddressed recipient’ (Gibson, 2003; Traum, 2004), and had time to look for interesting topics that they may wish to pivot to when a natural break in the conversation occurred. However, it was also clear that fully manual selection will not work in all places. In addition to the outlined issues on browsing at the start of interaction, manual selection often led to breakdowns during conversation. Digital Selfs acted as an invisible ‘layer’, providing resources to the conversation. Although participants were explicit when cueing information that came from the Digital Self (through explicit utterances or gesturing towards the HMD), participants were often not all viewing the same Digital Self. This led to a collapse of common ground, requiring management work to access the correct Digital Self to ‘repair’ (Clark and Brennan, 1991) and re-establish common ground. In such cases an automatic approach, or ability to quickly ‘sync’ a common Digital Self amongst group members, would be beneficial. Further work on mechanisms to do this is required, incorporating sensors to determine automatically who is in the participant’s group, and what the user’s current role is (e.g. is he or she currently talking) is one approach. Existing CSCW work (such as (McCarthy et al., 2004; Chen and Abouzied, 2016)) has considered only one-on-one interaction, and has not yet considered how groups form, reform and interact. Our work here expands on this to identify how digital augmentation is used in these phases, and will help develop mechanisms that support automatic switching between augmentations. Such mechanisms are not trivial, but are essential to further study Digital Selfs in the larger group scenarios we previously discussed.
6.3 Future work
Our future work is largely focused around addressing the issues in the discussion. By developing better support to synchronise and move between Digital Selfs, we will be able to study their use in larger groups of individuals and augment existing events. By encouraging participants to generate richer and more ambiguous Digital Selfs, we will be able to improve our understanding of how they can be used in browsing for individuals at the start of interaction. However, such events are largely a mixture of strangers and individuals with a prior relationship (e.g. friends). Prior work, including our own, has largely focused on strangers, yet an interview study that formed our initial ideas and motivated this work identified that that there is a need to tailor contents for different audiences (McGookin and Kytö, 2016), as does existing work on social media management (Farnham and Churchill, 2011). Individuals may wish to create different Digital Selfs for different relationships, yet interaction may occur in a mixed-group. Further study is required in different situations and prior relationships between individuals to strengthen and deepen our understanding of digital augmentation of interaction. Whilst we argue there is value in general interaction, it is clear that Digital Selfs may be valuable in a number of more focused, work related environments. Although the content individuals choose would obviously differ from the more general social scenarios we used here, there is value in applying our approach to situations where personal relationships between team members are both important, but where the work situation constrains the time to develop them (e.g. (Wong and Neustaedter, 2017)).
Our work has been the first to consider how digital, user-curated, representations of self can be incorporated into multi-party face-to-face interaction. Unlike prior work in both HCI and CSCW, that focuses on algorithmic matching, potentially causing ‘filter bubbles’, and does not consider how users would wish to be represented, we identified that media users choose is often ambiguous and comes from outside the existing digital and social media services that algorithmic matching employs. Our study also identified that user curated representations are effective at all three stages of face-to-face interaction, but did not dominate conversation. Prior work has only investigated a subset of these and has not considered the importance of their ‘non-use’. Our findings have significantly advanced knowledge into the emerging field of how face-to-face interaction can be augmented with digital media to support interaction.
- Abouzied, Azza; and Jay Chen. (2014). Commonties: A context-aware nudge towards social interaction. In S. Fussell; W. Lutters; M. Ringel Morris; and M. Reddy (eds): CSCW Companion ’14 . Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, Baltimore, Maryland, 15–19 February 2014. New York, NY, USA: ACM Press, pp. 1–4.Google Scholar
- Altman, Irwin; and Dalmas A. Taylor. (1973). Social Penetration: The Development of Interper-sonal Relationships. Oxford, England: Holt, Rinehart & Winston.Google Scholar
- Bilandzic, Mark; Ronald Schroeter; and Marcus Foth. (2013). Gelatine: Making coworking places gel for better collaboration and social learning. In H. Shen; R. Smith; J. Paay; P. Calder; and T. Wyeld (eds): OzCHI ’13. Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, Adelaide, Australia, 25–29 November 2013. New York, NY, USA: ACM Press, pp. 427–436.Google Scholar
- Borovoy, Richard; Fred Martin; Sunil Vemuri; Mitchel Resnick; Brian Silverman; and Chris Han-cock. (1998). Meme tags and community mirrors: Moving from conferences to collaboration. In S. Poltrock and J. Grudin (eds): CSCW ’98. Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work, Seattle, Washington, USA, 14–18 November 1998. New York, NY, USA: ACM Press, pp. 159–168.Google Scholar
- Bussmann, Uwe; and Silvia Schweighofer. (2014). Group Dynamics: The Nature of Groups as well as Dynamics of Informal Groups and Dysfunctions. Hamburg, Germany: Anchor Academic Publishing.Google Scholar
- Chen, Jay; and Azza Abouzied. (2016). One led is enough: Catalyzing face-to-face interactions at conferences with a gentle nudge. In D. Gergle; M. Ringel Morris; P. Bjørn; and J. Konstan (eds): CSCW ’16. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco, California, 27 February – 2 March 2016. New York, NY, USA: ACM Press, pp. 172–183Google Scholar
- Clark, Herbert H.; and Susan E. Brennan. (1991). Grounding in Communication. In L. B. Resnick; J. M. Levine; and S. D. Teasley (eds): Perspectives on Socially Shared Cognition.: APA Books, pp. 127–149.Google Scholar
- Devendorf, Laura; Joanne Lo; Noura Howell; Jung Lin Lee; Nan-Wei Gong; M. Emre Karagozler; et al. (2016). “I don’t want to wear a screen”: Probing perceptions of and possibilities for dynamic displays on clothing. In J. Kaye; A. Druin; C. Lampe; D. Morris; and J. P. Hourcade (eds): CHI ’16. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Santa Clara, California, USA, 7–12 May 2016. New York, NY, USA: ACM Press, pp. 6028–6039.Google Scholar
- Farnham, Shelly D.; and Elizabeth F. Churchill. (2011). Faceted identity, faceted lives: Social and technical issues with being yourself online. In P. Hinds; J. C. Tang; J. Wang; J. Bardram; and N. Ducheneaut (eds): CSCW ’11. Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, Hangzhou, China, 19–23 March 2011. New York, NY, USA: ACM Press, pp. 359–368.Google Scholar
- Goffman, Erving. (1963). Behavior in Public Places: Notes on the Social Organization of Gatherings. New York, USA: The Free Press.Google Scholar
- Goffman, Erving. (1969). The Presentation of Self in Everyday Life. London, England: Allen Lane The Penguin Press.Google Scholar
- Hall, Edward T. (1966). The Hidden Dimension. New York, USA: Anchor Books editions.Google Scholar
- Jarusriboonchai, Pradthana; Thomas Olsson; Vikas Prabhu; and Kaisa Väänänen-Vainio-Mattila. (2015). CueSense: A wearable proximity-aware display enhancing encounters. In B. Begole; J. Kim; K. Inkpen; and W. Woo (eds): CHI EA ’15. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, Seoul, Republic of Korea, 18–23 April 2015. New York, NY, USA: ACM Press, pp. 2127–2132.Google Scholar
- Jarusriboonchai, Pradthana; Thomas Olsson; and Kaisa Väänänen-Vainio-Mattila. (2014). User experience of proactive audio-based social devices: A Wizard-of-Oz study. In Z. Zaslavsky; S. W. Loke; L. Kulik; and E. Pitoura (eds): MUM ’14. Proceedings of the 13th International Conference on Mobile and Ubiquitous Multimedia, Melbourne, Victoria, Australia, 25–28 November 2014. New York, NY, USA: ACM Press, pp. 98–106.Google Scholar
- John, Oliver P.; and Sanjay Srivastava. (1999). The Big Five Trait Taxonomy: History, Measure-ment, and Theoretical Perspectives. In L. Pervin and O. P. John (eds): Handbook of Personality: Theory and Research 2nd ed.. New York, USA: Guilford.Google Scholar
- Kan, Viirj; Katsuya Fujii; Judith Amores; Chang Long Zhu Jin; Pattie Maes; and Hiroshi Ishii. (2015). Social Textiles: Social affordances and icebreaking interactions through wearable social messaging. In B. Verplank; W. Ju; A. Antle; Mazalek. A.; and F. Mueller (eds): TEI ’15. Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, Stanford, California, USA, 15–19 January 2015. New York, NY, USA: ACM Press, pp. 619–624.Google Scholar
- Kao, Hsin-Liu (Cindy); and Chris Schmandt. (2015). Mugshots: A mug display for front and back stage social interaction in the workplace. In B. Verplank; W. Ju; A. Antle; Mazalek. A.; and F. Mueller (eds): TEI ’15. Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, Stanford, California, USA, 15–19 January 2015. New York, NY, USA: ACM Press, pp. 57–60.Google Scholar
- Lampinen, Airi. (2014). Interpersonal Boundary Regulation in the Context of Social Network Services. Ph.D. dissertation, University of Helsinki, Helsinki, Finland.Google Scholar
- Lykourentzou, Ioanna; Robert E. Kraut; and Steven P. Dow. (2017). Team dating leads to better online ad hoc collaborations. In C. P. Lee; S. Poltrock; L. Barkhuus; M. Borges; and W. Kellogg (eds): CSCW ’17. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, Oregon, USA, 25 February – 1 March 2017. New York, NY, USA: ACM Press, pp. 2330–2343.Google Scholar
- Mayer, Julia M.; Starr Roxanne Hiltz; Louise Barkhuus; Kaisa Väänänen; and Quentin Jones. (2016). Supporting opportunities for context-aware social matching: An experience sampling study. In J. Kaye; A. Druin; C. Lampe; D. Morris; and J. P. Hourcade (eds): CHI ’16. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Santa Clara, California, USA, 7–12 May 2016. New York, NY, USA: ACM Press, pp. 2430–2441.Google Scholar
- Mayer, Julia M.; Starr Roxanne Hiltz; and Quentin Jones. (2015). Making social matching context-aware: Design concepts and open challenges. In B. Begole; J. Kim; K. Inkpen; and W. Woo (eds): CHI ’15. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, 18–23 April 2015. New York, NY, USA: ACM, pp. 545–554.Google Scholar
- McCarthy, Joseph F.; David W. McDonald; Suzanne Soroczak; David H. Nguyen; and Al M. Rashid. (2004). Augmenting the social space of an academic conference. In J. Herbsled and G. Olson (eds): CSCW ’04. Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, Chicago, Illinois, USA, 6–10 November 2004. New York, NY, USA: ACM Press, pp. 39–48.Google Scholar
- McGookin, David; and Mikko Kytö. (2016). Understanding user attitudes to augmenting face-to-face interactions with digital and social media. In J. Hakkilä and T. Ojala (eds): MUM ’16. Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, Rovaniemi, Finland, 12–15 December 2016. New York, NY, USA: ACM Press, pp. 285–296.Google Scholar
- McWeeny, Kathryn H.; Andrew W. Young; Dennis C. Hay; and Andrew. W Ellis. (1987). Putting Names To Faces. British Journal of Psychology, vol. 78, no. 2, pp. 143–149.Google Scholar
- Nguyen, Tien T.; Duyen T. Nguyen; Shamsi T. Iqbal; and Eyal Ofek. (2015). The known stranger: Supporting conversations between strangers with personalized topic suggestions. In B. Begole; J. Kim; K. Inkpen; and W. Woo (eds): CHI ’15. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, 18–23 April 2015. New York, NY, USA: ACM Press, pp. 555–564.Google Scholar
- Ofek, Eyal; Shamsi T. Iqbal; and Karin Strauss. (2013). Reducing disruption from subtle information delivery during a conversation: Mode and bandwidth investigation. In W. E. Mackay; S. Brewster; and S. Bødker (eds): CHI ’13. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 27 April – 2 May 2013. New York, NY, USA: ACM Press, pp. 3111–3120.Google Scholar
- Pakanen, Minna; Tuomas Lappalainen; Paula Roinesalo; and Jonna Häkkilä. (2016). Exploring smart handbag concepts through co-design. In J. Häkkilä and T. Ojala (eds): MUM ‘16. Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, Rovaniemi, Finland, 12–15 December 2016. New York, NY, USA: ACM Press, pp. 37–48.Google Scholar
- Pariser, Eli. (2011). The Filter Bubble: What the Internet is Hiding From You. London, England: Penguin.Google Scholar
- Persson, Per; Jan Blom; and Younghee Jung. (2005). Digidress: A field trial of an expressive social proximity application. In M. Beigl; S. Intille; J. Rekimoto; and H. Tokuda (eds): Ubicomp ’05. Proceedings of the 7th International Conference on Ubiquitous Computing, Tokyo, Japan, 11–14 September 2005. Berlin, Heidelberg: Springer-Verlag, pp. 195–212.Google Scholar
- Resnick, Paul; R Kelly Garrett; Travis Kriplean; Sean A Munson; and Natalie Jomini Stroud. (2013). Bursting your (filter) bubble: strategies for promoting diverse exposure. In A. Bruck-man; S. Counts; C. Lampe; and L. Terveen (eds): CSCW ’13. Proceedings of the 2013 Conference on Computer Supported Cooperative Work Companion, San Antonio, Texas, USA, 23–27 February 2013. New York, USA: ACM Press, pp. 95–100.Google Scholar
- Rubin, Zick. (1974). Lovers and Other Strangers: The Development of intimacy in encounters and relationships: Experimental Studies of Self-Disclosure Between Strangers at BUS STOPS and in Airport Departure Lounges can Provide Clues about the Development of Intimate Relationships. American Scientist, vol. 62, no. 2, pp. 182–190.Google Scholar
- Sacks, Harvey. (1992). Lectures on Conversation, Vols. I and II. Oxford, England: Blackwell.Google Scholar
- Svensson, Marcus Sanchez; and Tomas Sokoler. (2008). Ticket-to-talk-television: Designing for the circumstantial nature of everyday social interaction. In K. Tollmar and B. Jönsson (eds): NordiCHI ‘08. Proceedings of the 5th Nordic Conference on Human-Computer Interaction: Building Bridges, Lund, Sweden, 20–22 October 2008. New York, NY, USA: ACM Press, pp. 334–343.Google Scholar
- Traum, David. (2004). Issues in Multiparty Dialogues. In F. Dignum (ed): Advances in Agent Communication: ACL 2003. International Workshop on Agent Communication Languages, Melbourne, Australia, July 14, 2003. Revised and Invited Papers. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 201–211.Google Scholar
- Wong, Stephanie; and Carman Neustaedter. (2017). Collaboration and awareness amongst flight attendants. In C. P. Lee; S. Poltrock; L. Barkhuus; M. Borges; and W. Kellogg (eds): CSCW ’17. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, Oregon, USA, 25 February – 1 March 2017. New York, NY, USA: ACM Press, pp. 948–961.Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.