Towards Interactional Symbiosis: Epistemic Balance and Co-presence in a Quantified Self Experiment
In the frame of an experiment dealing with quantified-self and reflexivity, we collected audio-video data that provide us with material to discuss the ways in which the participants would work out social synergy through co-presence management and epistemic balance – accounting for their orientation towards the familiar symbiotic nature of human interactions. Following a Conversational Analysis perspective, we believe that detailed analysis of interactional behaviors offers opportunities for socially interactive robots design improvements, that is: identify and reproduce human ordinary skills in order to make the machines more adaptable.
KeywordsQuantified self Conversational analysis HRI Epistemics
Face-to-face conversation between humans implies a moment-by-moment organization of turns at talk, artifacts use and body postures that provide for the accountability of what is going on, what has been done, and what one could expect to be done. By ‘accountability’ is meant that social practices contain their very intelligibility as they occur in the here-and-now of activities. Analysts can rely on this situated intelligibility to describe and analyze how social activities are organized . Social activity is structured as an emergent product of interrelations between sequential organization of talk, gestures, cognition and objects from the environment [22, 34, 43]. Therefore, social activity fait système and becomes a unit for analysts, because it is accountably treated as one by the social actors in the first place. Following Charles Goodwin’s definition of symbiosis , we will consider that through social interaction, humans are meant to organize some ‘wholes’ that are both different from, and greater than their parts, and constructed through the mutual interdependence of unlike elements.
There are many ways to describe the social ballet that is wound up each time people are co-present – and that would inspire research in Human-Machine Interaction. In this paper we present data collected from an experiment where epistemic balance was at stake. Essentially, in the frame of a musical quiz, we created situations where the robot would at some delimited occasions step into a person’s epistemic territory. The intended encroachment was triggered through a reflexive utterance produced by the robot during the interaction, that is, a turn-at-talk that would reflect traces of the person’s own activity . The emergence of the reflexive turns relied on the participants’ physiological measurements that were measured with a connected Empatica E4 wristband [11, 16]. This epistemic encroachment was regularly responded to by the recipients. A close analysis of the following sequential environment ‘Reflexive Turn - Response’, will lead us to account for a collection of resources used by the participants in order to create a familiar social solidarity/synergy.
2 CA, Quantified-Self and HRI
Leaning on human-human (or animal) interactions to support conversational agents or robots’ design, relies on a robust literature [5, 10, 13, 14, 28]. In the light of the development of social robots or socially interactive robots , concepts drawn from sociology such as connection, co-presence and cooperation, figure in definitions of engagement as ways to describe commitments or partnership in Human-Machine/Robot interactions  (HRI, HCI). However, such concepts are rarely scrutinized in their practical and sequential achievement. For the few researchers in HRI or HCI who have used it, Conversational analysis (CA), as a descriptive and naturalistic approach using interaction itself as a resource for analysis , provide analytical and methodological tools either to account for the moment-by-moment accomplishment of Human-Machine situated interactions [40, 41], either to design systems, and therefore anticipate on further interactions . The present work is a contribution to this approach that aims at accounting for the sociality of robots as a practical accomplishment and therefore proposes an interactionist perspective on symbiosis. We assumed that giving access to the robot to something about the personal territory of an individual, like physiological data, could be questionable as something that helps, or somehow has an effect on the relation between this human and that robot.
The concepts of Quantified self and self monitoring are novel ideas to the field of HRI and we (the authors) are not aware of any experiments where this concept has ever been used in human-robot interaction. There are however several studies in the domain of human-computer interaction where the concept of quantified self has been employed. Li, Dey and Forlizzi  discuss how we can develop tools for analyzing the data that we collect about ourselves and how we can better perform self-reflection using this data. Human-computer interaction researchers have applied the concept of Quantified Self and self-tracking to diverse domains: to study electricity consumption , transportation habits , eating habits , and exercise habits . While the use of smartphones and wearable devices continues to increase, we are finding new uses for these devices for self-monitoring such as measurement of our physical activity [12, 48], tracking our sleep patterns  and even looking at changes in our mood over time  among others which have become commercial successes. The E4 sensor and its predecessors have been used in a variety of experiments by researchers. Pieper and Laugero  used features collected using the Q sensor (predecessor to E4) in a study on preschool children and their emotional eating habits. Hernandez et al.  utilized the E4 to study stress during driving. In our experiment, we utilized only three of these measures: galvanic skin response [3, 7], pulse rate and peripheral skin temperature and two derived measures: slope of the galvanic skin response and change in pulse rate.
3 Experiment’s Set up
3.1 Woz Set up
3.2 Activity’s Scenario: The Musical Quiz
The interaction was based on a straightforward musical quiz. After a short greetings sequence, the game started. We asked participants to play at least 3 rounds of the game, so that we could use different profiles (see below). Each round of the quiz contained 4 short extracts of music selected randomly from a music library containing music from various genres. After the extract was played (through the same integrated Nao’s speakers), the participant had 30 s to guess the artist and the title of the song. During this search, the participant could be interrupted by the robot producing a reflexive turn if significant physiological variations were detected. Besides, at the beginning of the game as well as at the end of each round, the robot produced a quantitative reflexive turn by uttering the measurements.
3.3 Turn Design and Robot’s Profile
The robot’s reflexive utterance design relies, first, on an exploration of pragmatic dimensions. That is, we wanted to encompass an occurrences’ spectrum from «giving straightforwardly piece of information» to «provide the participant with a warning, a council». Second, we paid attention to the distribution of the turns along an epistemic gradient [24, 30]. That is, some utterances would display a primary knowledge access from the robot’s point of view, whereas others would be more balanced towards the participant. Third, we have drawn from the pair Warmth/Competence that is used to study the believability of AI , to build the robot’s profile. That is, some utterances would be supposed to display a Competence feature (e.g. «your heart beat is X»; «you are stressed»; «your heart beat is rising»), while others would rather display some empathy or care (e.g. «are you stressed because you can’t handle it?»).
3.4 Data Collected
In the first session of our experiment, the robot operator was visible to the participants but the participants were not aware if the operator was controlling the robot or was just observing the functioning of the robot. In the 2 following sessions, the operator was hidden from the participants and therefore the experiment followed the Wizard of Oz paradigm. The number of subjects is 12 (5 men and 7 women), recorded over 3 sessions (3 participants in the first session, 5 in the second, and 4 in the third respectively) making up around 3 h of data. In the third session we proposed to a participant to attend to the interaction with her friend and colleague. 97 reflexive sequences were selected out of which 28 have been accurately transcribed following the CA methods.
4 Interactional Symbiosis as a Moment-by-Moment Achievement
Largely, we observed that reflexive turns had recurrently an effect on participants interacting with the robot. That is to say, when the robot claims, one way or another, an access to the participant’s personal territory, the latter displays some reaction to it. We intend to show that those reactions account for socio-organic solidarities as proofs of interactional symbiosis. First, there are behaviors that are general to social interactions, namely co-presence management issues and preference for agreement. Second, there are more specific practices related to the epistemic encroachment that participants undergo in the experiment.
4.1 Working Out Social Synergy (1): Co-presence Management
As the robot gives precise measurements, we observed in the data that participants would regularly display body postures, that could be described as ‘on-rendering-process faces’. Here is an example with an accurate multimodal transcription1:
There are indeed ways to display that one is taking informations into account without disturbing the ongoing accomplishment of the speaker’s turn. What is striking here, is that the participant is not only displaying this, but she’s also managing a basic co-presence problem : through a meticulous to and fro eye (and head) movement (L03), she operates the possible actions enabled by turn constructional units organization [21, 35, 45] to keep track of both the participation frame and the delivered information on herself. In other words she’s considering the robot’s turn as a component of a larger organizational process: we can see here how body posture, eyes movements and speech are entangled, in order to structure an intricate event like ‘receiving information about oneself’. Moreover, such methods that consist in moving body orientation from the speaker’s ‘face’ to an alternative (imaginary) space where one can accountably think over (i.e. showing a process of thinking), illustrate how treating the robot as a socio-interactional partner could be achieved, in the present of interaction.
This phenomenon can be even more intricate. In the following extracts two participants interact with the robot, P2 is attending to the quiz, P is the one officially ‘connected’ to the robot:
As we’ve seen that different kinds of semiotic resources are used in order to manage co-presence and to structure ‘wholes’, we can now turn to other phenomena, more specific to the experiment, for they have to do with epistemic (re)configuration.
4.2 Working Out Social Synergy (2): Epistemic Balance and Preference for Agreement
We found that participants display more elaborated reactions in face of reflexive turns that, instead of giving a precise measurement, produce other kinds of actions like interpretations, assessments, warnings. Extracts (3), (4) above present two different methods that participants can use to respond.
In extract (3), P couldn’t find the name of the music played. N produces a declarative turn that points out the participant’s emotional state. This is a kind of turn that call for agreement or disagreement [2, 47, 51]. P in return, displays an affiliation towards the reflexive turn with «ouais», and attaches it with an account that provides an explanation regarding why she may be indeed stressed – namely because she doesn’t know much about classical music. As we can see in the transcript this turn is simultaneously accomplished with a «no» head shake (L03). This way P associates a negative valence to her turn. While we can’t say for sure what object this valence refers to, this is at least a conduct that contributes to the affiliation of the participant towards the reflexive turn. Moreover, this way of building a turn is congruent with the way two persons preferentially behave in agreement sequences: first you refer to what is projected in the previous turn and then you deliver your position [26, 42, 44]. In extract (4), the participant failed in finding out the song title (and was quite sorry about that). N produces a warning that refers to an analysis of her heart beat increasing. Here again, we can observe an affiliation towards the robot’s turn through an assessment – namely that being stressed, is bad news .
Informations exchange is not only a matter of input-output mechanism. Informations such as that we’re concerned with, lie on a domain, or territory on which social actors have stratified access. That is, they occupy during the interaction an epistemic position on a gradient that extends from knowledgeable to less (or no) knowledgeable [24, 25, 30]. And during social interactions, participants may claim, negotiate, confirm, discard epistemic positions. The ways participants react to Nao’s turns, demonstrate that they configure those turns as components of a specific work, and that is: dealing ‘together’ with epistemic configurations.
Finally, whereas people consider having privileged access to their own experience, and so consider having specific rights to say something about it [2, 26], disagreement or confrontation may be problematic for the participants’ face, in a Goffman sense:
In line 01, N produces a polar question that embodies an explanation as a candidate for being stressed. This candidate is rejected by P (L03) but followed with an assessment (L06) that displays an interesting analysis of the very candidate: by showing a moral perspective on the emotional state mentioned in N’s turn, she analyzes it as an account of shame. Hence, without denying N’s epistemic authority to claim that she might be stressed, P challenges the robot’s account candidate in order to justify an epistemic re-balancing . This challenge is allowed by the very format of N’s turn – a polar question in which the recipient is entrusted with a knower stance. Moreover, what is striking here is the way the participant manages the juxtaposition of balancing the epistemic configuration with the problem of the preference for agreement: this is in fact not trivial that the «no» on line 03 is followed by a mitigation mark «pas trop», and an account (L06) that justifies the disagreement as a way to preserve both hers and the robot’s face [18, 29]. That is to say, P behaves as if the robot was indeed a ritually delicate object. Therefore, this peculiar extract, shows that the notion of symbiosis may encompass a moral dimension.
In this experiment we used a quantified-self device to provide the robot with the presupposition of a specific epistemic authority vis-à-vis the participant, and we tested this authority through reflexive sequences. Largely, we found (1) that reflexivity taken care of by the robot, has an effect on the participants’ behavior, as a step into their personal epistemic territory, and (2) that the persons display practices that show the analogous commitment as in human-human interactions regarding the preferential organization of turns-at-talk in terms of adjacency, agreement, and epistemic balance. Even if they are aware of the robot’s limitations, participants display an attention to organize a participation framework (with rights and obligations), in which the robot is treated like a participant in its own right. Organization is what binds elements, events or individuals in a symbiotic relationship, that is, a potential synergy in which different sign systems work together to build relevant action and accomplish consequential meaning. Therefore, the scope of turn-design must not be limited to stream of speech phenomena (grammar, speech acts), but must encompass structures providing for the organization of the endogenous activity systems within which strips of talk are embedded : 370.
As Foster  put it, detailed analysis of interactional behaviors offers opportunities for socially interactive robots design improvements, that is: identify and reproduce human ordinary skills (perception, practical reasoning, gesture…) in order to make the machines more adaptable regarding interactional situations (assistive robots in the home environment, companion robot). We observed in the data that the participants treat the robot as a body, a presence. It shows that social interaction, as Goffman identified it decades ago, is first, before talking, a matter of hand-to-hand management: entering in a mutual perceptive field, focusing on a common object – depending on the participation footing of the participant. Moreover, we introduced the problem of epistemic balance, and observed some occurrences of plasticity of the epistemic configurations. Epistemic balance phenomena analysis points out that social synergy is a dynamic process: status in the interaction are to be defined and may be negotiated. Epistemic configurations play also a fundamental role in the way higher-order classes of action such as suggestions, proposals, or offers are dealt with in the interaction. We believe this is an entire issue to explore in HRI.
Hence, we observed all sorts of practices that are grist to the interactional mill and accountable arguments for a view of humans working at establishing and maintaining socio-organic solidarities with a robot. This was done in a short term and a quite artificial setting. Prospectively, we would need a larger amount of data to extend the analysis towards (a) more natural interactions, (b) more differentiation in reflexive turn-design impact, (c) better understandings of the usefulness of physiological measurements as resources for Human-Machine situated symbiosis.
In this paper we use Conversational Analysis transcription convention from ICAR, Lyon 2: http://icar.univ-lyon2.fr/projets/corinte/bandeau_droit/convention_icor.htm.
- 2.Bergmann, J.: Veiled morality: notes on discretion in psychiatry. In: Drew, P., Heritage, J. (eds.) Talk at Work, pp. 137–162. CUP, Cambridge (1992)Google Scholar
- 5.Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., Yan, H.: Human conversation as a system framework: designing embodied conversational agents. In: Cassell, J., Sullivan, J., Prevost, S., Churchill, E. (eds.) Embodied Conversational Agents, pp. 29–63. MIT Press, Cambridge (2000)Google Scholar
- 6.Consolvo, S., Everitt, K., Smith, I., Landay, J.A.: Design requirements for technologies that encourage physical activity. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2006), pp. 457–466. ACM, New York (2006)Google Scholar
- 8.Delaborde, A., Tahon, M., Barras, C., Devillers, L.: A wizard-of-Oz game for collecting emotional audio data in a children-robot interaction. In: Proceedings of the International Workshop on Affective-Aware Virtual Agents and Social Robots (Affine 2009), New York (2009)Google Scholar
- 10.Dubuisson–Duplessis, G., Devillers L.: Towards the consideration of dialogue activities in engagement measures for human-robot social interaction. In: IROS2015 International Conference on Intelligent Robots and Systems, Designing and Evaluating Social Robots for Public Settings Workshop, pp. 19–24, Hambourg, Germany (2015)Google Scholar
- 11.Empatica. https://support.empatica.com/
- 12.Fitbit. https://www.fitbit.com/
- 14.Foster, M.E.: Natural face-to-face conversation with socially intelligent robots. In: Proceedings of the IROS 2015, Hamburg, Germany (2015)Google Scholar
- 15.Froehlich, J., Dillahunt, T., Klasnja, P., Mankoff, J., Consolvo, S., Harrison, B., Landay, J.A.: Investigating a mobile tool for tracking and supporting green transportation habits. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2009), pp. 1043–1052. ACM, New York (2009)Google Scholar
- 16.Garbarino, M., Lai, M., Bender, D., Picard, R., Tognetti, S.: Empatica E3: a wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In: EAI 4th International Conference on Wireless Mobile Communication and Healthcare, pp. 39–42. IEEE press, Athens (2014)Google Scholar
- 17.Garfinkel, H.: Studies in Ethnomethodology. Prentice-Hall, Engelwood Cliffs (1967)Google Scholar
- 18.Goffman, E.: Interaction Ritual: Essays on Face-to-Face Behaviour. Penguin books, London (1967)Google Scholar
- 19.Goffman, E.: Behavior in Public Places: Notes on the Social Organization of Gatherings. The Free Press, New York (1963)Google Scholar
- 20.Goodwin, M.H.: Byplay: participant structure and framing of collaborative collusion. In: Framing Discourse: Public and Private in Language and Society. Meeting of the American Anthropological Association, Washington (1985)Google Scholar
- 21.Goodwin, C.: Interactive footing. In: Holt, E., Clift, R. (eds.) Reporting Talk: Reported Speech and Footing in Conversation, pp. 16–46. CUP, Cambridge (2007)Google Scholar
- 27.Hernandez, J., McDuff, D., Benavides, X., Amores, J., Maes, P., Picard, R.W.: AutoEmotive: bringing empathy to the driving experience to manage stress. In: Proceedings of the Companion Publication on Designing Interactive Systems (DIS 2014), Vancouver, Canada (2014)Google Scholar
- 28.Janssoone, T., Clavel, C., Bailly, K., Richard, G.: Using temporal association rules for the synthesis of embodied conversational agents with a specific stance. In: Traum, D., Swartout, W., Khooshabeh, P., Kopp, S., Scherer, S., Leuski, A. (eds.) IVA 2016. LNCS (LNAI), vol. 10011, pp. 175–189. Springer, Cham (2016). doi: 10.1007/978-3-319-47665-0_16 CrossRefGoogle Scholar
- 29.Kerbrat-Orecchioni, C.: Théorie des faces et analyse conversationnelle. In: Colloque de Cerisy, pp. 155–195. Les Editions de Minuit, Paris (1989)Google Scholar
- 30.Labov, W., Fanshel, D.: Therapeutic Discourse. New York Academic Press, New York (1977)Google Scholar
- 31.Li, I., Dey, A., Forlizzi, J.: Understanding my data, myself: supporting self-reflection with ubicomp technologies. In: Proceedings of the 13th International Conference on Ubiquitous Computing (UbiComp 2011), pp. 405–414. ACM, New York (2011)Google Scholar
- 32.Maynard, W.: Bad News, Good News. Conversational Order in Every-Day Talk and Clinical Settings. University of Chicago Press, Chicago & London (2003)Google Scholar
- 34.Morin, E.: La Méthode. Tome 1. La Nature de la Nature. Editions du Seuil, Paris (1977)Google Scholar
- 35.Ochs, E., Schegloff, E.A., Thompson, S. (eds.): Interaction and Grammar. CUP, Cambridge (1996)Google Scholar
- 36.Pélachaud, C., Glas, N.: Definitions of engagement in human-agent interaction. In: International Workshop on Engagement in Human Computer Interaction (ENHANCE), The Sixth International Conference on Affective Computing and Intelligent Interaction, pp. 944–949, Xi’an, China (2015)Google Scholar
- 37.Pélachaud, C., Glas, N.: Topic transition strategies for an information-giving agent. In: Proceedings of the 15th European Workshop on natural Language Generation, pp. 146–155, Brighton (2015)Google Scholar
- 39.Pierce, J., Paulos, E.: Beyond energy monitors: interaction, energy, and emerging energy systems. In: CHI 2012 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 665–674. ACM, Austin (2012)Google Scholar
- 40.Pitsch, K., Wrede, S.: When a robot orients visitors to an exhibit. Referential practices and interactional dynamics in the real world. In: IEEE ROMAN 23rd International Symposium on Robot and Human Interactive Communication, pp. 36–42. IEEE press, Edinburgh (2014)Google Scholar
- 41.Pitsch, K., Kuzuoka, H., Suzuki, Y., Süssenbach, L., Luff, P., Heath, C.: The first five seconds: contingent stepwise entry into an interaction as a means to secure sustained engagement in human-robot-interaction. In: IEEE ROMAN 18th International Symposium on Robot and Human Interactive Communication, pp. 985–991, Toyama, Japan (2009)Google Scholar
- 42.Pomerantz, A.: Agreeing and disagreeing with assessments: some features of preferred dispreferred turn shapes. In: Atkinson, J.M., Heritage, J. (eds.) Structures of Social Action, pp. 57–101. CUP, Cambridge (1984)Google Scholar
- 43.Rollet, N.: Analyse conversationnelle des pratiques dans les appels au Samu-Centre 15: vers une approche praxéologique d’une forme située «d’accord». Ph.D. Sciences du Langage, Sorbonne Nouvelle Paris 3, Paris (2012)Google Scholar
- 44.Sacks, H.: On the preferences for agreement and contiguity in sequences in vonversation. In: Button, G., Lee, J.R., (eds.) Talk and Social Organisation, Multilingual Matters, Clevedon, UK (1987)Google Scholar
- 49.Tahon, M., Delaborde, A., Devillers, L.: Corpus of children voices for mid-level social markers and affect bursts analysis. In: LREC 8th International Conference on Language Resources and Evaluation, Istanbul, Turkey (2012)Google Scholar
- 50.T2 mood tracker. https://play.google.com/store/apps/details?id=com.t2.vas&hl=en
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.