Interdisciplinary Explorations of Digital Touch
This chapter introduces and reflects on the multimodal, sensory and interdisciplinary methodological stance of this book, and the InTouch project more broadly. We introduce our main framework, which combines multimodality and sensory ethnography. We outline the collaborations and interdisciplinary dialogues that we have engaged with to explore digital touch, and argue that this approach brings different aspects of touch to the fore in ways that are productive for research and design. Finally, the different ways in which we use prototyping to gain access to, and to generate, digital touch experiences and imaginations for the purposes of research is outlined.
KeywordsDigital touch Interdisciplinary Methodology Multimodal Prototype Sensorial Sensory
As discussed in Chap. 1, the exploration of digital touch communication provided in this book is situated within a broad social revaluing of people’s sensorial experience and re-evaluation of the roles of the senses, a part of which is a technological awakening to the sensory and changing social configurations to notions of connection, and possibilities for touch enabled through technological innovation. This is driving a new wave of digital sensory communication devices and environments.
We approach digitally mediated touch as a (emergent) communicative mode, a sensorial experience entangled in the materiality and sociality of the body, the environment and technologies. In this book, the sociality of digital touch is our starting point, rather than the physiology of touch to move away from a concern with sensory receptors, tactile perception and neurological processes. We understand the physical, material, and sensory aspects of touch as a part of when and how touch-based resources are taken up (or excluded) and how they can shape – or are shaped by – people to become semiotic resources. While, we draw insights on the dimensions of touch from a psycho-physical and neuropsychological account of the physical experience and perception of touch, these accounts are limited through their focus on the individual, understanding of the senses as universal (and somewhat ‘fixed’), and not recognising “the role that culture plays in the modulation of perception senses function” (Howes 2011: 161). From our perspective, touch ‘sensation’ is much more than a biological and physiological process, insights on the physical dimensions of touch and the physiological processes through which ‘signals’ or tactile sensations (e.g. pain, temperature, pressure) are perceived are the ‘stuff’ of semiotics.
That digital touch communication is both a rapidly evolving area, and at the same time ‘state-of-the-art’ touch technologies are necessarily at an early stage of development and readiness, poses a number of methodological challenges for those who research it. In this chapter, we explore two such challenges.
First, the challenge of researching digital touch technologies that are unstable, lab-bound, and not yet domesticated. We have responded to this challenge of studying how people interact with such technologies, by using a range of methods and selecting a mix of technologies to enable ‘naturalistic’ interaction ‘in the wild’ to be observed; some early stable prototypes that can be demoed; alongside the lab-bound observation of unstable experiments and speculative early designs.
Second, the significant challenge of researching digital touch with under-developed methods and theories. We address this challenge through methodological innovation and an interdisciplinary approach using appropriate tools from multimodality, ethnographic tools attuned to the senses, art and design-based methods, and Human Computer Interaction (HCI) approaches. While multimodality and sensory ethnography provide the primary theoretical frame for this book, and InTouch more generally, our case studies all, albeit in different ways and to different extents, involve interdisciplinary collaboration and dialogue.
In the next section, we introduce our main framework, and make a case for our combination of a multimodal and sensory approach to the sociality of digital touch. We then turn to discuss our use of prototyping as a way to gain access to and generate digital touch experiences and imaginations and to support interdisciplinary dialogues on touch.
2.2 A Multimodal and Sensory Lens on Digital Touch Communication
Understanding bodily knowing through research on and with the body is a founding feature of the authors’ work within multimodality (Jewitt et al. 2016; Jewitt 2014; Kress et al. 2005, 2014) and sensory ethnography (Leder Mackley and Pink 2014) especially as articulated in relation to digital communication and interaction (Jewitt and Price 2019; Price et al. 2016). We bring multimodality and sensory methods together to explore touch in response to the methodological challenge of how to understand the changing social landscape of touch, and the need for embodied methods to help gain insight on socially orientated understandings of digital touch. In this section, we offer a brief overview of multimodality and sensory ethnography (for a discussion of the challenges of doing so, see Jewitt and Leder Mackley 2018).
Within multimodality, objects and sequences of interaction are understood as meaningful signs – the outcome of a person’s or people’s actions, imbued with the maker’s interests mediated through the environment in which the sign was produced or encountered (Kress 2010). Meaning is understood as socially situated choice from a (dynamic) set of available resources; the affordances of which are shaped through their historical, cultural and social usage and their materiality – all of which relate to and are shaped by technologies. Here we use affordance to refer to the idea that different modes offer different potentials for making meaning – a form of action. Modal affordances affect the kinds of semiotic work a mode can be used for, the ease with which it can be done, and the different ways in which modes can be used to achieve broadly similar semiotic work. Modal affordances are connected both to a mode’s material and social histories, that is, the social purposes that it has been used for in a specific context.
Multimodality enables us to describe, categorize and understand the material and social resources and affordances of touch, the principles that underpin them, how they are shaped and used. For instance, how signifier materials (e.g. temperature, pressure, texture) are, through their social usage (the work of people, communities and societies), made into signs shared by a community and used to communicate, and to shape, establish or maintain social norms and conventions (e.g. notions of gendered touch, see Chap. 4). As an approach to communication it stresses the relationship between meaning systems and the social needs they are used to serve (Bezemer and Kress 2016; Kress 2010). InTouch uses a multimodal approach to ask what is counted as touch by participants in a given context, what semiotic meanings they associate with the dimensions of touch (e.g. location, temporality), and how these are used and interpreted by people to make meaning. For instance, placing one’s hand on the shoulder of another person and holding it there for a long time, with pressure, to communicate intimacy and reassurance, or power and control (depending on the context). In this way, multimodality raises issues of power and agency, for instance in relation to who can touch. Using multimodality, we are starting to map the emergent dimensions of digital touch and the social conditions and contexts that shape it as well as to characterise people’s use of touch for communication with attention to the cultural and social norms and power relations that shape their use (Jewitt 2017).
If multimodality asks how meaning is made and communicated, what meanings are made, and by whom, sensory ethnography sets out to account for the experiential, how meaning is perceived, the sensorial and often unspoken dimensions of everyday life and human activity (Pink 2015). It presents a set of phenomenological approaches that are attuned to people’s sensory worlds and exist in theoretical-methodological dialogue with wider theories and concepts around human perception, place, knowing, memory, imagination, affect, and movement (Leder Mackley and Pink 2013). A key methodological feature of sensory ethnography is shorter-focused encounters with participants and the notion that much of what is important about our feelings and activities is not easily observed or put into language – tacit, embodied, and unspoken. This approach enables us to find routes through which to share or imaginatively empathize with the actions of people, collaboratively exploring with participants their ways of knowing, being and doing, whilst drawing on their own embodied and emplaced understandings (Dicks 2014). Video is used to generate ethnographic encounters in ways that account for their multi-sensoriality and functions, through a ‘form of acquaintance rather than description’ (Pink 2015: 2), as a way for the researcher to feel their way back into the research context. The video re-enactments used in our In Touch with Baby case study illustrate this ‘empathetic encounter’ with the participants’ sensory worlds with an emphasis on understanding the place of touch.
The analysis of the case study data uses a multimodal and multisensorial approach to explore how participants know and tell through touch and bodily interaction. Our engagement with the case study videos as data begins through re-viewing the recordings as a team, making notes of interactional details and tensions, reflecting on our own embodied experiences of the research activities and materials, and revisiting and handling prototypes with attention to their sensorial and social properties. We examine when and how touch, the sensorial and materiality are brought into the scope/discursive space of making meaning, as a way to generate understanding of participants’ conceptualizations and realizations of digitally mediated touch. This approach focuses in on the modal and sensory choices that people make to represent and communicate, how these choices are shaped by people’s interests, social position and context, and seeks to understand the social implications of their choices for meaning, communication and knowledge. These conceptualisations and organisations of touch are understood as framed by social-cultural historical contexts as dynamic and changing over time under the influence of new social factors. In this way, analysis of digital touch communication is grounded in the broad social framing provided by both approaches to emphasize the social-cultural embedded-ness of digital touch and to capture the nuances of the lived sensory accounts that shape the digital design and use of touch.
2.3 Interdisciplinary Dialogues of Digital Touch Communication
InTouch brings multimodality and sensory ethnography into conversation with a range of other disciplines. The emergent state of digital touch communication often means it is not possible to observe these technologies in use in a naturalistic setting (e.g. the home). This is especially problematic for multimodal and sensory ethnography, which usually observe technologies in naturalistic or ‘in-the-wild’ contexts, in order to explore issues of agency and power through uptake and the domestication of technologies (Rogers et al. 2013). However, HCI, design and the arts can create new digital touch communication environments through making, speculative prototyping, staged scenario work, and generating artistic experiences, enable us as social scientists to investigate new forms or potentials of social touch.
Artists (e.g. The Art of Remote Contact, Virtual Touch case studies);
Neuroscience (e.g. Tactile Emoticon);
Designers (e.g. Designing Digital Touch case study);
HCI, computer science and engineering (e.g. In Touch with Baby, Tactile Emoticon, and Virtual Touch case studies);
This interdisciplinary mix is also reflected in the InTouch team, we have backgrounds in Art, Design, Fashion, HCI, Media and Communication Studies, Psychology, and Sociology. We seek to achieve a rich and nuanced account of digital touch communication, working across our methodological and conceptual differences to understand points of difference, and, where useful, how they can be productively brought into dialogue by contrasting and layering disciplinary understandings.
Disciplinary differences in where and how touch research takes place (Berker et al. 2006) offers potential for productive interdisciplinary collaboration. At a most basic level, we have had to explore how our different conceptualisations of touch and the digital: do we mean the same thing when we talk of touch? Each discipline has distinct, and sometimes incompatible conceptualisations of touch. These are embedded in disciplinary histories and methodological approaches. Within Tactile Emoticon, for example, whilst as social scientists, HCI interaction designers, computer scientists and neuroscientists we all understand touch as a complex phenomenon involving the body, brain, and the social environment, we conceptualise and research touch in ways that emphasise and attend to these elements differently. From a multimodal and multisensorial ethnographic approach, we account for a broad context of participants’ engagement with emoticons and digital communication, and their semiotic and sensory interactions with the device and one another, always reading this in relation to social and cultural norms. The HCI and computer science researchers are focused on designing and understanding interaction between the users and the device. The neuroscience researcher approaches touch in relation to the individual, physical realizations (the brain and the body systems), mechanisms and processes of touch perception, and the relationship between stimuli and the sensations and perceptions they affect.
It has been useful for us to understand the different views of technology across our interdisciplinary conversations and collaborations: the extent to which the digital is foregrounded or valued, the expectations placed on it, and the sense to which technology is mutually shaped through its use. For example, Invisible Flock, collaborators in the Art of Remote Contact case study, described themselves as having an ‘agnostic approach to technologies’, sometimes working with stable consumer technology and sometimes using physical computing systems to prototype their own, and the importance of not having a ‘default go-to technology…So we try to stay open to technology…the idea comes first”. The technology is introduced at a later stage of their research process than, for example, was the case in the Tactile Emoticon study – which had specific communicative ideas underpinning the design and therefore suggested the technologies to be used at an early stage.
Where and how the social comes into researching touch, has been a central aspect of our interdisciplinary work on digital touch communication. In the case of the Tactile Emoticon case study, for instance, while we all engage with the social, we did so by focusing in on different levels of the social. The social is at the heart of a multimodal and multisensory approach and its conceptualisation of touch a form of communication realising and realised through the social functions of a society or group, which are always present at the level of the individual. While HCI is a broad field, it is inherently social in the recognition of the user and their relationship with a device or technology, and places them at the heart of its research, ‘always looking at people and scenarios of use, not just how the person works’ (Field-note). Our neuroscience colleagues engage with the social in relation to understanding what prior experiences or pre-existing beliefs participants may bring to an experimental context.
Across the other case studies, both the artists, and differently so, the designers critiqued the social expectations of touch. A member of Invisible Flock spoke of trying to create ‘a digital layer of friction between these normal interactions to then perhaps begin a conversation around it. So, you make holding hands a little bit more complicated so that maybe you stop and think about it a little bit more and we can begin a new conversation.’ In other words, they work to actively extend and problematize a felt social experience for a visitor, they seek to create a new social touch moment, while we as social scientists attempt to capture, interrogate and understand it. Our interdisciplinary collaborations with art, design and HCI to create such environments or experiences, enable us as social scientists to investigate new forms or potentials of social touch have proven invaluable: opening up discourses of touch, and prompting speculation and imaginaries on digital touch desires and futures.
The difficulties many people experience in articulating bodily experiences, imaginations, and tacit knowledge raise challenges for research (Tarr et al. 2017: 1). We bring prototyping into the frame of social science as a way to engage research participants in exploring touch and digital touch communication. As social researchers exploring the multimodal and multisensorial qualities of touch, the ways prototyping enable the body to play a central role in generating qualitative data are significant (Jewitt et al. 2019).
With its origins in product development within Engineering, Design, Computer Science, and Human Computer Interaction, prototyping has typically been concerned with developing ‘an idea about a product, system, service or policy to meet human needs and devising a plan for executing that idea’ (Binder et al. 2011). It is also associated with Design Thinking, which advocates for ‘thinking with your hands’ as a way of quickly and practically exploring an idea and the feasibility and development solutions, to pre-empt wasting time and money on something that might not work or might not be ‘user’ centred, in a quick, cost-effective and contextually aware way (Dunne and Raby 2013). The re-orientation of prototyping to high level concepts and ideas, rather than design products and skills, has enabled it to travel across the boundaries of engineering and design into the humanities and social sciences, including anthropology (Salazar et al. 2017), and more recently, sociology (Lupton 2018).
The methodological migration of prototyping to social research has partly been fuelled by a desire to research ‘emerging and uncertain worlds’ (Myers and Dumit 2011; Salazar et al. 2017), notably imagined digital futures, ‘configuring future imaginaries that may not be expected to come to pass’ (Lupton 2018: 5). We have found, this method aligns particularly well with researching the unstable, uncertain, future-facing technological devices and environments associated with digital touch. By bringing prototyping into the frame of multimodal and multisensorial work, we situate it within a wider move towards innovative and creative social science methods (Jewitt et al. 2017).
Observing participants from within HCI, engineering and industry demo their prototypes as part of an interview process
Deploying existing prototypes as research probes
Facilitating prototyping workshops with research participants
Collaborating with artists and HCI designers to inform the design and build of digital touch prototypes and observing interaction with these designs
These approaches to prototyping varied in relation to the function of the prototype, who had access to it, who was involved in making, the material and/or technological resources involved, and the degree of conceptual or technological ‘finish’. They provided an opportunity for case study participants, and us as researchers, to externalize unrefined concepts in material ways and, in the process, to identify and clarify key aspects of ideas, to make present new scenarios for digital touch communication, provoked questions and surfaced differences (e.g. in conceptualisations of digital touch and/or communication). The prototypes were reflected on, assessed, and refined, and provided a prop for participants (and researchers) to enact the experience of using a proposed artefact.
We found prototyping an effective research tool for exploring digital touch communication with participants from design, art, and HCI who are familiar with it, as well as with others with no prior design experience. The use of prototyping, in these different forms, helped to facilitate interdisciplinary dialogues and collaboration on digital touch communication across the social sciences, the arts, HCI and neuroscience.
Sharing and exploring concepts with collaborators from other disciplines through processes of making, touching, and manipulating materials and objects promoted collaborative interdisciplinary dialogues and thinking towards gaining new knowledge about relevant phenomena (Camburn et al. 2017): in this case, digital touch.
This chapter has made the case for making the sociality of digital touch our starting point and the focus of this book. We have discussed the methodological challenges of researching digital touch communication at a time when technologies are evolving rapidly and are not yet ‘domesticated’, and methods and theories remain under-developed. Throughout the chapter, and the book more generally, we have argued for the need to attend to the multimodal and multi-sensorial aspects of touch and the significance of interdisciplinary dialogues with, for instance, art, neuroscience, HCI, and design. We have made the case for an approach to digitally mediated touch as a communicative mode, a sensorial experience entangled in the materiality and sociality of the body, the environment and technologies. We have discussed the potential of four different uses of prototyping to bridge interdisciplinary differences in order to gain access to and generate digital touch experiences and imaginations for research purposes. In the next chapter, we map the complex terrain of digital touch technologies for communication.
- Berker T, Hartmann M, Punie Y, Ward KJ (2006) Domestication of media and technology. Open University Press, MaidenheadGoogle Scholar
- Bezemer J, Kress G (2016) Multimodality, learning and communication: a social semiotic frame. Routledge, LondonGoogle Scholar
- Design Council (2007) Eleven lessons: managing design in eleven global companies. Design Council, LondonGoogle Scholar
- Dunne A, Raby F (2013) Speculative everything: design, fiction, and social dreaming. The MIT Press, Cambridge, MAGoogle Scholar
- Howes D (2011) Chapter 9: Hearing scents, tasting sights: toward a cross-cultural multimodal theory of aesthetics. In: Bacci F, Melcher D (eds) Art and the senses. Oxford University Press, OxfordGoogle Scholar
- Jewitt C (2014) Handbook of multimodal analysis. Routledge, LondonGoogle Scholar
- Jewitt C, Leder Mackley K, Atkinson D, Price S (2019) Rapid prototyping for social science research. In: Pauwels L, Mannay D (eds) The SAGE handbook of visual research methods, 2nd edn. Sage, London. (in press)Google Scholar
- Kress G (2010) Multimodality: exploring contemporary methods of communication. Routledge, LondonGoogle Scholar
- Kress G, Jewitt C, Bourne J, Franks A, Hardcastle J, Jones K (2005) Urban English classrooms: multimodal perspectives. Routledge, LondonGoogle Scholar
- Kress G, Jewitt C, Ogborn J (2014) Multimodal teaching and learning. Continuum/Bloomsbury, LondonGoogle Scholar
- Leder Mackley K, Pink S (2014) Framing and educating attention: a sensory apprenticeship in the context of domestic energy research. In: Arantes L, Rieger E (eds) Ethnographien der Sinne: Wahrnehmung und Methode in empirisch-kulturwissenschaftlichen Forschungen. Transcript, Bielefeld, pp 93–109Google Scholar
- Lupton D (2018) Towards design sociology. Sociol CompassGoogle Scholar
- Myers N, Dumit J (2011) Haptics: haptic creativity and mid-embodiments of experimental life. In: Mascia- Lees F (ed) A companion to the anthropology of the body and embodiment. Blackwell, London, pp 240–261Google Scholar
- Pink S (2015) Doing sensory ethnography, 2nd edn. Sage, LondonGoogle Scholar
- Rogers Y, Yuill N, Marshall P (2013) Contrasting lab-based and in-the-wild studies for evaluating multi-user technologies. In: Price S, Jewitt C, Brown B (eds) The SAGE handbook of digital technology research. Sage, LondonGoogle Scholar
- Salazar J, Pink S, Irving A, eds SJ (2017) Anthropologies and futures: researching emerging and uncertain world. Bloomsbury Academic, LondonGoogle Scholar
- Zhang EY, Cheok AD (2016) Forming intimate human-robot relationships through a kissing machine. In: Proceedings of the fourth international conference on human agent interaction (HAI ’16), New York, NY, USAGoogle Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.