Keywords

1 Feeling

Take a moment to notice the chair you’re sitting in, or the floor you’re standing on. You can stop reading anytime you want. The text won’t care. Are you reading a paper book? If so, imagine you’re reading on a screen instead. Better yet, go read something on a screen and notice what your eyes feel like, your shoulders, your mouth, your sense of self. Then come back.

Now imagine you’re in a large lecture hall, sitting on a comfortable dark orange seat. There are a few dozen people in here. A few friendly colleagues sit next to each other. A middle-aged woman with bright magenta hair, a patterned dress, and magenta tights walks up to the lectern. Most people look up and maintain eye contact. A few look down at their laptops or phones from time to time, even for prolonged periods of time, but this is perfectly normal, polite behaviour at an academic conference. Now the words you are experiencing are not just text. This woman is performing, and that performance depends just as much on you as it does on her.

How would your eyes feel now – maybe more comfortable from being able to look around? Your shoulders, maybe looser from being able to sit back? Your mouth, maybe tighter from having dutifully smiled at a bad joke because everyone else was chuckling? Your sense of self, maybe a bit of a knot in your stomach from the thought of talking to all these people you don’t know at the coffee break? Maybe your legs have been crossed for too long but you don’t want to risk bumping into the person next to you.

2 Background

I work in human-computer interaction (HCI) and experience design, the first of which (arguably) traces its lineage along self-styled scientific, functional lines (Norman 2002). Early work in HCI focused primarily on doing work more efficiently and effectively. The key word there is ‘doing’. Topics of research tended to be task-based, results-oriented, cognitive, or physical interactions with technologies. An example is Fitts’ Law (Goktürk, ND), a simple equation that predicts how quickly and accurately a typical person will be able to move a cursor to a particular target. I do not argue that there is no feeling in doing this – especially if you can’t get the cursor to land where you want it to! – but the focus tended towards the accomplishing of tasks.

In the past couple of decades, though, research has expanded into a realm dominated by feeling. Research questions have started to revolve around internal perceptions and affect produced by interactions with technologies. Designs were created to prompt emotional and/or aesthetic responses. These had cognitive and physical components as well, but the emphasis was less on using the mind to tell the hand how to move the cursor to the target and more on the embodied nature of the human interacting with technologies. Some of the technologies themselves were also demanding to be used in different ways – squeezed, for example, or kept in peripheral vision (Hassenzahl et al. 2012).

I argue that feeling comes into play any time that digital technology mediates an interaction between people, or when people interacting with a technology can sense or imagine others witnessing their interactions. For several years I had to explain what I meant by this, but the recent global encounter with Covid-19 means that anyone with access to digital technologies and the ability to shelter at home will know it. That feeling of connection, seeing a friend’s face on the screen, and at the same time loss, not being physically with them. That odd shift when live face-to-face contact takes a detour via a screen, virtual reality, or any other technological mediation – that is the type of experience that my research explores.

3 Performance

This is where performance comes in, because what is performance if not a means of mediating interactions between people? Walk into a big, ornate theatre showing Hamlet, and you’ll either know exactly what to expect or you’ll feel your lack of knowledge keenly (so keenly that you probably won’t be there in the first place). Walk up a plain staircase to a deserted office space for a night of queer performance art and don’t be surprised if you have to scramble across the floor to avoid being bled on. Buy a ticket to an online performance and know that whatever you see, it will likely be bounded by the edges of your screen but may be available for later viewing. Do something different and watch the interactions shift before your eyes: for example, place two human performers in a huge ‘aquarium’ with a touchscreen surface and see how passers-by respond to the changes in lighting and sound they cause by touching the screen, barely half a metre from the performers’ eyes (Taylor et al. 2011).

In terms of performance theory, the ways of valuing and analysing different elements of a performance lend themselves to the more recent addition of phenomenology to the traditional semiotic interpretation of what the audience member is seeing and hearing (Fischer-Lichte 2008) – especially as performance is continuously broadening its reach into site-responsive, game-based, promenade, immersive, intermedial, virtual, ‘Zoom’ as a generic term… And each of those genres invites its own preferred or provocative theoretical lenses as well. However, the task of conveying these theories in the confines of a single article or chapter intended for an HCI audience, or even an experience design audience, is often too great to justify the reward.

Therefore, I tend to step back in time and refer to theorists that are accessible and meaningful to both performance studies and HCI / experience design. My meaning of ‘performance’ draws from the likes of J.L. Austin’s ‘speech acts’ (Austin 1962), John Dewey’s pragmatist approach to aesthetics (Dewey 2005/1934), and Erving Goffman’s delineation of shared ‘onstage’ spaces where the way we appear to each other matters and our private ‘backstages’ where they probably do not (Goffman 1959). In other words, you do not have to be staging Hamlet to engage in a performance. Anyone in that lecture hall could say or do something to alter the performance, from a problem with the slide deck to an inappropriate question from an audience member to a fire alarm shrieking.

4 Performativity

It would be unfair to try to expand the meaning of ‘performance’ without recognising the impact that the word ‘performativity’ has, especially within academia, and especially when a female, feminist writer is doing the defining. However, it would take far longer than the space allotted for this chapter simply to offer a full rationale for the various defensible positions. For the purposes of the concepts and examples presented here, I will nod gratefully to Judith Butler for her ideas around gender performativity and the insights that these ideas have made possible. Then I will set the term ‘performativity’ to one side, reserving it as much as possible to Butlerian (Butler 2002) connotations around acts of performance.

5 Performative Experience Design

Yet here it is: ‘performative’. Performative Experience Design (PED) ‘is the setting of technological and social parameters to create opportunities for performative experiences with interactive technologies’ (Spence 2016, p. 5). This definition allows for solo experiences from singing along to music in your bedroom to witnessing the excruciating durational works of Tehching Hsieh (www.tehchinghsieh.net), though performative experiences nearly always involve other people more directly. What the definition highlights, though, is that the technology is on a par with the design of the forces that can guide its use. It also highlights the agency of the user – performer, audience member, bystander, or other (Reeves et al. 2005) – over that of the designer. Finally, it celebrates the looseness of the term ‘performative’. PED does not try to exclude other types of design. Rather, it provides a set of theoretical lenses and methodologies for creating and/or analysing virtually any interactive experience.

While background in performance and their analyses can help, and a novice can benefit from works such as the succinct one already mentioned (Fischer-Lichte 2008), in the end I think that what stands out to your designer’s eye – or design researcher’s eye – is most important. Keep in mind your line of enquiry or purpose in doing this work in the first place, and engage all your senses and imagination into exploring every technological and social parameter that the artists would have set, accepted, or left to chance in order to achieve their effects.

6 Example

6.1 VRtefacts – performance analyses

From 2017–2020 I worked on a project that aimed to make visits to museums more personally meaningful by using digital technologies to incorporate gifting in various ways, making visitors feel that they had more of an emotional stake in museums and their collections. No mention was made of performance in the project plans or agreements, and I was not a specialist in either gifting or cultural heritage studies. However, museums lend themselves well to performance in its broader sense. They have a space with a set of unique, well-established social norms. They have objects and spaces to direct your attention and action. Finally, of course, they are public, whether that is a throng of people trying to catch a glimpse of a famous painting, a shuffling group of elderly friends on their way to the café, the imagined impatient sigh of your companion as you linger just a moment longer in your favourite room, or the near certainty of a security camera catching you in its dispassionate stare.

Much of the early emphasis in this project was in capturing and sharing images of museum pieces that a visitor felt would appeal to a particular friend or family member. The tactic of having visitors decide on a ‘gift’ receiver before starting their museum visit was proving to be very powerful, but the early implementations were smartphone apps. Easy to understand, easy to share, but literally and figuratively flat.

I set out to imagine the opposite of what had been done so far – and please let me emphasise that this was out of academic curiosity, not criticism, because those results had already outstripped what we had hoped for! In my imagination, the opposite would be private, tactile, and three-dimensional. So, using my own PED methodology, I followed my line of enquiry back to earlier works I had analysed. Many of the one-on-one performances of the late Adrian Howells (Heddon and Howells 2011), for example, used tactility to create a space for physical and emotional closeness with the strangers who made up his audiences, one person at a time. For example, he might wash your feet, offer you a strawberry to eat with calm concentration, or simply hold you silently in a (fully clothed) embrace. These are such unusual things to do with a stranger, though, that he could not simply take your ticket and instruct you to take off your shoes and pop a strawberry in your mouth. His performances were as much a series of quiet invitations and unspoken negotiations, bringing audience members to the point of accepting what he offered, as they were the offers themselves.

I also felt that conversational storytelling – simply chatting – would provide an adequately low barrier to entry for museum visitors with no subject matter expertise or performance training. Again I used my own previous analyses of works such as Mike Pearson’s Bubbling Tom (2000), a conversational guided tour of his hometown. As with Howells, Pearson established a more intimate rapport with his audience than in a typical theatrical production. Here, though, the relationships he had with specific audience members changed the dynamic of the performance completely. Those who did not know him personally ‘acted’ like ‘proper’ audience members – they followed him and either kept quiet or spoke to companions in hushed voices while walking so as not to interrupt the ‘performer’, Pearson. This was not the case for at least some of those who also lived in his hometown, especially his friends and family members. They had no compunction interrupting him to offer their own recollections or opinions on topics he raised, even when their stories conflicted with his own. This would be nearly unthinkable in a traditional performance, of course, but also in the private worlds that Howells created for his individual audience members.

6.2 VRtefacts – Design

At this point, the ‘I’ who wondered about alternative directions for personally meaningful museum experiences and conducted performance analyses gives way to the ‘we’ of the team that brought VRtefacts to life, most notably Dimitri Darzentas and Harriet Cameron. Together, we sought to design technological and social parameters that would encourage museum visitors to offer up their personal thoughts or reminiscences, no matter how tangential or untrained, regarding items from a city museum. The technology that we wanted to explore was the convergence of a virtual reality (VR) space, which blinds the user to the physical world around them, with three-dimensional versions of the items they saw. These items were 3D-printed white plastic in the real world, but in VR, the scans of the real exhibits were overlaid on the physical items so that it looked and felt to the user as if they were holding and moving the actual item in their hands. This description covers only the essentials of the project, which is covered in more detail in Spence et al. (2020) (Fig. 1).

Fig. 1.
figure 1

Host guiding visitor

Full size image

Technologically, the novelty and importance of the project lay in the bringing together of VR imagery and resized 3D-printed museum objects. This alone might have made for experiences different from the VR or museum norm in a number of ways, but we doubted that on its own it would reliably achieve the goal of a personally meaningful encounter with the object. The lynchpin of the design, therefore, was to give the researcher running the experience some duties of a performer. We called this researcher the Host. Luckily, our research team included two skilled performers. We wrote and used a script, adaptable to visitor questions and comments, that gently altered the expectations of the interaction from research data-gathering, through light back-and-forth conversational storytelling, to the surprise at touching a physical object in what they believed to be a purely virtual space. At this point visitors would nearly always ‘take the stage’ and tell a story, reminisce, or muse over past experiences the item brought to mind. The ability to manipulate, feel, and spend time with an object led to some touching recollections, fascinating connections, and even the identification of a previously unidentified piece of machinery (Fig. 2).

Fig. 2.
figure 2

Visitor feeling aero engine that prompts family story

Full size image

Of course, I can only speculate on what would have happened without the Host, whose role is detailed further in (Spence et al. 2021). Between the combined experience of myself, my colleagues, and the literature we consulted on storytelling (e.g., Wilson 2006) and conversational storytelling (e.g., Langellier and Petersen 2004), we feel strongly that the ‘performative’ elements of the design, handled by the Host, altered a visitor’s expectations of the event from something akin to ‘go into VR and get surprised by touching something that may or may not have any personal meaning for me’ to something more like ‘get taken into VR by someone I can trust, who primes me to think of my thoughts and feelings as valuable to the museum – then get surprised by a physical object that I attach personal meaning to’. Thankfully, design research is not a linear process of proving what is empirically right or wrong. Rather, we made small analyses and judgements during our non-linear, iterative design process (e.g., Wolf et al. 2006), each of which showed how to move one step closer to our goal of personally meaningful interactions with physicalised museum objects.

7 Experience

To return to theory:

‘Experience in the degree in which it is experience is heightened vitality. Instead of signifying being shut up within one’s own private feelings and sensations, it signifies active and alert commerce with the world; at its height it signifies complete interpenetration of self and the world of objects and events.’ (Dewey 2005/1934, p. 18).

This has always been arguably true of the performing arts even when thinking of a West End or Broadway play performed on a proscenium stage. In traditional stagings such as these, an important theoretical and practical concern is ‘mise-en-scène’, or the arrangement of the elements of the production on its stage and how the show is directed. The implication is that the audience member sits still and directs their audio and visual attention to a rectangular spot at the middle of the space (Zoom, anyone?), where events will be staged for them to perceive. This may, indeed, constitute an experience, even an extremely moving one.

A strong argument has been made for a shift in the past couple of decades, though, away from mise-en-scène and towards ‘mise-en-sensibilité’ (Lavender 2016). To put his argument briefly, more and more performances are being conceived with the audience physically and experientially in the middle of the action. The scene is not held within a rectangle over there but is all around you, possibly engaging your senses of smell, touch, even taste, and challenging you exercise your agency in new ways. His book offers a number of detailed analyses of fantastic new works, some of which I have experienced myself. I encourage everyone to think of performance in this way, as a mental challenge if nothing else. I also encourage those who are less familiar with live performance to seek out some non-traditional works that appeal to you or your professional curiosities. The performing arts are simply too rich a resource to ignore, especially when accompanied by a methodology for applying the insights they provide to design research processes.

8 Closing

Feeling through experience, then, is a bit like feeling your way through a dark room. We are surrounded by digital technologies to the point where sometimes we can’t even see them. We only notice them when we bump up against them. My work aims to make these points of contact into personally meaningful experiences. I would hazard that the same could be said of my brilliant colleagues at the Mixed Reality Lab as well, without whom my body of work would never have come to fruition. PED therefore owes much to them, and to anyone who finds it interesting enough to try. I wish all of you – designers, HCI researchers, UX professionals, anyone invested in this area – the joy of bringing your own corner of the performing arts into your work. The everyday world can use more foot washes and hometown reminiscing.