Keywords

Opening

In Chapter 5, the ability of our algorithms to grasp and compose everyday life in the train station and airport came under significant scrutiny. Problems in classifying objects and their action states, issuing alerts and demarcating relevant from irrelevant footage were major concerns for the project participants. This built on the problems experienced in Chapter 4 with the deleting machine that seemed to always leave behind orphan frames. Taken together, this suggests that our algorithms might struggle to become the everyday of the airport and train station at least in their current form. The system architecture, the individual components, the relevancy detection algorithms, the IF-THEN rules might all need more work. And for the computer scientists from University 1 and 2, this was no more or less than they expected: their work in this project built on a decade of research carried out by themselves and colleagues that would extend beyond the fixed time frame of this project into future efforts. Our algorithms might live on in modified form in whatever the computer scientists or other colleagues chose to do next.

The project coordinators faced a different question. For the coordinators of the project—a European-based consulting firm—the possibilities of developing an ethical, algorithmic surveillance system to take to the market, had provided a compelling reason for their involvement in the project. Deletion , relevancy detection and algorithmic experimentation each had a partial orientation for the coordinators towards a future market. Building a value for the technology following trouble with relevancy detection, object classification, object tracking, background subtraction, the issuing of alerts and the deletion system appeared challenging. The coordinators instead looked to switch the basis on which the future of the technology was settled. Recognising that the system’s results in demonstrations to end users (see Chapter 5) and the deletion system’s audit log (see Chapter 4) would generate a continuing output of demonstrative partial failure, the coordinators instead sought to build an alternative basis for relations with the world beyond the technology. This set of relations would seek to map out a new market value for the technology. In place of technical efficacy as a basis for selling the system, willing customers were constituted as a means to attract others to (potentially) invest in the system. In this chapter, I will suggest that building a world of (potential) customers to attract investors required a broad number of participants, with market trends, sizes and values separated out and made subject to calculation . To do market work and build an investment value required this careful plaiting of relations. I will suggest that the efforts required to shift the focus from technical efficacy to investment can be considered through ideas of performativity .

The chapter will begin with a brief digression through recent writing on performativity , before looking at the coordinators’ work to draw investors into new relations with the algorithmic system. I will suggest that these relations operated in a similar manner to the object classification of our algorithms: investors, territories, future sales and market size had to be separated out and qualified, calculated and pacified in order that these new relations of investment might be developed. The chapter will end with a discussion of where we have reached in the everyday life of our algorithms.

Performativity

Performativity has played an important part in the recent science and technology studies (STS ) turn towards markets and marketing (see, for example, MacKenzie et al. 2007; MacKenzie 2008). The argument draws on the work of Austin (1962) and his notion of a performative utterance or speech act. Cochoy (1998) suggests a performative utterance can be understood as a statement ‘that says and does what it says simultaneously’ (p. 218). MacKenzie suggests a distinction can be made between utterances that do something and those that report on an already existing state of affairs (2008: 16). The most frequently quoted example, drawing on the work of Austin (1962), is the utterance ‘I declare this meeting open’. Such an utterance is said to describe and bring into being the state that it describes—it is a speech act.

Developing this further, Cochoy (1998) suggests: ‘a performative science is a science that simultaneously describes and constructs its subject matter. In this respect, the ‘performation’ of the economy by marketing directly refers to the double aspect of marketing action: conceptualising and enacting the economy at the same time’ (p. 218). From this, we could understand that marketing brings the matter it describes into being. For other STS scholars, the focus is attuned to markets rather than marketing. For example, Callon suggests: ‘economics in the broadest sense of the term performs, shapes and formats the economy’ (1998: 2). Araujo thus suggests that performativity involves market and marketing type statements making themselves true by bringing into being the subject of the statement (2007: 218).

In relation to financial markets , MacKenzie looks at the ways in which the work of economists brings markets into being. MacKenzie (2003) suggests that traders use market models to inform their trades, creating market outcomes that match the models. Furthermore, economists’ market equations embody a world of relations, prices and outcomes that the use of an equation effectively constitutes. The work of economists can be understood in a similar manner to a Kuhnian problem–solution exemplar; the complexity of the world can be rendered more or less coherent through models and equations which appear to work (i.e. to bring a solution to a problem) and can thus be employed again in other similar situations. The models and equations become paradigmatic couplings of problems and solutions for others to use. As a result, the risks faced by market actors in an otherwise complex, messy and uncertain world become reconceptualised as more or less manageable.

However, MacKenzie suggests that performativity is not a uniform phenomenon; instead he presents three approaches to performativity . First, there is ‘generic’ performativity in which: ‘an aspect of economics (a theory, model, concept, procedure, data set, etc.) is used by participants in economic processes, regulations, etc’ (2008: 17). Second, there is ‘effective’ performativity which involves: ‘the practical use of an aspect of economics that has an effect on economic processes’ (2008: 17). Third, drawing on the work of Barry Barnes, there is ‘Barnes-ian’ performativity in which: ‘Practical use of an aspect of economics makes economic processes more like their depiction by economists’ (2008: 17), and actions change in order to ‘better correspond to the model’ proposed by economists (2008: 19). We can see these approaches to performativity as moving from weakly formulated to more thorough forms of performativity . However, MacKenzie is clear that such models of performativity do not only operate in one direction. MacKenzie also introduces ‘counter performativity’ whereby: ‘practical use of an aspect of economics makes economic processes less like their depiction by economists’ (2008: 17).

Although this provides a provocative set of ideas for thinking through how market value for the algorithmic system might be built, performativity has been critiqued for buying too readily into, or merely confirming, the terms of market participants (Riles 2010; Dorn 2012; Bryan et al. 2012; Foucarde 2007; with a response from MacKenzie and Pardo-Guerra 2013). For Lee and LiPuma: ‘The analytical problem is how to extend what has been a speech act-based notion of performativity to other discursively mediated practices, including ritual, economic practices, and even reading’ (2002: 193). To switch attention to economic processes requires an expansion of the remit of performativity and a rethinking of the centrality of communication (such as Austin’s utterances) and an incorporation of acting and doing. Incorporating this broader set of entities would move us towards an approach developed by Barad (2003) who suggests shifting performativity away from its starting point in studies of language use and questions of representation, towards action (a similar extension is proposed by Butler 1997, 2010).

Although Barad is not focused on markets and forms of economic exchange in her discussion of performativity, the questions she raises appear to resonate with concerns posed to the STS move to engage with markets , calculation and measurement; that performativity might problematically narrow the focus for analytical action. Callon’s (2006, 2010) response to the critiques of performativity is that they continue (what he suggests is) Austin’s (1962) mistake of assuming statements are in some way separable from their social, cultural or political context. Instead, Callon argues for a need to explore the worlds performed into market action. This will be the starting point for our exploration of the project coordinators’ market work: just how do they perform a world of investment into being and what does this tell us of the everyday life of our algorithms?

Building a Market Value for the Algorithms

In the absence of reliable evidence of technical efficacy and given the apparent difficulties of putting on a convincing demonstration of the algorithms’ ability to grasp or compose everyday life, the coordinators drew together a variety of entities to participate in the building of a putative world into which investors could be invited. Building such a world was a complex task requiring calculative dexterity in order to render the emerging world convincing and legible in a document that could be sent to investors. It also required imagination to conjure the entities to be calculated and a compelling narrative into which they could be woven. Still this would be nothing more than a putative world of potential investment . For it to be given performative effect required buy-in from the investors.

First, complex, dextrous and imaginative preparation work took place. The project coordinators segmented the world into geographical regions to be accorded more value (Central and South America with strong predicted growth rates in video-based surveillance ), even more value (Canada and Europe with a growing interest in video-based surveillance and a burgeoning privacy-interested legislature and lobby) or less value (the USA with apparently less interest in privacy and a saturated market place for smart video analytics). These segmented geographies were not left as vaguely valued territories, but transformed into specific and precise calculations of Compound Annual Growth Rates (CAGRs) derived from a combination of expensive industry reports the coordinators had purchased and online sources. In this way, the market for video-based surveillance analysis was calculated to have a CAGR of 15.6% between 2010 and 2016. This was then broken down into the more and less attractive geographical segments previously described.

This provided a very hesitant initial set of calculations on which to build an investment proposition: geographies were segmented and calculated. However, this dextrous and imaginative work to separate and calculate did not end here. Customers were treated in much the same way. Hence governments were identified as a particular type of customer, tied to more or less attractive geographies. The more attractive governments were calculated as accounting for 17.59% of the video surveillance market and as more likely to be compelled into buying a deletion technology in order to promote their own privacy sensitive credentials. Transport firms were another customer type segmented and calculated as accounting for a further 11% of the video surveillance market with a predicted CAGR of 13.39% between 2010 and 2016. Major transport-based terror attacks were invoked as a basis for this growth in investment , but transport organisations were also identified as another potentially privacy-concerned customer (this despite the transport companies involved in this project seeming to lose interest in privacy as the project developed). Specific technological developments were also given the same treatment, with pixel numbers, high definition cameras and algorithmic forms of data analysis all separated and calculated as growth areas. Finally, video-based surveillance processes such as data storage were also separated out and calculated as a growth area, but with a growing storage cost—the kind of cost that could be reduced through deletion . Although this separation and calculation work was directed towards building a putative world into which investors might become enwrapped, the coordinators also worked to distinguish entities as outside or external to this world of potential relations. Hence 44 competitors were also identified, ranked according to size and spend, and their particular video-based, algorithmic data analysis systems were presented in terms of their inferior capabilities. This despite our algorithms continually running into problems.

The work here by the coordinators was similar to that carried out by our algorithms. Separating out, calculating, preparing and qualifying some entities while disqualifying others (such as competitors), grasping features of the world out there and bringing them to the system, provided the basis for building a potential world of investment relations. Alongside segmented geographies, everything from governments to pixel numbers became entities of this putative market work. The entities segmented and qualified (and disqualified) were drawn together into the world of relations in a document entitled ‘The Exploitation Report’. Here the qualified (and disqualified) entities made sense as providing a basis for investment . At the centre of this world of relations, however, sat our algorithms, the system architecture, its components, and the deleting machine as an investment vehicle whose technical efficacy remained absent from accounts. Technical capabilities remained silent, rendering the Report’s content accountably certain and ordered. The preparatory calculations embedded in the Report and the censure of any uncertainty in terms of the demonstrable proof of technical efficacy would now provide the basis for performatively accomplishing an effect: building a world of investors. Through convincing investors that the Report was compelling proof of the viability of investment and that the technological system qualified as a reasonable investment risk, the coordinators hoped to also build investors into the world of the algorithms.

Inclusions, exclusions and careful calculation provided the means for the coordinators to try and build a compelling narrative that would achieve this performative effect. Rather than relying on a single utterance (as in Austin’s illustrative examples of performativity), accomplishing this effect relied on the Report’s extended narrative as a means to provide a particular kind of evidence (not of technical efficacy, but of investment potential) on a particular scale (across industries and geographies). In place of uncertainty derived from 44 competitors came the assertion that none of the competitors could deliver as sophisticated a solution as that promised by the project. In place of a concern with governments cutting budgets in times of austerity came the assertion that governments must look to cut costs and therefore should look for the kind of cheap storage solutions that auto-deletion technologies could provide. In place of a concern that a new surveillance system might attract privacy-based criticism came the assertion that this system carried with it and provided a response to that privacy criticism. And in place of any concern from among project members that the technology didn’t work came nothing; technological inadequacies were excluded from the Report and its audience. Building this compelling narrative (Simakova and Neyland 2008) was central to accomplishing the performative effect.

From the preceding analysis, we can see that our algorithms are not left to fend for themselves, abandoned as a result of their technical inefficacies. Neither are they exactly excused from any further role in the project. They are in the Exploitation Report, but their lack of efficacy is excluded. To accomplish the performative effect, they need to be present as an investable entity, at the same moment as key features of their activity are absent. The orderly world of the investment proposition is as much dependent on these absences as the presence of the algorithms. Understanding performativity is not then restricted to single speech acts or even the content of the Exploitation Report alone, but requires understanding the concerted efforts to segment, calculate, and prepare a world of people, things, processes, resources and relationships that the investors can enter. Preparing the putative world for investors involved these presences and absences, but also the possibility of accumulating something further. This built on the segmentation, calculation and preparation work to narrate future returns on investments from building an ethical, algorithmic surveillance system. The system could be invested in and might go on to do the work that might be required of companies in the emerging and changing Data Protection and privacy landscape where such matters as a right to be forgotten (see Chapter 4) have gained momentum. Complying with policy requirements and customer expectations of privacy , and delegating this compliance to our algorithms (or at least, future renditions of our algorithms), might become a marketable good and attain a value .

Following many weeks of labour by the project coordinators in producing ‘The Exploitation Report’, the preparation work of segmentation, calculation and absenting of certain forms of data (on technical efficacy) was hidden. Making sense of the performativity through which an investment proposition is given effect requires an understanding of this preparatory work, but also cannot ignore the compelling narrative in which it is subsequently involved. Market value here achieves its potential through the segmentation of geographies, technologies, competitors and customers, the apportioning of a calculative value (or non-value ) to these entities and evidence from third parties to support the values evidenced. This work is only partly evident in the Report. The outcomes rather than the means of calculation , for example, are made prominent. However, the Report itself also needs consideration. The preparation work to segment, calculate and value entities had to be drawn into a compelling narrative that supported the future development of the algorithmic system. Work was thus done to connect things we all know are happening now (such as government austerity measures and the need to cut budgets) with features of the technological future (such as deletion ), to generate a compelling narrative for investment in the algorithmic technologies (in this instance, that austerity measures and cost-cutting could be achieved through deletion by cutting data storage costs). And other things that we know are taking place (such as the introduction of the EU General Data Protection Regulation) could be connected with a range of required activities (compliance with the legislation) that could be accomplished via our algorithms. Certainty in the narration of problems (that these problems exist and will be faced by these customers) and solutions (that this system will address these problems) might prove compelling. At the same time, producing a compelling narrative also required that some numbers (technical efficacy) and forms of calculation (how the world of the Report was prepared) remained absent. This continual switching between temporalities—the world as we know it now and the investable future—and accounts—things to be made available and things to be absented—became the means to attempt to compel investors to join the world of relations being built into the algorithmic system; that its market value would arrive.

The Everyday Life of the Algorithm

Where does this leave our algorithms? As the slightly embarrassing and incapable project partner to be excluded from financial calculations , a waste of time and money? And what does this tell us about the drama played out in current academic writing and in the media (see Introduction), in which algorithms are expected to take over our lives, run wild with our data or operate in ways that we cannot see? To address these questions, we need to step back and take a look at the everyday lives of our algorithms as they have developed throughout the chapters of this book. We need to see just where our algorithms have got to in life to make sense of their proposed future, their social, economic and technical prospects.

In the Introduction, we met the abandoned luggage algorithm and its IF-THEN rules. Little more than a set of step-by-step instructions that set out some conditions and consequences, these rules seemed far removed from the drama of artificial intelligence, big data and the opaque and inscrutable algorithm. Indeed scrutinising these IF-THEN rules appeared to offer little prospect of a great step forward. They were not about to leap off the page and create great change in the world. In order to understand this algorithm and the drama in which it was expected to participate, we needed to get close to its everyday activity. We needed to know just how this algorithm participated in everyday life, grasped or even composed everyday life and participated in the production of effects . We needed to know something about its prospects of becoming the everyday . It seemed clear that the IF-THEN rules alone would have little consequence. We needed to know who and what the algorithms were working with. Rather than treat the non-human as an incidental figure (as much of the sociological writing on the everyday has tended to), the algorithm would be accorded a specific kind of status. As a first move, we needed to de-centre the human as we know it from the middle of the drama. We could not afford to assume that this was primarily a story to be told by people. We needed to give the algorithm and its technical partners, at least in principle, the same potential agential status as the humans and then we needed to make sense of how they each participated in the composition of effects . We then needed to enter into the varied and only partially integrated everyday lives of the algorithm .

In Chapter 2, the human-shaped object and luggage-shaped object (among other objects) provided a focal point for our engagement with the algorithms’ everyday lives. Computer scientists in the project sought an elegant solution that was concise (using only the minimum amount of processing power required) and could solve the problems posed by the project to the satisfaction of various audiences. Here we could get a first glimpse of how the algorithms might engage in the everyday. Was this grasping everyday life (as if its major constituents were there prior to the work of algorithms, just waiting to be collected and displayed) or composing everyday life (a more fundamental working up from scratch of the objects to be made)? As a surprise to me, an ethnographer with an inclination towards composition , it turned out to be both. The algorithms were in the business of composing the everyday, with models built from scratch of the parameters of what it meant to be human-shaped, classifying small segments of streams of digital video data into putative humans, and then offering those forward as a means to classify the action states of those objects. Even articulating the everyday on these terms seems like a new form of composition , at least in contrast to how we might go about our everyday lives. But these algorithms also needed to grasp the everyday . They were not free to compose without limits as if there was no a priori world from which these objects could be mustered. The life of the airport, the people and objects in it had to be given a life within the algorithmic system that could be traced back through the airport and train station. Actual distances in centimetres, speeds at which people walk, distances covered, the angle and zoom of cameras among many other features of the everyday had to be accorded a form that enabled them to be grasped. And they had to be grasped in such a way by the algorithms that the journey could be made back in the other direction, from algorithm back to train station and airport. These were the demands that an elegant solution had to meet.

So our algorithms were beginning to be competent in grasping and composing everyday life. But their own lives were not without constraints. They were not just in the business of producing results, but demonstratively proving that they had produced the right kind of results. These were outputs that accountably and demonstrably accomplished the project’s three ethical aims, to see less, to store less and to do so without creating any new algorithms. Elegance alone was insufficient. To an ethnographer assessing their ethics , to an ethics board and later in ethical demonstrations , our algorithms had to continually and accountably prove their capabilities. The abandoned luggage, moving the wrong way and movement into a forbidden area algorithms had to work with other system components in Chapter 3, the User Interface, the Route Reconstruction system, probabilistic trees, algorithmic children , parameterisation, classification of objects and action states, to collectively demonstrate that everyday life could be improved by the emerging system. This was composition of everyday life, then, but one that was also morally improved. The world was not just grasped, but ethically enhanced. The accountable order that the algorithms could participate in, while in their experimental activities, had to intersect with a more formal sense of accountability . An opportunity had to be developed for future data subjects of algorithmic decision-making and their representatives to question the system. The algorithms also had to engage with the ethics board to begin to give effect to the ethically enhanced world. Unfortunately for our algorithms , these effects and the confidence with which they were demonstrated, began to dissipate as the system moved beyond experimentation .

In Chapter 4, it became clear that the system’s ethical aims might have a value beyond experimentation , in accomplishing compliance with new regulatory demands to delete data. Deletion might provide a means to accomplish a market value for our algorithms . Yet it was here that problems began to emerge. As preparations were made to use the algorithms to distinguish between relevant and irrelevant data and provide demonstrative proof that irrelevant data could be effectively and accountably deleted, project members started to disagree. Just what should constitute adequate deletion? Changing the route by which a user connects to data, overwriting, corrupting or expunging data from the system? As the project coordinators sought the most thorough means of deletion possible, as a prior step to developing a market for the system, the computer scientists struggled to match their demands. A system log was developed to produce accountable reports for humans of the algorithms’ ability to delete. But the system did no more than continually report the failures of the system: data was not deleted in its entirety, orphan frames were left behind, and the demarcation of relevant from irrelevant data came under scrutiny. The production of nothing (the deleted), required the production of something (an account of deletion), but the failure to successfully accomplish nothing (with deletion undermined by the stubborn presence of orphan frames) created a troubling something—a continually disruptive presence that questioned our algorithms’ abilities to produce nothing. Much of everyday life—somewhere between 95 and 99%—it turns out is irrelevant and can be deleted. By failing to grasp all this irrelevance and instead leaving a trail of data and reports that attested to this failure, the prospects of our algorithms becoming the everyday of the airport and train station were diminished.

This was the start of some escalating troubles for the algorithms . As they continued their journey from experimentation , they had to enter into the ever greater wilds of everyday life. From experimentation in settings with matching flooring and lighting, project participants acting out the roles of normal humans, and cameras supplying data from the right angle and height and distance, at the right frame rate for our algorithms to see, our algorithms now had to grasp real space, in real time. Here people, things and events unfolded in a naturally occurring away, across different floorings and lighting conditions, at different frame rates, with humans who now acted in oddly normal ways. Children went this way and that way, adults stood still for too long, luggage did not behave as it ought and humans wore the wrong kinds of outfits that looked just like the airport floor. Grasping and composing this everyday was too challenging. Under test conditions, in place of 6 items of potentially abandoned luggage came 2654 items. The relevant and irrelevant intermingled in a disastrous display of technical inefficacy. What had seemed like reasonable demonstrations of the algorithms’ capabilities to ethical audiences, now had to be questioned. Questions of the material integrity of these demonstrations (and the extent to which a relation of undoubted correspondence could be maintained between the system put on show and the world to which it pointed) were only matched by questions of their visual integrity (of who and what was in a position to see who and what). These questions continued and even grew for a time as our algorithms moved towards their final demonstrations to research funders. The king of Event Detection—abandoned luggage—could only be demonstrated through a careful whittling away of confounding variables. The flooring, lighting, luggage-type, positioning, behaviour of the luggage’s human owner, frame rate of the camera and other human-shaped objects of the airport each had to be closely controlled. In place of the algorithm going out into the world grasping or composing real time, real space everyday life, a more modest and controlled everyday had to be brought to the system.

And so we find in our final chapter that the algorithms are somewhat quiet. Away from the drama of contemporary academic writing and popular media stories, the algorithms take up a meek position in an Exploitation Report. In place of any fanfare regarding their technical efficacy, comes a carefully composed account, depending on imaginative and dextrous calculative work. Here, more and less valued geographical regions, customer types and inferior competitors stand in as proxies for our algorithms . The calculations , instead of talking about current technical efficacies, point towards a future potential of market value that could be achieved with investment . The performative accomplishment of the investment proposition negates the need for our algorithms’ everyday life to be put on display. At the end, they are not entirely absent from our story, but from the Exploitation Report their grasp and composition of everyday life, their prospects of becoming the everyday of the airport and train station, are deleted. Goodbye algorithm.