1 Introduction

Digitised collections and born digital items, such as photos or video, exist beyond the scale of human viewing. New methods are required to read, understand and work with the data, resulting in computation becoming increasingly central to both creation of a cultural reality and as the interpretative tool and practice. If artists look, then how might a machine see as a critical tool? Developing work on computational culture and the unstable digital object, my aim in this paper is to consider how the role of the medium affects computational critical practice. Situated in a critical Digital Humanities (Dobson 2018; Berry and Fagerjord 2016) perspective, this paper considers form and how this affects materiality. Heidegger’s (2012) consideration of causality to consider what computational criticism reveals about itself. Considering the computer as the first metamedium (Manovich 2013) allows us to question the fourth causality, that of the maker, and how this affects cognitive practice through interfaces.

This paper develops a previous consideration of the computational object (Emsley 2019) as a digital pharmakon, an object that is both poison and medicine. This is particularly useful when thinking of the Next Rembrandt (2016) and the superficiality of the image’s surface, hiding its constructed nature in plain sight. The project is a machine-generated image based on a reading of digitised Rembrandt portraits from museums and collections and is created by TU Delft, J. Walter Thompson (JWT) Amsterdam, Mauritshuis, ING and Microsoft. The project was intended to raise questions about the power of data for Microsoft’s customers and as a playful way of demonstrating ING’s support for Dutch cultural institutions, working in tandem with other Dutch institutions. Versions of the image are available digitally on the project’s website and it was printed using a 3-dimensional printer. The latter material form echoes painting but is manufactured from ink and a computational object.

Its surface might provide an appealing form, either aesthetically or epistemically, whilst simultaneously hiding issues and ideologies in its structures. Code can embed an argument deep within its structures and through its manner of assembly, one that requires software and code studies skills to begin to read and interpret. This assumes that the code or the training data are available to be read. The image output may require a supporting human reading, such as gender and class, and a computational one. Yet the surface is perhaps too perfect and requires readings of its substrate and a material understanding of the medium of the creators and their role. When studying an image using digital methods, we use a machine to see and remediate it from bits to models and grammars. These structures both create, and are created by, the interface effect (Galloway 2012) resulting in a grammatology (Derrida 2016a). I want to think about how the techne is created through a series of patterns that generate patterns, moving from an abstract concept to a concrete epistemic object.

By reconsidering the critical assemblage and using it to think with, we see the human and the machine as critical partners. Attending to the materialities of the reading through a playful approach that decentres potential meaning encourages us to glimpse beneath the surface. The reading is considered within a wider economy but to begin recognising those constraints and that theory might be embedded in tools that are deployed as critical practice. In reflective moment, we might consider our own positions as consumers but also as producers and to think about how we can be critical of critical tools and consider that they may have their own positions built into their assemblages. From this, I contend that the condition of the post-digital underpins the analytical process and reading computational culture that considers both human and machines in the process.

1.1 Situating the computational in culture

Situating potential readings as both computational and human reveals a potential issue. The tools for machine seeing are embedded in the same social and cultural economy as the object under study. These issues present a case for learning how to engage with and understand how the discourses merge and represent a cultural reality.

A consideration of the way the cultural object is created presents challenges in reading them. The consideration that the work of art is created within a series of codes (Greenblatt 1984) can be useful extended to consider the code itself as both containing and presenting code of culture. In learning how to read them, Anderson and Pold’s (2014) considerations of writing being dangerous are echoed as they may contain hidden purposes from their creation. Viewing the Next Rembrandt as a piece of art provokes not only questions about the artistic work but also the purposes in creating it. How might machines engage with Rembrandt’s own art to create a new image? Given that this image uses computational material, how might it be read and with what tools? Manovich’s conception that computers are the first metamedium, that the medium can be used to build other mediums and tools, raises a potential issue in the notion in the act of inscribing the data which I will attend to through Derrida’s grammatology. It provokes questions to what is being written and how is it being translated whilst making the object.

Having created this epistemic object, we need to reflect on what the new ways of seeing might bring to culture and computational culture more widely. While the object cannot be removed from the conditions that creates the object, we can take a critical view of these condition to understand its effects on culture (Liu 2012). Acknowledging the role of the computation goes to the making the role of criticism less incomplete and to begin reconsidering the way that computation creates a series of objects as a grammatology. Having identified this gap, a change of tack is required to begin a closer reading of the digital object.

2 Creating a digital object

The Next Rembrandt and critical approaches to it are both pharmakon and designed objects. What I want to consider is this portrait as digital object derived from a distant reading of the portraits and, in particular, the agential workflows that enable it to exist. The painting’s very name suggests a new Rembrandt, created 350 years after his death. It may be derived from Rembrandt’s work but clearly cannot be by him or his school. Its real creators are not identifiable from the name but requires some reading of the secondary material. The name’s provocation suggests that viewers may be wary of the object’s superficiality.

The image’s existence as a digital object suggests a functional transformation from images collected from different museums to becoming a digitised collection in storage. This representation is enabled through the function of digitisation and the workflows that it enables, suggesting that this data is the final form that emanates from a series of software processes and workflows. Each of these workflows suggest organisational and technological influences and patterns to make the work possible.

This digitisation of a collection of physical artworks into a digital object suggests a notion of art or culture as a potential computational form. A painting becomes a mathematical matrix where the colours and their positions create an alternate materiality. This transposition leads to a loss of information and a forgetfulness (Ellul 1965) as they convert the painting into a numeric form and language. The parts that cannot be digitised are discarded. The images require digitisation so that the final portrait could be created. It can only read the data in a way that the thing to be read supports. A critically meaningful response may come from an intentional, playful challenge but this still requires an understanding of the material (Berry 2014). To create a reading, one must develop a tool to create the machine representation and one that is written in the medium being read, so requiring us to consider how and what is being written and read with a machine.

The final image was created using facial recreation algorithms to determine the patterns that bring the image together. The data was derived from the digital reading using facial recognition algorithms. Using a model, the cultural images are converted into a digital format as paint becomes a pixel, representing a colour using a standard and a position. The mapping of a human epistemic object to a technical epistemic object requires a translation. Not only is one conceptual pattern being translated from a human concern, but the form is also translated into a technical one; one that can see within the generated numeric symbols. A human reading of the eyes in a portrait may be computationally read as a measurement of the varying colours in the pixels of an eye, its circumference and the distance between the two eye features. Within the context of the training data given to it, the reading may understand that the eyes are typically placed either side of the nose feature and has a mathematical relationship to other salient features. The salient features might be derived from the same training set or assumptions made. The computational process views the picture through technical relationships that are reliant on the underlying digital object. The Next Rembrandt team used 6000 points to classify the features (Dutch Digital Design 2018) to create a typical Rembrandt face from the data, having analysed specific features such as ears, nose and mouth. In part, this moves from the concrete form of the face to an abstract mathematical model of one. Converted into a model, the idea of face can be read at scale.

The abstraction allows the machine to create the pattern of the face. At this moment, the use of these features is a form of inference. The facial creation draws on the patterns identified facial recognition to guess at how the face might look before the proportions, also machine calculated, were applied. This concrete pattern for the machine is used to create an abstract concept of a face that is represented as a picture. In the act of the computational becoming a maker, the created symbols become signifiers and are interpreted and re-presented. The face exists as a numeric model in a space made by the pixels but it only becomes viewable once the pixels are converted from numbers to colours in a position. The viewable image is represented through a file which can be printed out digitally or as a 3-dimensional printed image. Rather than leave the image in 2-dimensions, a height model was created to simulate the paint layers for the printed object. The act of printing raises questions about the remediation of the digital image into a painted one and the embedded interpretations that might be interpreted by viewers. This version of the digital object is not one that enables a potential feedback loop into the underlying data but imposes itself. The paint simulation echoes the original medium but cannot be of it.

Our ability to read the pattern is affected by the use of a human set of experts who helped determine the final image and the group who guided the final form. A human team determined that the image would be a male within demographics. Whilst the machine is able to determine light and shape as patterns and associate these with features, even testing them by writing unexpected wrinkles into the eyes, it is reading the computational object. While it has its uses, the pattern has limitations and are challenging to learn to read. One way of reading patterns is that they identify ways of writing software or interaction design as a learned way of working with the machine. As machines learn from an abstract pattern, it then perhaps considers its own pattern to create and write. These patterns then create a new sense of interaction but one at a cognitive practice. By recognising the difference between the makers behind the provided responses, we might begin to see that the sense of self in this work is made by both humans and machines. Unpicking this process provides a space for reflection and interpretation to engage with the object.

2.1 Reading with the machine

Instead of being fixed in the manner of dried paint or a printed text, data points and models can be manipulated and queried. This form of data enables the grammatology of the digital model through its existence, but it also erases itself and its construction in the representation. We might begin to reveal this form through using software to read the image in and then translate it between formats, to query it or to write in errors and glitch it. These processes use patterns of concepts to approach the forgotten materiality though, in some cases, the operation may be quite easy. An image might have a simple file conversion from a JPG to a PNG format than can be achieved either through a script or a desktop programme, such as Preview on an Apple machine. These conversions abstract the protocol and material transformation models from the user to change the form. The very ability to be able to alter the underlying structure with little apparent changes to the final form suggests that there is a remediation of these forms to show the image. Although these changes hint at the existence of an object, it still does not allow it to be read. It suggests a superficial model that is already hiding the software processes that brought it into existence. The image is partially revealed as a shell. Before turning to its creation, I briefly consider critical approaches through glitch studies.

Glitching (Menkman 2011) can be seen as a critical material approach to deconstructing form. If the glitch is deliberate, rather than exploiting an unforeseen situation, then it requires an understanding of the potential for the symbol of the interface and might be seen as a pattern. In its playfulness, the application of the glitch reveals the remediating model though it contains its own understandings of the symbols and what they might signify. It is a critical reading of the computational surface that deconstructs the model and its remediations.

The Rembrandt image relies on a close reading of 346 paintings to create the units of distance as a ‘form of knowledge’ (Moretti 2007). This scale of reading may be beyond the human ability to read these in a scholarly fashion. Where this project created one final image as their output, Manovich (2012) cultural analytics project on Manga scanlation pages points to different ways of reading. Using scans of Manga pages, tagged by a community, the project used digital image processing to create a computational reading. Using the numeric forms of the colours to extract features based on them, they were able to take a quantitative approach to the pages. The resulting data is shown on a 2-dimensional graph, such as a scatterplot or image plot, to show the new form of relationship. Where Next Rembrandt focuses on human features to extract through computational patterns, the Manga project uses the computational data to explore relationships between pages using the numerical representation of colour and how the machine creates it in the file format.

Using visualisation as a descriptive system (Manovich 2012), the project uses the medium as metamedium to communicate what the critical tool is discovering. Computation is being used as a tool to translate a question into a pattern search and then to build its own response from the same medium. Although visualisation is not limited to machines, these of it here makes the computational reading sensible to humans. The type of visualisation affects this through its layout model and how the data points are constructed. Where Next Rembrandt falls more into an art historical recreation of cultural objects, the Manga project presents an analysis of the project’s underlying data. Yet without knowledge of the underlying data and processes, it is hard to read the scatterplot and appreciate the arguments involved in it. One has to learn the language of graphs and visualisation to begin questioning and reading them.

Echoing the relationship between recognition and creation, the question about the maker—machine or human—is raised. The computational tools work with existing media, such as painting and print, once they are digitised. Menkman’s glitching is another expression of this where the computational materiality enables tools to work with it and to alter it, but it is still the artist that creates the glitch. Within the other two projects, the computational tool is more apparent as a joint maker with humans. The assemblage becomes the site of theorising, either through testing the materiality or using it to read data. Using the patterns within the computational object in different ways to an intended purpose, the tools write an epistemic object that it is created as a visualisation to make it sensible for a human. The machine enables the reading to take place through the application of a mathematical model as the data is not really human readable and it can provide a reading. This must then be interpreted to consider what and how the response is created. It points towards a material issue with computational techniques where the reading is constructed from the same medium as the subject under test. It also demands a computer literacy to either code or to understand the response. As the computational object is approached, it hides itself behind the projected form as an entity derived from the form that is requested either through the purposes, such as a projection of economic or technical power through culture, or the critical question posed about it.

2.2 Patterns as forms

Before considering the issue of reading and writing, a consideration of patterns as forms that both mould and are moulded from the materiality of the digital is useful. I want to suggest here that the pattern is a way of thinking as well as affecting the material, demanding a consideration of the materiality of the pharmakon to begin to read it. The word pattern needs a consideration here, as both a thing that is found as well as something that is desired to be found. I think that it is useful to consider these in the light of the mix of functionality and representation that Anderson and Pold (2014) argue for but to consider it in the light of using machine driven tools. Patterns can be seen as a generative act (Derrida 2016a) or, in the sense of dark patterns, to create a misreading (Dieter 2015). Using a pattern, such as facial recognition, a machine learning algorithm uses a model against the data structures. In many ways, the machine can read the patterns more closely than a human to a generate a new model, or a sign. Knowing of these models and how they are constructed sets up a series of links that tie in the sets of models. These links might be considered as ways of generating meaning and as a way to divert attention from questions of purpose. By trying to identify the patterns used, we can begin to approach the concept, the form, of the thing that is to be made. As the abstract concept of the form, such as a theory or pattern, combined with materiality towards the presented object. A consequence of the transposition through models is the creation of a digital double of the original object.

In digitisation, the location of memory and understanding is moved from the human to the computational as the reading is only made possible through the translation of the cultural into a technical form. Rembrandt’s portraits had to be scanned and digitised for the algorithms to complete their viewings, yet the digital is not aware of the portrait’s life before the scanning process. It is only when paint is transposed into pixels that their computational use is made possible and that a new critical reading is required to understand the technical and social means of production of the final form. As the material is fundamentally changed, the digital must make a copy of the image rather than translate it. A successful reading of the doubling understands this materiality and the languages and logics that are required to interact with it (Stiegler 2019). These logics are not only computational to understand the new material form but also cultural to understand the remediated object.

When a new version of the reading is begun, existing results and data may be irrecoverably lost. It becomes a “redoubling” (2019), where Stiegler points to the forgetfulness or lack of history in the reading. Unless a pattern is used to act as a form of storage, either of the results or the provided parametric model, the reading loses its own history and context. This redoubling continues the move to the technical through allowing it to create its own present and historical realities. The tradition of comparing readings of the same object or artist requires access to these readings, not only in the final paper form but to also understand the forms used in the reading. Although this points to the open knowledge and data discourses, this paper’s concern is the way that the post-digital reading creates a grammatology within the digital object. Each reading becomes a micro epochal moment, an interruption that becomes a new moment in time, through the presentation of itself but forces a recognition of this use of time as model. Such forgetful re-presentations may alter the perceived grammatology and patterns used in the reading.

Critical computational methods are created to understand this computational double. Having explored the surface understanding of the translations between human and machine form to create the conditions for a reading, I want to turn inwards to the computational to consider the next challenge. The form that is being considered is remediated through the interface, a meeting point of two components, where the individual parts translate the model or data given to them. Processed in order, the concept is continually remediated through a series of models, either as data structures or implemented algorithms, to create a pattern matching machine. These patterns can either read a pattern as it is or statistically infer it.

As such, we might read the Next Rembrandt as a series of statistical inferences based on existing historical data that reveals the hidden computational judgements. The inferences are judgements based on existing knowledge and readings. Unlike a human critic who may raise some of these in the final output or leave enough traces for the reader to infer the judgement, the machine uses a logic to make an inference based on a set of models. We may raise a different consideration raises a different question: who is the maker, the entity that causes this to come into being? In Heidegger’s example, the chalice has one maker, the silversmith, who crafts the metal into the chalice (2012). The digital object may have at least two makers: the computational assemblage that receives and creates the object and a person, or team, who design the assemblage. The designer or coder creates the possibilities for the computational when the workflow is designed and built. This object may then be remade to read it critically.

The person who uses the interface, either through code, graphics or haptics, creates a conception of the desired form through the provided options. These may be through search terms, filters of percentages (such as network closeness) or a representational model such as a sonic or visual way of mapping the data. The computational assemblage reads these through interfaces that translate the concept into a technical form, which is then remade and mapped to the underlying data structures to create the computational object to be read. The human is an original maker who creates the initial form to be found and either provides the options within an interface or constructs a machine to process the form. The constructed machine accepts the input and creates the object according to the logics given to it by the builder and the data. Humans and machines become cognitive extensions of each other. The underlying machine processes are required to operationalise the theoretical concept and concretise the model as error or representation, depending on its fit to the interface. A cultural object is rendered artificial for the technical process. Simultaneously, the machine requires a model to begin the process of concretization from an abstract concept of a system, a scientific image (Sellars 1962) or representation (Simondon 2017), and the cultural one that might be considered as a natural object. The artificial object that is accepted is re-presented as a natural object. The materiality that is shown to be interpreted is derived from the concretisation that might not reach the abstract image that either party has in mind. It is the result of an imperfect set of transpositions and translations but it is embedded in the creation of culture.

The critical reading provided by the critical user as form becomes a negotiation with the underlying data. Once the transposed double is made, it is read by machines to be presented as seeing through a series of patterns that standardise the natural, cultural object behind the surface. As the cultural computational form is queried, it may be altered into desired response. It is a made object and there are makers that become revealed when the causality is questioned, revealing the technology involved. Having considered the forms and makers who create it, the materiality’s form can be considered as being created and altered through grammatology.

The digital object is created from the reading and writing processes that convert the original computational form from digitisation into the relevant models. An effect of the concretisation is the transposition and translation of concepts through a series of languages. This points to the medium as a site of transcoding.

The change of medium enables the processing of the data, from the finding of patterns to reformatting and presentation, but demands a change in the way that it is written. The machine reading, hiding itself as a model, uses interfaces to mediate the translation of the cultural data between the software components. The construction of the interface defines what signifiers it can accept and so what symbols that will be created. Any deviation from these definitions suggest an (un)intended break. I use this to point to the materiality of the symbols and the role of interfaces in computational reading and writing to create the digital object. The interfaces abstract the role of the digital in interpreting the given signs through the divergent ways that they are written as code and as computer language, which itself maybe general purpose or a domain specific. The code written to determine aspects of the data, such as dominant colours or features, uses a numeric way of creating the pattern that is defined by what the underlying language allows through its own grammar. Read these against computational forms before presenting them as a model, either as a table or a colour, the pattern requires translations of the underlying data to be able to access the code to read it.

The surface image of the Next Rembrandt is derived from a remediated pattern used in a file format that is created by a reading. This format is derived from algorithmic patterns using digitised data through a series of patterns. Reading and considering these patterns relies on a different pattern language: networking. The series of services, switches and routers to allow the computation and data to meet is hidden but should be understood. Given the example that has been used of the Next Rembrandt, this network might be considered through the physical network of museums and galleries who has allowed the pictures to be digitised. The physical form can be read as an artistic work but it hides the nature of computation required in its surface. Using a series of measurements, the painting has a series of raised areas where the image may be thicker, either by choice, such as overpainting, or accident, such as a drop. In this act, the concrete behaviour of the artist might be read as the algorithmic imagination of a study of paint surfaces.

Building on the concept of the location of reading and writing, the symbols are written and developed through human and digital interactions. Having considered this through the medium, it is useful to think of this through deconstruction. There is a gap between the signifier and the symbol that creates the meaning but can also through the different types of creation machine that creates a sense of the pharmakon. The realisation of the poison and remedy is revealed in this reading. The image that is constructed is a visual representation that might use colour, which potentially has a significance that can be read into it. Made from either a pixel or collection of pixels as a numeric entity that was derived from an internal process, the material form of a colour is altered and represented as numeric symbol on the screen but the human symbolic reading is removed. The human reading is reliant on the machine reading and the symbols that it creates and has to be recreated by the viewer. The symbol’s context might be altered through a change in the underlying computational model as the computation returns the form that it has gained from its reading of the data.

A deconstructionist reading here that can reveal the processes where patterns are read and written. This has the consequence of using the underlying computational models as texts to be read and interpreted. As part of différance, Derrida (2016b) uses the concept of play as a deliberate destabilising of meaning can be used to consider the way that meaning is being constructed. Engaging with the languages and structures to understand how they are written and read through iteracy (Berry 2014) to view how the data becomes an image or a graph. I see the act of interrogating the symbols and forms through the interface objects as part of computational thinking to understand how the code reads and writes. As such, it becomes possible to consider how the original images are constituted as a new cultural object using the material nature of the computational object. Rather than seeing culture as material object or text that can be rewritten, we might approach the role of the medium and techné in the process of a creating a cultural and epistemic object.

3 Conclusion

Computational culture is approached in this paper through questions of causality and grammatology to engage with the materiality. A computational image, created from a reading of transposed images, is used as a way of understanding the nature of the computational to make the image through patterns and models as well as being constructed as a model to find models. I contend that machines seeing can be used to augment human seeing as a critical act but that it requires a critical reading not only of the material but the tool that creates it.

In the act of seeing culture through a computational lens, we need to understand how the medium affects the material. I build on the idea of a post-digital criticism through considering the patterns used in tools and how this affects the grammatology. Although we might not see the underlying computational object, approaching it using critical tools and destabilising the presented form provides critical insights into how the cultural text is being actively created. Yet we need to also contextualise the object within the wider role of not just the techniques but also from the entities who have caused it to be constructed to consider why the tools have seen the way that they might. Questions of causality and grammatology enable the reader to understand the techne that underpins the cultural image as epistemic object.