Introduction

Historically, human imagination is ingrained with dreams of automates and automation that cut through fiction, social discourses, and science (Chassay 2010). These imaginaries instigate hope of improvement for human working conditions; although ‘convenient and seamless these technologies might appear’, they open new ‘questions relating to shifts in power, control, and autonomy’ (Selwyn et al. 2021: 2). Digital automation also provokes worries, often ethically founded, concerning the meaning of being human in the era of big data and machine learning (Lagerkvist 2017). In the present ever-complexifying ‘eduscapes’ (Appadurai 1996), these imaginaries re-actualize the question of how automation and educational practices will turn out together (Selwyn et al. 2021).

Research on digital automation in education calls for a critical inquiry into how scholars study algorithmic-decision making (ADM) technologies (Selwyn 2019; Prinsloo 2020). Narratives about automation in education often pledge new, sophisticated solutions and glamorous opportunities that promise to transform educational practices by improving and facilitating learning and knowledge production. Such ‘grand narratives’ (Markham 2021: 385 tend to adapt to the premises of available technology rather than teachers’ or students’ needs in everyday practices. In this terrain, scholars in education need to question research assumptions imbued in current glorious narratives on automation, because it is not only edtech or educational policies that contribute to such narratives but also research (cf. Pink et al. 2022).

This paper examines the relevance of research methods oriented toward futures in automating teachers’ assessment practices.Footnote 1 As put by Ross:

[a]utomation is a site of multiple ‘not-yetness’: it is thought to be necessary to cope with the increasing need for scaled-up, on-demand teaching; the vast and complex technical and pedagogical challenges associated with automating teaching, feedback, and assessment have not yet been solved; and the implications of reshaping education to fit the capabilities of a partially automated system have not yet been conceptually or empirically understood (Ross 2017: 222).

We argue that the emerging status of digital automation in education calls for applying research methods and sensitivities able to engage with the ‘not-yetness of digital education’ and the inherent 'complexity, uncertainty, and risk … of technologies and practices which are unknown and in flux' (Ross 2017: 214). Engaging with digital technologies that are new (employ new concepts and methods), innovative (they promise new and potentially superior solutions to problems), expected to have a significant socioeconomic impact on the education sector, and underdeveloped (they are not fully developed), is a démarche oriented toward the future.

This argument guides us when examining the relevance of research methods to studying automated futures (Pink 2022). In particular, we pay attention to ‘future-focused social science methods’ (Pink 2022: 750) that invest in a focal point in the future without eradicating the present and the past where true and imagined experiences are made. Our interest in such methods lies in understanding what aspects of emerging sociotechnical practices they can illuminate in contrast to social science research methods most often applied in education and technology studies (e.g., mixed-method studies, case studies, ethnography).

Examining how ADM systems are currently studied and exploring the value of future(s)-oriented social science methods constitutes a research strategy to unpack ‘dominant narratives about automation (Pink et al. 2022: 15). Such a strategy is driven by the commitment to 'consider alternate forms of automation. … and [the] possibility of designing alternates’ (Selwyn et al. 2021: 9). In doing so, we reflect on the 'knowledges' research in education contributes to via its methodological choices (cf. Haraway 2020).

Against this backdrop, this article is organized as follows: By taking the case of ADM technologies in teachers’ assessment practices, we explore the relevance of research methods oriented toward the future to identify narratives of automation that are less discussed in current debate (Selwyn et al. 2021). This exploration first situates the case of automating teachers’ assessment practices, a specific sociotechnical practice emerging in many universities and secondary schools worldwide, partly due to the constant development of systems such as automated essay scoring (AES) and automated short-answer grading (ASAG).

Research on these specific technological developments is discussed here to illustrate the current knowledge gaps about the automation of assessment practices in secondary and tertiary (university) levels of education. Following this, we examine the current research methods for studying automated grading technologies. We introduce the distinction between representational and future(s)-oriented research methods. We characterize such methods and describe their differences. We offer a tentative classification of future(s)-oriented research methods based on previous literature in the field. After that, our first-hand experience with future(s)-oriented research methods is recounted to reflect on their relevance to studies on automation in education. We discuss our insights in relation to temporality, epistemology, performativity (Ross 2017), and materiality (Sørensen 2009), research qualities that future(s)-oriented methods bring to the fore. We conclude by reflecting on the limitations of future(s)-oriented research methods and the implications for educational and research practices.

Knowledge Gaps in the Study of Algorithmic-Decision Making in Assessment Practices

We motivate the interest in the issue of exploring the relevance of research methods for the study of automation, in the specific case of assessment practices with algorithmic-decision making technologies, and more specifically automated grading systems, for several reasons.

First, the focus on automated grading is related to the central place assessment practices have in today’s educational institutions. Undoubtedly, assessment and accreditation are crucial for educational institutions and the legitimization of their educational practices. Most often, assessment practices are studied in connection with a variety of purposes, such as ‘prompting and sustaining learning’, and ‘certifying achievement’ (Dawson et al. 2020: 3). Such goals are pursued in studies about how assessment constitutes a university praxis where ‘judgments [are made] about what someone is capable of, based on some sort of demonstration or product’, and where feedback is provided (Dawson et al. 2020: 3). While much has been examined about improving assessment, assessment literacy, assessment design (Bearman et al. 2016), assessment strategies (Price et al. 2011), and effective feedback (Boud and Molloy 2013), little has comparatively been written about assessment as a site of power relationships.

As technologies, and in particular ADM in automated grading, ‘are not merely functional objects they also … mediate the relations between human beings and the world, and thereby shape human experiences and existence’ (Verbeek 2005 : 236), there is a need to engage with questions such as: How do the teacher (institution)-student power-relations get amplified or shortened with automation in their pedagogical relationships? Such other dimensions of automating assessment call for inquiring about the relevance of the research methods we apply to gain a broader understanding of central aspects that are affected by automating assessment practices (i.e., automated relations, automated educational judgment, desocialization) (see Selwyn et al. 2021: 4–5).

Second, research on ADM and automated grading has exploded during the last few years due to the latest developments in machine learning, the growing number of students attending universities, and the increasing digitalization of educational practices (Valenti et al. 2003). For instance, machine learning techniques like deep learning affect syntactic and semantic features differ from traditional, engineered approaches, as the model learns the text represented in the feature space (Filighera et al. 2020). This technical novelty affects how teachers design and assess students’ assignments and exams. However, we still know little about automated assessment systems’ influence on student learning, writing, and pedagogical relationships (Hsu et al. 2021).

Critical views on automation in education (Prinsloo 2020; Selwyn et al. 2021) do not deny the potential relevance of such emerging technologies but call for reconsidering how assessment practices are configured and shaped in systems such as automated grading. For instance: Which are the teachers’ and students’ actions prompted by automated grading, and which are discouraged? How are these sociotechnical configurations envisioned, and, how are they researched?

Third, research on assessment and grading ‘primarily [focuses] on the student experience rather than staff perception and practices’ (Barton et al. 2020). This research focus is also discussed by Chan and Luo (2022) in their work on teacher feedback literacy. Chan and Luo (2022) observe that ‘not much research has revealed teachers’ perceptions of feedback practices’ and that ‘when it comes to feedback provision, teachers often have to navigate through a complex landscape and make compromises if needed’ (Chan and Luo 2022: 63). Despite the little known about the teachers’ perceptions of assessment and feedback practices along with their increasing automation, systems such as automated grading of text production, ranging from short answers to essays, including formal, stylistic, and content-wise features, are steadily developing.

For instance, automated essay scoring (AES) and automated short-answer grading (ASAG) are systems that have lately gained much research attention. The former system targets coherence and argumentative organization of language, while the latter assesses the correct content and semantic correlation between answers and questions (Ramesh and Sanampudi 2022). The sustained development of such systems actualizes imaginaries in education fuelled by ideas about automation as the solution to the teachers’ workload (Selwyn 2021), fatigue, time constraints to provide students with immediate feedback, and impartiality (Schneider et al. 2022; Ramesh and Sanampudi 2022).

In this sense, there is a tendency to believe that ‘existing and emerging technologies are starting to play a role in changing assessment and could help address such issues both today and looking further ahead into the future, to make assessment smarter, faster, fairer and more effective’ (Pauli and Ferrell 2020: 6). In this context, it is imperative to engage with the challenges of automation (i.e., ‘parseable’ pedagogies; teachers having to ‘work to the algorithm’, etc. (see Pauli and Ferrell 2020), to contribute to imaginaries reflecting the intricacies of everyday assessment practices (Selwyn et al. 2021).

Summarizing our inquiry about the relevance of methods applied for the study of automation in assessment practices is motivated by observing: (1) dimensions of assessment that are currently overlooked, although they are key to focus on, given the increasing deployment of automation in secondary and tertiary education, (2) lack of knowledge about the influence of novel machine learning techniques on everyday teachers’ assessment practices, (3) dominant grand narratives about automation that need to be confronted with the messy and mundane realities of assessing students with sophisticated tools that are often hard to appropriate.

These observations have motivated the present explorative investigation where we take distance from positions that revolve around grand promises or perils and risks with automation in education. Instead, our work seeks to contribute to the current debate by reflecting on the research methods and knowledge mobilized in discourses on digital automation in education.

Which Research Methods for the Study of Algorithmic-Decision Making in Education?

Our inquiry regarding the choice of methods for studying ADM in education brings us to investigate the research methods applied for examining emerging sociotechnical practices characterized by the uncertainty and risks of the 'not-yetness' (Ross 2017). In the literature consulted about ADM, the research methods applied often aim to improve the performance (Lu et al. 2021) and quality of the feedback (Haldeman et al. 2021) provided by technical developments (Vittorini et al. 2021) and models (Çınar et al. 2020). Research methods are also applied to assess the feasibility of using ADM to evaluate the quality of student-written essays (Kumar and Boulanger 2020), increase accuracy and trust in the system (Schneider et al. 2022), and gain an understanding of how technical prototypes perform on actual tasks or on how teachers and students perceive the use of systems still under development (Hsu et al. 2021). These types of studies most often apply experimental research strategies oriented toward what works (Ross 2017).

In that regard, the choice of research methods is aligned to investigate how to enhance ADM system accuracy (Hsu et al. 2021), performance, and human trust, as well as to account for ethical considerations (Schneider et al. 2022). Such an experimental research strategy often includes surveys and interviews to identify students’ attitudes and/or perceptions of systems like ASAG or AES that they know very little about or have not used. Thus, the research methods applied in current studies on automated grading systems have primarily focused on the technical improvement and the user evaluation of such systems in controlled test beds (Burrows et al. 2015; Hsu et al. 2021).

The growing body of scientific research on ADM presently contributes knowledge on automated assessment grounded in results, most often obtained from studies conducted in controlled research situations. It also focuses on the users’ present or past experiences of emerging ADM technologies. Interestingly, it overlooks the study of the ‘not-yetness’ dimension of automation in everyday assessment practices (Selwyn 2021) and hardly questions the raison-d’être of automating assessing practices in education. Yet, issues regarding how automated grading systems are intended to be used in the future or how the current design of ADM ‘smuggles’ normative accounts of how emerging ADM technologies should be further developed in secondary and tertiary education remain open questions.

These observations bring us to differentiate, on the one hand, the research methods currently applied for the study of automated grading systems and, on the other hand, the research methods that could be used for pushing a critical research agenda in our field. The following section exposes such differences by tentatively distinguishing representational and future(s)-oriented methods.

Representational Methods

By representational methods, we refer to quantitative and qualitative social science research methods such as surveys, interviews, case studies, and ethnographies used in digital education to gather data from a known and experienced phenomenon. The use of representational methods in studies focused on educational practices with technology often mobilizes ‘folk theories’ about the functioning of social-technical systems (Kempton 1986: 75-76). On this note, Kempton (1986: 75) explains that ‘people routinely develop their theories to explain the world around them. These theories can be useful even when they contradict conventional technical wisdom.’

The concept of folk theories echoes the concept of ‘mental models’ (Johnson-Laird 1980) which means that people interact with the world via representations (i.e., images or models and processes — or propositional representations) of the ‘external’ and tangible reality. More precisely, Johnson-Laird (1980) refers to Kenneth Craik (1943) as the precursor of the term mental models by explaining the following:

If the organism carries a ‘small-scale model’ of external reality and its possible actions within its head, it can try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilize the knowledge of past events in dealing with the present and the future, and in every way to react in a much fuller, safer, and more competent manner to the emergencies which face it. (Craik in Johnson-Laird 1980 : 73)

Representational methods embed a particular understanding of what knowing in research entails. We find such an understanding assumes that knowing consists of generating a representation of a phenomenon by someone that exists independently of the phenomenon to be known (Varela 1989; Varela et al. 2017). Following this line of thinking, applying representational methods for the study of automation helps researchers to capture the users’ (e.g., teacher, student) representations of ADM technologies. In particular, users’ responses to surveys and interviews can be argued to gather users’ folk theories or more precisely, mental models which develop from the users’ representations of both the emerging technologies (e.g., the images that they associate with them) and the users’ interaction with them (i.e., representation of the processes, possible actions with them). Representational methods collect data after the fact; data are extracted from the users’ rational representation of emerging technologies and are analyzed to be primarily shared in academic publications. Representational methods convey to researchers the feeling of being in control.

The knowledge gained from using representational methods in education contributes thus to scholarly discussions on descriptions and explanations of known sociotechnical phenomena. They often fail to include the inherent proleptic nature of experience and learning (Callard 2018) and present methodological challenges for uncovering and engaging with the implications of emerging technologies in changing everyday practices (Pink et al. 2022).

In contrast, social science methods oriented toward the future issued from anticipation (Adams et al. 2009; Miller 2012), speculation (Auger 2013; Ross 2017, 2022), and critical future studies (Miller and Sandford 2018; Oomen et al. 2022) are in this respect promising for the qualitative research of emerging ADM practices in education. We call these methods future-oriented social science methods — or just future-oriented methods — as they are oriented toward crafting futures and enticing people ‘to become conversant with futures-in-the-making’ (Light 2021: 1). They assume a circular understanding of what knowing is, based on the idea that knowing (human cognition) is embodied action (Varela 1989); which refers to how knowledge is enacted in the encounter between the researcher and the reality or world in which the researcher takes part, judges its meaningfulness, and contributes to change it.

Future-oriented methods invite researchers to playfully ‘experiment with possible futures’ in material and experiential (i.e., sensorial, emotional) ways while pushing us to choose ‘conditions to work towards’ and ‘factors we might have to contend with’ (DiSalvo et al. 2016: 150–151). Future-oriented methods collect data about visions of the future (i.e., facts from the past blended with imagination); data are extracted from the users’ speculative thinking to drive societal change. Future-oriented methods convey to researchers the feeling of not being in control.

Future(s)-Oriented Methods

Future(s)-oriented methods aim to generate speculative accounts of users’ educational experiences with technologies where the past is blended with the future points in time. They aim to reflect on our constitutive relationships with our present and past. Instead of mobilizing users’ representations and mental models, future (s)-oriented methods stimulate imagination, which is an essential part of a research inquiry (Suoranta et al. 2022).

Future(s)-oriented methods aim to generate performative accounts of users’ educational experiences, too. By performative, we mean that they are applied to generate meaningful narratives seeking to enact change in education. In this sense, future(s)-oriented methods are ‘critical and creative and engage with speculative thinking in research in performative ways as they act to create the futures they portray. In part they do this through their focus on engaging publics.’ (Ross 2017: 221) They also call to action to know what is not yet known (see an experimental 'call to action' in Lindberg 2019).

The future(s)-oriented methods we refer to are non-predictive methods interested ‘in identifying and understanding the many different images of the future which exist, understanding why certain people have certain images rather than others, how their different images of the future lead to specific actions, or inactions, in the present, and how present actions or inactions themselves create certain aspects of the future’ (Dator 2002: 7). In this vein, we emphasized future in plural due to the commitment to imagining more than one future, in an attempt to include multi-voiced presents and pasts.

In the field of digital education, Jen Ross (2022) dedicates a whole chapter to speculative approaches to research and teaching that have, according to the author, ‘emerged as a result of the increasingly influential edtech imaginaries that are shaping policy and practice’ (Ross 2022:66). Based on Michael (2021), Ross differentiates between speculative fabulation, which is linked to the creation of stories, scenarios, vignettes, and speculative fabrication connected with artifacts, and objects enticing people to explore new ‘inventive problems’ Michael (2021:80), or designed artifacts in the form of design fiction (Bleeker 2009). Following such a distinction, speculative fabulation methods have, for instance, been applied in the study of datafication and automation in K-12 education (Selwyn et al. 2020); Hillman et al. 2020), and higher education (Suoranta et al. 2022; Dator 2002). Such methods often use social science or techniques from the field of literary studies.

Speculative fabrication methods, also called speculative fiction but better known as design fiction, have been underscored by Cox (2021) in education. Design fiction aims to question the present with glimpses from the future (Bleeker 2009) and is often defined as both a method for envisioning new futures and technologies and a tool for communicating innovations to researchers and the broader public (Tanenbaum 2014). Described as ‘a conflation of design, science fact, and science fiction’ (Bleeker 2009: 6), design fiction can be used ‘to pose critical questions about the impact of technology on society and to actively engage wider publics in how technology is designed’ (Cox 2021: 2). Cox (2021: 3) underscores that design fiction ‘ha[s] the potential to change the scope of the debate by shifting attention away from the existing literature and its focus on developing and testing specific AI applications (Zawacki-Richter et al. 2019) to weighing up more or less desirable directions of travel for society’.

Design fiction has been applied by Ross (2017) via Teacherbot, an automated agent participating in Twitter conversations problematizing and intervening in automated university teaching; Artcasting, a digital mobile prototype for art evaluation; and the Tweeting book, a ‘prototype created to problematize the emphasis in learning analytics on human activity and data’ (Ross 2017: 222).

Cox (2021) offers another example of work contributing eight design fictions about the potential use of AI and robots in learning, administration, and research. The eight-crafted design fictions seek to debate issues ‘as how [AI and robots] might enable teaching of high order skills or change staff roles, as well as exploring the impact on human agency and the nature of datafication’ (2).

Critically questioning discourse about automating education based on the perceived gains such as education efficiency, teachers’ time, and institutional costs, Gallagher and Breines (2021) provide us with an insightful example of how speculative events can be combined with qualitative work to study automation as a component of the teacher function and profession. Through the narratives generated by students and teachers, the authors contribute to a needed debate about the reconfiguration of the teacher function when automation is around us.

The fictional education catalogs from the Royal Institute of Technology (2027–2028)Footnote 2 and Lund University (2040)Footnote 3 in Sweden are concrete outcomes of using future(s)-oriented methods and speculative fabrication in the form of design fiction (Bleeker 2009). Yet, the past EU-funded project SpecultaiveEDuFootnote 4 illustrates how speculation fabrication methods and tools have been discussed in relation to educational issues. A set of speculative design resources are detailed and explained in the open repository created by Salvatore Iaconesi for the SpeculativeEdu project.Footnote 5 In the same vein, Otto Tähkäpää provides a series of resources for schoolsFootnote 6 to make thinking about the future visible. Such examples reflect the interest in researching educational futures by crafting — via critical narratives or designed artifacts — alternative imaginaries of digital educational technologies that get inspiration from science fiction or speculative design (DiSalvo et al. 2016).

The rich and increasing interest in discussing futures is also reflected in work conducted by Unesco’s Future Literacies LabsFootnote 7 and chairs. They aim to educate people to become ‘more skilled at “using-the-future,” more “futures literate,” because of two facts. One is that the future does not yet exist. It can only be imagined. Two is that humans can imagine. As a result, humans can learn to imagine the future for different reasons and in different ways. Thereby becoming more “futures literate”’.Footnote 8

The current speculative turn in education also regards the epistemic potential of speculation in research on digital education, as Ross (2017) explained in an article that has been a source of inspiration in the present work.

Understanding Different Types of Future(s)-Oriented Methods

Against this background, we exemplify the following future(s)-oriented methods’ categories: speculative fabulation, speculative fabrication, speculative (futures) literacy development, and speculative knowing. (See Table 1 for examples and descriptions of each category).

Table 1 Four categories of future(s)-oriented methods for education and technology studies based on Michael (2021) and Ross (2017)
Full size table

While different, these examples have in common the interest in shaping and influencing current educational imaginaries (Rahm 2021) to provoke change. These methods aim to produce actionable knowledge in education; instead of only representing knowledge about a fixed, external educational world. Inviting participants to imagine and speculate about the future is a way to engage with the unknown in research.

A Lens for Analyzing Outputs Generated with Future(s)-Oriented Methods

We select the qualities of temporality, epistemology, and performativity suggested by Ross (2017) as an analytical lens to work with the data and insights collected with future(s)-oriented methods. These qualities are described as follows:

By temporality, Ross (2017) refers to the relationship that future(s)-oriented methods entertain with the complex interplay of past, present, and future. For instance, emerging technologies such as ADM can be said to carry ‘multiple overlapping temporalities and is never simply about the future’ (Ross 2022: 59). On this note, Bendor et al. (2021), in the field of sustainable human–computer interaction, provide a comprehensive framework to guide thinking and analysis of temporality in futuring studies. These authors argue that ‘the past, much like the future, can be approached as a plurality and a repository of potentiality, and engaging with it can help extend both the purview and the methods used by futurists’ (2).

Bendor et al. (2021) illustrate their argument by describing two future-facing methods: forecasting and backcasting, and two-past facing methods: recasting and past casting. What is compelling in their work is that it not only provides us with a vocabulary to think about temporality when conducting research with future(s)-oriented studies but also entices us to think about the relations between past(s) and future(s). In particular, Bendor et al. (2021) stimulate us to think about the past as already figuring the (under-researched) futures in education.

By epistemology, Ross (2022) refers to the circular, embodied reasoning mobilized with future(s)-oriented methods in education research. The author emphasizes that it is less the interest in the future per se that brings scholars doing qualitative research to experiment with these methods but rather the necessity to challenge the ‘linearity, fixity, and the tendency of research to underplay the extent to which it is involved in creating the reality that uncovers’ (59). The epistemological quality of future(s)-oriented methods, explained by Ross, is central to our inquiry about how to study emerging automated assessment practices as it allows us to reflect on how a specific research problem or its space is defined. This quality echoes Varela’s critique of cognitivism. In the 1990s, Varela (1989) strongly criticized cognitivism by arguing that it postulates that human intelligence and intention operate on representations of an external, fixed world instead of human experience as constitutive and constituted of a world in continuous change. As such, the epistemology quality makes researchers in education become reflective of their assumptions about the future of education and automation while defining the research problem space.

By performativity, Ross (2022) explains the quality of future(s)-oriented methods to be 'part of what produces the problem or object of study’ or, in other words, ‘act to create the futures they portray’ (63). In this respect, applying future(s)-oriented methods not only invite people to generate fictive narratives or artifacts but also enact real change in their practices. In the case studied here, the use of future(s)-oriented methods with teachers and students will, for instance, not primarily seek to describe their challenges and dreams regarding automated assessment in their practices. Instead, they will aim to change the teachers’ and students’ present to deal with such challenges and move them to pursue their imaginaries. In this respect, engaging with future(s)-oriented methods demands, we researchers carefully reflect on our views of ethical stances and responsibility in research. As Ross put it (2017):

Speculative methods in education may tend to blur boundaries between research, design, and teaching, and therefore, provoke questions about how best to understand them as methods and the nature of the researcher’s responsibilities when adopting such approaches. (Ross 2017: 215)

To deepen the understanding of how future(s)-oriented methods unfold in research practice, we present an example from an exploratory workshop we run with a group of teachers in the following section. The insights from this illustrative example contribute to discussing how future(s)-oriented methods complement representational ones in the study of emerging automated assessment practices in education.

Putting Future(s)-Oriented Methods into Practice

To experience the use of future(s)-oriented methods for studying ADM in everyday education, we conducted an explorative workshop with seven high school teachers in March 2022 in the south of Sweden. This specific workshop came to deepen our past experiences with applying future(s)-oriented methods in conference workshops and university courses.

The teachers we invited belong to the International Baccalaureate (IB) program, where they teach different school subjects. We selected this specific group because of their past experiences with ADM in the students’ exams. The participants’ experience with automation in their assessment practices was an important selection criterium. This is because we aimed to enact speculations anchored in an individual’s experience of automation instead of an abstract idea or pure speculation.

In the invitation to the workshop, which was part of their professional competence development time, we specified our interest in having a conversation on assessment and the role of ADM in their practices. We also invited them to learn about social science fiction methods for writing fictive narratives.

The workshop lasted three hours and was held in their school. More specifically, the workshop introduced the topic of automation for the examination of student assignments/exams. We also introduced the teachers to the social science fiction guidelines (Lackey 1994). After that, we described the activity consisting of group discussions to stimulate speculative thinking and individually to write postcards from the future year 2032 (i.e., backcasting futuring techniques). The workshop concluded with a 'show-and-tell' session where the participants read aloud and commented on their postcards.

More specifically, writing the postcards involved the following three main steps:

  1. 1.

    Envisioning the setting level that is the physical environment of the school work in the year 2032.

  2. 2.

    Imagining the relational level, that is, the social environment of 2032, such as relationships with colleagues and students.

  3. 3.

    Creating the character, the one writing the postcard (i.e., behavior, actions, attitudes).

In what follows, we describe how we shaped the teachers’ speculative fabulation using the backcasting technique. Our description follows Elsden’s et al. (2017) structure for communicating speculative enactment in practice.

The Speculative Frame

We created a backcasting scenario to support thinking about positive and desirable changes. We asked the participants to look back at a successful educational transformation concerning student examination and automation and write a postcard where they had to tell what decisions were vital between 2022 and 2032 to reach the future situation. The prompt for the postcard was formulated through the question: What are you thankful for in your work? In this way, the exercise included engagement in conversations about how it is possible to do things to achieve desirable changes in the future. This activity was inspired and adapted from the Future week 2021 at Lund University in Sweden (Framtidsveckan 2021).

The Enactment

In small groups (i.e., two or three teachers), the participants discussed the questions we provided and wrote notes regarding their school and profession in 2032. In particular, they debated visions of future physical settings and social relationships with their students and/or colleagues and shared the envisioned characteristics of the main character (i.e., the one writing the postcard), sometimes themselves, sometimes someone else. This was an opportunity for the teachers to formulate and share their views on ADM and the challenges with assessment from their everyday practices with student examinations. To help stimulate the discussions in each step of the performing phase, we provided them with questions that worked as prompts for their conversations. We also provided them with post-its, pens, and paper. After three rounds of group discussion sessions (on settings, relations, and characters), we invited the participants to write their postcards individually. We printed free photos with images evoking the school, exams, technologies in the classroom, and people, for the participants to choose from when writing the postcards. The individual postcards consisted of a selected photo and a short narrative (half of an A4 page).

Reflections

The teachers’ speculative activity regarding student examination was a real space for the participating teachers to reflect and discuss assessment and digital practices. They engaged enthusiastically in conversations about ADM that remained private. We walked around in case they needed clarification or guidance, but otherwise, we decided not to listen to the discussions as we did not want to disrupt the flow of the topic of their discussions. We learned about their conversations afterward when they presented their postcards to the whole group and shared the highlights of their group discussion. Asking the participants to imagine the year 2032 in Sweden, their school environment, and, specifically, ADM and assessment resulted in visions that included challenges and possible solutions. Situating the participants in the future and inviting them to reflect on the past enacted a ‘space of possibilities to emerge’; ‘a repository of potentiality’ (Bendor et al. 2021: 2) that was key in enticing participants’ unleash of imagination. At times, we also experienced challenges in making the activity relevant and meaningful for the teachers. We explain later on, in conclusion, the limitations encountered with backcasting in relation to ‘discursive closure’ (Markham 2021).

From the teachers’ presentations and readings of their postcards, we agreed with the participants on mentioning 17 topics that shared both preferable and possible futures for the teachers. In particular, the topics described visions of the school in 2032 that reflected an educational system shaped by the decisions made in the timespan 2022–2032 regarding ADM in education. In this vein, the teachers shared visions of possible implications of automated grading in school; such implications went beyond discussions centered on the efficiency and effectivity of ADM in teachers’ assessment practices and student performance. Instead, they described general visions that extrapolated some of the challenges the teachers experience today with emerging ADM technologies in their practices. For instance, they spoke of ‘a radicalized school’ with less room for creative and critical thinking development, ‘less diversity of opinions’, and ‘more surveillance’. The teachers also situated the school in 2032 in a world with a ‘food shortage’ due to significant 'environmental issues'.

They mentioned a ‘shortage of competence in mother tongue languages’ as a consequence of the prevalence of English in school and society, and also both more and less ‘segregation’ due to the possibilities of (adaptive) technologies to help teachers to form groups based on students’ different abilities and knowledge needs. At the same time, they discussed visions in which ‘AI takes care of the student essays’ and ‘grading is centralized,’ contributing to the fairness of grading tools implemented nationally. They also discussed the possible transformation of the teaching profession in terms of the emergence of a new class of teachers, ‘consultants’, operating outside the physical school from a distance. They referred to ‘Siri’, the virtual assistant, using natural language in English as an accepted, legitimate participant in the Swedish school.

Ethics

The workshop did not require ethical approval according to the university research ethics service. The participants were able to provide informed consent. Before and during the workshop, we carefully informed the workshop participants about our research purposes on future methods and automation. During the presentation of the pre-formulated questions, which acted as prompts for the participants’ speculative activity, we were attentive to delivering value for the participants. In this sense, we carefully listened to the language and the topics mentioned. And we adjusted our prompt questions on the go to come closer to the participants’ interests in writing the fictive postcards.

Discussion

This work originated from an interest in determining the relevance of research methods for studying automation in secondary and tertiary education. After our exploration, informed by the literature and first-hand experiences with future-oriented methods, we reflect on the insights gained. We present such a reflection adopting Ross’s qualities of speculative methods (i.e., temporality, epistemology, and performativity). Such qualities provide the structure to this section. We also suggest a fourth quality called: materiality.

First, temporality was in our first-hand experience with speculative fabulation and the backcasting technique, enacted in the participants’ exploration of alternative futures situated in 2032 but also alternative future pasts situated in the timespan 2022–2032. Bendor’s et al. (2021: 3) suggestion about the ‘past as such is open and multiple’ resonated in the various pasts (or decisions) evoked in the teachers’ postcards (e.g., centralizing grading; accepting the English-speaking ‘Siri’ in the Swedish classroom, increasing adaptive learning). In this sense, instead of trying to identify a common pattern among the teachers’ narratives generated, we, the researchers, focused on describing the plurality of pasts, presents, and futures generated. We noted that the participants’ narratives are critical for socially discussing the relationships between pasts, presents, and futures and exercising agency toward the future. By creating the postcards, the teachers were also building worlds together and, by doing so, creating a local context contributing to the meaningfulness of the speculative exercise. Instead of conveying ideas and thoughts about the future in the abstract, the possibility to speculate makes it possible to situate ideas and thoughts in everyday stories where ‘the implications of change are presented through the subjective perception of (imagined) people' (Lund University 2040: 23). We also paid attention to the participants’ challenges in formulating positionalities in the future, which was revealed through questions posed in their discussions, such as, whose future is it? Which future?

In this vein, we noted tensions in the participants’ conversations since they systematically hesitated between composing preferable, possible, or probable futures. This tension could have been better addressed during the planning and execution of the workshop, for example, by inviting the participants to analyze and contrast the different types of futures evoked. Preferable, possible, and probable futures are dimensions of speculative activities necessary to marry to reach beyond normative thinking about ADM and assessment and strike a balance between utopian and dystopian educational statements.

Furthermore, we noted that the participants needed to be carefully guided throughout the different workshop phases — the backcasting method, although fun and creative, needs to be shaped and implemented to avoid superficial, flat conversations deliberatively. The embeddedness of materiality and the enactment in-between researchers and participants becomes critical here. This brings us to discuss the epistemology quality (Ross 2017), demanding to reflect on the epistemological value of the topics mobilized in the teachers’ postcards and discussions, and consider our relationship with the material generated.

Second, the epistemology quality brings us to reflect on how we researchers experience what knowing is with future(s)-oriented and representational methods. Although both representational and future(s)-oriented methods are applied for the study of sociotechnical practices in education and particularly ask about how actors interact and engage with the socio-material world they inhabit, they differ significantly in how such a socio-material world is understood and analyzed. Taking the case of automated assessment, we observe that representational methods understand the potential transformation of automation on assessment practices as manifestations of actors’ interactions with ADM technologies. They seek to understand steps and response strategies as guided by a ‘folk theory’ of actors’ fixed intentions, motives, objectives, and values. It is thus an underlying assumption of representational methods that the material world and its practices are guided by underlying (implicit) rules and theories and that the role of research methods is to discern and represent these rules to explain the material world and practices in education. Representational methods are also instrumental in conceptualizing change based on what has happened so far in a stable world, a world to be known. They are thus inherently retrospective in their efforts to understand (emerging) practices and sociotechnical transformations in society.

In contrast, future(s)-oriented methods’ perspectives on the material world and the practices embedded are different as these are not seen as guided by a set of intentions, motives, objectives, and values of an underlying social reality. Instead, for future(s)-oriented methods, the material world and its practices are neither in the external past nor the predicted future but 'in-the-making' (Light 2021).

As John Dewey argued in his seminal critique of the ‘Reflex Arc’ model of behavior (Dewey 1896/1998), actors — and we can add: researchers studying enaction — grasp the meaning as functions of actions in a broader, dynamic context which include ends-in-view, aims, and interests that are directed towards future states of affairs. Where representational methods tend to reify past and present actions and events to explain their interlinkages according to underlying rules grasped by theoretical accounts, future(s)-oriented methods seek to envision ‘what if’ situations as they are enacted in the light of the actors’ future(s)-oriented concerns and hope. Future(s)-oriented methods thus transcend Cartesian dualist representationalism in invoking the embodied, embedded, and enacted character of action, cognition, and research. They resituate action (and research) in time (Gallagher 2020).

In the first-hand experience with backcasting, we realized the importance of paying attention to the world-building the teachers conveyed in their sociotechnical imaginaries. More specifically, we became attentive to the implicit relations they drew between the school and the students (e.g., surveillance; care), the role of the school in society (e.g., segregation, cultivating creative and critical thinking), and the teachers with their profession (e.g., a consultant from a distance). Such relations, we understand, speak of implications of automation in education that provide new insights and questions regarding aspects like the weight of social relations in the school, the school role in society today and 10 years from now, the teacher’s professional identity, the teaching profession, among others. These aspects differ profoundly from those reflected in discourses obsessed with efficiency and technology-enhanced learning of automating assessment practices in education.

The epistemological quality of future(s)-oriented methods complements representational methods as they open a space for reflection and imagination in an abductive fashion (Tavoy and Timmermans 2014). Such a dialogical space allows new conditions for unexpected discourses of a plurality of everyday futures to emerge. The abductive, non-linear logic that future(s)-oriented methods introduce makes them different from the representational methods underpinned by deductive or inductive reasoning. In other words, such abductive reasoning is instrumental in questioning ‘assumptions about our knowledge of the future that are rarely challenged’ (Curry 2021: 1). Following Curry (2021), we note that the differences in the logic operating in engaging with the future can also be related to a ‘positivist school’ that values deductive or inductive methods (e.g., futurology, scenario planning, probabilistic trend analysis, cross-impact analysis) and a ‘critical futures school’ that values abductive ones. Such schools consequently differ in views about the future(s). While the positivist school aims to represent the future understood as being out there, pre-determined, in front of us, the critical school seeks to create the future as ‘the future is not something to be predicted but to be made’ (Montfort 2017: xii). In this sense, future(s)-oriented methods present a performative quality that makes them different from other methods in how they can potentially help think about the world, reinvent it, and impact it through the participants’ narratives and voices generated.

Third, we connect the performativity quality of speculative methods with how educational researchers support others (e.g., teachers) to enact visions of a common future and actions that matter to them in view of living a good life. As such, futures can be represented, recounted, and predicted, but they can also be viewed as not being 'out there' but 'in us' (Curry 2021). Following this reasoning, we can postulate that representational research methods, including those asking about the future in education and digital education, present a ‘form of epistemological closure … paralleled by normative closure’ linked to presumptions about a single common past and present (Curry 2021). This normativity can be critiqued with the use of future(s)-oriented methods, which remains key for research and practice in our field, particularly in studying emerging technologies such as automated assessment.

However, applying future(s)-oriented methods in research does not mean it is an antidote to normativity. On this note, Markham (2021) speaks of the limits of the social imaginary and the challenges to intervening in future speculations of memory, data, and algorithms. Markham (2021) suggests ‘discursive closure’ to explain how participants in speculative workshops naturalize, neutralize, and legitimize certain infrastructures and values that ‘close off discussions of alternatives that might counter current hegemonic power’ (382). This echoes discussions we participated in with the teachers during the workshop about the influence of Edtech in school practices, AI determinism, and narratives of the future as ‘unchangeable’ (Markham 2021: 384). Markham (2021) underscores ‘although the capacity for imagining something new or different resides in all of us, the available material for any imaginative act is greatly influenced by prior imaginations’ (383). This often results, as Ross (2022) notes, in ‘the reproduction of an understating of technology as inevitable and people as powerless to generate different kinds of futures’ (65).

In our experience with speculative fabulation, we noted that the quality of the material could have also influenced the teachers’ ‘discursive closure’ (Markham 2021) at times; more specifically, we refer to the prompts designed to unleash their imagination and the relatively homogeneity of the group participating in the workshop. As Ross (2022) reminds us, ‘when considering speculative methods in teaching and research, researchers and educators need to critically question the dynamics of voice and engagement they are mobilizing’ (66).

Fourth, we suggest materiality as the fourth quality of speculative fabulation. This quality is deeply connected to the fundamental role that the design of the backcasting exercise and prompts plays in configuring research knowledge and participants’ imaginaries. Tools matter. They are not neutral (Cerratto-Pargman et al. 2015). For instance, the choice of postcards as the discursive space to shape futures and pasts, the guidelines selected to guide the participants to write the fictive narratives, and the language used in the questions formulated by the researchers to scaffold participants to discuss futures and automation, the images we pre-selected for the postcards, and the general atmosphere and safe space created during the speculative workshop are, among others, designed materials, research tools that configured the teachers’ speculative fabulation experience. Such designed materials and tools are part and parcel of how the participating teachers valued the general activity and felt compelled to participate. In this sense, the materiality of the backcasting exercise conducted was associated with the epistemological and ontological assumptions we researchers made when planning and crafting the workshop in all its steps. In this sense, we learned that discussing and reflecting on our epistemological and ontological assumptions connected to automation and the futures of education in Sweden, before or after hand is an essential component of the materiality quality of speculative fabulation. As mentioned elsewhere (Cerratto Pargman and Jahnke 2019), ‘[e]ngaging with the material conditions of educational practices means paying attention not just to the material or thing but also to the multiple relationships or mediations that are afforded by and constrained in the interactions among teachers, learners, and school materials, such as artifacts, strategy documents, and policies. … Following Sörensen (2009), “material conditions of educational practices also invoke the relational character of materiality”’ (7). Developing sensitivity toward the materiality of speculative methods takes time, effort, and research reflexivity, generally not sufficiently exercised in studies about education and technologies. As Lackey (1994) put it so well, ‘it takes time to become comfortable with one’s own creativity, when accommodation to norms usually is the operative mode’ (Lackey 1994).

Conclusion

Current studies on emerging ADM technologies in assessment practices need to be attentive to the methodological choices made in selecting research methods so we can contribute knowledge and change in education that matters for those doing education every day and from the ground. This is crucial as automated technologies are emerging, and their role and function in educational practices are unsettled and partially dependent on the anticipations and valorizations of educational practitioners’ engagements. Practitioners’ relationships with emerging, novel technologies are not settled in advance, and the educational practices are open-ended and, at any stage in time, ripe for alternative enactments. This means that practitioners — students, teachers, administrators, and technology providers — envisioning the role of emerging technologies play a significant role in how they approach, enact future educational practices, and gain agency through their narratives. Neglecting to address the non-linearity, abductive, and emergent qualities of practitioners’ experiences and actions constitutes a significant flaw in studying the role played by emerging automation technologies in education. Nevertheless, future(s)-oriented methods present limitations too. Such limitations relate to (1) the materiality of the prompt questions and activities to inspire people to anticipate, aspire, and imagine (cf. Appadurai 2013), (2) the analytical lens to be able to embrace the richness and diversity of the narratives generated, (3) the participatory aspects of these methods can be better understood in terms of who is invited to participate in envisioning the future and what skills these future(s)-methods demand, not only from the facilitators/researchers but also from the participants, (4) the scientific value of the narratives (data) about the futures generated according to how science is currently understood, institutionally practiced, and legitimized.

Finally, we invite others to engage with these limitations so new knowledge(s) about how to engage with the future(s) of education and automation can be creatively, socially, and bottom-up generated.