1 Introduction

The public release of AI-based chatbots, such as ChatGPT and Gemini, has caught considerable public attention. The first reactions to using LLMs often range between awe and disbelief. Users tend to be impressed by how quickly LLMs generate vast amounts of text that appears trustworthy, seems to display some capacity of logic and integrates specific (possibly incorrect) references and numbers. Furthermore, the texts generated by LLMs are thought to be impressive in virtue of their resemblance to human penmanship. At the same time, users also spot that LLMs produce outputs with misleading information and flawed logic, resist taking a stand on issues and refuse giving an output on ethically sensitive topics [1]. These issues are endemic to LLMs in general, as we saw when starting to write this manuscript in 2021 and being originally primarily focused on OpenAI’s then flagship GPT-3 LLM. The rapid development and growth of various LLMs since then shows mounting public debate - and evidence - regarding their potential to undermine democratic values, such as informed engagement and deliberation [2]. Such worries about the negative democratic impact of LLMs even resulted in urgent calls for temporarily banning their development and regulating their use [3, p.7]. Because of the rapid development of LLMs, we do not focus on any particular empirical instantiation of LLMs in this paper. Rather, we focus on the algorithmic and design features similar across different generations of LLMs that facilitate a specific type of political engagement, as we show in Sect. 2.

Early studies suggest that LLMs pose a risk of diluting the individual and public capacity to critically evaluate information and hinder the meaningful exchange of ideas [4,5,6,7]. This is because LLMs allow to quickly spread curated content on the internet at large or on social media platforms in specific. For instance, they can be used to spread polemic messages or can act as “stochastic parrots” [4] that promote certain agendas or provide misinformation. In doing so, LLMs essentially spread the same messages in perpetually different linguistic variations. Furthermore, LLMs can have more direct political effects, because they can be easily used to consistently target specific groups of people, initiate harmful conversations [5], promote toxic and biased content through the intentional overuse of trigger words [6], or even to induce people to enrol in extremist and terrorist organizations [7]. Public institutions worldwide struggle to regulate and guide the development of LLMs both ethically and legally [8-9]. A common starting point for decision-makers across different sectors is to attempt to increase people’s digital literacy, such that theyare better equipped at informed decision-making when interacting with the output of LLMs [8,9,10,11]. This can be done by providing people with relevant knowledge about how LLMs are trained and how they generate the most probable response to a user query based on the training data. Development of digital tools to verify authenticity of the generated content features here as a way to enable people to make informed decisions and promote trust in this technology [12-13]. While such efforts are helpful in raising awareness about the opportunities and limitations of LLMs, we suggest that the political significance of LLMs goes beyond the spread of misinformation. In this paper, we explore this broader significance in terms of political de-skilling and argue that LLMs impede the ability to engage in political action. Being able to engage in political action, as we will show in the paper, involves being able to form oneself as a political subject who can recognize relevant actors in specific situations and contexts, resist, deliberate, form preferences, vote, represent, and revise own commitments [14]. Instead of focusing on specific types of political actions that a person can engage in, this paper draws from the work of Arendt and Foucault to explain how and under which conditions the constitution of political subjectivity is at all possible. This enables us to clarify why LLMs give rise to political de-skilling and hence pose a threat to political subjectivity, and by pointing to how the practice of parrhesia enables one to form one’s political subjectivity in relation to LLMs.

The paper proceeds as follows. In Sect. 2, we analyze the political significance of LLMs, and suggest that they bear the potential of becoming an important part of political practice. At the same time, however, we suggest that LLMs bear the threat of political de-skilling. In Sect. 3, we will use the work of Arendt—particularly her distinction between speech and language—to specify in more detail in what sense LLMs pose a threat to politics. In Sect. 4, we build on Foucault to articulate why the power exercised by LLMs shapes political subjectivity, as well as conceptualize how resistance to this power is possible. In conclusion, we show how our synthesis of Arendt and Foucault enables to take seriously that LLMs can give rise to de-skilling without falling back into a naïve technological determinism.

2 Large language models and politics

Arendt’s [15] understanding of politics is the starting point for our analysis of LLMs. Arendt’s political thought is deeply influenced by ancient Greek philosophy, particularly the idea of the polis. A polis can be understood as a self-governing community, where people freely come to discuss and decide on matters of public importance. Arendt builds on this tradition and interprets politics as being together with others, thus creating a shared public place that is receptive to human plurality and open debate. Under these conditions, a shared meaningful world can come into being. LLMs play a role in such a political context by mediating both the content and the space where people engage with one another. As we will show in this section, of ethical and political significance here are both the algorithmic structures underlying LLMs and the user interface.

With regard to the algorithmic structures, an important concern with LLMs lies in disproportionally relying on their output, while this output may not always be trustworthy. Hence, one of the chief political dangers of LLMs concerns misinformation. This concern is directly related to the technical properties of LLMs that produce strings of words based on statistical approximation [16–17]. That means that LLMs do not understand words as humans do, but instead process them as statistical patterns in a database, without regard for context or flexibility in meaning. So when you ask ChatGPT a question, it predicts the answer as the one most likely to resemble its training data, even though this answer might not actually be true. This is very problematic because it means that the use of LLMs requires constant user vigilance and expertise to be able to verify the outputs, complicating the deliberative capacities of people [16–17]. This worry is mirrored in the information-focused digital literacy strategies that we discussed earlier.

We contend that next to the focus on the generated output, it is equally important to examine how LLMs mediate people’s predispositions to engage with information. Specifically, the chatbot interface design is not neutral and promotes an idea of credibility and reliance in LLMs, as well as enrols the user into their continuous use. These design features co-shape people’s perceptions, consequently translating to their political actions: expressing opinions, making decisions, and taking collective actions.

Consider the clean and open look of the interface of LLMs. It relies on prompt-based interactions that are presented in a dialogical manner. This suggests a form of information exchange with a knowledgeable partner - the user asks, the machine answers. Chat-based LLMs help the users to summarize vast amounts of information, translate pieces of text or interpret them from a specific disciplinary angle, devise plans and strategies, and generate long pieces of text. At the same time, it is difficult if not impossible to verify the sources based on which the responses are provided. A user is expected to either be the domain expert who can verify the credibility of the generated information, or to simply trust it. The user can, however, rate the generated output with a thumbs up or down button (e.g. in ChatGPT), which subsequently invites the user to provide additional open feedback and classify the output as either harmful, false or unhelpful. At the bottom of the page, there is usually a disclaimer in small semi-transparent letters. For instance, ChatGPT’s website says, “ChatGPT may produce inaccurate information about people, places, or facts” [18] or Google’s Bard saying “Bard may display inaccurate or offensive information that does not represent Google’s views” [19]. Next to similar disclaimers, LLMs present a visually blank slate open to any questions. The visual look of chat-based LLMs thus also has a political relevance as it predisposes the users to trust the output and co-shapes the risks of overreliance and less critical attitude to the produced information.

Another way LLMs’ visual design preconditions users’ engagement in politics relates to anthropomorphism, a tendency to intuit human qualities in non-human others. People tend to anthropomorphize many objects and non-human animals, trying to make sense of them; it is a psychological feature of being human [20]. At the same time, this can induce unconditional acceptance of the anthropomorphized objects, their negative effects notwithstanding [20].

One famous example of anthropomorphism relates specifically to chatbots, and is known as the ELIZA effect. ELIZA was the first chatbot program built by Weizenbaum in the 1960s. It managed to engage the users in extended conversations and convinced many of the users of its intelligence, even when they knew they had been interacting with a computer rather than a human [21]. The ELIZA effect, as originally introduced, denotes “the susceptibility of people to read far more understanding than is warranted into strings of symbols– especially words– strung together by computers” [22, p.157]. It describes the effects of ascribing machines with human-like understanding and emotional responsiveness, such that humans relatively easily form bonds with conversation-style computer programs and trust their output. In the context of political action and connecting to the earlier mentioned algorithmic structures of LLMs, ELIZA effects are problematic because they risk to reduce active political engagement and agency of users due to uncritically embracing the biased, tailored or otherwise manipulative interaction patterns with LLMs.

Modern-day chatbased LLMs profit from the original ELIZA effect, intensified by additional design features. For instance, some chat-based LLMs, such as Meta’s Llama2, personify chatbots and encourage users to customize the chatbot’s personality [23]. In general, the frictionless user interface, the way in which large pieces of text appear in response to users’ prompts, without further references, and the extent to which the responses resemble human authorship all exacerbate the ELIZA effect. LLMs elicit misleading ideas of machine intelligence, the ability to understand and reason, to use language as people do, thus promoting the ideas of trustworthiness. The ELIZA effect may be partly responsible for the uncritical attitude of users towards LLM-based technologies, such as ChatGPT. Anthropomorphizing LLMs, with consequent ELIZA effects, is politically relevant because this mediates the context in which the users engage with information, form inclinations and beliefs, and even follow the questionable and harmful advice of LLMs [24].

These concerns have both individual and broader political implications that we will further explore in this paper. As has been argued, LLMs form a threat to political life because they potentially decrease the agency of individuals or undermine trust in existing political institutions. These political threats have predominantly been connected to the epistemic capacities of individuals and the emergence of filter bubbles and echo chambers [25–26]. The epistemic skills of acquiring, interpreting, and evaluating information are at risk because LLMs seem to decrease the capacity for evaluating information and increase the tendency to conflate factual information and political statements. Additionally, since LLMs can produce intelligible texts with the right prompts, users of LLMs no longer need to connect different issues or structure their thoughts about a certain topic when writing a text.

While we agree that LLMs can lead to a decrease in individual agency in this way, we aim to interpret these dangers specifically as threats to the capacity of individuals to engage in politics. Since the ability to evaluate information, to be critical about the trustworthiness of sources, or to distinguish fact from ideology are skills that are indispensable for active citizenship, we propose to interpret the threat of LLMs in terms of political de-skilling. By this token, LLMs form a threat because they bear the potential to rob citizens of the skills essential for engaging in political action [27].

Being able to engage in political action, as we show in the next two sections, involves being able to form oneself as a political subject. We draw from the work of Arendt and Foucault to explain how and under which conditions the constitution of a political subjectivity is possible. Doing so enables us to clarify why political de-skilling poses a threat to political subjectivity, as well as to point to opportunities to resist political de-skilling.

As Ucnik [28] has pointed out, the projects of Foucault and Arendt both attempt to offer an account of how to develop an ethico-political self that forms the basis of the possibility to undermine existing political and moral systems. The development of this self involves, for both thinkers, establishing a particular relationship to oneself through which one’s political subjectivity is constituted. The underlying assumption here is that insofar as one is a passive recipient of power, one is unable to engage in political activity. The constitution of political subjectivity makes it possible to engage in political action. In other words, the ability to develop a critical relationship to oneself emerges as a condition of possibility for political action. Political de-skilling, then, can be understood as a threat to politics because it undermines the formation of a critical self-relationship that is necessary for political subjectivity. The rationale for drawing from both Arendt and Foucault when analyzing the political threats that LLMs pose is threefold. First, Arendt’s analysis of the relationship between speech, action and politics enables us to articulate in what sense LLMs provide a threat to the subject’s precondition to engage in politics. Second, Foucault offers a lens to analyze LLMs as technologies of power and to carve out a space for establishing relationships with the powers being exercised. Third, a conjunction of Arendt’s and Foucault’s approaches to political subjectivity shows how resistance to the threat of LLMs to deprive individuals of the capacity of speech is possible by collective counter-conduct. We will start by unraveling these assumptions further by turning to the work of Hannah Arendt.

3 Arendt, politics, and the functionalization of language

The work of Hannah Arendt, specifically The Human Condition (1958) and Eichmann in Jerusalem (1964), helps to problematize the pervasive use of generative language algorithms in the context of political activity and potential political de-skilling. To demonstrate the relevance of Arendt’s political theory for LLMs, we will further elaborate on her view of politics and particularly focus on the distinction between speech and language.

Arendt considers politics to be the pinnacle of human activity that enables to partake in public life and to meaningfully relate to others [15]. This activity is grounded in the capacity of having critical self-knowledge and being prepared to reveal it to others. Arendt’s notion of politics takes human plurality as a starting point. On the one hand, plurality is grounded in a shared condition of being human and introduces a principle of equality in public debate. On the other hand, it underlines the uniqueness of each person: no human is exactly similar and hence potentially puts forward a distinct perspective and potential for change. Hence, Arendtian politics essentially consists in meaningful exchanges with others taking place in a shared space where one can express oneself in word (i.e., speech) and deed (i.e., action). This activity enables one to distinguish oneself from others and to contribute to lasting changes.

Action and speech are the cornerstones of politics, and are essential for meaningful exchange and debate with others.Footnote 1Action refers to the ability of collectively creating something new and meaningful, characterized by initiative, creativity, and responsibility. Speech concerns the exchange of ideas and the ability to communicate with others in general, characterized by an openness to learn and persuasive communication of one’s own viewpoints. These two faculties are interrelated insofar as speech is important to mobilize action and foster meaningful creation and change [15]. Parallel to action, speaking is a distinct political activity that allows people to reveal themselves to others and see other people as distinct selves. Together, speech and action create a space of appearances, which makes the practice of politics possible. Thus, action and speech are conducive to both the content and the process of politics.

Arendt worries that the principles of productivity, efficiency, and instrumentalization emphasized in Modernity restrict the human capacity for speech and action. According to her, “[w]henever the relevance of speech is at stake, matters become political by definition, for speech is what makes man [sic.] a political being” [15, p.3]. Modernity bears the threat of reducing speech to mere language, i.e. to functional words that reflect no distinct human self, no initiative, desires, or aims, and as such, reduces the potential for new beginnings. Below, we unpack this distinction between speech and language further and explore how it relates to LLMs.

3.1 On speech, language, and LLMs

The faculty of speech and action are essential for politics because they enable persons to position themselves, re-invent themselves, and actively shape the world they want to live in. This is particularly relevant for contexts that seem to preclude challenging the status quo due to, for instance, a tyrannic rule or an overload of decision-making points. It is through speech and action that people have a potential to evoke change even when it seems that change is impossible:

“With word and deed we insert ourselves into the human world. […] It is in the nature of beginning that something new is started which cannot be expected from whatever may have happened before. […] The fact that man [sic] is capable of action [and speech] means that the unexpected can be expected from him, that he is able to perform what is infinitely improbable” [15, p. 178].

The faculty of speech is essential for action and change as it represents the potential for “new beginnings”: through speech, people reveal who they are and what they want from the others, making action and change possible. However, the potential for new beginnings through speech only manifests in the presence of others. When exercising the faculty of speech collectively, individuals weave together a common space of appearances, thereby creating a public realm where people come together, disclose themselves and produce shared meaning. This insight enables us to sharpen our initial question regarding the political impact of LLMs: What is the political effect when that which seems as the speech of others is a product of generative AI, for instance, enacted by ChatGPT or synthetically generated personas on social media?

At first sight, LLMs seem to suppress the political faculty of speech because they jeopardize both its communicative and disclosing functions. For Arendt, speech is inherently social as the disclosure of one’s self requires the presence of other people. It is in dialogue with others that people can be persuaded to change their initial position, and where action and the ensuing possibility for change originate. LLMs have a bearing on both the communicative and the disclosive function of speech. They restrict the communicative function of speech by creating a false impression of interacting with another human being. This makes it that people are invited to invest in a reciprocal relationship through an exchange of intentions and desires, and have an idea of jointly creating an identity and shared community, whereas this reciprocity is, in fact, absent. LLMs also change the purpose and nature of the disclosing function of speech. Interaction with LLMs is not about enabling others to recognize themselves as unique individuals embedded in a particular lifeworld because LLMs are not capable of any interest in the user as a distinct person. Instead, LLMs process the input from the user and the consequent user profile as strings of code, and use it as a resource for updating their database.

Regardless of whether people know that they are engaging with LLMs and not a real person, their capacity to exercise their speech, and hence to engage in politics in Arendt’s sense, is challenged. When a person suspects or realizes that they have been engaging not directly with another person but with LLMs (scripted by another person), this may hamper future digital communicative practices. At play there may be a feeling of being fooled, an uncertainty in further interactions, attempts to “game” or sabotage LLMs, exploiting their vulnerabilities, etc. Not realizing that one engages in an exchange with a machine can be harmful for speech, because the anthropomorphization of LLMs effectively leads to misattributing agency to LLMs and misrepresenting their capacities for understanding and communicating like people would do.

One illustration of the danger of LLMs in sedating the reflective capacities of people can be demonstrated by a study of McGuffie and Newhouse that investigates how GPT3 facilitated online radicalization [7]. Through experimentation, they identified the significant effectiveness of the GPT3 model in curating AI-generated content that promotes engagement with extremist communities, and potentially helps recruiting new members. They show how GPT3 helped to produce personalized polemic messages that resemble certain target narratives (e.g. promoting Nazism) and could exchange messages with people that resemble interaction patterns of a specific persona (e.g. a QAnon follower). When targeted individuals join these AI-synthesized forums, they are met with enthusiasm, regularly confronted with the group’s concerns, and are invited to exchange ideas on relevant topics. The authors point out that the main risks of LLMs lie in creating a strong group identity with what seems as a genuine interest in the newcomers and their opinions (in contrast to the self-disclosure function of speech in Arendt), and in the simulation of frequent and consistent engagement that propagates the dominant viewpoints in ever different ways (in contrast to the condition of plurality and the communicative function of speech in Arendt). In short, a combination of technical features underlying LLMs and the extent to which LLM-generated output resembles human speech, with all its flaws and complexity, complicates the sense-making practices of people, and shapes their political activities.

Arendt’s work is also useful to highlight another politically significant feature of LLMs, namely how they seemingly flatten speech into merely functional and logical language sequences. In Eichmann in Jerusalem (1964), Arendt introduces a distinction between speech and language. Speech is connected to thought and reflection and signifies the capacity to reflect on one’s position in the world and one’s relation with others. Language, by contrast, is what Arendt calls an unthinking speech that consists in the blind following of syntax and grammar, resulting in the excessive use of bureaucratic expressions that reify existing narratives. Only relying on language makes one unable to speak for oneself and to be aware of one’s own position in the world.

By analyzing how Eichmann, a senior Nazi official, justifies his choices in the war and the Nazi regime, Arendt points out the banality of his language. No matter where he appeared, he predominantly spoke in cliches and “officialese” [30, pp. 48–49]. Later on, Arendt formalized her criticism, leading up to her definition of language: “Clichés, stock phrases, adherence to conventional, standardized codes of expression and conduct have the socially recognizable function of protecting us against reality, that is, against the claim on our thinking attention which all events and facts arouse by virtue of their existence” [31, p.160]. Arendt connected Eichmann’s conventional manner of speaking, traced through the general bureaucratic communication of the Nazi regime, to the evil of sedating the public to the horrors of the Holocaust throughout the war. The functional use of language(i.e., relying on stereotypical portrayals and overused phrasing) precluded Eichmann from distinguishing himself from the Nazi party, to reveal his own judgments and own self as a participant in those events. This also enabled him to distance himself from the events, and seemingly justified his inability for critical thinking. This is reflected in how he considered himself a mere representative of the collective dominant position, and as one of the many, was incapable of resisting what the dominant regime commanded to do.

Arendt’s analysis warns us of the danger of flattening the inherently political speech to a functional language. By enrolling in a formalization of language in an unreflective way, one creates a barrier between reality and oneself, thereby reducing one’s reflective capacity to position themselves as part of a world in which action is possible. Because speech is connected to reflective thinking, it is political, and because language is devoid of thinking and is merely functional, it precludes political action. It is therefore a duty in a democratic society to ensure the possibility of speech, such that the use of functional language, albeit socially important, does not become a default means of interaction.

To be clear, we are not suggesting that functional language is unimportant. On the contrary, it is essential for carrying out key human activities, such as labor and work. It is the overall system of grammar and words that constitutes a shared system of communication that can be learned and transferred to others. But not all communication is political, according to Arendt. Speech is a form of communication that allows articulating one’s distinctness through expressing opinions, and engaging in debates that are meaningful to us. In this sense, it is more active and dynamic, more personal and creative, and irreducible to the patterns and rules that govern language. This irreducibility is what lends speech its political importance, and indicates how LLMs might invite people to rely, even more than they ordinarily do, on functional language. For instance, LLMs may flood a public space with seemingly unique messages that in their own ways promote the same specific ideology, or promote the same set of values. This potentially reduces human capacity for speech, further prompted by the inability to contest or verify these messages in an open dialogue, and increases the likelihood of enrolling into another agent’s agenda and pattern of expression because of its dominating position. This analysis allows us to draw parallels between the famous reflections of George Orwell on writing filled with vagueness, passive voice, and stock phrases, and modern writing and speaking, mediated by LLMs:

“[It] does not consist in picking out words for the sake of their meaning and inventing images in order to make the meaning clearer. It consists in gumming together long strips of words which have already been set in order by someone else” [32, pp.7–8].

Thus, from this Arendtian analysis follows that LLMs promote the unreflective use of language, making us less prone to recognize that instead of distinguishing ourselves and our thoughts against others, we are enrolled into the political speech program of someone else without reflection. We will further explore this preliminary finding by expanding our analysis with Foucault’s work on subjectivity and counter-conduct in power relations.

4 Foucault and the possibility of counter-conduct

In Security, Territory, Population, Michel Foucault [33] introduces the conceptual pair of “conduct” and “counter-conduct” to analyze the dynamic of exercising power and resisting it. Introducing the notions of conduct and counter-conduct in this paper has a twofold purpose. On the one hand, it serves to highlight that LLMs exercise power that can be potentially resisted. On the other hand, it helps to appreciate how resistance to LLMs - as counter-conduct - can become collective by orienting to a shared and collective desire to be conducted differently.Footnote 2

Foucault links the notion of conduct to that of power and states that the exercise of power in Western society typically proceeds by imposing a specific form of conduct. He introduces the notion of conduct because of its internal ambiguity:

Conduct is the activity of conducting (conduire), of conduction (la conduction) if you like, but is equally the way in which one conducts oneself (se conduit), lets oneself be conducted (est conduit), and finally, in which one behaves (se comporter) as an effect of a form of conduct (une conduite) as the action of conducting or of conduction (conduction) [33, p. 193].Footnote 3

According to this definition, conduct has several meanings: it entails to conduct someone or something else, to be conducted by someone or something else, but also to conduct oneself. All these meanings indicate that the activity of conducting (a) is relational at its core and (b) results in a specific kind of behavior of the person that is being conducted. Every attempt to govern the conduct of others therefore always implies to stand in a relationship with how a subject wishes to govern himself. This is why, according to Foucault, power is always exercised on subjects that have a certain degree of freedom, and it is precisely because of this that power must be distinguished from violence, coercion, or domination. Freedom, for Foucault, does not imply the ability to conduct oneself outside of any sphere of power but instead consists in the ability to critically relate to the forms of conduct that are imposed onto someone and to translate this relation into particular forms of behavior.

Importantly, and what is well-known for those familiar with Foucault’s work, power never takes the form of absolute domination. Rather, power

is a structure of actions brought to bear upon possible actions; it incites, it induces, it seduces, it makes easier or more difficult; in the extreme it constrains or forbids absolutely; it is nevertheless always a way of acting upon an acting subject or acting subjects by virtue of their acting or being capable of action [35, p. 789].

This is why resistance to power remains always possible: while power is exercised by inviting people to engage in possible actions or embrace certain ways of conduct, “every power relationship implies, at least in potential, a strategy of struggle” [35, p. 794]. Counter-conduct, then, is one of those strategies of struggle.

Foucault gives several examples of counter-conduct. These examples reveal that counter-conduct does not necessarily consist in developing a specific individual relation to oneself in which certain modes of conduct are deemed (un)desirable, but also points to the possibility of the formation of collectives that are grounded in a shared will to be conducted differently. One form of counter-conduct that Foucault identifies is the asceticism found in monasteries. Through ascetic practices, individuals exercise a strict regime on themselves through which they attempt to obtain full mastery over themselves, their bodies, and their sufferings. Although asceticism is firmly embedded within Christianity, it evades pastoral power because it ultimately consists in the exercise of self on self [33, p. 205]. As a result, there is no immanent external power to which one needs to obey to become successful in mastering one’s temptations. This placing oneself under a different regime can thus be considered a particular way of conducting oneself against - and this is what makes it counter-conduct - the activity of conducting exercised by the Church. A contemporary equivalent of this practice can be dubbed digital asceticism: to deliberately withdraw into an offline environment and to find ways to organize one’s life independent of any digital interactions, including those with LLMs.

A second form of counter-conduct, which can be differentiated from the largely individualizing exercise of asceticism, is the formation of communities. These communities originate from the rejection of particular aspects of pastoral power and the desire to develop novel practices to replace those. For example, certain communities challenged the pastor’s privilege to power and authority on the basis of a definitive and inerasable sign, and replaced it with the idea that a pastor’s power cannot be independent of his moral character and worldly conduct [33, p. 208]. So, the point of such communities is not so much to remove the pastor from view, but instead to install different pastors that are better suited for providing religious guidance. Such communities thus remain firmly embedded within Christianity, but attempted to work towards a different religious practice to be exercised as a collective. Applied to LLMs, we can think of this form of counter-conduct in terms of rejecting the values of the corporations developing LLMs, and to attempt to develop models that are more in line with one’s own political concerns.

These twoFootnote 4 examples show that counter-conduct can take the form of both individual and collective practices. Underlying counter-conduct is the desire to be conducted differently, leading to different behaviors. This desire can manifest through the attempt to develop a particular relationship with oneself in which another form of conduct arises, but also through the development of a collective that embodies another form of conduction, and through which a relation with another forming power becomes possible. This is why Foucault insists that counter-conduct does not imply the negation of power per se, and is not grounded in the desire to be not conducted at all and to live in a complete freedom of possibilities. Rather, what unites the activity of counter-conduct is that the “objective is a different form of conduct, that is to say: wanting to be conducted differently, by other leaders (conducteurs) and other shepherds, towards other objectives and forms of salvation, and through other procedures and methods” [33, pp. 194–195].

A second characteristic of counter-conduct that is clarified through the above examples is that it remains to stand in a relationship with the power(s) that it intends to resist. The examples of asceticism or the building of new religious communities still occur within the scope of the Church that exercises power on subject, and the new forms of conduct that subjects desire can, according to Foucault, only be imagined in virtue of the existence of the conduct prescribes by the pastoral regime [33, p. 215]. Analogously, as we will explain in more detail below, also any form of counter-conduct in the context of LLMs takes place against a general horizon of digital technologies, and is never completely exterior to a given technological order.

4.1 Parrhesia as counter-conduct and subjectivation

How, then, can the notion of counter-conduct be made fruitful for resisting the political de-skilling that LLMs give rise to? In this section, we suggest, in line with recent Foucault scholarship [36,37,38,39], that the relevance of this notion stems from the fact that it enables connecting Foucault’s analysis of technologies of the self to the realm of political action. Foucault’s work on the technologies of the self is widely discussed in contemporary philosophy of technology [40,41,42,43,44], but is also criticized for paying little attention to political and institutional constraints on subjectivation [45–46]. We contend with the latter critique, and therefore show in this section how political de-skilling should not be addressed in terms of developing individual ways of self-governance. Instead, resisting de-skilling presupposes a shared desire to be conducted differently, such that alternative ways of political action become imaginable or possible. After this, we identify a particular political technology of the self, namely that of parrhesia - frank, fearless truth-telling - to point to how LLMs can be integrated into political subjectivation.

In the philosophy of technology, Foucault’s ethics of subjectivation is typically introduced to highlight that technologies, because they mediate human behavior, are amongst the forces that act upon the direction in which the subject develops and therefore have profound ethical impact. Developing an engaged relation with technologies, then, is understood in terms of individuals being able to recognize that technologies do shape subjectivity, and to exercise power on the ways in which they are shaped by technologies [44, p. 86].

It is tempting to equate the technologies of the self that Foucault writes about with particular technological artefacts. However, for Foucault, the term “technologies” in “technologies of the self” does not necessarily refer to the material artefacts that people use to reach certain goals in the external world, but refers to those activities that give an implicit answer to the fundamental questions of “What should one do with oneself? What work should be carried out on the self?” [47, p. 87]. Answering these questions, so Foucault maintains, requires engaging in activities of self-governance that enable someone to move in a particular direction while remaining part of a field of contingent forces. This “moving in a particular direction” is an attempt to acquire stability, such that one is able to pursue one particular path within a manifold of possibilities. The direction that this pursuit takes is what Foucault attempts to capture with the term subjectivation. Reminiscent of how Foucault speaks of counter-conduct, employing and developing particular technologies of the self is not something that is the result of a pure autonomy that enables one to give direction independent of any constraints. Instead, technologies of the self are a response to being situated in an already existing field of powers. In Foucault’s words, technologies of the self

permit individuals to effect by their own means, or with the help of others, a certain number of operations on their own bodies and souls, thoughts, conduct, and way of being, so as to transform themselves to attain a certain state of happiness, purity, wisdom, perfection, or immortality [48, p. 225, our emphasis].

From this definition, three elements stand out: technologies of the self (1) are effects of individuals on their own selves, which (2) transform the self’s current state, and (3) imply movement towards a certain (desired) goal (i.e., the transformation has a direction).

Whereas Foucault identifies a variety of ways in which work can be carried out on oneself throughout history, all of these have three things in common. First, they point to the subject’s capacity to resist being a passive target of contingent forces and emphasize the potential of the subject to actively steer their own development. Second, subjectivation implies cultivating an ethos that enables one to both identify certain powers that stand in the way of a desired transformation and the ability to position oneself in relation to those powers. Third, subjectivation is described in terms of a struggle without guaranteed success: bodily desires and other forces are always lurking in the background and potentially stand in the way of transforming oneself in a particular direction.

As indicated, Foucault uses the term “technology” in a much broader sense than only to refer to material artifacts. This is also why he can introduce speech as a technique that contributes to taking care of the self. The political implications of taking care of oneself become especially apparent in his lectures on the practice of Parrhesia: the practice of frank, fearless, trustworthy truth-telling. The person practicing parrhesia is not necessarily speaking true sentences, when truth is being understood as a property of a proposition in terms of its correspondence with an external reality, but rather is someone who “constitutes himself and is constituted by others as a subject of a discourse of truth [...] in which he presents himself to himself and to others as someone who tells the truth” [49, p. 3].

In this notion of Parrhesia, as a specific political technology of the self, we find a close connection between Foucault’s work with political ideas of Arendt, as it emphasizes the importance of disclosing one’s ideals in the presence of others. Insofar as we consider parrhesia an important aspect of political or democratic lifeFootnote 5, LLMs can indeed potentially undermine a central aspect of political subjectivity. This is because practicing parrhesia requires the presence of a particular connection between the statements that one utters and the person that one is or desires to be. This connection is absent in the authorless yet authoritative statements generated by LLMs. This gives rise to the question of whether the practice of parrhesia as a form of subjectivation is still possible in the particular discourses that LLMs give rise to [50]. Our suggestion here is that, when this potential undermining and the political de-skilling it can give rise to are recognized, the practice of parrhesia can also form the ground for developing a form of counter-conduct in relation to LLMs.

The practice of parrhesia is best explained by contrasting it with rhetorical techniques. Foucault defines rhetorics as

an art, a technique, a set of processes which enable the person speaking to say something which may not be what he thinks at all, but whose effect will be to produce convictions, induce certain conducts, or instill beliefs in the person to whom he speaks [49, p. 13].

The practice of parrhesia, in contrast, is not a particular technique determining how something is said in order to obtain certain effects, but instead consists in the practice of establishing a particular relation to oneself by means of which one is constituted as a truth-telling subject. The relationship with others established in the practice of parrhesia is not one of instrumentality that intends to put certain constraints on the possibilities of others, but instead one in which one puts oneself at risk by opening up the possibility for others to reject the truth that one is exposed to.

For a subject to be able to engage in such a relationship with oneself and others is not something that one can take for granted, but requires one to take care of oneself in a particular way, very much reminiscent of how Foucault speaks about technologies of the self. After all, the practice of parrhesia presupposes giving a particular answer to the questions of “What should one do with oneself? What work should be carried out on the self?” [47, p. 87]. Furthermore, whereas, as indicated above, the parrhesiast is always at risk of generating disagreement, another possibility is that one’s truth is recognized by other speaking subjects, such that it becomes possible to form a collective around a particular truth that is being told: the practice of parrhesia is a particular form of subjectivation. That is, it enables individuals to carry out work on themselves that enables them to become (a) subjects capable of engaging in truth-telling, and (b) subjects that are committed to particular ideals and truths that can be articulated publicly. Cultivating these skills and sharing a collective desire to engage in the practice of parrhesia can be a particular answer to the question what work on the self should be carried out in response to the power exercised by LLMs.

5 Discussion: criticizing LLMs beyond digital literacy

Our discussion of Arendt’s work on politics and Foucault’s work on the constitution of (political) subjectivity brings us in the position to better articulate how LLMs give rise to political de-skilling. Political action, following Arendt, presupposes the capacity to exercise one’s faculty of speech because only in this manner does it become possible to disclose one’s interests in the company of others: speech is the condition of possibility for dialogue. She contrasts the faculty of speech to mere language use, which is, according to her, primarily oriented towards the instrumental realization of particular predetermined goals. In this case, the goals themselves are not questioned, and language use merely functions as a vehicle for efficiency. Moreover, since engaging in political action presupposes plurality, reducing or eliminating the capacity to exercise one’s faculty of speech suppresses the possibility of fertilizing plurality as a ground for creating a meaningful world with others.

Whereas drawing on Arendt primarily allows to point to the challenges that LLMs pose for doing politics, Foucault’s conceptual pair of conduct and counter-conduct suggests that– if we indeed consider LLMs as technologies of power– humans are not bound to accept the instrumental use of language when interacting with LLMs. Foucault suggests that individuals need not uncritically embrace a particular way of being conducted but can organize themselves collectively around a shared desire to be conducted differently. Doing so does not require developing a place extraneous to the powers present in the digital world– in fact, as we saw, it is impossible to carve out a space that exists entirely outside the forces exercised on individuals, but instead to find novel ways of integrating LLMs into political life. We have suggested that the practice of parrhesia is a fruitful starting point for developing a potentially more desirable way of being conducted.

Thus far, a dominant rhetoric about responsible integration of LLMs into society concerns increasing digital literacy in (potential) users as a key solution [11, 13]. On this view, the main underlying issue that needs to be addressed is the extent to which LLMs contribute to the spread of misinformation. While we agree that misinformation forms a critical problem, and that an increased digital literacy puts users in a better position when interacting with (the results of) LLMs, our analysis suggests that focusing on digital literacy alone is insufficient to address the problem of political de-skilling. Whereas a focus on misinformation targets the ability of individuals to recognize particular content or to be able to distinguish accurate from inaccurate information, a focus on political skills concerns the process(es) by means of which political engagement becomes possible in the first place.

We have suggested that combining insights of Arendt and Foucault can be of help when analyzing how LLMs contribute to political de-skilling. We take it to be the main advantage of this combination that it helps navigate between being in awe of the possibilities offered by LLMs, and an overly critical evaluation that considers them only to be an increase of the already problematic aspects of the digital world such as misinformation, isolation, and the decrease of critical skills. On the one hand, drawing on Arendt and Foucault enables pointing to the threats LLMs pose to the development of a critical political subjectivity. This is because, as indicated, LLMs have the potential to lead to political de-skilling, and since much of political action takes place online or is digitally mediated, LLMs affect a key aspect of political activity. The result of this de-skilling, so we have suggested from the perspective of Arendt, is the prevalence of a functional language that increasingly turns reality into a homogeneous space in which the plurality needed for democratic interaction is suppressed.

On the other hand, and this is how our proposal forms a counterweight to overly critical or deterministic perspectives of LLMs, Arendt maintains that since human beings are always embedded in a social world, the possibility of a new beginning is always latently present. Even when LLMs are said to suppress our speech, and hence our capacity of political engagement, they cannot suppress the potential of a new beginning to be born. And even though there is a risk of homogeneity, there is also a possibility of recognizing heterogeneity in the plurality of human existence. After all, people do not choose the world they are born in, and do not live in it as isolated individuals, but find themselves in the company of others who bear a similar latent potential to start something novel. As a result, life always involves unpredictability that can never be reduced entirely or flattened out by the functionality of LLMs or other technoscientific developments.

However, the potential of a new beginning cannot simply be taken for granted. This is because Arendt singles out plurality as a condition of possibility for political action. Since human beings are thrown into a world that is beyond their control and become a particular subject in virtue of this thrownness, their subjectivity is shaped by how the world is constituted before their presence. And if it is plausible that LLMs in particular are partly constitutive of the world in which human beings find themselves, to exist in the presence of such technologies also implies operating against a horizon of certainty and homogeneity, which constitutes a threat to plurality. Here, we can make a connection between Foucault’s notions of counter-conduct and the technologies of the self.

As we saw, power is, for Foucault, not a repressive or alienating force that comes from above, but rather is productive in that it helps create a particular kind of subject with particular desires, particular problems, and particular possibilities of resistance. A new beginning can therefore never emerge out of nowhere, but is always the result of one’s desire to be governed differently, or an attempt in finding a new way to govern oneself. In other words, a new beginning is only new relative to already existing governmental structures and can only occur within an already existing horizon of governance. Rather than capacities of human subjects that exist a priori, specific technologies of the self or particular forms of counter-conduct are always circumstantial, and come into existence in relation to the particular forms of power.

We have highlighted the practice of parrhesia as one particular form of counter-conduct that resonates well with an Arendtian conception of politics. Engaging in this practice means attempting to develop the skill(s) to position oneself as a truth-telling subject and to employ language in a non-instrumental manner, i.e., not as a means to obtain a certain end. Developing these skills encompasses more than the possibility of recognizing misinformation, and is not fostered primarily by labelling the digital content, including in the texts generated by LLMs. Upon our analysis, the ability to critically evaluate information is the result of a certain form of political subjectivity instead of its cause. Considering that LLMs are here to stay, and taking the practice of parrhesia as a starting point for political subjectivity, resisting political de-skilling consists in the development of a shared desire to conceive of LLMs as a vehicle of speech that can recognize the need for plurality, and can propagate it. This need can instigate a shared form of counter-conduct grounded in a desire to be governed differently, and in a different conception of the power of words.

In this paper, we have primarily focused on the processes by which subjectivity is constituted and have left it open to what kind of skills are precisely needed for integrating LLMs responsibly into the context of politics. This is partly because what kinds of skills are relevant depends on the kind of political subjectivity that emerges as a result of the counter-conduct inspired by the practice of parrhesia. However, an essential starting point that can be distilled from our analysis is that we can begin to understand LLMs as parrhesiastic technologies that generate texts that are often authoritative in nature and by implication present themselves as truth-tellers. Engaging in a non-instrumental way with such technologies (e.g., not just as technologies that produce blueprinted texts, but relating to those texts as a way to better understand or even challenge our own intentions and practices), implies positioning oneself in similar parrhesiastic terms and developing one’s own skills at truth-telling. Doing so is a form of counter-conduct directed at creating new beginnings, which will allow one to navigate between the optimistic and doomsday narratives about LLMs, and find new ways to politically engage with the world in which one finds oneself.