Keywords

1 Introduction

In a classic paper, Herlocker et al. define one of the core tasks of recommender systems as follows: Find good items (Herlocker et al. 2004, 9). This definition, however, is deceptively simple. What is a good item? And good for whom? For the user or for the provider of the recommender system? Or even for a third party? These questions are much more complicated than they might seem, and they are not as frequently dealt with as one might expect. As Jannach and Adomavicius note, a “question that is rarely asked explicitly in recommender systems research, is: What is a good recommender system? (Or: What is a good recommendation?)” (Jannach and Adomavicius 2016, 8).

If we take, for example, e-commerce as one of the most widespread applications of recommender systems, a simple answer that first comes to mind is this: A good recommendation – the recommendation of a good item – would simply be the one resulting in a purchase. After all, if we did not think the recommendation was a good one, we would not buy the recommended item. However, even though purchases may in some cases serve as an indicator for good recommendations, purchases and good recommendations are not necessarily equivalent. If they were, cases in which consumers are somehow manipulated by a recommender system into buying things would have to count as good recommendations as well. The same goes for click-through rates (CTR) as an indicator for good recommendations. Clicking through links can be the result of a user having been presented with user-relevant content, and therefore possibly constitute an example of a good recommendation. But the all too familiar cases of clickbait clearly represent examples of misleading and deceptive recommendations (Burr et al. 2018, 743). Clickbait may seem harmless when a user is “merely” presented with irrelevant content. Some critics argue, however, that it can become a serious problem when it contributes to creating so-called “filter bubbles” and “echo chambers” (Bozdag and van den Hoven 2015; Flaxman et al. 2016), although their actual significance and precise impact is controversial (Dubois and Blank 2018).

These are only some of the ethical challenges raised by recommender systems. There are many others (Milano et al. 2020). In this contribution, I examine how recommender systems affect our practical reasoning and whether they pose a threat to autonomy, i.e., what influence recommender systems have on our capacity for making our own choices. I will argue that a basic requirement for integrating recommendations in autonomous decision-making consists in being able to identify the rationale behind recommendations: only if we understand why we are being presented with certain recommendations is it possible for them to be integrated into decision-making in a way that preserves autonomy.

The aim of this paper is thus to answer two questions: (1.) What role do recommender systems play in our decision-making? (2.) How can automated recommendations be integrated in practical reasoning such that it yields autonomous decisions? To answer the first question, I explore in a first step what is generally involved in making recommendations and integrating them in decision-making (Sect. 7.2). In a second step, I examine the influence of automated recommendations on our decision-making by critically discussing a popular proposal in the literature, which conceives of recommender systems as a form of digital nudging. In doing so, I highlight what automated recommendations and digital nudges have in common but also point out significant differences (Sect. 7.3). To answer the second question, I use the conceptual tools developed in the previous sections and argue that if our decision-making based on automated recommendations is to be autonomous, then the features used in principles of selection and filtering must be transparent because otherwise the recommendation’s rationale cannot be identified and hence the recommendation cannot be integrated in practical reasoning (Sect. 7.4).

2 Practical Reasoning, Choices, and Recommendations

Recommendations and choices are elements of practical reasoning. Practical reasoning is the capacity for weighing reasons to decide on a particular course of action among various alternatives (Wallace 2020). Choices are the result of particular pieces of practical reasoning. Recommendations guide other people’s choices by supporting their practical reasoning. They are issued, accepted, and rejected in pretty much every area of life, for example, if we ask a salesperson what product to buy or if we ask a friend what to do about a certain moral conundrum. Recommendations can thus be understood as offering decision-making support in answering questions of the general form “What should I do?” That makes recommendations a species of normative or value-judgments, judgments whose subject matter is not only concerned with matters of fact but are also closely connected to action (Rosati 2016).

Richard Hare, in his The Language of Morals, argued that the ultimate purpose of moral judgments – in fact, all value-judgments – is to guide choices rather than making truth-apt statements (Hare 1952, 29, 127). Non-cognitivist approaches such as Hare’s dominated metaethics until the 1980s but became increasingly controversial since (Schroeder 2010). Nowadays cognitivism represents the mainstream (Bourget and Chalmers 2014, 476).

Fortunately, Hare’s main thesis, his wider (meta-)ethical views and the controversies surrounding them need not concern us here. What makes Hare’s reflections particularly suitable for my purposes is that in arguing for his thesis he provides a general analysis of what is involved in guiding choices in both moral and non-moral contexts, an analysis that is independent of his main thesis and the rest of his theory. So, it does not matter whether his main thesis is correct. Since the point of recommendations is to guide choices, Hare’s analysis will prove useful to shed light on what is involved in making recommendations and integrating them in decision-making.

Hare starts with the observation that choosing – understood as an instance of practical reasoning as opposed to picking out something at random – is intrinsically linked to standards. Standards provide norms, rules, or principles of selecting items in order to decide one way or another:

We only have standards for a class of objects, we only talk of the virtues of one specimen as against another, we only use value-words about them, when occasions are known to exist, or are conceivable, in which we, or someone else, would have to choose between specimens. (Hare 1952, 128)

The classes of objects – also called “class[es] of comparison” (Hare 1952, 133) – that are the targets of deliberation can be diverse: Hare’s examples comprise cars, pictures, billiard-cues, and fish bait. It does not matter much which classes of comparison we consider, or whether the context of choice is actual or counterfactual. In theory, at least, a context of choice in which we can or need to decide between different specimens within a particular class of comparison can be thought of for virtually every class of objects. And calling a certain specimen of a class of comparison a good one is tantamount to suggesting it should be chosen (Hare 1952, 127).

According to Hare, such contexts of choice involve standards, which specify characteristics allowing us to compare items with one another and thus providing us with reasons to choose one specimen rather than another. For example, telling someone or being told by someone “This is a good car” implies that the car possesses certain characteristics on the grounds of which it is called a good car. These so-called “good-making characteristics” (Hare 1952, 133) are descriptive properties forming the basis for value-judgments to guide choices, whether they are our own or those of others. Depending on various factors such as (consumer) ends, values, and preferences, good-making characteristics may include, for example, facts about the car’s safety, speed, stability, or sustainability. Whatever the characteristics may be in the particular case, these characteristics form the standard according to which the cars under consideration are judged, and they can therefore figure in the reasons for either choosing or recommending a particular car. That is why questions of the form, “Why are you calling this X a good one?”, “Why should I choose X?”, and “Why are you recommending X to me?” can be answered by referring to the good-making characteristics as a particular standard of judgment.

Now, standards of judgment introduce another logical element in contexts of choice:

As we shall see, all value-judgements are covertly universal in character, which is the same as to say that they refer to, and express acceptance of, a standard which has an application to other similar instances. (Hare 1952, 129)

Value-judgments such as “That is a good X” are “covertly universal” for the simple reason that the standards entailed by this kind of judgments are in principle applicable to other members of the same class of comparison – in fact, the same standard must be applied to other members of the class of comparison on pain of inconsistency. If I tell someone “This is a good car”, and my reason for this judgment is that the car in question is stable on the road, then it would be inconsistent to say of another car with the exact same characteristic “This is a bad car” – other things being equal. Since the good-making characteristics form the basis of my value-judgment, it would be inconsistent to make a contradictory judgment about another object possessing the same good-making characteristics. Of course, usually we consider several potentially good-making characteristics and weigh them against one another to arrive at an “all things considered” judgment. Thus, one specific property serving as a good-making characteristic in one case – e.g., stability on the road – may be outweighed by other negative characteristics – e.g., poor energy efficiency. Still, unless the good-making characteristics of a particular object are somehow affected by other relevant factors, logical consistency requires that value-judgments about other relevantly similar objects cannot differ unless the good-making characteristics also differ. In other words, a difference regarding the value-judgment implies a difference regarding the good-making characteristics (Hare 1952, 131).

In sum, choosing involves standards of judgments providing us with the resources to compare and choose among specimens of a particular class of objects based on reasons. Standards are universal, not in the sense of being applicable always and everywhere, but rather in the sense that, once they are employed to judge the merits of a particular object, they automatically apply to other objects that are similar in the relevant respects simply in virtue of the objects’ sharing the same characteristics on which we base our corresponding value-judgments. The upshot of this analysis of value-judgments is that judgments of the form “This is a good X” are never just about a single object but also, implicitly, about other objects sharing relevantly similar characteristics due to the involvement of universal standards.

With these considerations in mind, we can draw certain conclusions with respect to the role recommendations play in decision-making. As we have seen, choices and recommendations involve universal standards of judgment. These standards of judgment are essential for understanding the reasons on which choices and recommendations are based. If a salesperson at a car dealership recommended a certain car to me by saying “Take this car (it is a good one)”, then I would have to know the standard – the good-making characteristics – informing this recommendation in order understand its rationale. After all, the good-making characteristics of it may vary significantly depending on different customer ends, values, and preferences. Maybe it is a good car in terms of speed but not in terms of sustainability; maybe it is good for commuting but not for long-distance drives; and maybe it is a good car in virtue of the size of the commission the salesperson will receive – whatever the good-making characteristics, and hence the standard of judgment, the point is that unless I know the standard and thus can understand the rationale behind the recommendation, I am in no position to integrate it in my decision-making. For, I simply cannot assign a role to something I do not understand. In short, standards of judgment provide the identity conditions for the rationale behind a recommendation; I cannot understand the recommendation’s rationale if I do not know the standard (and hence the reasons) informing it.

The importance of this point lies in the fact that even a good reason for a particular recommendation may not necessarily constitute a good recommendation for me if the rationale behind the recommendation is inconsistent with my ends, values, and preferences (but possibly with those of someone else). Only if I can adopt the reasons behind a recommendation as my reasons for accepting the recommended choice can we truly speak of a good recommendation capable of being integrated into my decision-making.

In general, the good-making characteristics of an item and the ends, values, and preferences on which they depend constitute the rationale behind a particular recommendation. Basing a decision on a recommendation, or at least including it as a relevant factor in decision-making, presupposes knowing the rationale behind the recommendation in order to understand its potential role in decision-making. And only if I understand its role can I determine what it contributes to my decision. One important consequence of this, which will be relevant in Sect. 7.4, is that decisions based on recommendations whose rationale we do not understand carry the danger of making our decisions opaque to us, posing a severe threat to autonomy – and this is so regardless of whether the recommendations in question are well-intentioned or not.

In this section I explored the role of recommendations in practical reasoning in the analogue world. In the following section, I will explore the specifics of recommendations under the conditions of a digital environment, in particular with respect to recommender systems.

3 Recommender Systems and Digital Nudging

Navigating the online world we encounter recommendations at every turn. On streaming portals, shopping sites, social media platforms, and online newspapers we are confronted with recommendations regarding what video to watch, what products to buy, what posts to like, and what articles to read. The automated systems behind these recommendations promise to prune back the digital jungle of content and guide us to the things we are actually looking for.

As noted, the primary purpose of recommendations is to guide people’s choices, and this is no different in the digital world. Whatever the motives and reasons behind the recommender systems’ operators, whether they have their users’ best interests at heart or not, it seems rather uncontroversial that recommender systems are designed to guide people’s choices in one way or other by affecting their decision-making. That is why the employment of recommender systems has been linked to digital nudging, the broader online practice of influencing people’s behavior with digital means (Burr et al. 2018; Jesse and Jannach 2021; Milano et al. 2020). However, the proposal to conceive of recommender systems as tools of digital nudging is rarely fleshed out and the ethical questions recommender systems raise are rarely addressed. In this section, I will discuss this proposal in more detail.

Weinmann et al. define digital nudging as follows:

Digital nudging is the use of user-interface design elements to guide people’s behavior in digital choice environments. (Weinmann et al. 2016, 433)

Digital nudging is a practice that structures people’s digital environment in ways that affects their decision-making, and thus the resulting decisions. The design elements responsible for influencing choices can assume a variety of forms and usually draw on diverse psychological mechanisms known to be involved in decision-making. For example, one comparatively simple psychological phenomenon is the middle-option bias: given the choice between three or more options, people display a propensity towards the option situated in the middle (Schneider et al. 2018, 70). A more complex phenomenon is the decoy effect: people are more likely to choose an option if the option next to it is highly unattractive (Schneider et al. 2018, 69). Another powerful nudging mechanism involves setting default options. Default options exploit the status-quo bias, i.e., the disposition to stick with preset options out of inertia (Caraban et al. 2019, 4). Square, an app for making online payments, for example, includes “tipping” as the default option, and users have to opt-out if they do not want to tip, which increases the probability of tips (Carr 2013). The list of such psychological phenomena and the nudging mechanisms employing them is enormous. In their survey paper, Jesse and Jannach have identified and categorized 58 psychological phenomena and 87 nudging mechanisms (Jesse and Jannach 2021).

There are two important things to note about the concept of digital nudging. First, the concept of nudging was originally developed by Thaler and Sunstein and applied first and foremost to offline contexts. Second, Thaler and Sunstein embedded nudging as the central tool in their policymaking approach they dubbed libertarian paternalism, which is designed to help people make decisions that are better for themselves but do not restrict their freedom of choice (Thaler and Sunstein 2008). Although the background of libertarian paternalism is frequently mentioned in the literature on digital nudging, more often than not it plays only a subordinate role. This should be kept in mind when considering the ethical implications of digital nudging because the kind of goals pursued by digital nudges is highly relevant for assessing them, yet not all digital nudges are implemented with the aim to benefit users.

To bring digital nudging, particularly as exemplified by recommender systems, into sharper relief, let us compare it with the original concept from Thaler and Sunstein. Thaler and Sunstein build their libertarian paternalism on insights provided by the behavioral sciences, according to which people are less than perfect decision makers (to put it mildly). For one thing, they do not always make decisions that are in their best interest. For another, particular types of decision-making are frequently prone to a whole array of fallacies (Thaler and Sunstein 2008, 8). Practical reasoning as depicted in the previous section – in which a well-informed individual carefully balances reasons for and against the purchase of a certain kind of car, comparing its strengths and weaknesses with other cars in the light of considered ends, values, and preferences, reflecting on and integrating the expertise of a seasoned salesperson – represented only one type of decision-making. The other type consisted in decision-making that works much more quickly and without extensive reflection. This process relies more on intuition, habits, and rules of thumb, which enable us to act faster and nearly automatically in comparison to explicitly reason-based decision-making. But it is also much more error-prone (for example, the susceptibility to the middle-option bias and the decoy effect). In the established terminology introduced by Kahneman, decisions are made employing either the fast thinking of System 1 or the slow thinking of System 2, depending on what the circumstances require (Kahneman 2012, 20–22). For example, thinking through carefully the purchase of a car is a job for System 2, whereas driving it home from the car dealership on an empty and familiar road is a job for System 1.

The key idea behind nudging is that the deficiencies of System 1 can be utilized to help people make better decisions and thus increase their quality of life. Unlike conventional paternalism, libertarian paternalism tries to achieve this goal without bans and incentives, but simply by intervening in what Thaler and Sunstein call “choice architecture”, i.e., people’s decision environment. Whereas conventional paternalists typically restrict people’s choices by prohibiting certain options, libertarian paternalists present the options in ways that make it more likely for people to choose what is supposedly better for them anyways – they nudge (Thaler and Sunstein 2008, 6).

Most importantly, then, nudges target the presentation and not the content of options with which people are confronted – nudges modify the how, not the what, which is essential for libertarian paternalists because they adamantly insist on preserving freedom of choice. The standard example is a cafeteria where fruit is put at eye level of customers to increase the probability of their choosing this healthy option rather than a less healthy one. Crucially, the less healthy options remain available. It is just that they are assigned a less attractive position in the choice architecture of the cafeteria. With freedom of choice thus ensured, libertarian paternalists promise to respect people’s autonomy because they can still choose according to their preferences and thus pursue their ends unimpeded. And libertarian paternalists intend to help those whose psychological vulnerabilities – for example, weakness of the will or inertia – would otherwise prevent them from choosing what is in their own best interest.

How does this analogue form of nudging in offline context compare to digital nudging in online contexts? Generally speaking, most recommender systems employ either content-based recommendations or collaborative recommendations (Jannach et al. 2011). Content-based recommendation systems filter options by searching for items sharing specific features. This involves past user behavior because in order for the recommender system to suggest similar items it needs some point of reference, for example, items the user liked in the past. The rationale behind this recommendation technique is closely related to the role of standards in practical reasoning explored in the previous section. If I judge a certain product to be good, this judgment is usually based on certain features the product possesses (the good-making characteristics). Since these features on which I base my value-judgment form a general standard that applies to other items exhibiting the same features, it is therefore reasonable to assume that similar products will receive the same assessment. Content-based filtering thus appears to ensure that if I am presented with items similar to the ones I previously liked, then I will probably like these items as well.

The other type of recommender system frequently employed is based on collaborative recommendations. Here users are simply presented with items other users with similar user preferences have liked in the past. This “crowdsourcing” method for generating recommendations has the advantage of not needing data on the content of items as well as not needing much data on a particular user to make predictions of, and subsequently recommendations for users.

Now, in light of the definition of digital nudging there is an obvious sense in which the employment of recommender systems qualifies as a form of digital nudging: recommender systems are designed to guide people’s choices by modifying their digital choice architecture. They select and order options, customize information, and suggest alternatives (Jesse and Jannach 2021, 7). Automated recommendations can thus be understood as digital nudging insofar as they influence our decision environment.

There is, however, at least one important difference between recommendations and nudges that makes it somewhat misleading to categorize recommender systems as digital nudging tools. Recall that one of libertarian paternalism’s central doctrines puts the utmost emphasis on freedom of choice. Nudges are supposed to modify only how options are presented, and not to alter the range of options itself. But many recommender systems do precisely that. And given their central task – finding good items – this is rather unsurprising. The whole point of a recommender system is to filter out options that are deemed irrelevant for the user, just as a salesperson at the car dealership also does not show you all the cars they sell but only recommends to you a subset of them based on your preferences. This constitutes a significant difference between nudges and recommendations. Both are used to guide choices – but nudges arrange a given space of options in a way that makes it more likely for people to pick the option choice architects think is in their best interest, whereas recommendations create a space of options by presenting only those options purportedly tailored to people’s preferences.

Even if a recommender system were to display all available options, in most cases there would be so many options that it would be practically impossible for users to review them all. Consider, for example, Google’s widely used search engine. Millions of search results are displayed and ranked on countless pages for every query, yet most people only browse through the first few search result pages (Pasquale 2006). This is an example of the so-called positioning nudge, according to which different visual arrangements of options significantly impact our choices (Caraban et al. 2019, 5). Although freedom of choice is theoretically preserved because a user could browse through all search results, in practice this is unfeasible. This is distinctive of many recommender systems given the huge amount of data they process. Thus, in effect, a recommender system presents users only with a subset of possible options, and many of them use personalized information to shape and limit the set of possible options even further. For example, simply the location of a user – whether the user logs on from Europe or from the U.S. – will make a difference for the search results on Google (Bozdag 2013, 212).

In sum, recommender systems can be understood as a form of digital nudging in the broad sense expressed in the definition by Weinmann et al., according to which any practice counts as digital nudging that aims at guiding people’s choices by modifying their digital choice architecture. But since libertarian paternalism is an integral part of the original concept of nudging, the term “digital nudging” may be misleading if it is taken to be in the service of the very same agenda. To avoid confusion, it should be kept in mind that the stated goals of libertarian paternalism – helping people make better decisions but preserving their freedom of choice – are not necessarily elements of digital nudging.

Does this mean recommender systems violate freedom of choice, and thus autonomous decision-making? This question somewhat distorts the issue since recommendations, unlike nudges, are expressly designed to reduce our options for our own benefit. Given the huge amount of online information, we want recommender systems to present us only with a relevant subset of possible options, just as we want the salesperson to show us only products of potential interest. Narrowing down possible options is not a bug, it is a feature of recommendations. How this affects our decision-making and which conditions have to be fulfilled for automated recommendations to be integrated in practical reasoning in a way that preserves autonomy, then, is the topic of the next section.

4 Autonomy in Practical Reasoning with Recommender Systems

Heinrichs and Knell have recently described nicely a feeling of alienation that may emerge from engaging with recommender systems, a feeling that does not normally occur during conversations with salespeople:

Think about the bookseller again: If she gives you a recommendation, maybe you would ask for more details. Or after a reading, perhaps you would tell her your impressions and discuss the book with her. This is not possible with an AI recommendation system – and this feels weird. When someone tells you something, you expect to be able to ask questions and make comments. If this is not possible, it is a profound deviation from our common discursive practice. (Heinrichs and Knell 2021, 1575)

The difference between the recommendation of a salesperson and a recommender system is that engaging in an exchange of reasons regarding the recommendation’s rationale is impossible with a recommender system. That is why, Heinrichs and Knell argue, we must not (at least not yet) consider recommender systems bona fide participants in our discursive practices, but rather only as complex tools (Heinrichs and Knell 2021, 1578). I agree with this assessment. Reasoning with a recommender system in the same way as with a bookseller may be impossible (so far). But could it be possible to reason with it in the sense of using it as a supporting tool in decision-making, i.e., to integrate the output of recommender systems in one’s decision-making in a non-alienating way? In the following, I will argue that being able to identify the rationale behind recommendations is a basic requirement to assign them a meaningful role in our decision-making and thus for the resulting choices to be autonomous.

As I argued in the previous section, recommender systems can be understood as tools of digital nudging insofar as they guide people’s decisions by modifying their choice architecture. Arguably, the most profound ethical problem that has been raised about nudging practices concerns threats to autonomy (Engelen and Nys 2020). The term “autonomy” is highly contested and many different interpretations have been proposed (Dworkin 2007; Jennings 2007). This is also true of debates on the ethics of nudging, but most participants are concerned with what Vugts et al., in their literature review, have summarized under the umbrella term “agency”:

Apart from a context that allows choice, autonomy also requires a capacity to choose and decide, and this refers to agency. Agency involves being able to lead one’s life and act on the basis of reasons and intentions […]. This presupposes that the person has relatively stable ultimate goals, can reason about what options are preferable given those goals, and can reflect on the choices he or she makes and has made. Practical reasoning is a necessary capability in agency. (Vugts et al. 2018, 116)

In short, autonomy consists in the capacity to set your own ends and to achieve them by exercising practical reasoning – by considering possible choices and weighing reasons to make decisions. Note that this notion of autonomy first and foremost concerns personal autonomy and not moral autonomy, i.e., it concerns the ends of people considered as individuals, not the relation between people pursuing their possibly different and conflicting ends (Waldron 2005). I do think it makes a difference for the ethical assessment of a recommender system whether the type of recommended item touches on societal matters, and thus that issues of moral autonomy need to be considered in such cases. For example, it seems to make a difference whether a recommender system is designed to help you find an exciting crime novel or a means of transportation with low carbon emissions to protect the environment. The societal dimension is beyond my focus here, but I have dealt with it elsewhere (Bartmann 2022).

Among the biggest threats to autonomy is manipulation (Noggle 2018; Schmidt and Engelen 2020; Vugts et al. 2018). Just as the term “autonomy” the term “manipulation” is also contested, and there are different accounts of what constitutes manipulating someone (Noggle 2022). For present purposes, I will draw on the following definition:

[M]anipulation is hidden influence. Or more fully, manipulating someone means intentionally and covertly influencing their decision-making, by targeting and exploiting their decision-making vulnerabilities. (Susser et al. 2019, 4)

Against the backdrop of the definitions of autonomy and manipulation, threats to autonomy can arise roughly at two different levels: at the level of the ends people pursue (i.e. at the level of what goals to pursue), and at the level of the means they employ to achieve these ends (i.e. at the level of how they pursue them – decision-making). I will review both levels with respect to recommender systems in turn.

Let us start with the level of ends. Consider once again, for example, an e-commerce recommender system and consider the viewpoint of the provider and the viewpoint of the consumer. A recommendation resulting in a purchase may be a good one from the perspective of the provider – because the purchase increases profits – but maybe not necessarily a good one from the perspective of the consumer – because the purchase may not really reflect the consumer’s preferences. Or consider a certain type of business model popular with many online services employing recommender systems. Most social media platforms, for example, make money not primarily with their users but rather with other corporations paying for targeted advertising based on the users’ data (Koene et al. 2015). In such cases, the potential conflict of interest between a service and its users is built into the business model because it is the corporations and not the users who are the actual (paying) customers of the service, which thus has a substantive incentive to align its interests with its customers rather than with its users.

The potential mismatch of ends, values, and preferences just described is a species of the so-called value-alignment problem and occurs when a recommender system is “competing”, rather than “collaborating” with users (Burr et al. 2018, 742). As Burr et al. elaborate:

Importantly, one can describe the goal of the ISA [intelligent software agent] as either “maximising the relevance for the user”, or as “maximising the CTR”. These two quantities are often conflated in the technical literature, but they are not necessarily aligned. (Burr et al. 2018, 743)

The misalignment of values between providers and users of recommender systems thus represents an obvious source of ethical problems. However, the misalignment does not necessarily have to be intentional but may be the result of the fact that different parties are involved in the engagement of a recommender system. In the simplest case, the parties involved represent opposite end points of a recommender system divided into providers and users, and it seems obvious that their respective ends, values, and preferences do not necessarily coincide. As I argued in Sect. 7.2, even if a recommendation is well-intentioned and based on reasonable standards, this does not ensure the recommendation is a good one because the standard used might not be relevant to the user. The task “Find good items” can simply be realized in many different ways depending on the users’ respective ends, values, and preferences. For example, from the user’s viewpoint, a good recommendation may consist in items matching long-term preferences, in items representing relevant alternatives to a reference item, or simply in providing satisfying user experiences; from the provider’s viewpoint, a good recommendation may consist in changing user behavior in desired ways, in increasing demand and sales, or simply in learning more about customers (Jannach and Adomavicius 2016, 8). Thus, just as being well-disposed towards one another does not rule out misunderstandings, in the same way a user and a recommender system can work at cross-purposes even if it is not designed to “compete” with its users. It may just very well be that a recommender system misidentifies the ends, values, and preferences of users due to its limited data or because of a misinterpretation thereof.

Let us now turn to the level of means. Assuming the ends, values, and preferences of users are respected and not in conflict with the operational goals of a recommender system, is the influence on people’s choice architecture to support them in achieving their ends unproblematic? Jesse and Jannach, for example, imagine (but do not discuss) a “nudging-enhanced” recommender system generating recommendations in which recipes for healthy meals are highlighted after having identified the tendency of its user to choose predominantly unhealthy recipes (Jesse and Jannach 2021). Let us flesh out this example further. Imagine the nudging mechanism works successfully. The user chooses and prepares more healthy meals and even experiences a significant increase in health and well-being after some time, all the while not knowing how this improvement in quality of life came about. Would we consider the employment of such recommender systems ethically unobjectionable? After all, the choice for more healthy meals is made by the user, and the user receives real benefits from the engagement with such a recommender system. What is wrong with that?

The idea behind these kinds of arguments is rooted in a particular understanding of decision-making. A “pristine kind of decision-making that is purely deliberate and reflective” was an “unattainable mirage” (Engelen and Nys 2020, 145). Decision-making was always subject to external influences not fully under our control anyways. Even a completely random decision-environment was still a choice architecture influencing us one way or the other. The point they make is that it is impossible to present options in a neutral way. What is true of the food items in the cafeteria was also true in general: options must be arranged and presented in some way or other. This means choice architecture is inevitable – and since people’s ends are respected, why not design choice architecture in such a way so as to help decision-makers achieve their self-chosen ends (Sunstein 2015)?

As I argued elsewhere, I doubt that the inevitability of choice architecture gives one license to modify it (Bartmann 2022). Rather, the inevitability of choice architecture puts a particular responsibility on nudgers because the changes made to it may have an effect on people’s decision-making. But regardless, what seems to me ethically problematic is neither that there is no neutral design of choice architecture, nor that we rely on automated recommendations in decision-making. The problem arises when we are not aware of factors playing a significant role in our decision-making. This is precisely the problem with the above example of recommendations of healthy recipes because they would exert hidden influence and thus be instances of manipulation. If users are covertly nudged into preparing healthier meals by tapping into their cognitive biases and psychological vulnerabilities, they can no longer make sense of the choices made because they are unaware of the recommender system’s rationale for the recommendations. If, on the other hand, users were aware of the rationale – by receiving some indication from the recommender system or by being able to adjust its settings themselves – then the manipulative aspect would disappear. Making the automated recommendations’ rationale transparent that way would enable genuine agency because people would not be manipulated but could rather nudge themselves in a self-determined way. Of course, it would not make users’ vulnerabilities disappear; however, it would address them in a way that allows for integration in practical reasoning and subsequent action.

In general, being able to discern the standards behind automated recommendations to identify their rationale is essential to ensure they can be integrated into practical reasoning. The danger here is even more profound than the one already mentioned at the beginning of this section (the feeling of alienation one may experience because of the impossibility of engaging in a genuine exchange of reasons with a recommender system). And this more profound danger is present even if the influences on users’ choice architecture through recommender systems are not intended to exploit decision-making vulnerabilities: it is the danger of a recommender system’s opaque standards to induce self-alienation in users by disrupting the connection between users’ decisions and their reasons for making them. If I make a decision based on a recommendation whose rationale I do not understand, then I do not really understand why I made the decision because I do not know the reason for it. Decision-making becomes a black box, and this makes the resulting decision alien to me because I cannot recognize the decision as my own.

How can the problem of self-alienation be prevented? As I argued in Sect. 7.2, only if I can identify the rationale behind a recommendation that is aligned with my ends, values, and preferences can I assign it a role in my decision-making and integrate it in practical reasoning. One way of making an automated recommendation’s rationale identifiable is by providing an explanation for how the recommender system works, i.e., the principles of selection and filtering behind the generation of recommendations such as: “Customers who bought this item also bought…” (Tintarev and Masthoff 2011, 479). Do explanations like this give a satisfying answer to the question why I am being recommended some item?

Consider, for example, a streaming portal recommending movies to you. The presentation of recommended movies often includes captions such as “Because you watched movie X” or “Others who watched movie X also liked movie Y”, examples of content-based filtering and collaborative filtering, respectively. Now, it seems as if these captions provide the recommendations’ rationale by giving a reason as to why these movies, and not others, are presented to you. But do they? Even if a particular movie is cited as the reference item used as a basis for recommendations, that still leaves open the movie’s particular features used in filtering. What precisely are these features? The specific genre, specific filming locations, specific actors, directors, or screenwriters? It makes a substantive difference if I recommend to you a certain movie because it is an action movie or because the very same movie features your favorite actor. That is why a satisfying answer to the question why I am being recommended a certain item requires the disclosure of an item’s features relevant for the principles of selection and filtering involved – the good-making characteristics. Otherwise, an identification of the recommendation’s rationale would not be possible. This applies even more to collaborative filtering because they generate recommendations based on other people’s preferences I do not know, such that the degree to which the recommendation is opaque to me is even higher.

One may object against this high standard of transparency by pointing out that even if we do not know how a recommendation was generated we are still able to assess the value of the recommended item independently. After all, I could, for example, simply read the abstract or the first few pages of a recommended book to determine whether buying it would be a good choice. That may be true. But recall that, as I argued in the previous section, (automated) recommendations reduce rather than merely rearrange possible options. So, even if I can assess the set of books recommended to me independently, we must not forget the fact that different principles of selection and filtering would have yielded different recommendations as my starting point of assessment, a fact of which I can only make sense if the underlying principles are identifiable. Given the amount of data recommender systems process, this preselection does impact our starting point, and hence our possible choices significantly. Therefore, if opacity and the associated danger of (self-)alienation is to be dissolved and not simply pushed back a step in practical reasoning, the good-making characteristics composing the standard behind a recommendation must be discernible to ensure autonomous decision-making.

5 Conclusion

Recommender systems profoundly affect our practical reasoning by influencing our decision-making. Filtering processes employed by recommender systems shape our choice architecture, not just by selecting and reducing the possible options among which we can choose, but also by presenting the remaining options in specific ways. Given the enormous amount of information available online, recommender systems can support our decision-making by providing us with relevant options. But for decision-making to be autonomous and to prevent the danger of self-alienation, we must be able to recognize our choices as our own. I have argued that integrating automated recommendations into our practical reasoning in a non-alienating way requires that we can identify the rationale behind a recommendation to understand a decision based on it. This, in turn, makes it necessary that the good-making characteristics of a recommendation be made transparent, i.e., those features used in the principles of selection and filtering to generate recommendations. Only if a recommendation’s rationale is identifiable and thus provides a satisfying answer as to why I am being presented with a particular recommendation can I integrate it in practical reasoning and make autonomous decisions.